Malware traffic analysis is the investigation of recorded network communications for suspicious activity and potential indicators of compromise. The usual raw material is a PCAP, a saved capture of packets observed on a network. The job is to assemble several artifacts into an investigative lead, not declare a host infected because one domain name looks dodgy.
A PCAP can preserve network requests and connection details for later review. Encryption limits what the capture exposes: a TLS Client Hello marks the start of an encrypted connection and can expose connection-start metadata, while the encrypted contents can remain hidden. Traffic analysis is useful evidence, not omniscience.
Start with the PCAP, not a verdict
Packets are units of data moving across a network; a PCAP records them. In a review, useful artifacts can include domains, URI paths, protocols, User-Agent strings, and the host associated with a connection. A recovered file or its hash can also become an investigative lead.
An unusual artifact needs context. In the June 2025 case study cited below, the analyst treated a browser-like PowerShell User-Agent, a Microsoft-themed lookalike domain, and an unusual URI path as signs of a likely automated script or malware beacon. That was the analyst's interpretation of that capture, not a general rule for attributing malware.
A practical PCAP workflow
- Open the capture in a packet-analysis tool. The cited case-study analyst used Wireshark and NetworkMiner.
- Reduce the packet pile. In that Wireshark case study, the analyst used (http.request or tls.handshake.type eq 1) and !(ssdp). http.request selects unencrypted HTTP requests; tls.handshake.type eq 1 selects TLS Client Hello messages; and !(ssdp) excludes SSDP multicast noise. It is a triage filter from one analysis, not a universal malware detector.
- Inspect the remaining artifacts. Review visible domains, URI paths, User-Agent strings, protocols, and the internal host tied to each connection. For TLS traffic, record only what the capture exposes at connection start; encryption may conceal the content.
- Preserve the details. Record artifacts precisely, including domain spelling, URI paths, protocols, and associated host information. Those details are what allow an investigator to compare related connections rather than rely on a single suspicious-looking item.
What suspicious traffic can look like
In the cited training-PCAP analysis, the analyst identified Microsoft-themed lookalike domains, unusual URI paths, HTTP and TLS 1.2 traffic, and an infected-host name as indicators. The value came from the combination of artifacts in that capture, not from a claim that any one domain or protocol proves malicious activity.
A separate training-PCAP write-up used NetworkMiner to identify a file presented with a .png name as an executable, then linked the download to an internal user and host. The filename alone was not the useful evidence. The connection between the object, requesting machine, user context, and network activity was.
Manual investigation and machine learning are different layers
Manual packet analysis lets an investigator examine visible requests and metadata and document the artifacts that support an incident lead. Malware-Traffic-Analysis.net describes itself as a site for sharing PCAP files and malware samples, and it provides PCAP-based exercises, tutorials, and workshop material.
Machine-learning work can instead transform PCAP data into flow features for model evaluation. One ACM study used the CIC-IDS-2017 dataset and CICFlowMeter to extract 80 features from PCAP files, including source and destination IP addresses, source and destination ports, and protocol. The researchers used Pearson correlation for feature selection and five-fold cross-validation while comparing five algorithms on a selected Friday-morning botnet subset.
That study's design does not establish a blanket claim that a model will identify malware in every network. CIC-IDS-2017 included benign traffic and categories including web attacks, denial-of-service, botnet, Heartbleed, and distributed denial-of-service traffic, collected over five days. Dataset selection, labels, extracted features, and evaluation design define what such a result can show.
Do not create new network activity while investigating old network activity
The case-study author disabled a virtual machine's network adapter as a precaution. The author also warned that extracting payloads, clicking links, replaying traffic, or allowing tools to resolve domains automatically can initiate connections to malicious infrastructure. Treat that as an operational warning from the analyst, especially when a PCAP contains material or destinations you have not yet identified.
Frequently asked questions
How do you analyze a malware PCAP in Wireshark?
Open the PCAP, then focus the review with display filters. One cited case study used (http.request or tls.handshake.type eq 1) and !(ssdp), which selects HTTP requests and TLS Client Hello messages while excluding SSDP multicast traffic. Review visible domains, URI paths, User-Agent strings, protocols, and associated internal hosts in context.
Which network artifacts can become indicators of compromise?
The cited analyses identify domains, URI paths, protocols, User-Agent strings, host information, and, when a transferred object is recovered, file-related details as investigation artifacts. An artifact is a lead to examine alongside related traffic, not proof by itself.
What can TLS Client Hello traffic reveal during an investigation?
A TLS Client Hello marks the beginning of an encrypted connection and can expose connection-start metadata. The cited case study used those messages to examine encrypted connections, while noting that encryption can conceal the traffic contents.
How is machine learning applied to malware traffic detection?
In the cited ACM study, researchers used CICFlowMeter to extract 80 features from PCAP files, selected features with Pearson correlation, and compared five machine-learning algorithms using five-fold cross-validation on a selected CIC-IDS-2017 botnet subset. That result is specific to its dataset and evaluation design.
Sources
- Malware Traffic Analysis — www.malware-traffic-analysis.net
- Digging Through a PCAP: an analysis of a PCAP file from malware-traffic — medium.com
- Malware Traffic Analysis using Machine Learning — dl.acm.org
- Malware Traffic Analysis | Steelcoffee Writeup | by Aaron Stratton - Medium — medium.com
