This is the second to last in 2023 edition of the “Advanced In The Wild Malware Test”. Once again, we reveal the differences between the tested security solutions for SOHO (Small Office and Home Office), and business. The test was carried out under the control of Windows 10 Pro. We used a variety of threats, and the most interesting ones include:
- Generic.BAT.Downloader
- Trojan.ShellStartup
- Trojan.Metasploit
- IL:Trojan.MSILZilla.RedLine
- Trojan.Fragtor
- Trojan.MSIL.Androm
- DeepScan:Generic.Malware.Lco
- HEUR.RoundKick
- Trojan.Lazy
- Trojan.NSISX.Spy.Gen.24
To classify threats, we use feedback from the antivirus engine of our technology partner, MKS_VIR sp. z o.o.
Our tests comply with the guidelines of the Anti-Malware Testing Standards Organization. Details about the test are available at this website as well as in our methodology.
How do we evaluate the tested solutions in 2 steps?
In a nutshell, the Advanced In The Wild Malware Test covers 2 large aspects that follow each other, and combines into a whole:
1. Selecting samples for the test and analyzing logs of malware
We collect malware in the form of real URLs from the Internet on an ongoing basis. We use a wide spectrum of viruses from various sources, and these are public feeds and custom honeypots. As a result, our tests cover the most up-to-date and diverse set of threats.
The analyzed samples in Windows are subject to thorough verification based on hundreds of rules: the most common techniques used by malware creators. We monitor system processes, network connections, Windows registry, and other changes made to the operating and file system to learn more about what malware was doing during the analysis. In the next step, based on the threat score, malware is either tested by the solutions or rejected.
2. Simulating a real-world scenario
In the test, we simulate a real scenario of threat intrusion into the system via a URL. It can be a website prepared by a scammer or a link sent to a victim via messenger that is then opened in the system browser.
We divide the results into two areas: when a website or malicious file is identified by a scanner in a browser, or when malware is launched on the system. Thanks to this, we are able to point out the exact differences between the solutions.
In addition to the results of changes made by malware samples, we monitor whether and how the protection solution responses to threats. We also measure the response time of incident recovery (the so-called Remediation Time).
Results in September 2023
In September 2023, we used 343 unique malware samples from the Internet.
Tested solutions to install at home and in a small office:
- Avast Free Antivirus
- Bitdefender Total Security
- Eset Smart Security
- F-Secure Total
- G Data Total Security
- Kaspersky Plus
- Malwarebytes Premium
- Quick Heal Total Security
- Sophos Home
- Trend Micro Maximum Security
- Webroot Antivirus
- Xcitium Internet Security
Solutions for business and government institutions:
- Emsisoft Business Security
- Malwarebytes Endpoint Protection
- Xcitium ZeroThreat Advanced
Key technical data and results are available at the RECENT RESULTS webpage, along with a threat landscape that has been prepared based on telemetry data.
Protection test in numbers
We have prepared the following summary based on telemetry data from the test in September 2023:
- 15 protection solutions took part in the test.
- Websites encrypted with HTTPS (in theory – safe) contained 160 malware samples.
- 181 malware samples were hosted with HTTP.
- During the analysis, each malware sample took on average 31 potentially harmful actions in Windows 10.
- Most malware originated from servers located in the USA, Netherlands, Sweden.
- The following domains were mostly used to host malware: .com, .org, .info.
- The average response time to a threat by all developers combined is: 133 seconds.
- The best average response time to tested threats was achieved by Sophos with time of 10 seconds.
Methodology
We systematically develop the methodology to be up-to-date with new trends in the security and system protection industry, which is why we provide a link to our methodology:
- How do we acquire malware for testing, and how do we classify it?
- How do we collect logs from Windows and protection software?
- How do we automate all of this?
We are a member of the AMTSO group, so you can be assured that our testing tools and processes comply with international guidelines and are respected by software developers.
Test conclusions: Multi-layered protection is the key to success
This edition was a record in terms of the number of 0-day files used to run the full test. Depending on a developer, the number of unknown files may have been different, as you can see in the telemetry data we provide in the CSV table.
Based on several recent editions of security tests, we have come to the conclusion that solutions with multi-layered protection are the key to achieving the best results in protection against malware. Why? Because the use of a variety of mechanisms and techniques is becoming more effective in protection against an increasingly diverse spectrum of threats from the Internet.
For example, when it comes to handling Trojans, ransomware, backdoors, and other forms of malware, the best approach is to combine traditional tools such as scanning webpages, keywords, hyperlinks, and IP addresses on websites distributing malware. As a result, many threats can be effectively prevented already at the stage of visiting a website.
Another essential element is heuristic scanning, which analyzes the behavior of files and applications after the launch, looking for suspicious activity that indicates the presence of malware. In addition, cloud-based analysis allows suspicious files to be uploaded to the developer’s servers that enables quick and more accurate identification of threats. It is worth noting that these files are often blocked before they are executed, which prevents the system from being infected.
It is also extremely useful to have dedicated protection against ransomware. This type of threat often disguises itself as Trojans or downloaders, so multi-layered protection is crucial because a single mechanism can fail.
Many solutions have special mechanisms to detect and block ransomware. Additionally, it is essential that a product has a firewall that monitors network traffic and blocks suspicious connections.
Multi-layered protection is designed to increase efficiency in protecting against a variety of threats and minimize risk. The more advanced endpoint protection solutions are, the better they can protect against different types of malware. That is why it is worth investing in comprehensive security solutions that are able to meet the growing challenges of cybersecurity.