In November, we tested software for protecting personal computers and workstations for the last time this year. It was the 21st edition of the initiative that aims to test solutions from different developers. Threat landscape is dynamic, so we do not slow down the pace, by improving the methodology and tools that are necessary for testing.
Starting from this edition, we will publish more statistical data. We share this information because we want to be the most transparent of all testing laboratories. We already know that our approach to analysis is appreciated by users who have knowledge in the field of computer security. On the other hand, we try to explain everything in simple way to non-technical users. Not without reason, the developers we work with appreciate such an analytical approach because we help them improve their software, which affects users security.
Tests of security software in November 2022
In November 2022 we used 400 malware samples to test threats in the wild originating from the Internet that every user can come across. Compared to the previous editions, it is not much, but let us explain why:
We changed the method of delivering malware samples to the test systems in such a way that we stopped hosting it on our own DNS and HTTP servers. Now we use REAL URLs that contain links to malicious files, and the entire procedure is automated.
In November 2022 all tested products achieved a maximum score of 100% blocked threats in the Advanced In The Wild Malware Test. We have prepared the following summary based on the logs collected from the Sysmon tool in Windows and database with saved information:
- 13 security solutions took part in the test.
- There were 3 solutions for enterprises and 10 for home and small businesses.
- Eventually, we used 400 unique URLs with malware with total of 1745.
- Analysis of usability of virus samples showed that 243 are not useful for the test due to the low harmfulness. In the moment of testing the remaining malware were too small (less than 20kb) or offline, so they were rejected.
- Exactly 315 malware samples were hosted with HTTP.
- Pages encrypted with HTTPS (safe in theory) contained 85 malware samples.
- During the analysis, each malware sample made on average 23 potentially harmful changes in Windows 10.
- Most malware were from servers located in the USA, Netherlands, and Germany.
- .COM, .CLOUD, .NET domains were mostly used to host malware.
- The average detection of URLs or malicious files on the PRE_LAUNCH level was 56,9%.
- On the POST_LAUNCH level, file detection was on average 43,08%.
- In this edition of the test, each product managed to block 100% of samples (regardless of the analysis level).
Interestingly, HTTPS can be used to host malware. Always remember that SSL certificate and so-called padlock next to the URL does not guarantee the security of the website!
Malware and URLs in the test
The so-called “existence of malware in the wild” is very short and ranges from a few seconds to several days. From experience we can say that the time is counted in minutes: harmful URLs with a link to a file vanish as quickly as they appear.
For this reason, as many as 868 of them were not available at the time of downloading to the testing systems with Windows 10 and protection software installed. If a file cannot be downloaded from the network (does not exist) it is impossible to test it. Then there is a new attempt to download another malware which is queued for testing. But before that in order not to complicate we verify every such file relying on statistical (Yara rules) and dynamic analysis (full run in Windows). We invite all interested readers to the methodology – step 3.
Tested solutions and results in November 2022
We intentionally do not provide version numbers because the testing system updates applications once a day:
- Avast Free Antivirus
- Avira Antivirus Pro
- CatchPulse
- Emsisoft Business Security (for enterprises)
- ESET Smart Security
- Kaspersky Internet Security
- Malwarebytes Premium
- Malwarebytes Endpoint Protection (for enterprises)
- Microsoft Defender
- Trend Micro Maximum Security
- Webroot Antivirus
- Xcitium ZeroThreat Advanced (for enterprises)
- Xcitium Internet Security
Unlike other testing laboratories, we try to make our tests fully transparent, so we provide a complete list of virus samples and never share unnecessary technical information.
Key information and all test results are available on the page RECENT RESULTS. A complete list of malware samples with the results for each individual developer is available in MALWARE COMPARISON TABLE.
Configuration of the system and security software
We usually use the default settings. If you want to learn more about capturing malware, analysis on Windows, and the entire algorithm of acting, please check out our methodology.
In the ranking for the best protection software, all developers have demonstrated the superiority of their solutions over a set of malware. Modern security solutions often come with machine learning for detecting threats without using signatures (this is the Post-Launch level), artificial intelligence that correlates events from the entire system, and sandboxes in the cloud.
The leaders in blocking malware at the Pre-Launch level over 90% are: Avast, Kaspersky, Avira, ESET, Malwarebytes. We mean the stage when a threat is blocked before it is run, with an average of the entire test being 56% of all developers.
The leaders in blocking unknown files are: CatchPulse (98,25%), Xcitium Advanced Endpoint Protection (98%), and Xcitium Internet Security (94%), with an average being 43% of all developers.
The differences are marginal, so we encourage carrying out your own analysis when choosing software. The effectives of protection is very important, but also other aspects such as warranty, technical support, user-friendly interface, performance, online banking protection, management console, EDR module, encryption with VPN, and other product’s features.