In May 2022, we completed the 18th edition of the Advanced In the Wild Malware Test. We carry out this analysis for our readers systematically in order to show the effectiveness of security from various developers in the long run. This time we used 1925 samples of malware to test mechanisms of protection against threats from the Internet which are verified before they are added into our test base.
In May 2022, we checked the following software (always the latest version – our testing system updates antiviruses and signatures once a day):
- Avast Free Antivirus
- Avira Antivirus Pro
- Bitdefender Total Security
- Comodo Advanced Endpoint Protection (for businesses)
- Comodo Internet Security
- Emsisoft Business Security (for businesses)
- Malwarebytes Premium
- Mks_vir Endpoint Security (for businesses)
- Microsoft Defender (Windows 10 with SmartScreen disabled – explanation below)
- CatchPulse (formerly known as SecureAPlus Pro, learn more)
- Webroot Antivirus
Unlike other testing laboratories, our tests are fully transparent, so we provide a complete list of virus samples and other technical details described in our methodology. Key information and test results are available at https://avlab.pl/en/recent-results.
Summary of the May edition of the test
The May edition has been a little different for us because since this month we have permanently introduced to virtual machines tagging downloaded files from the Internet by the operating system in order to maintain compliance with the system security feature MOTW (Mark Of The Web). It turns out that when comparing the results to previous months, this change did not have a clear effect on blocking malware. Nevertheless, it was necessary to adapt the methodology to the new requirements.
Naturally, malicious code creators can use certain file formats in order to bypass MOTW. Microsoft tries to make this difficult by providing protection for the Office suite against documents containing malicious macros. However, in most cases, executable files in Windows downloaded from the Internet are tagged with metadata e.g., Zone.Identifier with a specific value known more as Mark-Of-The-Web (MOTW).
Files tagged with MOTW metadata are isolated and cannot perform certain actions. That is good because, for example, MOTW files are blocked by the Microsoft Defender SmartScreen technology in Windows 10 and Windows 11 that compares them with a list of allowed executable files.
In the test, we used Chrome because it is the most popular browser, so if a file is not known/trusted, the embedded SmartScreen module should prevent it from running and warn a user:
Please note!
In May we experimented with Microsoft Defender. We disabled SmartScreen in order to verify the true effectiveness of protection against malware without the use of technology that is not part of the antivirus, but the operating system. Microsoft’s results were not satisfactory because 98,6%.
In July 2022, in the next edition the SmartScreen technology for Microsoft Defender will be enabled in order to compare results. Experience on the example will tell whether the level of protection is significantly higher. We will present the conclusions in the next summary.
New! Level 1 + Level 2 = Pre-launch
At the suggestion of our community, starting this month we introduce changes in the naming of security levels but keeping the previous classification just in case. So far, we have been identifying the detection levels in the following way:
Level 1 shows early blocking threats in a browser or on a hard drive.
If this fails, the next is Level 2: a virus is scanned by the antivirus based on signatures when moving from X to Y folder. Only if such protection exists. In this study, there are many test cases when samples have not been verified by developers yet, so the next level of analysis is crucial.
Level 3 represents modern protection without any local signatures because a virus is run in the operating system. It is the most dangerous situation but needed because it shows the true effectiveness of protection against 0-day threats that are unknown for developers of protection software.
Change from May 2022
Level 1 and Level 2 are combined into single level: Pre-launch (previously Level 1 or Level 2)
Level 3 remains the same level with new name: Post-launch (previously Level 3)
Reclassification of levels is for marketing to simplify the methodology and make it easier to understand the tests that users have suggested. Thank you!
So the new classification concerns detecting malware samples:
- Before they are launched in the system (Pre-launch)
- and samples which are blocked after launch (Post-launch). This is the most dangerous situation but experience shows that such cases require tests in the field.
Certain antiviruses of the next generation intentionally do not have protection in a browser (Pre-launch). Sometimes they do not have traditional protection based on signatures (Post-launch), so without proper interpretation, such tests could favor other protection solutions. Not ours!
In the tests, we do not award negative or positive points for early blocking threats. Security product must stop a threat in any way – provided by a developer. The final result is whether or not this has been achieved.