In January 2023 we conducted a next edition of the long-term Advanced In The Wild Malware Test in the new year. The test aims to select security products that respond well to threats over a long period of time – we avoid issuing opinions on the basis of one test.
Starting from this edition, in order to stand out more from other testing laboratories as precursors in the security industry, we introduce a new comparative feature – the so-called “response time to a threat and resolve security incident” – Remediation Time. We will measure it for each tested product for home and business to further highlight the differences between protection software when confronted with threats in the wild (coming from the Internet).
So far, as you might have noticed in previous editions, almost all protection products have achieved detection of samples in the wild at a level equal or nearly equal to 100%, and the differences concerned the so-called level of blocking malware which is divided into:
- The PRE-Launch level applies to detecting malware samples before they are launched in the system
- The POST-Launch level applies to concern the analysis when a virus has been launched and blocked by tested products
- The Fail level
We have introduced an additional indicator showing differences in protection between products to the Advanced In The Wild Malware Test since January 2023. It is the so-called REMEDIATION TIME which is the time of malware detection from the introduction to the system by a browser, through the launch to detecting and resolving security incident.
Remediation Time occurs only at the POST-Launch level, i.e. it applies to the analysis when a virus has been saved on a hard disk, launched, and blocked by a tested product. Note that we do not calculate the time to respond and resolve an incident at the early level of blocking in a browser (PRE-Launch) because it is always 0s (preventive protection).
Remediation Time for each sample is used to calculate the average time to respond and resolve an incident from the whole test in a given edition. In addition, we collect the maximum Remediation Time, as well as the shortest time to detect and resolve an issue with malware.
Let us explain this for Webroot Antivirus in the following example:
- In January 2023 we used a total of 447 malware samples
- For the calculation of Remediation Time we include 249 samples for the POST-Launch level.
- Average Remediation Time during of the analysis in memory or scanning in the cloud was 125,321 seconds (one hundred twenty-five seconds and three hundred twenty-one milliseconds).
- The longest time observed (Remediation Time Max) was 351,810s.
- The shortest time to response and resolve (Remediation Time Min) was 0.000s (a threat was blocked immediately after launch).
Why we investigate the time to response and resolve a threat? (Remediation Time)
Some tested security solutions offer IT teams responsible for security in an organization specific tools for searching threats (EDR-XDR) and making decisions about blocking an incident.
Remediation Time expressed in seconds will vary depending on the product, configuration, availability of the infrastructure with information about threats provided by a developer, as well as the stability of the network connection in which the test is carried out. It is worth considering these aspects when analyzing the results.
In our test, we try to configure an application to automate this decision, i.e. we need to assume that after taking a certain action by a protection product, it is considered successfully blocked for a given malware sample. It is necessary because this series of tests is automated, and only this way it is possible to check the effectiveness of security on hundreds of malware throughout the entire month without human interference.
Remediation Time indicates to you or your company that the threat has been at least identified during the attack phase and at least partially stopped.
We want to be more transparent than other testing laboratories, that is why we share such detailed information. We already know from the comments that our approach to analysis is appreciated by users who have knowledge in the field of computer security. On the other hand, we try to explain everything in simple way to non-technical users. Not without a reason, the developers we work with appreciate such an analytical approach because we help them improve their software, and thus increases end users security.
Test of security software in January 2023
In January 2023 we used 447 malware samples to test threats in the wild originating from the Internet that every user can come across.
Tested solutions:
- Avast Free Antivirus
- Bitdefender Antivirus Free
- Avira Antivirus Pro
- Emsisoft Business Security (for business)
- Immunet Antivirus (Free Edition)
- Malwarebytes Premium
- Malwarebytes Endpoint Protection (for business)
- Microsoft Defender
- Webroot Antivirus
- Xcitium Internet Security
- Xcitium ZeroThreat Advanced (for business)
Test of protection against Internet threats in numbers
We have prepared the following summary based on the logs collected from the Sysmon tool in Windows 10 and database with collected information from the test:
- 11 security solutions took part in the test.
- Ultimately, we used 447 unique URLs with malware.
- Exactly 381 malware samples were hosted with HTTP.
- Websites encrypted with HTTPS (in theory – safe) contained 66 malware samples.
- During the analysis, each malware samples took on average 28 potentially harmful actions in Windows 10.
- Most malware originated from servers located in the Netherlands, Russia and USA.
- .com, .org, .za domains were mostly used to host malware.
- The average detection of URLs or malicious files at the PRE-Launch level was 58%.
- At the POST-Launch level, file detection was on average 38%.
HTTPS can be used to spread malware. Remember that the SSL certificate and so-called padlock next to the URL does not guarantee the security of the website!
Malware and URLs in the test
The so-called “existence of malware in the wild” is very short and ranges from a few seconds to several days. From experience we can say that the time is counted in minutes: harmful URLs with a link to a file vanish as quickly as they appear.
For this reason, as many as … of them were not available at the time of downloading to the testing systems with Windows 10 and protection software installed. If a file cannot be downloaded from the network (does not exist) it is impossible to test it. Then there is a new attempt to download another malware which is queued for testing. But before that in order not to complicate, we verify every such file relying on statistical (Yara rules) and dynamic analysis (full run in Windows). We invite all interested readers to the methodology – step 3.
Configuration of the system and security software
We usually use the default settings. If you want to learn more about obtaining malware, analysis on Windows, and the entire procedure algorithm, please check out our methodology.
Detailed results
Key information and all test results are available on the page RECENT RESULTS.
Conclusions from the test for the so-called Remediation Time
Remediation Time is an individual measured value for determining the response time of security product to a threat. In the Advanced In The Wild Malware Tests, we need to configure a product to enable it to make the best possible decisions – it is usually moving a threat to quarantine or blocking connections using a firewall along with collecting logs. In organizations employing security specialists, Remediation Time can be significantly greater due to the decision-making process, so the less time it takes from the intrusion of malware into the system until it is noticed and resolved, the better. From the point of view of the end user, the most important thing is to detect a 0-day threat or suspicious behavior, and resolve incident implications.
We would like to point out that Remediation Time should be analyzed in combination with all features of security software as the individual protection technologies from developers may differ significantly.
The differences are in detail, so we encourage carrying out own analysis when choosing software. The effectives of protection is very important, but also other aspects such as warranty, technical support, user-friendly interface, performance, online banking protection, management console, EDR module, encryption with VPN, and other product’s features.