New: We have begun the process of migrating the Windows 11 test environment to the latest version, 25H2 Pro. Most security solutions will be gradually transferred to this version of the system. Some products are still running on earlier Windows builds and will only be fully migrated from January 2026. To ensure stable implementation, it is necessary to refine the tools and automation mechanisms that work with individual products, which is why the process has not been carried out simultaneously for all solutions.
New: After consulting with several vendors, we have decided to change the browser used in testing. Firefox has been replaced by Opera, which is based on the Chromium engine. Opera ensures full compatibility with modern web technologies and, at the same time (unlike other Chromium-based browsers), continues to use the system file save window (“Save as”). This is a key feature that allows us to perform fully automated, non-invasive sampling in a Windows environment.
Improved: In the November edition, we are expanding our monitoring of LOLBins and TTPs exclusively to techniques confirmed in real campaigns. We are observing over 70 binaries used in living-off-the-land, from administrative tools and interpreters to data transfer modules and cloud processes. At the same time, we track over 70 critical registry keys related to persistence, service configuration, and system manipulation. Such a wide range ensures precise detection of abuse and allows us to evaluate security products against real, current threats.
New: We have added extended logs in a human-readable format for software developers, which will allow them to quickly trace the activity of a malware sample in the system, as well as the response of the tested software to the launched threat. In the future, we will expand this functionality to collect even more telemetry data of various types.
Improved: New LOLBINs have been added to the processes used by malware. The configuration will now include additional legitimate processes and tools that can be used by malware to hide itself or download additional payloads. These will be used to build statistics as part of the Advanced In-The-Wild Malware Test series in the “LOLBins in statistical terms” section.
Improved: Updated the so-called malware response detection indicators for some of the tested security products, which we search for in logs to confirm detection or removal of the threat.
New: We have updated our website with new landing pages for all test categories.
Improved: We made numerous updates to the Sysmon configuration to more accurately detect changes in Windows made by malware and to better detect the response of security solutions under test.
Improved: Made one change to the Firefox browser to fix a sometimes occurring bug related to the lack of an active window for running malware samples from the Vmware API via legitimate Windows processes.
Improved: Updated Windows 11 to the latest version as of January 31, 2025.
Improved: Made some changes to the testing application to make technical logs more understandable to humans during analysis.
New: An OCR engine has been added to recognise text from screenshots, allowing us to better and more accurately identify potential malware files before they qualify for testing. This allows us to quickly reject installers, PUAs, PUPs and corrupted files.
We also use OCR to capture alerts and messages from security software. This gives us an additional opinion on how the tested solution reacts to malware. This provides vendors with further irrefutable proof that malware has or has not been detected.
Improved: We have added a SHA256 comparison of the malware sample on each machine with the security solution installed. This is to avoid errors where product X can be tested against another sample downloaded from the same URL.
Improved: The CSV reports provided to producers as evidence have been improved. They now include more detailed information from the database.
Improved: We have reorganised and improved the design of our home page and sub-pages relating to the services we offer.
Improved: We have added new features to the web application to make it quicker and easier for Testers to complete the test summary.
New: Added a delay to the download of a file in the browser so that some installed extensions can load correctly before starting the test on a malware sample.
Improved: Sorting test data on Linux is now better structured. A few additional changes will allow us to conduct technical investigations faster.
Improved: We have located a bug in the latest version of Sysmon 15.12 that causes the Sysmon service to crash. The bug has been reported to Microsoft. The developer does not have a fix for this yet, so we have implemented an individual workaround (after weeks the issue has been fixed in Sysmon 15.14).
Improved: We have changed the way the Remediation Time is calculated. This will not have a significant impact on the results, as so far every solution has been tested under the same conditions and on the same principles.
Namely, until now, the Average Remediation Time was calculated only for samples blocked at the Post-Launch level. From now on, the Average Remediation Time is calculated from the Pre-Launch level, and it is always 0 seconds (detection of an incident with blocking a sample before its launch) which is included in the average for all samples in the month.
New: At the request of one of the developers, we added a custom rule to the testing application which is responsible for generating additional logs in a machine with the installed solution, so that it is easier to make decisions about protection against a specific malware sample.
New: We expanded the testing application with a feature to generate a log with information about installed versions of software on Windows using Powershell.
New: Our web application for managing telemetry data from the tests through a browser was supplemented with additional functionalities to speed up the work for testers when summarizing results for a given period. For example, we added faster search for potentially unwanted samples such as PUPs, PUAs, Adware, and others.
Improved: for one of the developers, we improved a custom search of logs for selected threat indicators.
Improved: we improved Firefox’s automatic update for our everyday testing.
Improved: we added the Sysmon rules with additional detection indicators of several new solutions.
New: We implemented a new feature in the testing application that allows to collect telemetry data from selected logs of the tested solution. Developers can receive more feedback on blocked threats upon special request.
New: We integrated an external scanner of malware samples so that they are better classified even before being included in the test. This change should have a positive effect on the rejection of samples that are not suitable for testing for a variety of reasons. Scanning technology is provided by mks_vir (but also Arcabit) – Polish developer of security software.
New: It is now official that the May edition of the Advanced In The Wild Malware Test is compliant with the AMTSO standards, of which we have been a member since April 2023.
Improved: This month’s changes also include generating improved CSV reports from the test with additional telemetry data that we share with all developers.
New: Since the May edition of our test, we have added an individual summary view for each Vendor on their request and that coming from the community. Better marketing perception.
New: We have started the procedure to join a group of experts from AMTSO (Anti-Malware Testing Standard Organization).
New: Added so-called Remediation Time to comparison.
New: Joining a new cybersecurity expert group, known as Cyber Transparency Forum to develop new standards on transparency and endpoint cyber security.
Improved: Added Yara rule to the testing system to detect and classify malicious activity.
Improved: Added some rules to the testing system to detect malicious activity in Windows.
Improved: Added some new Antivirus Indicator Rules to the testing system.
Improved: Added new rules for better detection of malicious activity on Windows using Sysmon software.
New: Development of the so-called Threat Landscape from the test:
New: Replaced the old SHA-HTML table where we assigned a malware sample to a product and results. Instead, we simply provide a CSV file for everyone to download and browse for better transparency.
New: At the request of the community and Vendors, we have abandoned hosting malware for testing using a local DNS and HTTPS. We replaced these with URLs in the wild downloaded in a browser.
New: We changed the graphical presentation of the results.
New: At the request of our community, we simplified complex levels of blocking malware: Level 1, Level 2 at PRE-Launch, and Level 3 at POST-Launch.
New: Implementation of the visual admin panel for faster and better management of telemetry data from tests in the backend infrastructure.
New: Change of the Process Monitor software for logging changes in Windows to the Sysmon tool with a custom configuration.
New: Added Yara rules to pre-classify malware samples and reject invalid ones.
Improved: Replacement of the old NPM package for DNS with a new package.
Improved: Time zones in the database and files shared with Vendors as feedback from the test.
Improved: Changing file logs structure on backend infrastructure.
