One of the positive aspects of cooperation with developers, who provides security solutions for SOHO and Enterprise markets, is access to information about attacks and malicious software. From data obtained we learn that threats that are sent with HTTP and HTTPS protocols using drive-by download attacks are becoming an increasing problem. Avira software has blocked over 3 million malicious URLs in the last 12 months. One the other hand, from Kaspersky Lab product community, we learn that in the period from January to October 2018 the number of blocked attack attempts and malicious files using “WWW protection” module embedded in Kaspersky products is more than one and a half billion in the world and 19 million in Poland. The statistics are closely related to anonymous data provided by software installed on the end user devices so the larger community, the more accurate information. Detailed information about attack type were provided by Check Point: there are 325 and 458 cyberattacks related to HTTP and HTTPS protocols conducted each day respectively on organizations in Poland and in the world. In Poland as many as 68% of all attacks represent blocked attempts of communication between malicious software and C&C servers. Every third attack (30%) is responsible for exploiting vulnerabilities in applications or operating systems. The remaining 2% concern malicious websites.

Test przeglądarek
Threats statistics in Poland and in the world.

We observe systematically an increase in attacks through Internet protocols which is why we wanted to test solutions available in the Google Web Store. We realize that many users don’t have adequate protection or use products without malicious content filtering function in web browsers. Therefore, we wanted to test popular extensions for web browsers which capture and filter all network traffic blocking malicious content and software.

Tested solutions

The test was carried out from 10th to 23rd October 2018. During this time, we have checked a level of protection of the following solutions in the Chrome browser based on 1870 unique virus samples:

  • Avast Online Security
  • Avira Browser Safety
  • Bitdefender Trafficlight
  • Check Point SandBlast Agent for Browser*
  • Comodo Online Security
  • Malwarebytes Browser Extension
  • McAfee WebAdvisor
  • Panda Safe Web
  • uBlock Origin**
  • Windows Defender Browser Protection

**The following lists were used: Malvertising filter list by Disconnect​​​​​​​​​, Malware Domain List​​​​​, Malware domains, ​Spam404​​​​​​​​​.

*The „Check Point Sandblast Agent for Browser” solution is available in two variants. The free version only protects against phishing attacks. The commercial version also protects against downloaded malicious files, using threats emulation in a safe environment. Using the Threat Extraction analyzer (TEX), it provides a reconstructed version of a file which is devoid of e.g. malicious macro commands. Such a file can be run without consequences resulting in excellent protection against 0-day attacks.

Check Point Sandblast
Free version of the „Check Point Sandblast Agent for Browser” extension doesn’t have the most important functionality if it isn’t connected to the SandBlast service.

The „Check Point Sandblast Agent for Browser” extension must be connected to a SandBlast device or equivalent in the cloud, SandBlast Cloud services where unknown files scanning is carried out. Those files pose a potential threat to the organization. The extension can be configured in a manner which analyzed file isn’t downloaded or run automatically or by an employee until verification is completed as a result of the drive-by download attack. The extension has been developed to integrate with the leading Check Point service which ensures network security by blocking detection proof malicious software. Free version of the extension didn’t meet requirements for scanning downloaded threats, so we tested a commercial version at the request of the developer.

Description of the procedure

Algorytm postępowania w testach
The tests were carried out on the basis of this algorithm.

The task of our testing system is to automate the management of security solution, configured workstation, and captured malware samples from attacks on the honeypots network. The project core is based on the Ubuntu 16 LTS distribution, while testing system called PERUN has been equipped with modules for analyzing virus samples, correlating and parsing collected logs, as well as managing Windows 10 systems. The PERUN system is combination of NodeJS and Python programming languages. We hope that this solution will delegate the most time-consuming work to a computing power of workstations, allowing us to present the results of an antivirus protection from two areas: against threats in the wild and new security circumvention technologies.

The ideal source of samples is one that provides new and different types of malicious software. In this case, the “freshness” of the collected samples is very important, because it affects the real protection against threats that can be found in the wild. The samples used in this test come from attacks on our honeypots network which are very important tool for security experts. The purpose of traps for intruders, script kiddie or other scripts, is to pretend “victim” (in terms of systems, services or protocols) and save, among others, logs from attacks, including malicious software. We use low and high interactive honeypots that emulate services such as: SSH, HTTP, HTTPS, SMB, FTP, TFTP, MYSQL, and SMTP.

Symu kontrolne wirusów honeypot
Malware collected on the one of the honeypots.
Lokalizacja honeypotów na świecie
Here are the honeypots localizations that we use to obtain malware samples: Canada, USA, Brazil, Great Britain, the Netherlands, France, Italy, the Czech Republic, Poland, Russia, India, Singapore, Japan, Australia, South Africa.

A test that reproduces real user and malware behavior is the best from the point of view of Internet users and developers. Owing to the honeypots, a new collection of viruses is being gathered each day. Before each sample goes to machines with installed security products, it’s thoroughly analyzed. We need to ensure that only “100%” malicious samples will be tested. The situation when a virus will not work in the system because it was programmed for another geographic region will never happen in our tests. Due to this, readers and developers are confident that the malware that has been qualified for testing is able to seriously infect the operating system, regardless of which part of the world it comes from.

Before a potentially malicious sample is qualified for testing, one of PERUN's components checks if the malicious software is causing undesired changes. For this purpose, each virus is analyzed for 15 minutes. The human factor, excluded from the test, does not provide the ability to ensure that e.g. malware will shut down its activity after 60 seconds. We need to set a time threshold, after which we should interrupt the analysis. We realize that there is such a damaging software which, before it is activated, can delay its launch even up to a dozen or so hours. It can also listen for connections to the C&C server on the ephemeral port. There were also occasions when a malicious program was programmed to infect a particular application or waited for visiting a website. For this reason, we have made every effort to ensure that our tests are as close to reality as possible, and samples that are “unsafe” are not included in the test virus database.

After analyzing each malicious application, malware activity logs are entered into the system event log. This solution helps us to export the necessary information from the system log to CSV files. Based on the collected data, the algorithms developed determine whether a particular sample is undoubtedly malicious. We publish the partial information from each analysis on the CheckLab website in a user-friendly and manufacturer-friendly format. The details are provided to the manufacturers and, on request, to distributors.

Windows 10 Local Security Policy
Examples of audit policy settings. Windows 10 provides the access to very detailed information about system events.

We can determine with certainty whether a security product has stopped malware by means of signature or proactive security components. Analyzing logs is very time-consuming and therefore, the algorithms that implement this process have been developed.

Windows Event logi
Examples of logs collected from Bitdefender software activity.
Ransomware Windows 10 Event
Examples of logs collected from ransomware activity.

Registered logs, i.e. about changes to the file structure and directories, system registry, task schedule, logging to systems and network shares, as well as process network communications provide the necessary information about system changes introduced by malicious software. For example, if an infected Word document with several scripts (visual basic, cmd, powershell) will be started and a file from a website will be downloaded, the virus will be saved in %TEMP%, and launched, then the activated detection of each operating system component in the audit settings policy will forward such information to the event log. As a result, even the slightest modification performed by malicious software will be registered. Whether it's a keylogger, backdoor, rootkit, trojan, macro virus or ransomware – the system event log will record all actions.

In this test, the algorithm was limited to checking whether the malware was downloaded to the system. If not, it means that it has been blocked in a browser by a tested product.

  1. The Perun system launched every morning all workstations with installed products. It performed an update of virus signatures databases or files of a tested solution within 30 minutes.
  2. After making sure that workstations with installed products are ready for testing, snapshots were taken.
  3. All operating systems with installed protection products were launched.
  4. A malicious software sample selected for testing was downloaded by Google Chrome web browser on all workstations.
  5. If malicious software was blocked at an early stage, it was marked in the database with a special identifier.
  6. At the last stage, the testing system waited for completion of the analysis on all workstations, and then went back to previous points to analyze a next virus sample.

Results and conclusions of the test

A demand for free solutions which protects computers is high, so we couldn’t skip this type of security tools. Most of the tested solutions achieved a slightly different result which to some extent reflects the fact that developers share information about threats. However, there is no doubt that the Check Point SandBlast Agent for Browser extension has gained the leading position (remember that its free version protects only against phishing).

Test rozszerzen do przegladarek

The certificates were granted based on the following score:

  • 100-98% best+++
  • 97-95% best++
  • 94-92 good+

Blocking malware at the level of 95-100% is a very good result, but interpreting it differently shows the other side of the coin. Let’s take Avira Online Security solution as an example. 1797 blocked samples is a lot, but there are still 77 potential attempts to run malicious code. A browser-level protection is very important because it can filter out known virus types and malicious web applications which download additional droppers to an operating system. An installed active antivirus agent is necessary to protect against modern malware distribution vectors. Remember that not only HTTP/HTTPS protocols are involved in spreading threats. We include a large number of attacks on an account of email protocols (IMAP, POP3), file sharing, as well as IIS network services for Windows Server that can be remotely attacked using exploits, injecting a malicious load into the operating system. Securing all potential areas in the operating system is crucial therefore web browser extensions that filter threats are insufficient but at the same time necessary, so they should be supported by a local protection.


Was the test sponsored?

The test wasn’t sponsored neither by Check Point company whose solution achieved the maximum result, nor by any of the other producer.

When were the tests carried out?

The tests were carried out in the period of 10 – 23 October 2018.

Do tested protection software have access to the Internet?

Of course.

How did you choose products for the tests?

We were guided by the popularity. Unfortunately, we’re limited by the performance of the server, which we use as a platform for our tests. If the demand for the tests increases, we will for sure be able to test more products.

How to join your tests?

If you are a manufacturer, distributor, or developer and you would like to join our tests, simply contact us. In response, we'll ask you to provide guidance on the proper operation of your product. We will also arrange other details that are needed to develop an automated malware detection procedure.

Is it possible to join the tests informally?

Yes. If you think your solution is not fully developed or you are afraid of getting low evaluation, you can join the test for a trial period. The protection results will not be made available to the public. In addition, we will provide you with the necessary details to help improve the effectiveness of your product's protection.

Are the tests free?

It isn’t true that charging for the preparation and publication of tests is synonymous with manipulating results. Once caught up in fraud, the organization will never again be able to rebuild its position and credibility. The very small fee collected is treated by both parties as remuneration for work and improved user safety. Without financial help, maintaining the infrastructure, continually improving the procedures and necessary tools needed to conduct the tests would not be possible. In return, we offer access to detailed information and samples used in the test to each creator. The tests are conducted under the AVLab brand that exists for 5 years (soon CheckLab as well) – marketing benefits are the added value.

Is all information available publicly?

Not all. Manufacturers have insight into more detailed data. Other information that is necessary to visualize the results remains available to any reader.

Do you carry out other tests?

Yes, but we don’t have a test schedule developed. In large comparative tests, we focus on checking the protection against sophisticated cyberattacks. Preparing such tests, cooperation with developers to improve security and producing a final report takes far more time than automatic verification of the protection on the basis of malicious software samples.

Do you perform tests and prepare reviews at the request of the developer or distributor?

Of course. We can prepare detailed reviews that will be published on AVLab and CheckLab. Interested developers of software and hardware are encouraged to contact us.

Do you share samples of malware?

Yes. If you want to access the virus database, please contact us. This service is payable. The reliability of our tests is always at the top of the list, so the database that you have access to will be already checked by antivirus software.

In what environment do you carry out tests?

The tests are performed in virtual machines. Virtualization is increasingly being used in VDI (Virtual Desktop Infrastructure) work environments. We use scripts that further "harden" the system, which makes it more difficult for viruses to detect virtualization. We realize that some worms may detect their launch in a virtual system, so we only take into consideration those samples that have been thoroughly verified before. We don’t include malicious software that is able to detect well-hidden virtualization. This is not an ideal solution, but we are doing everything we can to approach the tests professionally and reconcile these aspects at the same time.

How do you make sure that a virus sample is really malicious?

On the basis of detailed logs. We have developed over 100 indicators that are likely to point out any malicious changes infect the system. The more such indicators are in the logs, the greater the chance that a particular sample is malicious.

Based on what data do you decide whether the product has blocked the threat?

Based on the collected data, the algorithms developed determine whether a particular sample is undoubtedly malicious or whether it has been stopped by a security product installed. We can with certainty determine whether the protection program has stopped malicious software using the signature or proactive protection components. Analyzing logs is very time consuming, so we have developed the algorithms that implement this process.

What are your plans for the future?

We want to provide users with an online platform for sharing information about threats. Systematic improvement of already developed tools and methodology is a natural process, so we are working on adding another protocol providing the machines with samples of malicious software and adding other types of honeypots into the network.

Can I use the tests published on AVLab?

Of course. Please appreciate our work in improving the security and provide the test source.

If you are a supplier of software or hardware designed for monitoring networks and end user devices, we can carry out so-called unit tests. We will adjust ways to circumvent a product to get a full view of the security effectiveness. In dedicated tests, we will use malicious software, tools, and techniques of bypassing that are used in real attacks. We invite you contact us.

Adrian Ścibor

Podziel się


Trang Pittsley pon., 11-03-2019 - 01:19

Excellent blog and Great information :) Keep this work, good luck.

Dodaj komentarz