Differences in software for PC protection in CheckLab.pl tests (January 2020)

27 lutego 2020

In January 2020 the CheckLab.pl organization prepared a list of popular solutions to protect home computers and work stations of micro, medium, and large enterprises. Among the tested solutions there are 8 specialized top-class products for end customers, and also 2 solutions for companies — Comodo Advanced Endpoint Protection and Webroot Business Endpoint Protection.

In the January comparison, the CheckLab employees would like to draw the attention to:

(1) Differences in blocking threats that arise from the protection mechanisms implemented in a tested security solution.

Products which take part in the test do not receive additional points for blocking threats at the early stage. In theory, it is better when a threat is detected early. However, in the final evaluation, the most important is the effectiveness of blocking samples of malware regardless of detection stage of a sample.

Solutions for computer protection can be built differently — for example, the product X can scan protocols HTTP/S, and the product Y may not be able to do that. That is not something bad, and certainly should not affect the final result because such differences arise from operation logic of security solution. Developers use various ways to secure the working environment, and that is why none of these technologies should be favored.

The CheckLab organization in January 2020 prepared comparative charts where you can see differences in blocking malicious software. Colors from dark to light blue, including red show average and totalized rating in blocking all samples that were included in the test.

Individual levels represent:

One of the sample combinations of computer protection solutions.

In the security report from CheckLab, interpretation of the whole test is important because experts think that differences between products should be shown, so whether modern technologies fulfill their role.

Giving a specific example — if the product X blocks all threats thanks to the huge portion of static metadata in the cloud, but one and unique sample of malicious software leads to data loss or computer infection with banking trojan, is such solution worth recommending.

(2) The effectiveness of signatureless protection is shown as the Level 3 in charts, and in the table.

The analysis of IT solutions should also be started with signatureless protection. Entries in the antivirus database are byte sequences that are characteristic for a specific sample of malware.

On the market of protection solutions there are products that do not use signatures. In reputable products, the signatureless protection is still effective, but works best as support for modern ways of defense against attacks. In the CheckLab tests such differences are shown, so anybody interested can check if antivirus product can detect a threat without signatures.

The signatureless protection is marked with the Level 3 in the charts and tables, whereby malware sample file goes into the operating system where a tested product has all options active, and is installed with default settings. The protection solution has a chance to stop a threat in a browser or after getting onto a hard drive, before launching a sample, and also after executing malicious file. None of the protection components are disabled by testers, because a product has been designed to operate with all security elements active.

Signatures are effective when it comes to known techniques of network attacks and detecting malicious files. However, creators of malware cloud them using various masking methods. For this reason, reputable solutions must have more advanced ways to detect attacks and threats. Developers of security solutions should know that CheckLab will not make it easier for them. Every aspect of the protection will be carefully checked.

Comparison of protection in January 2020

The test described below uses harmful software, tools, and techniques of bypassing security that are used in real campaigns, although additional modules have not been considered, such as: online banking protection, anti-theft module, encrypted VPN tunnel, home network monitor, storing and generating passwords, webcam protection, and other optional modules of good software to protect the operating system.

Explanation of the “Advanced In The Wild Malware Test”

The name of the „Advanced In The Wild Malware Test” perfectly reflects its character. The source of malicious software are honeypots located on all continents of the world. We collect malware, among other, for the Windows system. Samples captured in attacks are checked on the basis of over 100 patterns before they are qualified for testing. These patterns allow us to determine whether a potentially dangerous file is actually a threat to the Windows 10 Pro operating system.

In January 2020, in the third edition of security tests of the „Advanced In The Wild Malware Test” we verified the effectiveness of detecting and blocking malicious software of 10 solutions for protecting computers.

The tests lasted continuously the whole January 2020. The list of tested solutions is as follows:

  • Avast Free Antivirus
  • Avira Antivirus Pro
  • Bitdefender Total Security
  • Comodo Advanced Endpoint Protection
  • Comodo Internet Security
  • G DATA Total Security
  • Kaspersky Total Security
  • SecureAPlus Pro
  • Webroot Antivirus
  • Webroot Business Endpoint Protection

The results of January 2020

CheckLab as the first organization in the word shows such detailed information from tests to all interested people. We share checksums of malicious software by dividing them into protection technologies that have contributed to detect and stop a threat. According to experts, this type of innovative approach of comparing security will contribute to better understanding of differences between available products for consumers and enterprises.

A chart describing differences between individual products is available at http://checklab.pl/en/recent-results

Comparison of protection solutions in January 2020.

In the fifth edition of the test, we have granted the BEST+++ certificate to:

  • Avast Free Antivirus
  • Avira Antivirus Pro
  • Bitdefender Total Security
  • Comodo Advanced Endpoint Protection
  • Comodo Internet Security
  • G DATA Total Security
  • Kaspersky Total Security
  • SecureAPlus Pro
  • Webroot Antivirus
  • Webroot Business Endpoint Protection

Levels of blocking malicious software samples

The CheckLab employees are probably pioneers in this regard — they show more detailed diagnostic data than any other testing institution, including the largest such as AV-Comparatives and AV-Test. Blocking of each malware sample by tested protection solution has been divided into a few levels:

  • Level 1 (P1): The browser level, i.e. a virus has been stopped before or after it has been downloaded onto a hard drive.
  • Level 2 (P2): The system level, i.e. a virus has been downloaded, but it has not been allowed to run.
  • Level 3 (P3): The analysis level, i.e. a virus has been run and blocked by a tested product.
  • Failure (N): The failure, i.e. a virus has not been blocked and it has infected a system.

The results of blocking each sample are available at http://checklab.pl/en/recent-results in the table:

Published checksums of malicious software have a beneficial influence on transparency of the tests, and build trust to the testing organization.

The products and Windows 10 settings: daily test cycle

Tests are carried out in Windows 10 Pro x64. The user account control (UAC) is disabled because the purpose of the tests is to check the protection effectiveness of a product against malware and not a reaction of the testing system to Windows messages.

Additionally, the Windows 10 system contains installed the following software: office suite, document browser, email client, and other tools and files that give the impression of a normal working environment.

Automatic updates of the Windows 10 system are disabled in a given month of the tests. Due to the complicated process and the possibility of a malfunction, Windows 10 is updated every few weeks under close supervision.

Security products are updated one time within a day. Before tests are run, virus databases and protection product files are updated. This means that the latest versions of protection products are tested every day.

Malicious software

We have used 685 malicious software samples for the test, consisting of, among others, banking trojans, ransomware, backdoors, downloaders, and macro viruses. In the contrast to the well-known institutions that verify the security usefulness, the CheckLab tests are much more transparent because the organization share the full list of malware samples.

During testing, all solutions have access to the Internet. The experts of CheckLab use real working environments in a graphic mode that is why the results of individual samples may differ from those presented by the VirusTotal service. The CheckLab organization points that out because inquisitive users may compare our tests with the scanning results of VirusTotal. It turns out that differences between real products installed on Windows 10 and scanning engines on VirusTotal are significant. We have explained these discrepancies in the article “How we test antivirus? The making of CheckLab.pl, a website dedicated to security tests”.

Information about CheckLab

The CheckLab organization was founded in July 2019 by the AVLab.pl company operating since 2012 in the industry of informatics security. The primary objective of the CheckLab organization is to test security usefulness, and issuing certificates confirming the protection effectiveness against malware, and also provide results to public information while ensuring the maximum transparency of the tests. In the studies, the CheckLab employees use malicious software, tools, and techniques of bypassing security that are used in real cyberattacks. Even though the project called CheckLab has existed only for a few weeks, the organization already cooperates with the largest companies in the security industry.

PODZIEL SIĘ:

Share on facebook
Share on twitter
Share on linkedin
Share on email
AUTOR: Mateusz Kurlit
guest
0 komentarzy
Inline Feedbacks
View all comments

Newsletter

BĄDŹ ZAWSZE NA BIEŻĄCO!

Newsletter

ZAPISZ SIĘ NA POWIADOMIENIA
BĄDŹ ZAWSZE NA BIEŻĄCO!
zapisz się

Bitdefender GravityZone Webinarium

Dowiedz się, co eksperci mówią o GravityZone

POLECANE PRODUKTY

YUBIKEY

Klucze zabezpieczające

100% ochrony przed phishingiem

Newsletter

BĄDŹ ZAWSZE NA BIEŻĄCO!

Newsletter

ZAPISZ SIĘ NA POWIADOMIENIA
BĄDŹ ZAWSZE NA BIEŻĄCO!
zapisz się

Newsletter

BĄDŹ ZAWSZE NA BIEŻĄCO!

Newsletter

ZAPISZ SIĘ NA POWIADOMIENIA E-MAIL I ZAWSZE BĄDŹ NA BIEŻĄCO!
zapisz się

POLECANE PRODUKTY

YUBICO

Klucze zabezpieczające

SILNE UWIERZYTELNIANIE DWUSKŁADNIKOWE, WIELOSKŁADNIKOWE I BEZ HASŁA

Bitdefender GravityZone Webinarium

Dowiedz się, co eksperci mówią o GravityZone