Cyber Transparency Audit
General Methodology
Methodology Audit, Q1 2023 - Q2 2024
The testing organization plugs into the static infrastructure of the cybersecurity solution provider. He performs additional work and obtains permission to validate the results. It issues a certificate confirming compliance with the methodology.
General Description and Audit Inclusion Criteria
The methodology will change as often as it is available to everyone who wants to participate in the discussion of its development. This is the first version of the methodology for the audited producer in Q1 2023, which was prepared in cooperation with the Cyber Transparency Forum group and may change in the future with the participation of other producers, which we will inform about in subsequent audits.
Changes are necessary because, depending on the data that can be provided to us by producers, we want to adapt the methodology to all protection solutions as good as possible in order to extract the maximum benefit from the statistics for end customers as well as the producers.
Moving towards transparency is a natural direction set by the producers of the software with a public source code. As a testing unit, we want to set new trends in the area of cybersecurity, so we invite producers who want to take participate in the audit to contact us.
I. Requirements for Producers
- The Cyber Transparency Forum invites every Producer who has telemetry data from at least 100,000 (one hundred thousand) devices to participate.
- The testing organization will require access to data that meets at least the recommended security policy settings. These may include so-called hardened settings. It is not allowed to audit data from devices with disabled security modules, e.g. antivirus, SSL scanning, unless this is a recommended policy.
- It is allowed to participate in the audit of a device with the following settings assigned:
- Recommended policy: These are the required workstation security settings to meet the minimum security proposed by the producer. The IT solution provider must recognize these settings in the delivered static infrastructure so that there are no situations where the auditor checks data from policy devices with key product features disabled.
- Hardened policy: The protection settings have been increased by the administrator or focused on maximum security using a predefined hardened policy prepared by the producer.
- The producer will create access to statistical data through an API or a graphic panel. Access to the data should be possible in the following time periods:
- access to data from the last 1–3 hours,
- access to data from the last 24 hours,
- access to data from the last week,
- access to data from the last month,
- access to data from the last full quarter.
- The producer agrees to publish the results online. Through transparency, it is possible to better understand the strengths of the product and develop ways to detect and fix weaknesses in the product that may pose a high risk.
- The producer must prepare the data in accordance with the following section, “5. Required telemetry data” with at least the recommended telemetry data that will be part of the overall analysis performed by the Auditor.
- Other requirements are specified in the regulations of the Cyber Transparency Forum.
II. Requirements for Auditors
- Any entity that tests IT solutions, the so-called test lab, can become an auditor.
- An auditor, at the auditing stage, will communicate with the Producer and, if possible, will indicate errors that lie or prevent the completion of a full audit. The producer is obliged to repair them immediately, at the latest 2 weeks before the end of the audit.
- The auditor may propose adding new functions and data to the static infrastructure and using the new data in the same or subsequent edition of the audit.
Audit Test Component
This is the general information that must be met by the Producer and the Auditor in order to effectively validate the data.
I. Minimum Requirements of Data for Endpoint Security.
To carry out the transparency audit, we set the goal of validating data from at least 100,000 (one hundred thousand) end devices, regardless of the operating system. It is not required to include all devices in the statistics, because it may be a business secret, or it may be technically difficult for the Auditor to separate. Therefore, in order to carry out the audit correctly, we require:
- Telemetry data from a minimum of 100,000 end devices.
- Including only those endpoints that are secured with a protective agent with at least the default (predefined by the producer) or hardened (modified “up”) security policy.
The producer should care about making its customers aware of the applied protection settings. In specific situations, the system administrator may reduce the security level, so we want to exclude devices with a reduced level of protection from the audit.
III. Required data from the Producer about devices
Secured endpoints with the producer’s software must provide the central system with the following telemetry data that will allow the audit to be prepared:
- The total number of devices.
- The number of devices containing potential malicious activity.
- Number of devices with confirmed malicious activity.
- The number of devices that meet security standards (not infected).
II. File information and metadata from endpoints
We include the following file telemetry in the audit. The data provided by the Producer should be anonymized so as not to disclose confidential information.
- The total number of unknown files that turned out to be clean.
- The total number of unknown files that turned out to be malware.
- The total number of unknown files that turned out to be PUAs/PUPs.
Obtaining information about files and metadata will allow better interpretation of the audited data, which can contribute to:
- Searching for trends over time to better understand if a given threat or cyberattack is increasing or decreasing in the statistics.
- Seeing how threats increase over time, such as the type of malware detected or the number of malicious domains blocked.
- Repair the effects of the attacks so that the producer’s end customers can receive recommendations to improve the security fundamentals, such as increasing the use of SSL certificates or strengthening e-mail security.
- Comparison of data with industry averages or other sources to see how the producer and their customers are exposed to a given type of attack and how they deal with them.
Audit Test Process – Step by step
- The cybersecurity service provider provides an API or graphical interface with anonymous telemetry data to the testing organization.
- The required data must be able to be sorted according to a specific time schedule, e.g. last hour, week, month, quarter.
- Telemetry data should include basic information about devices and files.
- The producer will create a process whereby research organizations will have access to this data at any time during the audit. Data must be updated on an ongoing basis, at least once a day. The auditor can use test malware to ensure the authenticity of the data.
- The producer will provide the same set of data through the software API or the proposed data format (JSON, XML, CSV, etc.), so that the testing organization can connect to the back-end of the producer’s statistical infrastructure and download this data for further analysis.
- The time of the audit depends on the schedule agreed upon between the producer and the auditor. During this time, both sides cooperate and any aspects that require clarification are urgently resolved. There may be “spot checks” or additional questions to the producer about the data provided by the research organization.
- The testing organization will publish an audit report. The report will be available to everyone at the end of the reporting period.
- Finally, the research organization issues a certificate confirming the successful validation of the audited telemetry data.