Website vulnerability scanners benchmark

In the benchmark testbeds, vulnerabilities are defined based on established standards, including OWASP Top 10 and the Common Weakness Enumeration (CWE). In addition, issues identified by scanners that serve as effective defense-in-depth measures, such as anomalies in the Content Security Policy (CSP) header, were also included.

While this approach introduces a degree of subjectivity, it aligns with the practical responsibilities of security engineers who evaluate scanner reports during real-world assessments.

The validation of each vulnerability reported by the scanners was conducted manually to ascertain its accuracy as a true positive. We encourage stakeholders to contact us should there be any concerns or discrepancies identified in our measurement of results, as we aim for the highest level of precision in our evaluations.

In choosing benchmark testbeds, the preference for open-source options was guided by the goal of making it easy for independent testers to replicate the setup.

A benchmark is only as valuable as its level of transparency and the lengths to which it allows for results verification within the cybersecurity community.

The Active Scan policy was configured to a Default Alert Threshold of Low and Default Attack Strength to Insane. Additionally, injection was enabled in all the input vectors. At the time of writing, these were: URL Path, URL Query String (with the option to add parameters), POST Data, HTTP Headers, and Cookie Data.

The benchmark evaluates detection and accuracy rates across two targets - Broken Crystals and DVWA (Damn Vulnerable Web Application) - which include multiple vulnerable technologies. Acunetix from Invicti, the Pentest-Tools.com Website Vulnerability Scanner, and Burp Suite consistently reported higher accuracy in vulnerability detection.

Scanners were tested against Broken Crystals and DVWA (Damn Vulnerable Web Application), focusing on true positives, false positives, and false negatives to ensure a comprehensive and transparent comparison. The goal was to simulate the perspective of an application security engineer looking to evaluate a DAST (Dynamic Application Security Testing) tool.

Each web app scanner was manually configured to use their most comprehensive crawling strategy and to attempt to use all the vulnerability detections they have.

Where available, the REST API swagger files that defined the API were specified, as well as the GraphQL endpoint to be scanned.

Some vulnerabilities were present in endpoints protected by authentication, so, where possible, each scanner was configured to run the security scan as an authenticated user.

No scanner can guarantee to detect every vulnerability. The benchmark shows each tool's strengths and weaknesses across a varied range of technologies.

Scanning speed is not included as a performance metric in this benchmark.

Yes, the 6 selected website vulnerability scanners included in the benchmark are a mix of open-source (ZAP) and commercial scanners (Invicti Acunetix, Port Swigger Burp Scanner, Pentest-Tools.com Website Vulnerability Scanner, Qualys - WAS/Web Application Scanning, Rapid7 InsightAppSec).

This selection focuses on the most popular web application scanning tools in the cybersecurity industry.

The benchmark includes both on-premise website vulnerability scanners (Rapid7 InsightAppSec) and tools that run on a cloud platform such as the Pentest-Tools.com Website Vulnerability Scanner, with some vendors offering both versions for the same product (e.g. Invicti Acunetix, Port Swigger Burp Scanner).

This benchmark provides insights into which web application scanners are better at detecting vulnerabilities relevant to specific compliance requirements, especially those focusing on mapping and monitoring the attack surface of an organization.

Absolutely! The scan results in this benchmark help security teams select suitable scanners by highlighting detection capabilities, detection gaps, and vendor commitments to adding detection for high-risk vulnerabilities in complex web app architectures.

Yes, the evaluation includes a detailed analysis of false positives reported by each scanner. This aspect is critical as it impacts the trustworthiness and efficiency of the scanner in real-world scenarios.

High false positive rates can lead to unnecessary investigations, wasting time and resources. By assessing the rate of false positives, the benchmark helps security professionals understand which tools provide the most accurate and reliable results, ensuring better resource allocation and more effective vulnerability management.

The benchmark uses standardized criteria to ensure consistent and fair comparison across different scanners.

The benchmark includes both lightweight and scanners with more robust functionality, catering to various business sizes and web application security requirements.

This benchmark was first published on May 29, 2024 and will be updated annually.

The benchmark focuses on the most popular website scanning tools in the security industry and does not include pricing information, which is highly variable based on customer needs and can be subject to change depending on the vendors’ commercial decisions.

The benchmark does not address user interface and ease of use for each scanner because there are no objective criteria for comparing these factors, important as they may be for the decision-making process and overall performance of the tool.

This benchmark’s results are based on standardized tests across a transparently described setup. They provide a reliable comparison based on publicly available data about test results. Download the benchmark (PDF) for the complete details.

False negative rates are critical for risk assessment as they indicate the vulnerabilities that scanners fail to detect, which can leave web apps exposed to potential threats.

High false negative rates undermine the reliability of a scanner, as attackers can exploit undetected vulnerabilities, posing significant security risks.

The benchmark measures the false negative rates to provide a clear understanding of each scanner’s detection capabilities, ensuring that security teams can trust their tools to uncover as many vulnerabilities as possible and minimize overlooked risks.

This metric helps prioritize scanners that offer comprehensive coverage and improve overall security posture by identifying the most effective tools for vulnerability detection.

The benchmark does not address integration capabilities with other security solutions because there are no truly objective criteria for comparing these factors, important as they may be for the decision-making process and overall performance of the tool.

For those wishing to independently confirm the findings, it is essential to acknowledge that all scanners were updated with the latest detections as of February 2024, which is when the benchmark tests were run.