Network vulnerability scanners benchmark
Discover which network vulnerability scanners deliver both comprehensive CVE coverage and accurate findings against an independent testbed.
This benchmark compares the best network scanners to help you understand their detection capabilities and limits on a specific set of vulnerable environments.
Overview of this network scanners benchmark
Necessary
Why compare the best network scanners?
This benchmark fills a critical gap in the security community by providing a comprehensive, transparent, and up to date evaluation of network vulnerability scanners.
It gives security specialists visibility into the actual performance of the most popular network vulnerability tools and contributes to a stronger decision-making process.
Transparent
What kind of findings does this benchmark include?
This benchmark includes the results of rigorously testing 7 of the most popular network vulnerability scanners against 160+ vulnerable environments.
The key findings come with the full list of results, including the ports, technologies, and CVEs tested.
Realistic
How does this benchmark evaluate the scanners?
The benchmark examines detection availability and detection accuracy - and provides clear and transparent criteria to ensure a fair, standardized comparison across open-source and commercial tools.
Focusing on remote detections, the benchmark provides a realistic assessment that helps security specialists understand which scanners provide the most reliable remote detection capabilities.
Methodology for this network scanners benchmark
Testing period: January 2024
Detections: all scanners were updated with the latest detections as of January 2024.
167
vulnerable environments tested
17
instances used in the testing setup
7
popular network scanners evaluated
128
environments with remotely detectable CVEs
39
environments with non-remotely detectable CVEs
2 criteria
detection availability and accuracy
Criteria for evaluating network scanners in this benchmark
Testing period: January 2024
Detections: all scanners were updated with the latest detections as of January 2024.
Detection availability for all environments was calculated as
=* 100count of the detection existence
total number of vulnerabilities
Detection availability for vulnerabilities that can be detected remotely was calculated as
=* 100count of the detection existence
total number of vulnerabilities analyzed that can be detected remotely
Detection accuracy for all environments was calculated as
=* 100count of the vulnerabilities detected
total number of vulnerabilities
Detection accuracy for vulnerabilities that can be detected remotely was calculated as
=* 100count of the vulnerabilities detected
total number of vulnerabilities analyzed that can be detected remotely
Watch the benchmark breakdown
Watch David Bors, Security Research Engineer, break down the results of this benchmark in just 4 minutes, and see how the best network vulnerability scanners did on every test category.
Download the benchmark![Network vulnerability scanners benchmark [2024] - methodology & results](/_vercel/image?url=%2Fimages%2Fdavid-bors-benchmark-video.webp&w=1536&q=100)
The results of this network scanners benchmark
Detection availability vs. detection accuracy
Most commercial vulnerability scanning solutions state they have detections for the majority of vulnerabilities in the 167 environments tested for this benchmark.
However, the tests revealed inconsistent performance and notable disparities between detection availability and actual accuracy. This applies both to results from tests against all vulnerable environments and to those that exclusively focus on remotely detectable vulnerabilities.
Tool rankings
Overall detection
The Pentest-Tools.com Network Vulnerability Scanner consistently outperforms both commercial and open-source tools, coming in first, while Qualys Vulnerability Management takes 2nd place, with ProjectDiscovery Nuclei in 3rd.
Remote detection
Nuclei surpasses Qualys Vulnerability Management, indicating superior performance in detecting remotely exploitable vulnerabilities.
Commercial scanners
Except Nexpose, most commercial scanners show similar detection availability, supporting their claims about comprehensive vulnerability coverage.
Real world relevance
This benchmark covers a subset of each scanner's capabilities, so it is important to understand that it is not a perfect representation of the scanners’ global performance; factors such as user-friendliness, system integration, and support quality can be equally relevant to their overall performance but there is no unbiased evaluation method for them.
The most accurate network vulnerability scanners
When looking at their performance across all vulnerable environments, there is a similar level of detection availability among the major commercial key players, with two exceptions.
- Tenable’s Nessus displays the highest discrepancy between detection availability and actual accuracy: it claims 55.09% detection availability but achieves only 18.56% accuracy across all vulnerable environments.
- Rapid7’s Nexpose is the biggest exception, as it is not possible to differentiate between local and remote checks in their vulnerability database.
- Qualys’ Vulnerability Management and ProjectDiscovery’s Nuclei show lower variance, with actual detection rates ~25% lower than their stated availability, highlighting a more reliable detection capability.
- The highest performance across the entire range of vulnerable environments goes to the Network Vulnerability Scanner on Pentest-Tools.com, which stands out through its near perfect consistency between availability and actual detection.
The best network scanners for remote vulnerability detection
Remote checks (or black-box assessments) are the main scope of this benchmark because:
- they offer the only transparent and objective way of evaluating multiple network scanners in a way that’s publicly verifiable
- they are particularly attractive for threat actors
- they make up the majority of critical CVEs that pose major real-life challenges to both organizations and the cybersecurity community
- they align with the current needs of security practitioners, who need to simulate and understand an external attacker's viewpoint.
To evaluate the scanners’ performance in this context, this benchmark included 128 environments with remotely detectable CVEs in dozens of different technologies.
- There is a slight shift in ranking when moving from detections across all vulnerable environments to remotely detectable vulnerabilities.
- Nessus once again exhibits the highest discrepancy between detection availability and actual accuracy, identifying only 22.66% of the vulnerable environments it claims it can detect.
- When focusing exclusively on remotely detectable vulnerabilities, ProjectDiscovery’s Nuclei moves up to 2nd place, pushing Qualys Vulnerability Management down to 3rd. This indicates Nuclei has a slightly broader scope for this particular use case.
- The Network Vulnerability Scanner on Pentest-Tools.com is once again a high performer, coming up as the best network scanner for remote vulnerability detection.
See the full benchmark results and all the data behind them
Download the benchmark
10 things you can do with this benchmark of network scanners
1. Assess tool accuracy
Compare the precision of popular network scanners - especially in identifying remotely detectable vulnerabilities - to better understand their attack surface mapping capabilities and their detection limits.
2. Optimize tool selection
Choose the best network vulnerability scanner for specific environments based on detection rates. Improve the quality of your incident response plans and data for compliance audits to boost your overall security posture.
3. Evaluate detection capabilities
Assess the capability of the most popular network scanners to find remotely exploitable vulnerabilities and validate the extent of their CVE coverage. This also provides hard data on vendors’ investment and focus on adding new modules for detecting high-risk vulnerabilities.
4. Enhance data quality for security processes
Identify gaps in detection across the most popular network vulnerability scanners, both commercial and open-source to find opportunities to improve your security toolset and workflow.
5. Benchmark internal tools
Compare the effectiveness of your in-house tools against commercial and open-source vulnerability scanners regarded as industry standards.
6. Develop scanning strategies for vulnerability assessment
Tailor your network scanning approaches based on each scanner’s strengths and weaknesses to ensure maximum coverage and accuracy - and prioritize your time and resources.
7. Select tools for penetration testing and security testing
Choose network vulnerability scanners that can truly inform and complement manual testing, improving cybersecurity specialists’ ability to detect critical security issues and speed up remediation - especially in time-sensitive contexts.
8. Develop training materials
Create training scenarios using the data in this benchmark to help new-entry security practitioners develop stronger criteria for validating a network scanner’s capability and accuracy.
9. Vendor negotiation
Leverage the findings from this network scanners benchmark in negotiations with vendors or use it to provide feedback for product improvements you find necessary for your needs.
10. Community sharing
Share the findings in this benchmark to improve collective knowledge within your team, organization, or community. (Your feedback on ways to improve this benchmark is more than welcome!)
See the full benchmark results and all the data behind them
Download the benchmark
Benchmark FAQs
Which network vulnerability scanner is the most accurate?
The benchmark highlights which network vulnerability scanners have the highest detection rates for security vulnerabilities across 167 environments and dozens of different technologies.
The Network Vulnerability Scanner on Pentest-Tools.com is the most accurate across all these environments and throughout 128 environments with remotely detectable CVEs.
How were the network scanners evaluated?
The network scanners were tested against a predefined set of vulnerabilities and scenarios to measure detection availability and detection accuracy.
The benchmark involved testing 167 vulnerable environments across 17 instances used in the testing setup.
Which port ranges did you use for the tests in this benchmark?
Most tools were initiated with their default settings, targeting the entire TCP port range (1-65535).
Can these network scanners detect all types of security vulnerabilities?
No network scanner can detect every vulnerability in all network devices.
The benchmark shows each tool’s strengths and weaknesses across 167 environments and dozens of different technologies, comparing their coverage and accuracy.
Which network vulnerability scanner is the fastest?
The benchmark does not include scanning speed as a performance metric.
Are open-source network scanners included in the benchmark?
Yes, the 7 selected network vulnerability scanners included in the benchmark are a mix of open-source and commercial scanners. This selection focuses on the most popular network scanning tools in the cybersecurity industry.
Are cloud-based network scanners included in the benchmark?
The benchmark includes both on-premise network vulnerability scanners, such as Nessus Professional, Rapid7 Nexpose, and OpenVAS, as well as network vulnerability scanners that run on a cloud platform such as the Pentest-Tools.com Network Vulnerability Scanner and Qualys Vulnerability Management.
How does this benchmark help with compliance?
This benchmark provides insights into which network scanners are better at detecting vulnerabilities relevant to specific compliance requirements, especially those focusing on mapping and monitoring the attack surface of an organization.
Are these network scanners suitable for small businesses?
The benchmark includes both lightweight and scanners with more robust functionality, catering to various business sizes and network security requirements. This is why it evaluates 4 commercial scanners and 3 open-source ones.
How often is this benchmark updated?
This benchmark was first published on May 24, 2024 and will be updated annually.
How does pricing affect the choice of network scanners?
The benchmark focuses on the most popular network scanning tools in the security industry and does not include pricing information, which is highly variable based on customer needs and can be subject to change depending on the vendors’ commercial decisions.
How user-friendly are these network vulnerability scanners?
The benchmark does not address user interface and ease of use for each scanner because there are no objective criteria for comparing these factors, important as they may be for the decision-making process and overall performance of the tool.
Do these network scanners support automated scanning?
All listed network scanners - Nessus Professional, Nmap, Nuclei, OpenVAS, the Pentest-Tools.com Network Vulnerability Scanner, Qualys, and Rapid7 Nexpose - support automation through features like scheduled scans, scripting, and integration with CI/CD workflows.
How reliable are the benchmark results?
This benchmark’s results are based on standardized tests across a transparently described setup. They provide a reliable comparison based on publicly available data about test results. Download the benchmark (PDF) for the complete details.
Can these network scanners integrate with other security tools?
The benchmark does not address integration capabilities with other security solutions because there are no truly objective criteria for comparing these factors, important as they may be for the decision-making process and overall performance of the tool.
How does the benchmark handle updates and new vulnerabilities?
For those wishing to independently confirm the findings, it is essential to acknowledge that all scanners were updated with the latest detections as of January 2024, which is when the benchmark tests were run.