We brought AI Pentests to DEF CON. Here's what practitioners actually said

Nearly 9 in 10 security practitioners told us AI-generated findings still need real manual validation before anyone can act on them. If you've ever sat with a stack of AI-generated findings, not sure which ones to trust, you already know the feeling behind this post.

Our offensive security work goes back to 2017. The AI-specific part of it, figuring out where AI actually helps versus where it's just noise, is more recent, starting with our Machine Learning Classifier in 2025, engineered specifically to cut fuzzing false positives, a purpose-built model for one narrow problem. The state of AI pentesting survey, our study of 158 security practitioners, just confirmed, at scale, something we'd already been running into ourselves.

In August, we took the product built on that foundation and put it in front of the people the survey was actually about: the practitioners at DEF CON, and a wider look at the industry at Black Hat US.

Here’s the overview, for anyone who’d rather read it at their own pace and revisit certain parts later than rely on catching the livestream. 

Robert Tanase, our Lead Product Manager, joined our host Jan Pedersen on Office Hours #10 to talk through what happened at both events, and what it means for AI Pentests, powered by Specter.

The number that started it all

• 87.8% of practitioners who use AI to generate security findings say those findings need significant manual validation before they can trust them

• 62% rank report quality and developer-readiness as where AI pentesting tools already perform best

• 39% put data handling, where vulnerability data gets stored and who can see it, in their top three buying criteria

The pattern behind that first number is specific. Robert put it this way on the session:

LLM models are very good at the beginning of an engagement and at the end, in the reporting phase. […] The mid part, the hardest part, still requires a lot of work, because the LLMs do tend to hallucinate. - Robert Tanase, Lead Product Manager, Pentest-Tools.com

The quote describes where the risk actually sits. A model doesn't struggle with an unfamiliar language or business context at the start of an engagement, it can crawl an application and draft a clean report regardless. What it struggles with is the middle: telling a real SQL injection from one that's three authentication steps and a rate limit away from being reachable at all.

The data-handling question came up everywhere

The 39% who flagged data storage as a top concern weren't a fringe group. Robert heard the same thing directly from customers, and again during the conferences:

It came both at the two conferences we spoke about, DEF CON and Black Hat, but also from our internal customers that do ask us about where is the data processed and stored, and where those LLM models are hosted and running. - Robert Tanase, Lead Product Manager, Pentest-Tools.com

Our answer: "we are using frontier LLM models that are hosted and running in the US", which lines up with where a large share of our customer base already sits, almost half are US-based.

The full survey is free to read, no signup, no gate.

DEF CON and Black Hat US

If you've never been to either, the contrast is bigger than most people expect. Black Hat US runs closer to a trade show: big commercial booths, a lot of swag, sales conversations, badge scans. DEF CON is built around the security community itself, with a different atmosphere and mission, hacking villages on everything from medical devices to lock picking, a social engineering village with lines out the door, and a total of 10 exhibitor booths against Black Hat's hundreds.

We had a booth at DEF CON, and it mattered a lot to us. We were the first company from Romania to exhibit there. After almost a decade building the company without outside investment, from three people in 2017 to a team of 65 today, standing in that room as one of only 10 exhibitor booths felt like being taken seriously by an industry we'd been building toward for years. That's where AI Pentests got its first public run in front of the community. Black Hat was more of a listening exercise, walking the floor, seeing what the rest of the industry is building.

By Robert's read, the two crowds cared about different things. Some people wanted the outcome: a report with no false positives, full evidence, ready for compliance. Others wanted the mechanics: which LLMs, whether the team built its own harness, how it handles model misbehavior or context rot. Neither group was wrong to ask, they were just looking at the same product from different angles.

Why we built another AI pentesting tool

There are already a few tools in this category. The reason to build one more: a decade of deterministic tooling and offensive security work behind it, backed up by a few concrete things:

• Our Head of Professional Services holds the GIAC Security Expert certification, on top of a stack of other GIAC and OffSec certifications

• Our research team is an active part of the disclosure community year-round, our research lead alone is credited with over 120 CVEs, and this year’s work includes everything from a WordPress pre-auth chain to full database compromise, to a critical, unauthenticated authentication bypass in phpBB, alongside a second high-severity OAuth issue in the same software

The same team and research feed into AI Pentests. It's not a wrapper around a frontier model, it's a decade of offensive security knowledge with a model doing part of the work.

Four claims and what is actually new

We're standing behind four things: 

• Battle-tested in bug bounty programs and live engagements

• Delivers only validated exploits, with zero false positives

• Compliance-ready reports

• Deterministic coverage

The last two aren't new, customers already know us for deterministic tools and clean reports. 

Zero false positives is the new claim, specific to AI Pentests, and it’s backed by testing across our own private beta, public and invite-only bug bounty programs, and live engagements run by our services team. Confirmed finds along the way include PepsiCo, F5, and the US Department of War.

The mechanism behind this claim: an in-house LLM harness proposes and executes attack scenarios, and accuracy comes from controlled exploitation - nothing makes the report until that step actually confirms it.

How an AI pentest works

Set the scope: a target URL, black box or grey box, optional authentication or API keys, and any boundaries you want respected.

From there, the agents run recon, form hypotheses, and chain what they find. 

Robert's own example from the session: an SQL injection that extracts a username and a hashed password, then an attempt to crack that hash and authenticate with the cracked credentials to go deeper into the application. In other cases, that might mean signing up for a free account instead, to test what's only visible once you're logged in.

Another agent cross-checks every finding before it counts, evidence, reproduction steps, and all. Once it's done, one click produces a PDF report ready for the compliance programs you already work with.

We've put one line on everything from DEF CON: it runs the pentest, you run the show. Scope, authorization, and judgment stay with you throughout.

What is in the AI Pentests report

• Security teams get the full pentest results: findings, agent activity, and the reasoning behind each decision

• Engineers get remediation guidance mapped to CVSS scores and the OWASP Top 10

• Leadership gets a section built for a five-minute read: how many vulnerabilities, what they are, at a glance

• Auditors and customers get evidence that supports SOC 2, ISO 27001, and DORA, among others

Does AI Pentests replace the person doing the testing?

It's the question that came up at the booth, in the survey, and again during the live Office Hours Q&A, where one attendee asked it directly: 

Will AI Pentests eventually push far enough to replace the human in the loop entirely?

Robert's answer: "We don't think that humans are replaceable." Parts of the job don't fit the way LLMs currently work. Research still requires novel thinking a pattern-matcher can't produce. Complex, business-logic-heavy applications still need someone who understands what the client actually needs, not just what the code does. The goal is taking the repetitive parts off a tester's plate, so more of their time goes toward exactly that kind of work.

Where AI Pentests go next

We're now in closed early access. If you want a closer look at what it does and how it works, head to the AI Pentests page, you can request early access from there.

Watch the full session for everything that didn't make it into this recap.