Skip to main content
The ML Classifier uses machine learning to identify pages that return a 200 OK status code but actually represent error or non-existent content (soft 404s). This reduces false positives in scan results.

​
What is a soft 404?

A “soft 404” occurs when a web server returns a successful HTTP status code (typically 200) for a page that doesn’t actually exist or contains error content. Common examples include:
  • Custom “Page not found” pages that return 200 instead of 404
  • Error pages styled to match the site design
  • Placeholder pages with generic content
  • Pages that redirect to a homepage or parent directory
These soft 404 pages can pollute scan results with false positives, making it harder to identify real findings.

​
How it works

The ML Classifier analyzes HTTP responses to distinguish between:
  • Legitimate pages: Real content that should be reported as findings
  • Soft 404 pages: Error pages disguised as valid responses
The classifier uses multiple detection techniques:
Compares page content against known 404 response patterns for the target site.
Analyzes page content for common error indicators and patterns.
Uses trained models to classify ambiguous responses.

​
Usage in Website Scanner

The ML Classifier is integrated into the Website Scanner.

​
Enabled tests

The classifier runs automatically when the following tests are enabled in the Initial Tests section:
The ML Classifier is enabled by default for these tests.

​
How it improves results

Without soft 404 detection, these tests might report hundreds of false positives, pages that appear to exist but are actually custom error pages. The ML Classifier filters these out, so you only see legitimate discoveries.

​
Usage in URL Fuzzer

The ML Classifier is also integrated into the URL Fuzzer.

​
How it works

When fuzzing for hidden files and directories, the URL Fuzzer sends many requests that will return error pages. The ML Classifier:
  1. Analyzes each response
  2. Identifies soft 404 patterns
  3. Filters out false positives from the results
The URL Fuzzer doesn’t generate findings directly, but its results are cleaned by the ML Classifier to show only legitimate discoveries.

​
Data handling

The classifier runs on our own self-hosted models, on infrastructure we control in Europe. This is the case for both the Europe and United States data regions.

​
What the classifier receives

To decide whether a page is a soft 404, the classifier needs the HTTP response the scanner got back. That means:
  • The URL that was requested
  • The HTTP status code
  • The response headers
  • The redirect chain, if the request was redirected
  • The page content
Scripts, styles, comments, and most layout markup are stripped out before the model sees the page, along with repeated elements and very long attribute values.

​
Training and retention

Your scan data is not used to train our models. The classifier was trained on public data and de-identified scan results. These models classify rather than generate, so there is no risk of invented content in the result. The classifier returns a probability for each label and nothing else. For how this fits with our other AI features, see the AI data policy.