Every false positive we have shipped had the same cause

Four separate false positives, four different vulnerability classes, one root cause: the scanner accepted something it had caused as proof. A template matched a marker it had put in the request. An SSRF probe counted a callback our own client made. Here are the four, and the five rules that now stop them, each enforced by a test rather than by review.

A scanner is judged on what it finds and lived with on what it gets wrong. We keep a list of every false positive Crossfyre has produced, with the cause written next to it, and at some point the list stopped being a list of unrelated mistakes and became one mistake made four times.

In every case, the scanner accepted something it had caused as evidence that the target was vulnerable. Four different vulnerability classes, four different bits of code, the same error underneath all of them.

The four

A remote code execution finding against a page that says hello

Shellshock, the 2014 bash bug, is normally detected by putting a payload in a header and looking for evidence it ran. Our template put a marker in the User-Agent header and looked for that marker in the response.

Mutillidae's home page prints Browser: <your user agent>. So the page echoed the marker back, exactly as it echoes anything you send, and the template reported critical remote code execution against an application that had simply been polite.

The same shape, against a script that echoes a path

A WebLogic CVE template fired against a small Python script whose entire behaviour was to print the request path back. Different template, different year, same mechanism: the proof of the vulnerability was a string the scanner had supplied.

Seven file-read findings against a documentation page

Local file inclusion oracles often look for markers that indicate PHP source has been read. A static .txt documentation file, full of <?php code examples, contained those markers permanently, on every request, with or without a payload. The oracle never checked whether the payload had changed anything. Seven findings, all of them the page being itself.

An SSRF confirmed by our own web client

This is the subtlest one and the most alarming, because it defeated a technique we consider gold standard.

Server-side request forgery is confirmed out of band: you point a parameter at a listener you control and wait to see whether the target contacts it. A callback is meant to be unfakeable, because only the server can make it.

The endpoint under test was an open redirect. It answered 302 pointing at our listener. Our own HTTP client, which follows redirects by default, obediently fetched it. The callback arrived. The oracle saw a callback and reported SSRF.

The question that catches all of them

When we sat down to write rules, the useful thing turned out to be a single question to ask of any oracle:

What did the target have to do that it could not have done by echoing us back, and that our own client did not do on its behalf?

If there is no answer, you do not have an oracle. You have a mirror.

The five rules

All five are enforced in code. This matters more than the rules themselves, because a rule that lives in a review checklist is a rule that holds until the reviewer is tired.

1. A template may not match on a word it transmits

If a template sends a string in a header, a body or a payload list, it may not use that same string as its proof. A test walks every built-in template and fails the build when one does.

Some detections are legitimately about reflection: that is what cross-site scripting is. Those opt out explicitly with a reflection tag, which turns "I did not think about this" into "I decided this, in writing, and it is visible in the diff".

2. A leak must be a change from the baseline

Every branch of the file-read oracle now fetches the endpoint with no payload first and compares. A marker present in both the clean response and the payload response proves nothing about the payload. This one rule kills the entire class the documentation page belonged to.

3. A marker must be something the target had to compute

This is the strongest of the five. Instead of looking for a string we sent, send a problem and look for its answer.

Shellshock now proves execution with arithmetic. The request carries a multiplication of two large primes and never carries the result. A page that echoes the request back returns the multiplication, which we ignore. A server that actually ran a shell returns the product, which nothing else can produce without doing the work.

sent:      the expression
echoed:    the expression        -> not a finding
computed:  the product          -> a finding

4. A callback must not be one we caused

Out-of-band payloads go out on a client that does not follow redirects. If the endpoint answers with a 3xx pointing at our own listener, that ends the probe rather than confirming it. The redirect is the target telling us it will not be doing the fetching.

5. Never probe off-scope

The last one is not about correctness. During the same exercise a donate form on a deliberately vulnerable application had a field pointing at a well-known payment provider. The injector followed it and sent SQL injection payloads to a third party's live production service.

What this costs

Honesty about the trade: every one of these rules makes the scanner quieter, and some of that quiet is real vulnerabilities going unreported.

Rule 4 means an SSRF behind an open redirect will not be confirmed by the automatic path. Rule 3 means a target that is genuinely vulnerable but cannot reach the network, or cannot compute, produces nothing. Rule 2 means an oracle needs an extra request per check. We think that is the right direction, because a false positive costs a human being an hour and costs the tool its credibility permanently, but it is a direction and not a free win.

There is also a limit on how well we can measure any of this. Our benchmark scores precision against declared negative controls, and a false positive only counts if a control predicted it. During this exercise a critical false positive sat inside a run reporting 100% precision, because no control happened to cover it. Any precision number measured that way is optimistic by construction, including ours.

The full run, with the answer keys and the misses classified by cause, is in what a scanner finds when the answer key is written first.

A finding you cannot reproduce is not a finding. That is the whole product. See what survives