What a scanner finds when the answer key is written first

Nineteen deliberately vulnerable targets, each with a machine-readable answer key written before the scan so nobody could move the goalposts afterwards, plus one target rigged so that every finding is wrong by construction. Final score: 48 of 48. The unflattering part is that none of the six we clawed back came from teaching the engine to detect anything new. It could already find all six. It was busy breaking the applications it was measuring, timing itself against a brake it had set an hour earlier, and being scored by a harness that threw away the one field the score is matched on.

Point a scanner at nothing but broken applications and the only number you can produce is recall. Recall is easy. Report everything as vulnerable and you score 100%, which is roughly how this industry ended up with tools that fire on a stiff breeze.

So limeyard includes a target where the correct score is finding nothing at all. The rest is nineteen targets across eight kinds, each carrying a truth.yml answer key that was written before the scan ran.

Everything below is measured. Where a number is missing, it says so. Where a miss turned out to be our benchmark being wrong rather than our engine being wrong, it says that too, and that happened more often than we would like.

The result

48 of 48 in-scope answer-key entries found. No finding matched any declared negative. Every one of the twelve targets carrying in-scope entries is at full marks.

Before anyone gets excited: a full score against a fixed lab measures the lab at least as much as it measures the scanner, and this lab is finite and now completely solved. The number worth watching is what happens the next time somebody adds an entry. We are publishing this because how it moved is more interesting than where it stopped.

It moved from 42 of 48, and the honest version is a bit embarrassing. Not one of the six came from teaching the engine to detect anything it could not already detect. Hand it the right request and it found every one of them in under four seconds. What changed is that it started getting handed the right request, and stopped vandalising the applications it was supposed to be measuring on the way there.

Twenty-four entries sit in that second table and none of them counts in either direction. They are not propping the number up and they are not dragging it down. If one ever becomes scoreable it moves into the first table and the denominator moves with it.

When this was measured

A scorecard that quietly mixes runs is a scorecard reporting a number for work it never did. So every app now records when it was scanned, and the scorer refuses to shut up about the ones that do not. Every result feeding the figures above came from the same day, 11 September 2026.

Two are still flagged, and being able to write that sentence honestly cost a code change. AndroGoat carries no timestamp at all, and no in-scope entries either, so it contributes nothing in either direction. Mutillidae contributes six of the forty-eight and is only part stamped even after a re-scan, because one of the files it is scored from was written by a pass that no longer exists.

The scorer used to call that app current. It took the newest timestamp it could find per application, which answers "when did we last touch this" rather than the question a scorecard is actually asking. An application is only as fresh as its stalest half. It now takes the oldest stamp and names anything that comes back as a mix:

mutillidae   PART-STAMPED: no stamp in results_authed.json
             (scored alongside stamped data, so this app is older than it looks)
no timestamp: androgoat

That is a small bug in a measuring tool, found while writing up the measurement, and it had been quietly making the results look better maintained than they were. It was the fourth time in this exercise that the harness turned out to be the broken thing. We stopped being surprised somewhere around the second.

Precision, and what the number hides

mirage is the inverted target. Fifteen baits, and the correct score is falling for none of them. Five vendor SQL error pages with no database behind them, four reflection contexts that escape properly, a template-looking stack trace that never evaluates, an SSRF bait that never fetches, a traversal bait that never opens a file, a redirect that never sets Location, a command-injection transcript that never runs a shell, a public-by-design admin page, config-shaped JSON stuffed with fictional secrets, an always-slow endpoint, and a soft 404 that answers 200 for every path in a wordlist. None of the fifteen fired.

That sentence was true and it was resting on less than it sounded like. Until 13 September the injection oracles had never run against this target at all. mirage links every bait as a bare path with no query string, so the crawl found the paths and no parameters, and the injection stage skipped the whole application with no injectable endpoints discovered. "None of the fifteen fired" was measured by the vulnerability scan alone, while five fake vendor SQL errors, four correctly escaped reflection contexts, a template stack trace and an SSRF bait all sit there aimed squarely at oracles nobody had pointed back at them.

The nine baits that declare a method, a path and a parameter are now seeded straight from the answer key, so the oracles are aimed at exactly what the key says is there. First run: 2,715 requests across SQL injection, command injection, file inclusion, template injection, cross-site scripting, CRLF, SSRF and race conditions, and zero findings. The claim did not change. What changed is that it is the injection engine reporting nothing now, rather than an engine that was never asked.

It is not literally silent, though, and an earlier draft of this page said it was. That is the exact sin the rest of this report keeps complaining about, so here is the correction. mirage reports four things: no X-Frame-Options, no Content-Security-Policy, no X-Content-Type-Options, and nginx detected. Three info and one low. All four are true statements about the application, and not one of them is a bait. Zero findings would have read better on a slide and been wrong.

Five false positives were found and fixed earlier in the exercise, four of them one root cause: the scanner accepting something it had caused. A template matched its own echoed payload and reported remote code execution against an application that simply prints your browser string back. A file-read oracle matched a documentation page's own PHP examples. An SSRF probe counted a callback that our own redirect-following client had made against an open redirect. All are now blocked by rules a test enforces rather than by review.

What happened when we checked by hand

That callout is easy to write and much harder to act on, so we started acting on it. Every finding the answer key does not cover now gets read by a person, and the verdict is recorded together with the method used to reach it, because a verdict with no method behind it is just an opinion wearing a timestamp.

Fifty-six had been reviewed when this started. Forty-eight were real and eight were false. The rest were unreviewed, which is worth saying out loud, because an unreviewed finding is not a passing one. It is a finding nobody has looked at.

The first row is the one a vendor puts on a slide. It does not budge, because the controls behind it were being handled correctly the whole time. The second row is the one that told us anything at all. The gap between those two rows is the entire argument of this section, and the fact that only the second one moved is the proof.

Seven of the eight are one cluster on DVWA, and the cause is the one we keep running into. Our own scanner had submitted the application's password-change form. That queued a flash message in the session, and the boolean oracle read that message appearing and disappearing as the content difference it was hunting for. Checked by hand on an untouched copy, a tautology and a contradiction come back identical. The scanner had changed the target and then measured its own change.

The eighth is a different failure and deserves its own name: it is stale. A Shellshock finding against Mutillidae that the current template does not produce. Re-tested by hand, the template sends a multiplication and waits for the product, Mutillidae echoes the expression straight back, the product never shows up, nothing fires. The finding survives purely because result files pile up and that app's entry predates the fix.

Both causes got fixes: an inert control in the boolean oracle, and an engine that refuses to submit credential-changing forms at all. A fix nobody re-measured is a hope, so we re-ran the fleet.

What the re-run said

Seven of the eight are gone. DVWA now reports three SQL injection findings and every one sits on a real query parameter. Nothing on User-Agent, nothing on Referer, nothing on the page that sets the application's own difficulty level. The engine says out loud what it refused to touch, which is the part worth stealing:

inject: skipped POST /password_change.php without testing it: `password_curr`
alongside a confirmation field makes this a credential-change form, and
submitting it would set the credential rather than test it.

The eighth is still there, and chasing down why turned out to be the most useful hour of this section. Mutillidae got re-scanned once the application was free, and the fresh scan produces no Shellshock finding at all. The engine stopped making it. It survives in exactly one place: an old results file that no runner writes any more, orphaned when that pass was retired, still being read at scoring time.

So the obvious move was to stop reading it. The obvious move is always the one to be suspicious of. Scored without that file the run reports zero false positives and 100% precision over everything a person has checked. Recall does not change, because the file contributes no detections at all. A better number, obtained by deleting data.

The rule we settled on: retire the runner and you retire its output in the same change, in writing. An input nobody can regenerate is not evidence any more. It is sediment.

There was a quieter result at the time: findings the answer key does not cover fell from 189 to 161, and the high and critical ones among them from 85 to 62. Roughly a quarter of what this scanner had been saying was the scanner talking about itself.

Those numbers have since gone back up, to 194 unmatched and 88 high or critical, and we are not going to bury that. It is not a regression. Mutillidae used to get cut off by the deadline before it finished; now it finishes, and Mutillidae contains vastly more cross-site scripting than the answer key can be bothered to name. It accounts for 37 of the 88 by itself. XSS Maze, an application whose entire purpose is to be full of XSS, accounts for another 22.

A benchmark number that goes up when the scan gets more complete is a number you have to explain rather than quote. Which is the point: unmatched does not mean wrong. It means the answer key never made a claim either way, and until a person reads it, neither should we.

The scanner was breaking its own targets

Four of the misses in the last version of this report were one story told four ways, and it does not flatter us. The engine was altering the applications it was measuring. Every symptom pointed somewhere else, because a damaged target reports fewer findings, and fewer findings looks a lot like a clean result.

Mutillidae draws its settings as ordinary links. The crawler clicked index.php?do=toggle-enforce-ssl, which flips the session into SSL-enforced mode, after which every page redirects to https on a port that only speaks plain HTTP. Every fetch after that died in a TLS handshake. 141 of 223 fetches came back empty, and the crawl announced 223 pages as though it had done its job, because a transport failure and an empty page looked identical to it. The two pages carrying three answer-key entries were in that 141. The link sitting right next to it, do=toggle-security, raises the application's security level, so the crawl was also hardening the target before the scan got there.

Then the injection pass requested those same links itself, once per payload per class, so it kept re-breaking the session all the way through the scan. That is where "93 requests that never answered" came from, against an application that answers a hand request in 20ms.

It also submitted DVWA's password-change form, a GET carrying password_new and password_conf, and set that lab's admin password to the crawler's own sample value. Every run after that logged in as nobody and measured its findings against a login page. RailsGoat's account form went the same way. Both had been broken for an unknown number of runs, and the harness could not tell us, because its login helpers handed back whatever cookie jar they ended up holding, logged in or not. A cookie is not a session.

The fourth one deserves a wince. The blind command-injection probe injects a command that calls home. With the collector unreachable, a bare curl sits in a TCP connect for about two minutes inside the target's request handler, holding the worker that served it hostage. Ten of those per string argument is a denial-of-service test nobody ordered. Measured on DVGA, against a freshly restarted target each time:

A target that cannot reach the collector inside three seconds was never going to confirm the finding anyway. The deadline costs nothing and buys the application back.

Same mistake, different costume: the GraphQL authorization probe proved an operation was reachable by calling it, and decided what was safe to call based on the operation type. The comment in our own code said "read-only queries are always safe to probe". GraphQL promises nothing of the sort. DVGA files systemUpdate, deleteAllPastes and systemDiagnostics(cmd:) as queries, and systemUpdate shells out to python3 setup.py through os.popen. On a customer's API that same pattern is deleteAllUsers, filed under the reassuring word.

The guard we wrote to fix this was too blunt on the first try and cost us a real finding. It skipped any endpoint carrying a password and its confirmation. RailsGoat's account form carries user[password], user[password_confirmation] and user[id], and user[id] is an SQL injection the answer key names. The harm is setting a credential, not touching the endpoint. So the fields that could set one now get stripped out of the request and everything else is tested normally. A request with no password field in it cannot change a password.

The brake nobody could see

The most expensive bug of the whole exercise produced no wrong findings at all. It just made the scanner slow, and only sometimes, which is the hardest kind to notice and the easiest to explain away as a busy afternoon.

The engine paces itself per host. When requests start failing it backs off, and when they keep failing it stops overlapping them at all. That state lived in the daemon for the life of the process. So one pass that pushed a small application into failing left every later scan of that host crawling, with nothing anywhere on screen saying why.

Recovering through successes could not fix it, and this is the part worth keeping. A host the engine has stopped overlapping gets requested one at a time, each request waiting out the delay, so the forty consecutive successes that would release the brake are exactly the thing the brake makes slow. It holds itself down.

Pacing is a claim about conditions right now, so it should expire. A host nobody has spoken to for a minute is not evidence of anything, and the delay, the serialising flag and the latency window all get thrown away on the next request. In a daemon serving one customer after another, a target hammered at noon was still slowing somebody else's scan at four.

Three ways a benchmark lies to you

The rest of the movement was not in the engine either. It was in the thing measuring the engine, which by this point in the report is less a twist than a running gag.

A GraphQL command injection was scored as a detection failure for months while the engine was finding it on every single run. The harness stored two fields per GraphQL finding, its class and its name, and binned the rest. The scorer derives an endpoint path from the target, got nothing, defaulted to /, and the answer key says /graphql. One dictionary comprehension decided the engine could not do something it had been doing the whole time, and every run since nodded along.

A form is submitted whole or it does not run. Mutillidae's user-info page executes its query when user-info-php-submit-button is present, and renders an empty form when it is not, so a request carrying only the parameter under test landed on a page that did nothing and the endpoint read as clean. The body path had always sent every declared field with one of them carrying the payload. The query path sent one field. That asymmetry stood between the engine and three answer-key entries it finds in under a second each.

And an endpoint learned from a routing table turns up with no shape at all. RailsGoat publishes 176 routes on its development error page, POST /password_resets among them, and the action behind it calls Marshal.load(Base64.decode64(params[:user])) before validating anything. Nothing links to it and no form declares its fields, so a crawl can only ever produce the route and nothing else. The engine has had a field-mining operation since the day it was written. The harness had never once called it.

Recon, measured separately

Findings are not the only thing a scanner is for. The estate scenario is a routable subnet with authoritative DNS behind it, so discovery can be scored exactly rather than approximated. It runs on its own and was last measured on 9 September 2026, which is older than everything above.

The service rack is the one worth dwelling on: Redis with no password, MongoDB with authorization switched off, memcached, PostgreSQL trusting every connection that knocks, and MySQL root with an empty password. Each one identified by actually speaking the protocol, not by guessing from the port number and hoping.

Why every miss is classified

"The engine cannot detect this" and "the engine never got there" are different bugs with different fixes. A benchmark that mashes them together sends you off improving the wrong half of the product for a month. So every miss lands in exactly one bucket, and only one.

The auth row is why the separation was worth building. It held seven of the misses until recently, and not one of them was a detection problem. The login helpers were handing back a cookie and calling it a session, so the run reached those endpoints while being nobody in particular. Fixing the helpers moved all seven into the found column without anyone touching a single oracle. Filed as detection failures, they would have bought a month of work on entirely the wrong thing.

All four rows are empty now, and it is worth being precise about what that does and does not mean. It does not mean the engine detects everything. It means that of the entries this lab declares in scope, there is no longer one where the request arrived, the oracle read the answer, and the oracle got it wrong, and none the scan failed to reach at all. A table stops being useful the moment it looks like this. That is an argument for writing harder entries, not for opening anything.

What the benchmark got wrong

Recall went from 21.8% to 100% across this exercise, and most of that was the measurement rather than the product. The harness sent one engine its own command-line flag names instead of its daemon fields, so an entire DNS zone read as zero. It pointed an HTTP fingerprinter at database ports and filed the failure as a detection miss. It ran four injection classes while scoring the other six as misses, so six whole categories were unfindable by construction. And several answer-key entries pointed at paths that return 404, expected a property no scanner should ever report, or described a vulnerability the container did not actually have.

The reverse happened too, and that is the more useful half. One entry took five explanations before anybody landed on the true one, and four of those looked exactly like detection failures. The real causes were two general engine bugs: a transport failure being treated as a verdict in three separate places, and one pass wide enough to overload a small application until it stopped answering altogether.

How the lab is built

The lab, the answer keys and the scoring harness are one repository, and it is not published. Each target records its upstream author, its licence and the date somebody checked that provenance. Every container runs with capabilities dropped and no way to escalate, on the grounds that the entire point of these applications is that they are hostile.

The targets themselves are all public, deliberately vulnerable applications, so none of what makes this work is ours to keep. Write the answer key before the scan. Score against the key rather than against the output, so an entry the scan never reached is a miss with a cause and not a blank. Classify every miss by cause, and keep a target in the lab whose correct score is finding nothing at all.

The scorer prints every miss with its cause, and every unmatched high or critical finding with an instruction to go and check it by hand. That second list is where every false positive in this report was found. It is currently 88 entries long and 75 of them have never been read.

← All publications