Five of our authorization tests were passing for the wrong reason

A negative test asserts that a request is refused. Ours were refused, they went green, and five of them never reached the authorization check at all: the request body was malformed, so the server rejected the shape and returned before any permission was evaluated. A passing suite cannot tell that apart from a real refusal. Here is the audit that can.

Most of the tests that matter in an API are negative ones. Not "the owner can read this", which is easy and which the feature work already proved, but "somebody else cannot", which is the entire security model written down.

A negative test is also uniquely easy to get wrong, because it passes when something goes wrong, and there are many ways for something to go wrong. Only one of them is the one you meant.

The shape of the problem

Here is a negative test that looks completely reasonable. Sign in as a user who should not have access, call the endpoint, assert the call is refused.

r = client.post("/api/v1/workspace/members/remove",
                json={"member": other_user_id},
                headers=auth_as(not_an_admin))
assert r.status_code >= 400   # refused. green.

It goes green. It has proved nothing, if the handler expected a field called member_id and you sent member.

In that case the server never got as far as asking who you were. It tried to parse the body into the shape the handler wanted, failed, and returned 422 Unprocessable Entity from the framework layer. The authorization code did not run. It could be deleted entirely and this test would still pass.

We had five of them. They were found in August 2026 while growing our suite from 235 tests to 748, and none of them was found by reading the tests, because each one reads correctly.

Why an assertion cannot catch this

The obvious fix is to assert on a specific status. Not "refused" but 403. It helps, it is worth doing, and it is not enough. Plenty of handlers correctly answer 404 for an object you may not see, because a 403 confirms the object exists. Others answer 401 when a session expires mid-flight. Pin every negative test to one number and you either force handlers to lie or get a suite that breaks whenever somebody makes a sensible choice.

The deeper issue is that the status code is the wrong thing to inspect. What you want to know is not what the server answered, but how far it got before it answered. That is not visible from the outside of a single request, which is why no assertion inside the test can see it.

The refusal audit

So we stopped looking at tests one at a time. Setting an environment variable puts the whole suite into an audit mode:

CFX_TK_AUDIT=1 pytest suites/

In that mode the client records every (route, status) pair the entire run produces, and afterwards flags any denial that never reached an authorization gate. It can tell, because the gate is instrumented: a request that passed through it is marked, and a request that was rejected on shape before it got there is not.

The output is a tally rather than a pass or fail, which is the right shape for this. It is not asserting that your tests are good. It is showing you which of your refusals were real, so you can look at the rest.

308 (route, status) pairs observed
  0 shape rejections standing in for a denial

Zero is the current state. It was five.

The second gate: routes nobody tested

The audit answers "are my negative tests real". A different script answers "did I write one at all", by walking every route the server registers and checking each has authorization coverage. It runs on commit and reports two numbers, both of which must be zero: routes with a gap, and routes needing review.

All 323 mounted routes are accounted for, none of them has a gap, and the third bucket the report calls review is now empty. That bucket means the checker found a check but could not prove it was the right one, because it could not resolve the handler the route points at. It held one route for a long time, and it was reported as unproved rather than rounded up to a pass for as long as it did.

What the new tests actually found

Worth recording, because the point of all this is bugs and not test counts. Once the suite was real, it immediately found things:

• Four admin delete endpoints reported success for objects that never existed, and wrote an audit-log entry describing the deletion of nothing. An audit log containing invented events is worse than no audit log, because it will be believed.
• Two endpoints let a malformed email address reach our live mail provider. Validation existed on one path and not on the other.
• The request replay tool carried its own weaker copy of an SSRF guard instead of calling the shared one. Two implementations of a security control means one of them is out of date, and you find out which by being attacked through it.

That last one is its own lesson. A duplicated guard is not defence in depth. It is one guard and one liability wearing the same name.

What to take from it

• A green negative test is a claim about your server, not a fact about it. It asserts a refusal happened, not that the refusal was the one you were testing for.
• Instrument the gate, not the test. Whether authorization ran is knowable at the server. It is not knowable from a status code.
• Audit across the suite, not inside it. Every one of our five bad tests reads correctly on its own. The pattern only exists in aggregate.
• Send a body the handler accepts, then change only the identity. If a negative test differs from a positive one by more than who is asking, it is testing more than one thing and you will not be told which one failed.

The same reasoning drove a structural fix in our assistant, where the answer was to remove the model's ability to name a tenant rather than to test that it does not: the tenant parameter our AI assistant does not have.

We test other people's authorization by driving the same endpoint as several identities and comparing. It is the same idea, pointed outward. API authorization testing