How Crossfyre scans survive a crashed node

Long recon runs fail in boring ways. A spot instance gets reclaimed. A laptop lid closes. A target starts rate-limiting and a tool hangs until the SSH session dies. With most scanning setups, that is the whole run gone. You rerun from zero, pay for the compute twice, and hope it finishes this time.

Crossfyre treats a dropped node as a normal event, not a failure. Work that was in flight is redelivered to a healthy node and the run continues. This post walks through how that actually works under the hood.

The problem with "just run the tool"

When you SSH into a box and run a scanner, the run lives and dies with that process. There is no durable record of "this much is done, this much is left." If the process dies at 80 percent, the 80 percent dies with it. Tools that checkpoint to a local file help, but only if the box itself survives, which is exactly what is not guaranteed when you are renting cheap compute or scanning from disposable infrastructure.

The fix is to stop treating the run as one long-lived process and start treating it as a stream of small units of work with durable state behind them.

Workflows become operations on a durable queue

When you launch a workflow, Crossfyre decomposes it into discrete operations. Each operation is published onto a NATS JetStream stream, which is a persistent, acknowledged message log. A node pulls an operation, runs it, streams results back, and only then acknowledges it.

• If a node pulls an operation and then dies before acknowledging, the operation is never marked done.
• JetStream redelivers that unacknowledged operation to another available node.
• Results stream back as operations complete, so partial progress is real progress, not a number on a progress bar.

What happens when a node drops

Say you have five nodes chewing through a large content-discovery run and one of them gets reclaimed mid-operation. Here is the sequence:

1. The node stops sending heartbeats and its in-flight operations stop being acknowledged.
2. JetStream waits out the acknowledgement window, then re-queues those operations.
3. The four surviving nodes pick them up and keep going.
4. Completed results that already streamed back are untouched. Nothing is re-scanned that was already finished.

You do not babysit any of this. From the dashboard the run just keeps moving, slightly slower for a moment while one node falls out of the rotation.

Resuming a run that was paused or interrupted

Because operation state is durable, a workflow can be halted and resumed deliberately too, not just survive accidents. Pending operations sit on the stream until a node is available to take them. If your whole fleet goes offline overnight and comes back in the morning, the run picks up where it left off rather than starting over.

The billing consequence: failed work is free

This architecture has a direct effect on what you pay. Crossfyre bills on a reserve-then-reconcile model. When a workflow launches, an estimated cost is reserved against your credits. When it finishes, you are charged for the work that actually completed, and the reservation is reconciled against reality.

• Operations that never completed because a node died are not billed. They get redelivered and run elsewhere, and you pay once for the result, not once per attempt.
• If you cancel a run halfway, you pay for the half that produced results, not the full estimate.
• Credits never expire, so a reserved-then-released amount simply returns to your balance.

Why we built it this way

Offensive work happens in hostile, unstable conditions by definition. Targets fight back with rate limits and tarpits. Cheap infrastructure disappears. Networks degrade. A recon platform that assumes the happy path is a platform that loses your longest, most valuable runs precisely when they are most expensive to repeat.

Durable, acknowledged, redeliverable operations are not a feature we bolt on. They are the reason the distributed model is usable at all.