Content discovery at scale without getting rate-limited

Content discovery is brute force by nature: tens or hundreds of thousands of requests to find the handful of paths and parameters that matter. Run that from one box at full tilt and you get rate-limited, WAF-blocked, or simply IP-banned before the interesting results show up. The fix is not "go slower and pray." It is to spread the work across addresses, pace each one sensibly, and make the whole thing resumable so an interruption costs minutes, not the run. This post covers that with mach, the Crossfyre content-discovery engine.

Why one box at full speed fails

A defender sees content discovery as a flood of requests from a single source, mostly returning 404, climbing a wordlist. That pattern is trivial to rate-limit or block. When you respond by cranking concurrency higher you trip the threshold faster; when you crank it lower the run takes days. The single source is the problem, not the speed alone. Two levers solve it: spread the requests across more egress addresses, and keep each address under whatever threshold the target enforces.

A baseline mach run

Start with the engine itself. mach substitutes each wordlist entry into the ::FUZZ:: marker and records what comes back. Two flags are your pacing controls: --tasks (concurrent in-flight requests) and --interval (milliseconds each task waits between requests).

bash
# Single-node path discovery, paced to stay polite.
mach scan \
  --url "https://app.example.com/::FUZZ::" \
  --wordlist-path ./wordlists/raft-large-dirs.txt \
  --tasks 30 \
  --interval 40 \
  --success-status-codes 200,204,301,302,401,403 \
  --headers "User-Agent: Mozilla/5.0"

Effective request rate is roughly tasks / (interval + response_time). Lower --tasks and raise --interval to back off; do the reverse to push harder. Tracking the right status codes matters: a target that returns 403 or 401 for protected paths is telling you those paths exist, so do not filter them out.

Distribute the work across a node fleet

The single-source problem disappears when the run does not come from a single source. When you launch a content-discovery workflow on Crossfyre, the control plane splits it into discrete operations and publishes them onto a durable NATS JetStream queue. Each node pulls operations and runs the mach daemon against them, so the wordlist is chewed through in parallel across machines with different egress addresses.

• Throughput scales with the fleet. Five nodes mean roughly five times the completed work per minute, without any single node going five times harder.
• Load spreads across addresses. The target sees traffic from several sources at a moderate rate each, rather than one source hammering, which is far less likely to trip a per-IP threshold.
• Per-node pacing stays sane. You tune the per-node concurrency and interval to stay under the limit, and add nodes to go faster, instead of overdriving one box.

Layer proxy chains over egress

Nodes add address diversity at the machine level. Proxy chains add it underneath. Each Crossfyre node can route its outbound traffic through layered proxy chains and an isolated network namespace with a VPN/WireGuard tunnel, so scans leave from where you choose and never from the host network. Combine that with a fleet and you get many egress paths, each carrying a fraction of the total request volume.

• Spread, not hide. Splitting egress across proxies keeps any single address well under a per-source rate limit, which is what stops the run from getting throttled.
• Egress control. Because traffic leaves through a controlled namespace, you decide the exit point per node and keep the scan off the host’s own network.
• Composability. Fleet distribution and proxy chains stack: N nodes times their proxy paths gives you many low-rate egress points instead of one hot one.

Tune concurrency to the target, not the clock

There is no universal correct rate. The right setting is the most throughput the target tolerates without throttling, and you find it empirically. A practical loop:

1. Start conservative: modest --tasks, a real --interval.
2. Watch the response codes in the live TUI. A sudden wall of 429 or 503, or a cliff in latency, means you found the ceiling.
3. Back the per-node rate off below that ceiling, then add nodes to recover total throughput.
4. Trim the wordlist to the target. A focused list (right tech stack, right conventions) finds more in fewer requests than a giant generic one, which is the cheapest rate-limit avoidance there is.

Resume instead of restarting

Long runs get interrupted: a node is reclaimed, a target starts tarpitting, you pause to retune. mach persists scan state, so rerunning the same command continues from where it stopped rather than re-walking the wordlist. Pass --fresh-start only when you intend to discard saved state.

Across the fleet this is automatic. Operations live on the durable queue until a node completes them, so a node that drops mid-run has its in-flight work redelivered to a healthy one, and nothing already finished is rescanned. With reserve-then-reconcile billing you are charged for completed work, not for the attempt that died. The full mechanics are in how scans survive a crashed node.

Put it together: a tuned wordlist, polite per-node pacing, work spread across a fleet, egress layered over proxy chains, and crash-safe resume. That is how a six-figure content-discovery run finishes without getting your addresses banned halfway through. New to the engines? Start with meet the engines, and see pricing for node and concurrency limits per plan.