Methodology
Deterministic, hidden-ground-truth grading for bug-bounty reports: the BountyReady methodology
Most evaluations answer “can it find the bug?” We answer a different, downstream question: “is the report it produced actually submittable?”
By BountyReady · July 15, 2026
The problem: finding a bug is not the same as filing an acceptable report
The bug-bounty ecosystem is being flooded with reports that look right and are marked N/A — “AI slop.” The cost is asymmetric and reputational: a single N/A or spam closure can pull a researcher’s Signal down faster than several valid mediums pull it up, and programs have responded with stricter triage and outright bans. Independent work on agent evaluation reaches a compatible conclusion from the other direction: precision, not recall, is what makes an agent usable — an agent that files many false positives becomes a review burden regardless of how much it finds.
The gap this leaves is a decision, not a measurement: given one report, in hand, should it be submitted? Answering that well requires three properties an averaged, probabilistic benchmark does not need — and can actively work against.
Design goals
- Reproducible. The same report must always produce the same verdict. A researcher betting reputation on the result cannot tolerate a judge that answers differently on re-run.
- Un-gameable. The ground truth must not be memorizable. Public benchmarks on public repositories can be trained on; scores then reflect recognition, not capability.
- Actionable per report. The output must name why a triager would reject this specific report — missing proof, insufficient impact, an unproven chain, out of scope — not a score to compare against other agents.
- Proof over assertion. Claiming an exploit worked is not the same as showing it did. The method must tell the two apart.
The proof-token contract
Each scenario is a deliberately-vulnerable target whose endpoints emit a specific proof signal only when the vulnerability is actually exploited — a value that cannot be produced by a benign request or guessed from the outside. Cross-tenant access returns a record belonging to another tenant; a server-side request forgery returns an internal value the client should never see; a privilege escalation flips a role a normal user cannot change. The grader knows, per scenario, exactly which signals constitute proof, and matches the report against them.
The contract is symmetric and load-bearing: the target’s emitted signals and the grader’s matchers are two halves of one interface. This is what lets scoring be exact instead of interpretive — there is a right answer, and it was produced by the target, not inferred by a model. The proof signals are never published; they are the hidden ground truth, which is what makes the verdict un-gameable.
Negation-aware matching (the anti-slop core)
Naive keyword matching is trivially fooled: a report that says “I did not observe cross-tenant data” contains the same tokens as one that proves it. To prevent a report from earning credit for evidence it explicitly disclaims, the grader splits report text into evidence units and applies per-scenario detectors that reject claims phrased as negatives or hypotheticals. A report is credited for a piece of evidence only when it actually asserts, in the positive, that it obtained it. This catches the most common form of slop: confident-sounding write-ups that, read carefully, never demonstrate impact.
Log correlation: proven vs. asserted
A report can claim an exploit request was sent. To separate proven exploitation from narration, a submitter may attach a HAR or JSONL request log. When the log is present, the grader correlates it with the claimed exploit: if the report reads as accepted-like but the log does not contain the request that would have produced the proof signal, the verdict is downgraded. Proof beats prose.
The verdict ladder
Verdicts are assigned by a fixed precedence ladder, evaluated in order, so the result is a pure function of the inputs:
out-of-scope → ungradable → false-positive → N/A-risk → missed-chain → insufficient-impact → accepted-like
with a final downgrade of accepted-like when attached logs do not corroborate the required exploit request. Higher-precedence verdicts win: a report that is out of scope is out of scope regardless of how impressive the rest reads. The ladder mirrors how a real triager reasons — scope first, then validity, then completeness of the chain, then impact — which is what makes the “why” actionable rather than abstract.
What this method is not good for (the honest trade-off)
Determinism buys reproducibility and un-gameability at the cost of generality. It requires a curated scenario whose proof signals are defined in advance; it does not score arbitrary findings on arbitrary real targets. For that job — measuring and comparing agents across open-ended, real-world targets — a validated LLM judge (such as the one Ethiack’s EthiBench uses) is the appropriate tool, precisely because semantic matching is what handles the ambiguity of the wild.
So the two approaches are not rivals; they are complementary instruments:
- Measurement (LLM judge): which agent is better on average, across a corpus, including real targets.
- Gate (deterministic, hidden ground truth): is this report submittable, right now, reproducibly, before it costs reputation.
BountyReady is the second. It is deliberately narrow — a controlled, hidden, un-gameable rehearsal — because a gate you can bet your reputation on has to be.
Run a scenario, submit your agent’s report, and read the deterministic verdict — free.
Run a free scan →See a sample verdict →