Perspective

Grading the report, not the flag: what Ethiack’s pentesting-agent benchmark gets right

Ethiack made the case that validated discovery — not flag capture — is what matters. We agree. Here’s the second question their benchmark, by design, doesn’t try to answer.

By BountyReady · July 15, 2026

Ethiack recently published “Evaluating Pentesting Agents — Part 1”, along with an open-source tool (EthiBench) and a paper (“From Controlled to the Wild”). If you build or run security agents, read it. It is the most serious public argument yet for a claim we built BountyReady around:

Flag capture is the wrong way to measure a pentesting agent. What matters is whether it actually found and validated a real vulnerability.

We agree with almost all of it, so this isn’t a rebuttal — it’s a “yes, and.” Ethiack answered one question extremely well. There is a second question sitting right next to it that their benchmark, by design, does not try to answer. That second question is the one we built BountyReady around.

What EthiBench got right

For years, agent evaluations optimized for capture-the-flag: did the agent pop the box and print the flag? That is easy to score deterministically, which is exactly why everyone did it — and exactly why it stopped being useful. Flag-capture benchmarks like XBOW’s XBEN are now effectively saturated; when everyone scores near 100%, the number tells you nothing.

EthiBench moves the goalpost to the right place. Instead of “did you capture the flag,” it asks “did you discover and validate real vulnerabilities across a complex target?” It scores an agent’s findings against 108 expert-annotated ground-truth vulnerabilities on three open-source apps, using an LLM to semantically match each finding to the ground truth and a bipartite assignment step to resolve them into clean true positives. Out comes precision, recall, F1, a CVSS-weighted severity score, and — crucially — cost.

One finding in particular deserves a spotlight, because it is the whole ballgame:

Ethiack argues that precision — not recall, not F1 — is what decides whether an agent is usable in production. An agent with low precision quickly becomes unusable, even if its recall is high.

If you have ever triaged agent output, you felt that in your spine. An agent that finds ten real bugs and files forty false positives hasn’t saved you time — it has created a review queue. That is the “AI slop” problem that has had bug-bounty programs closing submissions and banning researchers. Ethiack just put a number on it.

The question EthiBench doesn’t try to answer

Here is the design choice that matters. EthiBench is a measurement: point many agents at a corpus, get back precision/recall/F1 so you can rank them. It answers “which agent is better on average?”

That is genuinely useful — if you are choosing a harness, comparing model backends, or writing a paper. But it is not the question a hunter or a builder asks at the moment that actually costs them something. That moment is:

“I have this specific report, in my hand, right now. Do I send it — or does it get marked N/A and cost me my reputation?”

That is not a measurement. It is a gate. And a gate needs three things a corpus-level benchmark isn’t built to give you: a verdict on this report (not an average over a hundred), the specific reason a triager would reject it, and a verdict you can bet your reputation on — identical every time.

The one thing we’d do differently: for a gate, don’t use an LLM judge

EthiBench’s matcher is an LLM (GPT-5.4 Mini at temperature 0.3). Ethiack validated it carefully — against 50 expert-triaged reports it misclassified about one — and for their job, that is the right call. When you are scoring arbitrary findings against arbitrary ground truth on real targets, you need the flexibility of semantic matching. Nothing else scales to the wild.

But a temperature-0.3 judge is, by definition, non-deterministic: run it twice, you can get two answers. For a benchmark averaged over a corpus, that noise washes out. For a gate on a single report, it is the wrong property. If you are going to decide “send / don’t send” and put your Signal on the line, you want a verdict that is identical every time and can’t be argued with.

So BountyReady made the opposite trade for the opposite job. We grade one report, before you submit, deterministically, with no LLM in the loop. Our targets emit specific proof signals only when a vulnerability is actually exploited, and the grader checks — with exact, negation-aware matching — whether the report proves it. Same report in, same verdict out, every time. Zero cost per run. And because the ground truth is hidden (unlike a public benchmark on public repos, which an agent can memorize or train on), the verdict can’t be gamed.

The output isn’t a score to compare against other agents. It is a decision with a reason: accepted-like, needs-triage, false-positive risk, insufficient impact, missed chain, out-of-scope. The exact thing a triager would tell you — except you hear it before you spend your reputation, not after.

Two tools, two jobs

These aren’t competitors; they are two instruments for two moments. Use EthiBench to pick and tune your harness across real targets. Use BountyReady as the pre-flight check on the report your harness just produced — a controlled, hidden, un-gameable rehearsal — before it touches a real program.

Ethiack made the case that validated discovery is what matters, and that precision is what makes an agent usable. We built the gate that enforces exactly that, one report at a time.

Keep reading

• Deterministic, hidden-ground-truth grading for bug-bounty reports: the BountyReady methodology
• BountyReady vs EthiBench vs flag-capture benchmarks