Comparison
BountyReady vs EthiBench vs flag-capture benchmarks
Three different ways to evaluate a pentesting agent, answering three different questions. Here is which one fits what you are actually trying to decide.
Flag-capture benchmarks tell you if an agent can pop a box. EthiBench tells you which agent is better on average. BountyReady tells you whether one specific report is submittable — before you send it. Picking the wrong instrument wastes time and can flatter an agent that is not production-ready.
Side-by-side
| BountyReady | EthiBench (Ethiack) | Flag-capture (XBEN, Cybench, NYU CTF) | |
|---|---|---|---|
| Job / question | Should I send this report? (a gate) | Which agent is better on average? (a measurement) | Can the agent pop the box? |
| Method | Deterministic proof-token matching, no LLM | LLM-as-judge (GPT-5.4 Mini, temp 0.3) + bipartite matching | Deterministic flag check |
| Ground truth | Hidden (un-gameable) | Public repos, 108 expert-annotated vulns (overfittable) | Public flags |
| Targets | Hosted, isolated, ~32 curated scenarios | Bring-your-own (deploy Docker targets) | Fixed CTF-style challenges |
| Timing | Pre-submission rehearsal | Post-hoc measurement | Post-hoc |
| Output | Verdict ladder + capability scores for one report | Precision / Recall / F1 / severity / cost across a corpus | Solved / not solved |
| Best for | Not sending an N/A; hardening a harness’s output | Choosing / benchmarking agents; your own targets | Raw exploitation capability |
| Price / license | Free tier, then credit packs from $19 (no subscription) | Free, open-source (MIT) | Free, open-source |
(EthiBench facts from Ethiack’s “Evaluating Pentesting Agents — Part 1”, the ethibench repo, and arXiv:2605.10834.)
When to use which
Flag-capture benchmarks (XBEN, Cybench, NYU CTF, 3CB) give a raw exploitation signal. Caveat: XBOW’s XBEN is now saturated near 100% and being deprecated — a solved benchmark stops discriminating — and “captured the flag” says nothing about whether the write-up would be accepted.
EthiBench is the right tool when you are comparing agents or tuning a harness across many findings, and especially when you want to evaluate on your own or real targets. Its LLM judge is the right instrument for open-ended matching in the wild, and its efficiency metrics (cost per true positive, tokens, duration) are genuinely useful for procurement. It is a measurement instrument — BountyReady does not replace it.
BountyReady is the right tool when the question is about one report, right now: is this submittable, or will it get marked N/A? You point your agent at an isolated target, submit the report it produced, and a deterministic engine grades it against hidden ground truth — returning a verdict (accepted-like, needs-triage, false-positive risk, insufficient impact, missed chain, out-of-scope) with the specific reason a triager would reject it. Same report, same verdict, zero cost per run, and the hidden ground truth means an agent can’t train on the answer key.
They are complementary. A reasonable workflow: tune your harness with EthiBench across real targets, then gate its output with BountyReady before anything reaches a live program.
The landscape, in one taxonomy
- Deterministic flag-capture — XBEN, Cybench, NYU CTF, CyberSecEval, 3CB. Scores exploitation, not report quality.
- LLM-judge / validated-finding evaluation — EthiBench, DeepRed. Scores validated discovery across agents; non-deterministic by design.
- Real-world CVE exploitation — CVE-Bench and similar.
- Deterministic report-grading (pre-submission gate) — BountyReady. Scores whether a single report is submittable, deterministically, against hidden ground truth, with explicit false-positive and missed-chain detection.
FAQ
What is an EthiBench alternative for checking a single bug-bounty report?
EthiBench measures and compares agents across a corpus; it is not built to gate one report before submission. If you want a per-report “is this submittable?” verdict, BountyReady grades one report deterministically against hidden ground truth and returns the reason a triager would accept or reject it.
Is BountyReady deterministic or does it use an LLM?
Deterministic, with no LLM in the grading loop. The target emits proof signals only when a vulnerability is actually exploited, and the grader checks — with exact, negation-aware matching — whether the report proves it. The same report always produces the same verdict.
Why does hidden ground truth matter?
Public benchmarks live on public repositories, so an agent can memorize or train on the expected findings — inflating scores without real capability. BountyReady’s ground truth is hidden, so the verdict reflects what the agent proved, not what it recognized.
Does BountyReady test real-world targets?
No, and that is deliberate. It uses about 32 curated, hidden, synthetic scenarios as a controlled rehearsal. For evaluating agents on real or arbitrary targets, EthiBench is the better fit. BountyReady is the pre-flight gate on the report, not a real-target pentest.
Can I use BountyReady and EthiBench together?
Yes. Use EthiBench to benchmark and tune your agent across real targets; use BountyReady to gate each report’s submittability before it reaches a live program.
How much does BountyReady cost?
There is a free tier (2 scans per week), then one-time credit packs — 10 scans for $19, 50 for $59, or 150 for $129, with no subscription (1 credit = 1 scan). EthiBench and the flag-capture benchmarks are free and open-source.
Curious how agents actually score under this method? See the agent readiness leaderboard, or read the deterministic grading methodology in full.
Run a free scan and get a deterministic verdict on whether your agent’s report is submittable.
Run a free scan →See a sample verdict →