SRE-Bench: a contamination-free reverse engineering benchmark for AI agents

What happened on this date

At 22:14 UTC on 2026-08-11, arXiv received v1 of "The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark" (primary category cs.CR). A v2 followed on 2026-10-01, and the figures below come from the abstract shown when we checked. The authors argue that a credible reverse engineering benchmark must stay out of model training data and reflect the scale and anti-analysis protections of real software. SRE-Bench was built from scratch by RE experts over more than 5,000 hours: 19 private, real-world-scale programs averaging 16.9K lines of code, 44 in-house anti-analysis mechanisms, 262 binary instances and 1,572 deterministically graded tasks, evaluated across 13 agentic settings and 11 models. Two frontier agents fully solved only 31.5% and 26.9% of graded instances. In an internal setting with no budget cap and safeguards disabled, one model reached a near-perfect pass@4, though reliably picking the correct candidate remained hard.

Why it matters for people protecting app logic

The most useful line for defenders is that agents were largely insensitive to compiler optimization and static linking. In other words, higher optimization levels or static linking should not be counted as protection against AI-assisted analysis. The second signal comes from comparing two results: two frontier agents fully solved fewer than a third of instances, while in the setting without a budget cap, repeated attempts solved almost everything. Protection is better described in terms of how much time and cost it imposes than in terms of whether it can be broken. The abstract also says ablations confirm that contamination control and realistic scale are essential, which is a reminder that "the AI could not read it" on public samples or toy programs cannot be extrapolated to your own product.

Concrete checks

First, remove items like "high optimization level" or "static linking" from your protection inventory, or label them explicitly as not being security measures. Second, write down a goal for each asset you protect, namely how much time or cost an analyst should at least have to spend, and use it to decide which logic must move to the server. Third, assess your own builds using real-scale release versions rather than demo programs, and record which tools and model versions you used. Fourth, re-test periodically as models change, because the same build faces different analysis capability at different points in time.

Your checklist

• Do not count optimization level or static linking as protection
• Set a target analysis cost for each protected asset
• Assess real-scale release builds and record tool and model versions
• Re-test the same build periodically as models change