Reveree (arXiv 2609.01185): stage-by-stage diagnosis of LLM reverse-engineering agents

What happened on this date

On 2026-09-01 (UTC), Hadjer Benkraouda, Hongyu Cai, Berkay Celik and Gang Wang submitted v1 of Reveree to arXiv. Their argument is that judging LLM agents on CTF reversing challenges by flag capture alone hides where an agent fails and whether a success came from analyzing the binary or from recalling a public write-up. Reveree scores trajectories at three levels: solve rate, progress through an eight-stage reverse-engineering schema, and a behavioral profile. Comprehension stages are scored by an LLM judge validated against a human expert and blinded to outcomes; the rest uses deterministic checks. Across nine frontier models, four prompting strategies and 88 picoCTF and NYU-CTF challenges, the base model mattered most and prompting had a smaller, model-dependent effect. Larger, newer or pricier models were not reliably better. Failures clustered in comprehension stages, and extra budget or reasoning effort helped little. Models could reproduce some picoCTF flags from descriptions alone, NYU-CTF showed little measurable recall, and most solves survived surface perturbations.

Why it matters for protecting app logic

When teams ask whether AI can clone their app, the usual mistake is to rely on a single score. Reveree shows that such a number can mislead in both directions: public challenges are inflated by memorization, while failures may simply reflect a current model's comprehension ceiling that the next generation may not share. The useful takeaway is that the hard part of reversing is understanding semantics, which is exactly what code transformation and server-side splitting try to make more expensive. The honest boundary is that the paper studied small CTF binaries, not protected commercial apps, so it cannot support a claim that any protection stops agents. Conversely, if your app reuses a lot of public open-source code, that part sits closer to the memorizable case; the logic worth protecting is the business rules you wrote yourself.

Checks you can run

First, when assessing your own release build, record how far AI-assisted analysis gets at each stage, such as whether it identifies module boundaries, key data structures and core rules, instead of logging only pass or fail. Second, split your code into public dependencies and in-house logic and focus the assessment on the latter. Third, treat a result like this model did not understand it as time-limited: rerun the assessment after each model generation or protection configuration change and keep the records. Fourth, keep the rules and keys that carry real value on the server, leaving only what the client needs. Fifth, when describing protection outcomes, state the model, budget and sample scope you used, and avoid overstating.

Your checklist

• Record stage-level progress of AI-assisted analysis on your own build, not just pass/fail
• Separate public dependencies from in-house logic and assess the latter first
• Re-assess and archive results after model generations or protection config changes