Table of contents
17
By Salman Khan, OSCP+, Founder of XHack, SRT (Synack Red Team member)
Read this in 30 seconds: No independent test has crowned one LLM for reverse engineering, and the best benchmark available is partly vendor-involved. What the evidence does show is what to measure, and how much the setup matters more than the logo.
- The leaderboard winner has an asterisk. On SRE-Bench (Columbia-led, 262 binaries from 19 private programs), GPT-6-Astra fully solved 82.3% of the instances that could be graded, but only 181 of 262 (69.1%) were gradable because of refusals. GPT-5.6 Sol solved 31.5% and Claude Fable 5.1 solved 26.9%.
- Safeguards change the score. The same paper reports 88.0% pass@1 for GPT-6-Astra and 82.1% for Opus 5.5 with cyber safeguards off and no budget cap, in an evaluation whose token-equivalent cost was roughly $566,000.
- Effort and price do not track the brand. GPT-5.6 Sol went from 5.8% at Medium effort to 55.8% at Max, and DeepSeek-V4.1-Flash cost $0.5 per instance while Sol cost $42.5.
- Small specialists can beat big generalists at one narrow job. In a 2024 paper, a 6.7B model scored 45.4% re-executability on HumanEval-Decompile against 16.0% for GPT-4o and 20.1% for Ghidra, for C on x86-64 Linux only.
- Humans still steer. Bishop Fox reports strong results with light supervision, and SentinelLABS found models declaring work done too early.
If you search for the best LLM for reverse engineering, you find an awesome list, a few leaderboards and a lot of opinion. This guide pulls the published numbers on each LLM for reverse engineering into one place, says who produced each one, and then gives you a five-test method to rank models on your own binaries, which is the only ranking that matters for your work.
A note on method. I read the SRE-Bench paper, the LLM4Decompile and GPT-4 reverse engineering papers, the Bishop Fox and SentinelLABS write-ups, and the vendors’ access-programme pages, all on 4 November 2026. I did not run a benchmark myself, so I make no claim that one LLM for reverse engineering is better than another. Figures are quoted as the sources print them, and I say where a source is a vendor or an author with a stake. I run XHack, so I also say where our agent fits.
The best LLM for reverse engineering depends on which job you give it, because reverse engineering is at least five different jobs.
Decompiler cleanup turns raw decompiler output into readable code that still behaves the same. Renaming and retyping recovers names and data structures from a stripped binary. String and crypto spotting identifies algorithms, keys and encoded data. Handling protected code means working through packing, obfuscation and anti-analysis tricks. Long-horizon investigation means following a sample across many files over hours and keeping the findings straight.
An LLM for reverse engineering that is excellent at the first job can be weak at the last. That is why one leaderboard number cannot answer the question, and why the first thing to ask of any claim is: which of these jobs did it measure?
For the wider picture, see AI reverse engineering and AI malware analysis. This article stays on model choice.
SRE-Bench is the most serious public attempt to compare an LLM for reverse engineering on realistic binaries, and it shows a wide gap between models and a wide gap between settings. The paper, arXiv:2608.11469, was submitted on 11 August 2026 and revised on 1 October 2026. It was led by Columbia University with Vals AI and UC Berkeley, and its author list includes people affiliated with OpenAI, Anthropic and Google DeepMind.
The benchmark has 262 binary instances built from 19 private programs written in C, C++, Rust and Go, covering network protocols, games, file formats, malware and firmware. The programs carry 44 protection primitives. Each instance has six graded tasks, and a full solve means all six pass. The container gives the agent standard tools such as Ghidra, radare2, GDB and angr.
In the public setting, every LLM for reverse engineering in the table ran in the same agent harness at its highest reasoning effort, capped at 500 steps and six hours, with cyber safeguards on where the model had them.
| Model | Graded (of 262) | Fully solved (% of graded) | Mean score (of 6) | Cost per instance | Time per instance |
|---|---|---|---|---|---|
| GPT-6-Astra | 181 | 149 (82.3%) | 5.55 | $13.5 | 18 min |
| GPT-5.6 Sol | 254 | 80 (31.5%) | 3.69 | $42.5 | 88 min |
| Claude Fable 5.1 | 223 | 60 (26.9%) | 2.58 | $34.7 | 123 min |
| Claude Opus 5 | 256 | 32 (12.5%) | 1.91 | $23.6 | 81 min |
| GPT-5.5 | 262 | 10 (3.8%) | 1.02 | $17.8 | 46 min |
| DeepSeek-V4.1-Flash | 262 | 2 (0.8%) | 0.85 | $0.5 | 61 min |
| Grok-4.5 | 218 | 2 (0.9%) | 0.45 | $13.5 | 66 min |
| GLM-5.2 | 262 | 0 (0.0%) | 0.21 | $26.6 | 149 min |
Four lessons for choosing an LLM for reverse engineering come out of this table.
First, read the “Graded” column before the “Solved” column. GPT-6-Astra was gradable on only 181 of 262 instances because of frequent refusals on cyber safety grounds. Its solved rate is a rate over the instances it did not refuse. The authors say the scores should not be read as success rates over all 262, and they flag the refusals as a concern for practical use.
Second, effort and setting move results more than the model name does. In the paper’s unconstrained evaluation, run without cyber safeguards or budget limits, GPT-6-Astra reached 88.0% pass@1 and 99.2% pass@4. Opus 5.5 reached 82.1% and 98.1% under its internal configuration. GPT-5.6 Sol moved from 5.8% pass@1 at Medium effort to 55.8% at Max. The authors note that a high pass@4 needs a private grader, and that practical use means choosing or validating a result without one.
Third, price is not capability. GPT-6-Astra was both the top scorer and cheaper per instance than GPT-5.6 Sol, and DeepSeek-V4.1-Flash cost almost nothing and solved almost nothing.
Fourth, source-code skill does not transfer for free. The authors write that strong source-code security results do not necessarily translate into effective binary analysis.
What the benchmark does not tell you about your LLM for reverse engineering. It uses 19 programs, which the authors call a small pool, and it does not include the largest real-world software. The GPT-6-Astra results are vendor-involved: the project site says the team worked with OpenAI to evaluate it, and the results also appear in OpenAI’s system card. The binaries are private, so you cannot rerun it yourself, and it measures recovering behaviour to pass a grader, not whether an explanation is correct. Treat it as strong evidence of relative capability under one harness, not as a ranking of each LLM for reverse engineering for your day job.
An LLM for reverse engineering that refuses your task has a score of zero on that task, whatever the leaderboard says. SRE-Bench made this visible by excluding refused runs from the graded set, but your project does not get that luxury.
Anthropic’s Cyber Verification Program page says its generally available models can still be used for secure code review, threat modelling, patching known issues, finding vulnerabilities in your own source code and triaging alerts. It also says other cyber work such as malware analysis or exploit validation may be interrupted by its safety classifiers, and it invites qualifying professionals to apply for adjusted access. Individuals, including independent researchers, apply as individuals, and only the Defense tier is available to them at this time. The programme requires data retention.
OpenAI has a parallel route. XDA Developers reported on 14 April 2026 that OpenAI was expanding its Trusted Access for Cyber programme and offering GPT-5.4-Cyber, described as a version that lowers the refusal boundary for legitimate work and adds binary reverse engineering. I could not load OpenAI’s own pages, so treat that as a secondary report of OpenAI’s claim, not a tested result.
The practical rule: when you compare each LLM for reverse engineering, record every refusal and interruption as a failed run, and check which access programme your account would need. Our guide on why AI refuses hacking requests covers the refusal side in more depth.
For one narrow job, decompiler cleanup of C code, a small open specialist beat a general model in the best published comparison, but that comparison is from 2024. LLM4Decompile, published at EMNLP 2024 (arXiv:2403.05286), fine-tuned models from 1.3B to 33B parameters to turn disassembly or Ghidra output into C. It scored the result by re-executability: the decompiled code is compiled and must pass the original test assertions.
| Setup (HumanEval-Decompile, average over O0 to O3) | Re-executability |
|---|---|
| Ghidra alone | 20.12% |
| GPT-4o | 16.01% |
| Ghidra output cleaned by GPT-4o | 35.22% |
| LLM4Decompile-End 6.7B (direct) | 45.37% |
| Ghidra output refined by LLM4Decompile-Ref 6.7B | 52.74% |
The authors report the same direction on the harder ExeBench set, where their 6.7B model scored 17.98% against 3.78% for GPT-4o. The repository now lists later models, including a 22B version at 63.6% and a 9B version at 64.9%, though its table does not say which benchmark those figures use.
Three caveats matter. The work covers C compiled for x86-64 Linux with GCC at O0 to O3, and the authors say neither their approach nor Ghidra can effectively decompile obfuscated code. The baseline was GPT-4o, a 2024 model, so it says nothing about today’s frontier LLM for reverse engineering. And the authors are the model’s developers.
An older study, arXiv:2406.06637 (June 2024), tested GPT-4 alone on explaining and renaming decompiled code. It rated suggested function names 2.83 out of 3 for meaningfulness but 2.29 for effectiveness, and in one yes/no scenario GPT-4 got 25.6% of its answers wrong. Those numbers are two model generations old, but they illustrate the point that a fluent explanation is not a correct one.
Practitioner write-ups agree on one thing: an LLM for reverse engineering is now useful on hard targets, and it still needs a human to notice when they are stuck.
Bishop Fox, 8 July 2026. Jon Williams described using Claude Code with Sonnet 4.5 and a Binary Ninja MCP integration to decrypt a SonicWall NSv firmware image from a single-line instruction. The model reproduced work a senior researcher had done by hand. It also got stuck: it kept grinding on a failing decryption until the human suggested running the software itself, and it needed a prompt about trying different hash algorithms. The post puts the total at a few sessions of light supervision and does not give a duration. Its conclusion is in the title, senior-level skill with junior-level autonomy. It tested one model on one target, and Bishop Fox sells related services.
SentinelLABS, updated 22 July 2026. Juan Andrés Guerrero-Saade and Gabriel Bernadett-Shapiro built an eight-stage investigation of a 2005 sabotage implant in IDA Pro 9.3 and ran several frontier models through it, using more than 23 billion tokens across all experiments. They report GPT-5.6 Sol as the only publicly available model to complete all eight stages. Their consistent finding is that models failed at sustained ownership of an investigation, not at local skill: they promoted work too early and declared checkpoints clean before later review found problems. They recommend supervised investigative agency, with humans defining objectives and controlling what evidence is released. They also caution that run counts were small and operator involvement was consequential.
A companion SentinelLABS post from 2 July 2026 found that compaction cut input tokens by about 86% with the aggregate score unchanged, but recovery of higher-level objects declined, so keep exact evidence outside the compacted context.
You can rank any LLM for reverse engineering for your own work in an afternoon with five tests, a fixed harness and a scorecard. This is a method I recommend, built from how the sources above measured things, not a published standard.
Use binaries you built or are authorised to analyse, and prefer ones that are not public code. SRE-Bench’s authors found that a lightly modified gzip was solved in every configuration because its source is widely available in training data, which suggests recognition can stand in for analysis. Handle any malware samples only in an isolated lab.
| Test | What you give the model | How you score it |
|---|---|---|
| 1. Decompiler cleanup | 20 functions from a project you compile with optimisation and strip | Cleaned C recompiles and passes your own tests (re-executability) |
| 2. Names and types | The stripped binary, with the symbol file kept aside | Exact, close or wrong against ground truth, plus a count of confident wrong answers |
| 3. Strings and crypto | Samples using algorithms and constants you chose | Correct algorithm and data flow, each claim backed by an address |
| 4. Hardened build | The same program with protections you are allowed to apply | How far test 1 and 2 scores fall |
| 5. Long investigation | A multi-file task with a time cap | Completion, plus every “done” claim later found false |
Then fix the variables so the comparison means something. Use one harness and one prompt for every model, set the same reasoning effort, run each task three or four times, and log these fields per run:
The key metric is cost per verified solve, not cost per run. An LLM for reverse engineering that is cheap per run and solves almost nothing, as in the SRE-Bench table, is the expensive one.
Match the setup to your constraint first, then use your own scorecard to pick the LLM for reverse engineering inside it.
| Your situation | What the evidence points to | Watch for |
|---|---|---|
| Samples cannot leave your network | A local model through Ollama or llama.cpp; specialist models such as LLM4Decompile for C cleanup; an agent front end such as LLNL’s OGhidra | No published head-to-head; small models struggle with obfuscated code; verify everything |
| Heavily protected or large binaries, with budget | A frontier hosted model at its highest effort setting inside an agent harness | Refusals, per-instance costs up to tens of dollars in the SRE-Bench public run, premature claims |
| Malware or exploit-adjacent work | Hosted models with programme access, such as Anthropic’s Defense tier or OpenAI’s trusted access | Interruptions without access, programme scope and data retention terms |
| Routine triage, naming and string work | A mid-priced model chosen by your own tests 2 and 3 | Fluent but wrong names; demand addresses as evidence |
| An IDA-based team | Hex-Rays’ own AI tooling | Check status: its March 2026 post lists Claude Code skills for IDA as shipped in 2025 and its Knowledge and Semantic Engines as beta |
On the local row, LLNL’s OGhidra (BSD-3-Clause with a commercial alternative, about 465 GitHub stars, latest commit 2 November 2026) connects Ollama models to Ghidra. Its README lists models such as gemma3:27b and gpt-oss:120b with their memory needs and makes no quality claims. That is honest, and it means you have to run test 1 to 5 yourself before trusting it. If you also want a Ghidra connection for a hosted agent, see our guide to Ghidra MCP with Claude Code once it is live.
These five errors cost people the most time when they pick an LLM for reverse engineering.
So yeah, here is where we talk about what XHack brings. XHack is a full offensive security company across three layers: a self-serve AI agent for individuals, a self-serve security platform for companies, and managed testing by our researchers. For choosing an LLM for reverse engineering, the first layer is the relevant one.
I will be direct about what we have not done. We do not publish an SRE-Bench score or any reverse engineering benchmark for the XHack agent, and I am not going to invent one. What we offer is a way to run the scorecard above on the LLM for reverse engineering you prefer.
Plans are $20, $49 and $150 a month. The 7-day free trial is one per person, and the agent finds vulnerabilities from the start with exploitation unlocking once your identity is verified. See pricing and the local and offline AI page. Everything the agent says about a binary still needs your own verification.
When is another route better for an LLM for reverse engineering? If your team is on IDA and Hex-Rays ships what you need, or you are happy with a frontier model plus a Ghidra MCP server, you may not need us. If you would rather hand a reverse engineering job to people, our researchers take scoped engagements. Prefer to talk it through first? Book a free consultation, even if that turns out not to be us. Brutal honesty is kind of our thing.
No independent test crowns one LLM for reverse engineering. On SRE-Bench’s public setting, GPT-6-Astra led with 82.3% of gradable instances fully solved, but it was gradable on only 181 of 262, and results with safeguards off were higher for several models. GPT-5.6 Sol and Claude Fable 5.1 solved 31.5% and 26.9%. Because setting, effort and refusals shift results, run your own five-test scorecard.
Not today, according to the practitioners who have tried an LLM for reverse engineering. Bishop Fox reports that Claude needed human prompts at two points during a firmware task, and SentinelLABS found models declaring checkpoints clean before later review found problems. Both recommend supervised use. Apart from GPT-6-Astra, no model in the SRE-Bench public table fully solved even a third of its graded instances.
For C compiled for x86-64 Linux, the best published comparison favours a fine-tuned specialist. LLM4Decompile’s 6.7B model scored 45.4% re-executability against 16.0% for GPT-4o and 20.1% for Ghidra alone. That comparison dates from 2024 and covers unobfuscated C only, so test current general models on your own targets before relying on it.
Yes. LLM4Decompile is open and LLNL’s OGhidra connects local Ollama models to Ghidra. The XHack agent can also point at a local model. The trade-off is quality: there is no published head-to-head of local models on realistic binaries, and small models are weaker on obfuscated code. Run the scorecard and keep the samples in an isolated lab.
Some interruptions are documented. Anthropic says malware analysis and exploit validation may be interrupted by its classifiers and points qualifying professionals to its Cyber Verification Program. SRE-Bench’s GPT-6-Astra was refused often enough that only 69.1% of its instances were gradable. Apply for the relevant access programme and treat each refusal as a failed run in your scorecard.
In SRE-Bench’s public setting, cost per instance ranged from $0.5 for DeepSeek-V4.1-Flash to $42.5 for GPT-5.6 Sol, under one harness at maximum effort. Your cost depends on binary size, effort level and how many retries you need. Track cost per verified solve, because a cheap model that rarely solves anything is the expensive choice.
The honest answer to “which is the best LLM for reverse engineering” is that the question hides three others: which job, which setting and which refusals. The strongest public benchmark shows large gaps between models and larger gaps between settings, the specialist decompiler evidence is a 2024 data point, and practitioners say the human still steers. Rank each LLM for reverse engineering on your own binaries with a fixed harness, count refusals as failures and price cost per verified solve. Do that and you will have a better answer than any leaderboard can give you.
Categories
Related articles
