Agent Evaluations and Benchmarks: SWE-bench

What Is SWE-bench?

SWE-bench is a benchmark constructed from real GitHub repositories: each instance pairs an issue (a bug report or feature request submitted to a popular Python project) with the commit that resolved it. The agent’s task is to read the issue description and a snapshot of the codebase, then produce a code change.

The scoring criterion is intentionally strict. The agent’s code change is applied to the repository, and the issue’s associated test suite must pass entirely. A plausible explanation or a similar-looking diff is not enough — the tests must actually run and turn green.

This design matters for a reason. Earlier NLP benchmarks measured text similarity or multiple-choice accuracy. SWE-bench measures the concrete outcome of real actions: the agent reads files, modifies code, and a test verifies that the modification works.

┌──────────────────────────────────────────────────────────────┐
│                 SWE-bench Evaluation Flow                    │
├──────────────────────────────────────────────────────────────┤
│  1. Input:  GitHub issue text + repository code snapshot     │
│  2. Agent:  loop runs (explore → modify → run tests)         │
│  3. Output: code change diff                                 │
│  4. Judge:  apply diff → run associated tests → pass/fail    │
│                                                              │
│  Note: evaluation environment is isolated from agent env     │
└──────────────────────────────────────────────────────────────┘

Variants of SWE-bench

SWE-bench comes in several official forms.

Variant	Issues	Description
SWE-bench (full)	2,294	original complete set
SWE-bench Lite	300	self-contained issues, fast evaluation
SWE-bench Verified	500	human-verified for problem clarity and solution quality

SWE-bench Verified is the most widely cited variant. Human annotators reviewed each issue to confirm the problem statement is unambiguous and the reference fix is reasonable. Leaderboard numbers depend on the model snapshot, evaluation harness, number of attempts, and time and cost budget, so they should not be quoted as a single measure of general ability.

SWE-agent and the ACI Concept

The most important contribution of the SWE-agent paper is not the score itself but the introduction of the ACI (Agent-Computer Interface) concept. SWE-agent’s initial version achieved 12.5%, a meaningful result by the standards of its time. What matters most is the design principle that made it possible.

Earlier approaches handed the agent unrestricted bash shell access: read files, modify them, run anything. The SWE-agent researchers argued this actually confused agents by overwhelming them with choice. Instead, they provided an interface deliberately narrowed and tuned for code-editing work.

The ACI design principles are as follows.

Bounded command set: Rather than hundreds of raw bash commands, the agent receives a small number of purpose-built tools: view_file, edit_file, search_code, run_tests, and a handful of others. This makes tool selection unambiguous and reduces parameter hallucinations.

Actionable error messages: When a tool fails, the error message is not a raw system string like “Error: permission denied”. Instead it reads something like “The line number you specified exceeds the file length. The file has 45 lines.” The agent can immediately decide its next action based on this context.

Context window management: Rather than loading entire files into context at once, the interface shows only the 50–100 lines around the current editing position. This reduces context waste and keeps the agent focused.

Evaluation Harness and Isolation

An evaluation harness fixes the dataset, repository snapshot, container image, test command, timeout, and retry policy together. It separates the agent-run environment from the final evaluation environment and preserves patches, tool output, and traces so failures can be reproduced. ACI-style narrow interfaces and test-execution loops are useful design choices, but their effect is comparable only under the same evaluation contract.

Caveats When Interpreting Benchmark Scores

SWE-bench scores are important indicators but should not be accepted uncritically.

Data contamination risk: Issues opened after 2024 are relatively safe, but older issues may have appeared in model training data. SWE-bench Verified’s human review partially mitigates this but does not eliminate it.

Hidden cost and time variables: A system allowed 100 API calls per issue and one allowed only 10 are not comparable at face value. When reading benchmark reports, always check iterations, total tokens, and wall-clock time alongside the resolve rate.

Language specificity: SWE-bench is entirely Python. Agent performance on TypeScript, Rust, or Go codebases cannot be reliably predicted from SWE-bench scores alone.

The next chapter shifts from evaluation accuracy to loop economics — the cost structure that accumulates with every iteration a loop runs.