Before the listing, an important calibration on the numbers \u2014 because one major benchmark shift happened mid-cycle and is not yet reflected in most tool comparison articles.<\/p>\n
SWE-bench Verified<\/strong> has been the industry\u2019s standard coding benchmark since mid-2024. It presents agents with 500 real GitHub issues drawn from popular Python repositories and measures whether the agent can understand the problem, navigate the codebase, generate a fix, and verify that it passes tests \u2014 end-to-end, without human guidance. It was a credible proxy. In February 2026, that changed.<\/p>\n On February 23, 2026, OpenAI\u2019s Frontier Evals team published a detailed post<\/a> explaining why it had stopped reporting SWE-bench Verified scores. Their auditors reviewed 138 of the hardest problems across 64 independent runs and found that 59.4% had fundamentally flawed or unsolvable test cases \u2014 tests that demanded exact function names not mentioned in the problem statement, or checked unrelated behavior pulled from upstream pull requests. More critically, they found evidence that every major frontier model \u2014 GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash \u2014 could reproduce the gold-patch solutions verbatim from memory using only the task ID, confirming systematic training data contamination. OpenAI\u2019s conclusion: \u201cImprovements on SWE-bench Verified no longer reflect meaningful improvements in models\u2019 real-world software development abilities.\u201d OpenAI now recommends SWE-bench Pro as the replacement for frontier coding evaluation.<\/p>\n This does not make SWE-bench Verified scores useless. Other major labs continue to report them, third-party evaluators continue to run them, and they remain useful for broad directional comparison. But any ranking that presents SWE-bench Verified scores as clean, objective measurements of real-world ability \u2014 without this caveat \u2014 is giving you an incomplete picture. All scores in this article are flagged accordingly.<\/p>\n SWE-bench Pro<\/strong> is harder to interpret than Verified because published results vary significantly by split, scaffold, harness, and reporting source. The benchmark contains 1,865 total tasks divided into a 731-task public set, an 858-task held-out set, and a 276-task commercial\/private set drawn from 18 proprietary startup codebases. When the original Scale AI paper<\/a> measured frontier models using a unified SWE-Agent scaffold, top scores were below 25% \u2014 GPT-5 at 23.3% \u2014 reflecting a genuinely harder evaluation. However, current public leaderboard<\/a> and vendor-reported runs now show substantially higher scores under newer models and optimized agent harnesses: OpenAI reports GPT-5.5 at 58.6% on SWE-bench Pro (Public), while Anthropic\u2019s comparison table lists Claude Opus 4.7 at 64.3% and Gemini 3.1 Pro at 54.2%. These numbers should not be directly compared with the original sub-25% SWE-Agent results without noting the scaffold and split differences \u2014 the benchmark has not changed, but the evaluation conditions and model generations have. When you see a 60%+ SWE-bench Pro score alongside a sub-25% one, they are measuring the same benchmark under very different conditions, not two separate tests.<\/p>\n Terminal-Bench 2.0<\/strong> evaluates terminal-native workflows: shell scripting, file system operations, environment setup, and DevOps automation. As of April 23, 2026, GPT-5.5 leads at 82.7% on this benchmark \u2014 confirmed in OpenAI\u2019s official release<\/a>. Claude Opus 4.7 scores 69.4% (Anthropic\/AWS-reported<\/a>), and Gemini 3.1 Pro scores 68.5%. An important methodological caveat: different harnesses produce different numbers for the same model. Anthropic\u2019s Opus 4.6 system card showed GPT-5.2-Codex scoring 57.5% on the independent Terminus-2 harness vs 64.7% on OpenAI\u2019s own Codex CLI harness \u2014 a 7-point gap from harness alone. When comparing Terminal-Bench figures across sources, always check which execution environment was used.<\/p>\n One final cross-benchmark caveat: agent scaffolding matters as much as the underlying model. In a February 2026 evaluation of 731 problems, three different agent frameworks running the same Opus 4.5 model scored 17 issues apart \u2014 a 2.3-point gap that changes relative rankings. A benchmark score labeled with a model name reflects the model and<\/em> the specific scaffold wrapped around it, not the model in isolation.<\/p>\n The current leader on SWE-bench Verified among third-party trackers is Claude Mythos Preview at 93.9%<\/strong>, announced April 7, 2026 under Anthropic\u2019s Project Glasswing<\/a>. It is not generally available. Access is restricted to a limited set of platform partners; Anthropic has stated it does not plan broad release in the near term, in part due to elevated cybersecurity capability concerns. It sits outside the main comparison below because developers cannot access it through standard channels. Its existence does, however, signal that the practical capability ceiling sits substantially above what any publicly available tool currently delivers.<\/p>\n SWE-bench Verified (self-reported):<\/strong> 87.6% (Opus 4.7) \/ 80.8% (Opus 4.6) SWE-bench Pro (Anthropic internal variant):<\/strong> 64.3% (Opus 4.7, #1) \/ 53.4% (Opus 4.6) Terminal-Bench 2.0:<\/strong> 69.4% (Opus 4.7, Anthropic-reported) CursorBench:<\/strong> 70% (Opus 4.7, Cursor-reported) Claude Code subscription:<\/strong> $20\u2013$200\/month | Opus 4.7 API:<\/strong> $5\/$25 per million tokens<\/p>\n Claude Code is Anthropic\u2019s terminal-native coding agent and the leader on code quality metrics across most self-reported and third-party evaluations as of May 2026. It runs from the command line, integrates with VS Code and JetBrains via extension, and is built around Claude Opus 4.7<\/a> \u2014 released April 16, 2026.<\/p>\n Opus 4.7 represents a step-change over its predecessor. SWE-bench Verified jumped from 80.8% to 87.6% \u2014 a nearly 7-point gain. On Anthropic\u2019s internal SWE-bench Pro variant, the model moved from 53.4% to 64.3%, an 11-point gain that puts it ahead of every current publicly available competitor on that harder benchmark. On CursorBench, Cursor\u2019s CEO reported Opus 4.7 at 70%, up from 58% for Opus 4.6. Rakuten reported 3\u00d7 more production tasks resolved on their internal SWE-bench variant; CodeRabbit reported over 10% recall improvement on complex PR reviews with stable precision.<\/p>\n Opus 4.7 introduced self-verification behavior: the model writes tests, runs them, and fixes failures before surfacing results, rather than waiting for external feedback. It also introduced multi-agent coordination \u2014 the ability to orchestrate parallel AI workstreams rather than processing tasks sequentially \u2014 which matters for teams running code review, documentation, and data processing simultaneously. The 1 million token context window can support much larger repository contexts than shorter-window tools, though very large monorepos still benefit from indexing, retrieval, or file selection strategies to stay within practical limits.<\/p>\n One important pricing distinction: Claude Code subscription tiers ($20\u2013$200\/month) are what individual developers pay to use Claude Code in the CLI and IDE integrations. The underlying Opus 4.7 API is priced at $5 per million input tokens and $25 per million output tokens \u2014 unchanged from Opus 4.6 \u2014 with a batch API discount of 50% and prompt caching reducing costs further. Teams building custom agents on top of the Anthropic API are not paying the subscription rate.<\/p>\n On Terminal-Bench 2.0, Opus 4.7 scores 69.4% \u2014 strong, but GPT-5.5 has since moved ahead on this specific benchmark at 82.7%. For pure terminal\/DevOps agentic workflows, that gap is worth considering.<\/p>\n Best for:<\/strong> Developers working on complex multi-file engineering tasks, large codebases, or long-horizon refactoring who prioritize output quality over speed.<\/p>\n Terminal-Bench 2.0 (GPT-5.5):<\/strong> 82.7% \u2014 current #1 SWE-bench Pro Public (OpenAI-reported, GPT-5.5):<\/strong> 58.6% SWE-bench Verified (third-party trackers, GPT-5.5):<\/strong> ~88.7% (OpenAI does not self-report) Pricing:<\/strong> Codex CLI is open-source (model usage requires a ChatGPT plan or API key); GPT-5.5 in Codex available on Plus ($20\/month), Pro ($200\/month), Business, Enterprise, Edu, and Go plans; API: $5\/$30 per million tokens (gpt-5.5)<\/p>\n An important correction to many comparisons of Codex: the Codex CLI is a local tool that runs on your machine<\/strong>, not a cloud-sandboxed system. The Codex CLI (available on GitHub as GPT-5.5 launched April 23, 2026<\/a> and is OpenAI\u2019s most capable coding model to date. On Terminal-Bench 2.0, it scores 82.7% \u2014 the current #1 position across all publicly available models, ahead of Claude Opus 4.7 (69.4%) and Gemini 3.1 Pro (68.5%). OpenAI describes Terminal-Bench as the more representative benchmark for the kind of work Codex actually does: \u201ccomplex command-line workflows requiring planning, iteration, and tool coordination.\u201d On SWE-bench Pro (Public), GPT-5.5 scores 58.6% per OpenAI\u2019s release data, behind Claude Opus 4.7 (64.3%) but ahead of earlier GPT generations. Claude Opus 4.7 still leads on code quality for multi-file, long-horizon software engineering; GPT-5.5 leads on terminal-native, DevOps-style agentic execution.<\/p>\n Note on SWE-bench Verified: OpenAI stopped self-reporting this metric<\/a> in February 2026 due to contamination concerns. Third-party trackers show GPT-5.5 around 88.7%, but OpenAI\u2019s official position is that this benchmark is no longer a reliable frontier measure. They report SWE-bench Pro instead.<\/p>\n GPT-5.5 is available in ChatGPT (Plus, Pro, Business, Enterprise, Edu) and across Codex (CLI, IDE extensions, and the Codex web product). API access was announced and is rolling out. API pricing: $5\/$30 per million tokens for gpt-5.5, a 2\u00d7 jump from GPT-5.4. More than 85% of OpenAI employees now use Codex weekly \u2014 a signal of internal confidence in the product beyond benchmark numbers.<\/p>\n Best for:<\/strong> Developers focused on terminal-native, DevOps, and pipeline automation workflows where Terminal-Bench performance is the primary signal; also the strongest choice for fire-and-forget execution via the Codex web product.<\/p>\n SWE-bench Verified:<\/strong> ~51.7% (default config; rises substantially with Opus 4.7 backend) Task completion speed:<\/strong> ~30% faster than GitHub Copilot in head-to-head testing ARR:<\/strong> $2 billion (February 2026) Pricing:<\/strong> $20\/month (Pro), $60\/month (Pro+), Enterprise tiers above<\/p>\n Cursor reached $2 billion ARR in February 2026 \u2014 doubling from $1 billion in November 2025 \u2014 and is reportedly in talks to raise approximately $2 billion at a $50 billion-plus valuation, with Thrive Capital and Andreessen Horowitz. These figures reflect real developer adoption, not benchmark-driven hype.<\/p>\n Cursor\u2019s SWE-bench figure (~51.7%) represents its default model configuration. Because Cursor is model-agnostic and supports Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro, and Grok, its effective benchmark ceiling scales with the model selected \u2014 a developer running Cursor with Opus 4.7 gets materially different performance from one using a default configuration. The 30% task completion speed advantage over Copilot reflects Cursor\u2019s editor-native architecture, which eliminates context-switching overhead between a terminal agent and a separate IDE.<\/p>\n Cursor is a VS Code fork rebuilt around AI at every layer. Its Plan\/Act mode gives developers a structured workflow: plan, review, then execute. Background Agents (Pro+ tier, $60\/month) run autonomous coding sessions on cloud VMs in parallel, without blocking the main editor. Per-task model selection \u2014 fast model for autocomplete, reasoning-heavy model for complex edits \u2014 gives fine-grained cost control.<\/p>\n Cursor is its own editor, not a plugin. Developers using JetBrains, Neovim, or Xcode cannot use Cursor without switching editors. That constraint is real and limits its enterprise footprint compared to Copilot.<\/p>\n Best for:<\/strong> VS Code-native developers who want the best AI-native IDE experience and are willing to pay for the integrated workflow.<\/p>\n SWE-bench Verified (Gemini 3.1 Pro):<\/strong> 80.6% Terminal-Bench 2.0 (Gemini 3.1 Pro):<\/strong> 68.5% Context Window:<\/strong> 1 million tokens Pricing:<\/strong> Free tier via Google AI Studio; Google One AI Premium for higher limits<\/p>\n Gemini CLI is Google DeepMind\u2019s open-source coding agent ( Gemini 3.1 Pro also scores strongly on several non-coding benchmarks: ARC-AGI-2 (77.1%), GPQA Diamond (94.3%), and BrowseComp (85.9%), making it a strong option for scientific computing, agentic research workflows, and tasks that mix coding with deep reasoning. For Google Cloud-native teams, Gemini CLI integrates directly with GCP, Vertex AI, and Android Studio.<\/p>\n The free tier is its most strategically distinctive feature. Solo developers, students, and open-source maintainers who cannot justify a $20\u2013$200\/month coding agent subscription have a legitimate frontier-quality option here. At 80.6% SWE-bench Verified \u2014 matching Claude Opus 4.6 and ahead of GitHub Copilot\u2019s default configuration \u2014 this is not a compromise free tier. It is a genuinely competitive product that removes cost as a barrier to entry.<\/p>\n Best for:<\/strong> Cost-sensitive developers, Google Cloud teams, and individual contributors who want frontier model quality without a monthly subscription.<\/p>\n SWE-bench Verified (Agent Mode, default model):<\/strong> ~56% Adoption:<\/strong> 4.7 million paid subscribers (January 2026) Pricing:<\/strong> $10\/month (Pro), $19\/month (Business), $39\/month (Pro+), Enterprise custom pricing; AI Credits billing transition on June 1, 2026<\/p>\n GitHub Copilot is not the most capable agent on this list by benchmark, but it is the most widely deployed. With 4.7 million paid subscribers \u2014 75% year-over-year growth \u2014 and 76% developer awareness per GitHub\u2019s Octoverse report, Copilot is the baseline AI coding tool at most enterprise software organizations. Microsoft CEO Satya Nadella confirmed in early 2026 that Copilot now represents a larger business than GitHub itself.<\/p>\n Two important updates for the current pricing picture: GitHub added a Copilot Pro+ tier at $39\/month<\/strong> that unlocks the full model roster and higher compute limits. More significantly, GitHub announced that Copilot is moving to AI Credits-based billing on June 1, 2026<\/a><\/strong>, which means certain agent actions, premium model calls, and background task execution will draw from a credits pool rather than being included in the flat monthly fee. Base plan prices are unchanged as of the announcement, but total cost for heavy agentic use may increase depending on how credits are consumed.<\/p>\n On model selection: in February 2026, GitHub made Copilot a multi-model platform<\/a> by adding Claude and OpenAI Codex as available backends for Copilot Business and Pro customers. The 56% SWE-bench figure reflects the default proprietary Copilot model. Configuring it to use Claude Opus 4.7 or GPT-5.5 would push that number substantially higher \u2014 though premium model calls draw from the credits pool under the new billing model.<\/p>\n At $10\/month for individuals and $19\/month for business seats, Copilot\u2019s price-to-capability ratio is the strongest entry point for enterprise teams that need predictable licensing, SOC 2 compliance, audit logs, and broad IDE support across VS Code, JetBrains, Visual Studio, Neovim, and Xcode. In enterprise procurement, compliance posture often outweighs a few SWE-bench percentage points.<\/p>\n Best for:<\/strong> Enterprise teams that need predictable licensing, compliance posture, and broad IDE support across multiple environments.<\/p>\n Performance:<\/strong> Higher on clearly scoped tasks; significantly weaker on ambiguous or complex tasks Pricing (updated April 14, 2026):<\/strong> Free, Pro $20\/month, Max $200\/month, Teams usage-based with $80\/month minimum, Enterprise custom<\/p>\n Devin holds a special place in this category\u2019s history. Its 13.86% SWE-bench Lite score at launch in early 2024 \u2014 the first time any AI system had autonomously resolved real GitHub issues at meaningful scale \u2014 was industry-defining. By today\u2019s standards, every tool above it in this ranking has surpassed that number by a factor of four or more.<\/p>\n Devin 2.0 is a substantially different product. It runs in a fully sandboxed cloud environment with its own IDE, browser, terminal, and shell. You assign a task; Devin produces a step-by-step plan you can review and edit; then it writes code, runs tests, and submits a pull request. Interactive Planning and Devin Wiki \u2014 which auto-indexes repositories and generates architecture documentation \u2014 address two of the original\u2019s biggest criticisms.<\/p>\n On well-scoped, well-defined tasks \u2014 framework upgrades, library migrations, tech debt cleanup, test coverage additions \u2014 Devin reports higher success rates, with independent developer testing consistently showing strong results on clearly specified work. Reliability drops sharply for ambiguous or architecturally complex tasks; one documented community test found far more failures than successes across 20 varied tasks, highlighting that task specification quality directly determines output quality.<\/p>\n On pricing<\/strong>: Cognition retired its older Core and ACU-based self-serve plans on April 14, 2026<\/a> and introduced cleaner tiers: Free, Pro at $20\/month, Max at $200\/month, Teams usage-based with an $80\/month minimum, and Enterprise with custom pricing. If you have seen the earlier \u201c$20 Core + $2.25\/ACU\u201d pricing in other articles, it is no longer current.<\/p>\n Cognition also partnered with Cognizant in January 2026 to integrate Devin into enterprise engineering transformation offerings, and launched Cognition for Government in February 2026 with FedRAMP High authorization in progress \u2014 signaling a deliberate push into institutional deployments.<\/p>\n Best for:<\/strong> Teams with clearly scoped, well-specified engineering tasks \u2014 migrations, test generation, framework upgrades \u2014 where the cost of reviewing AI output is lower than the cost of doing the work manually.<\/p>\n SWE-bench Verified:<\/strong> 72% GAIA Benchmark:<\/strong> 67.9% License:<\/strong> MIT Pricing:<\/strong> Free to self-host; pay only for model API inference<\/p>\n OpenHands (formerly OpenDevin, rebranded in late 2024 under the All-Hands AI organization) is the open-source community\u2019s answer to Devin. With strong open-source adoption visible through GitHub activity and community usage, and a 72% SWE-bench Verified score, it matches or exceeds commercial agents at several price points.<\/p>\n OpenHands supports 100+ LLM backends \u2014 any OpenAI-compatible API, including Claude, GPT-5, Mistral, Llama, and local models via Ollama. The CodeAct agent can execute code, run terminal commands, browse the web, and interact with web-based development tools inside a Docker sandbox. Its 67.9% on the GAIA benchmark confirms that web interaction capabilities are substantive.<\/p>\n The bring-your-own-key model means zero platform markup \u2014 you pay inference costs directly to your model provider. For open-source projects, budget-constrained teams, and developers who want full auditability of agent behavior, it is the strongest option in this tier. Self-hosting requires Docker and access to an LLM provider API; there is no hosted SaaS product.<\/p>\n Best for:<\/strong> Open-source teams, developers who want full control and auditability, and budget-conscious practitioners who already have API credits with a major model provider.<\/p>\n SWE-bench Verified (self-reported, Augment harness):<\/strong> 70.6% Differentiator:<\/strong> Full repository context engine; MCP-interoperable Pricing:<\/strong> Team and Enterprise tiers<\/p>\n Augment Code\u2019s 70.6% SWE-bench score is self-reported using Augment\u2019s own harness and published on Augment\u2019s engineering blog<\/a>. As with all agent-scaffolding-dependent scores, it should be read as \u201cwhat Augment + Opus 4.5 achieves with Augment\u2019s context engine,\u201d not a standalone model number. That caveat stated, the architectural insight behind the score is real and independently validated: in the February 2026 scaffold comparison described earlier, Augment\u2019s context-first approach outperformed other frameworks running the same model by 17 problems out of 731.<\/p>\n The core innovation is that Augment\u2019s engine indexes an entire repository before the agent begins work \u2014 rather than building context reactively from open files. For enterprise teams working in large, mature monorepos, this produces measurably better results on tasks that require cross-module reasoning. Augment also exposes its context engine via MCP (Model Context Protocol), making it interoperable with other agents. A developer could use Augment\u2019s indexing while running Claude Code or Codex for generation.<\/p>\n Best for:<\/strong> Enterprise teams with large, mature codebases who need deeper repository context than single-session tools provide.<\/p>\n Pricing:<\/strong> Free (open-source); pay for model API inference Architecture:<\/strong> Git-native terminal agent<\/p>\n Aider is the git-native coding agent: it operates directly in your local repository and structures its changes as a series of atomic git commits with descriptive messages \u2014 a workflow that meshes well with teams that do careful code review. It supports any OpenAI-compatible model, giving the same model-agnostic flexibility as OpenHands, and runs entirely in the terminal with no IDE dependency.<\/p>\n Where Aider lags behind higher-ranked tools is on complex, multi-step agentic tasks that require web access, browser interaction, or long-horizon planning. It is a powerful tool within a clearly defined scope \u2014 terminal-based, git-integrated coding \u2014 rather than a general-purpose autonomous agent.<\/p>\n10 AI Agents for Software Development<\/strong><\/h2>\n
A Note on Claude Mythos Preview<\/strong><\/h3>\n
#1. Claude Code (Anthropic)<\/a><\/strong><\/h3>\n
#2. OpenAI Codex (OpenAI)<\/a><\/strong><\/h3>\n
openai\/codex<\/code><\/a>) runs a local agent loop in your terminal, using OpenAI\u2019s API for model inference. The cloud execution surface \u2014 where tasks run in an isolated VM without touching your local environment \u2014 is the Codex web product and IDE integrations, not the CLI. This distinction matters for security, network access, and cost modeling.<\/p>\n#3. Cursor<\/a><\/strong><\/h3>\n
#4. Gemini CLI (Google DeepMind)<\/a><\/strong><\/h3>\n
npm install -g @google\/gemini-cli<\/code>). Its primary model is Gemini 3.1 Pro<\/a><\/strong> \u2014 released February 19, 2026 \u2014 which scores 80.6% on SWE-bench Verified and 68.5% on Terminal-Bench 2.0. Gemini 3 Flash (approximately 78% SWE-bench Verified) is the lighter, cheaper option within the same CLI. These are distinct capabilities and the Gemini 3.1 Pro number is the correct headline for what Gemini CLI can deliver at full configuration.<\/p>\n#5. GitHub Copilot (Microsoft\/GitHub)<\/a><\/strong><\/h3>\n
#6. Devin 2.0 (Cognition AI)<\/a><\/strong><\/h3>\n
#7. OpenHands<\/a> \/ OpenDevin (All-Hands AI)<\/strong><\/h3>\n
#8. Augment Code<\/a><\/strong><\/h3>\n
#9. Aider<\/a><\/strong><\/h3>\n
![\"\u26a0\"](\"https:\/\/s.w.org\/images\/core\/emoji\/17.0.2\/72x72\/26a0.png\")
Benchmark Alert<\/div>\n