SWE-bench Verified Dataset

SWE-bench
Verified

500 manually verified real software engineering tasks created by the Princeton NLP team, sourced from real GitHub repositories' bug fixes and feature requests, serve as the gold standard benchmark set for evaluating AI code agents (such as Claude Code, Devin, SWE-Agent).

Get the dataset now
500 verified tasks 12 Python repositories MIT License Princeton NLP
🔧
500
Number of verified tasks
📦
12
Python repositories
📜
MIT
Open License Agreement
🏛️
Princeton
NLP Research Team

Dataset Highlights

The authoritative benchmark for evaluating AI code agents' capabilities, with each task meticulously verified by humans.

🐛

Real GitHub Issues

Each task comes from real GitHub repository Issues, containing complete problem descriptions, contextual information, and code change history, reflecting real software development scenarios.

✅

Human-Verified Tasks

All 500 tasks have been individually reviewed and verified by humans to ensure clear problem descriptions, correct solutions, and effective test patches, eliminating noise from automated filtering.

🧪

Test Patch Verification

Each task comes with a corresponding test patch (Test Patch) that can automatically verify whether the AI-generated code fixes pass all relevant test cases correctly.

🏗️

12 Major Python Repositories

Covers well-known open-source projects such as Django, Flask, scikit-learn, sympy, matplotlib, spanning multiple fields including web frameworks, scientific computing, and data visualization.

🤖

Code Agent Benchmark

Industry-recognized evaluation standard for AI code agents, with cutting-edge systems like Claude Code, Devin, SWE-Agent, and OpenDevin measured against this benchmark.

📄

MIT License

Utilizes the MIT open-source license, freely usable for academic research, commercial evaluation, and product development, with no usage restrictions, and an active community for continuous updates.

Applicable Scenarios

Comprehensive assessment of AI programming capabilities from academic research to engineering practice

🤖

AI Code Agent Evaluation

Measure the performance of AI programming assistants like Claude Code, Devin, SWE-Agent, etc., in real software engineering tasks

🔬

Software Engineering Research

Provide standardized evaluation datasets for research directions such as automated program repair, code generation, and defect localization

🛠️

Automated Bug Fixing

Evaluate and train automated defect repair systems, validating their generalization capabilities across projects and domains

📊

Development Tool Benchmarking

Provide standardized performance evaluation schemes for IDE plugins, code review tools, and CI/CD systems

code benchmark software-engineering GitHub AI-agents

Data Preview

The following is a single task example structure from the SWE-bench Verified dataset

JSON
{
  "instance_id": "django__django-16379",
  "repo": "django/django",
  "base_commit": "a1e4e3f...",
  "problem_statement": "FileBasedCache has_key is susceptible to race conditions...",
  "hints_text": "",
  "created_at": "2022-12-04T17:43:25Z",
  "patch": "diff --git a/django/core/cache/backends/filebased.py ...",
  "test_patch": "diff --git a/tests/cache/tests.py ...",
  "version": "5.0",
  "FAIL_TO_PASS": ["test_has_key_race_handling (cache.tests.FileBasedCacheTests)"],
  "PASS_TO_PASS": ["test_cache_key_i18n (cache.tests.FileBasedCacheTests)", "..."],
  "environment_setup_commit": "4c76ffc..."
}

3 Steps to Get Started Quickly

Start in minutes from data acquisition to evaluating the AI agent

01

Browse Datasets

View dataset details on the Ace Data Cloud platform to understand metadata such as task structure, coverage repository, and licensing agreements.

02

Get Data

Obtain 500 validation tasks via API, each containing a problem description, code patch, and test patch, ready to use.

03

Run Evaluation

Compare the patches generated by the AI agent with test_patch for validation, assess the pass rate, and generate performance reports.

Get Dataset

Start Evaluating AI Code Agents

500 manually verified real tasks, MIT open license, available immediately. Whether you are a researcher or an engineer, this benchmark dataset is the preferred choice for evaluating AI programming capabilities.

Get Dataset Now