FrontierCode

Product · Data product / dataset by Cognition

The first benchmark to measure mergeability for production-quality code.

FrontierCode is a benchmarking product that measures the "mergeability" of AI-generated code against production standards. It provides AI researchers with 132 maintainer-crafted tasks to evaluate models on correctness, style, and scope discipline.

Launched
2026-06-08

Category: Software Testing

Delivery formats
Web-based leaderboard and task walkthroughs
Task Count (v2.0)
132 tasks
Maintainer Effort per Task
40+ hours
Reasoning Effort Levels
Low, Medium, High, XHigh, Max
Evaluation Standard
Mergeability
Task Categories
Security patches, performance optimizations, bug fixes
  • Measures "mergeability" by production standards
  • 132 tasks crafted by open-source maintainers
  • 40+ hours of maintainer effort per task
  • Evaluates correctness, style, and scope discipline
  • Includes security patches and performance optimizations

Overview

FrontierCode is Cognition's benchmarking product designed to evaluate AI models against production-quality software engineering standards. It is intended for AI researchers and developers to measure "mergeability," determining if a human maintainer would actually accept an agent's code into a repository. The benchmark utilizes a suite of 132 tasks crafted by open-source maintainers, covering complex areas like security patches and performance optimizations. Each task requires over 40 hours of maintainer effort to define grading criteria, which include unit tests, custom rubrics, and automated verifiers. FrontierCode sets itself apart by enforcing realistic internet usage restrictions and conducting manual reviews of every task by Cognition researchers to ensure high-quality evaluation.

Evaluation Framework

  • Mergeability Measurement: The benchmark evaluates whether a human maintainer would actually merge the pull request generated by an AI model.
  • Ensemble Grading Techniques: Grading is performed using an ensemble of unit tests, custom rubrics, and novel verifiers to check for correctness and test quality.
  • Code Quality Standards: The benchmark assesses adherence to codebase standards, scope discipline, and coding style.
  • Maintainer-Crafted Tasks: Every task is crafted by open-source maintainers of the repositories they originate from, with over 40 hours spent per task.

Data and Methodology

  • 132 Diverse Tasks: The FrontierCode 2.0 suite includes 132 tasks spanning diverse repositories and categories.
  • Advanced Task Categories: Tasks include complex engineering work such as security patches and performance optimizations.
  • Realistic Internet Usage: The benchmark restricts internet usage to mimic an engineer's workflow, such as reading documentation and searching for error messages.
  • Anti-Cheating Mechanisms: A scanner and model judge identify and zero out runs that use reference solutions or fetch content from upstream repositories.

Use cases

  • AI researchers use the benchmark to compare the mergeability of different frontier models against production standards.
  • Model developers evaluate how reasoning effort levels impact the cost and quality of generated code.
  • Engineering leaders analyze the Pareto frontier of AI coding capability to select models for autonomous workflows.
  • Maintainers define realistic coding tasks to test if AI agents can adhere to specific codebase styles and standards.

Specifications

Coverage
132 tasks spanning multiple repositories and categories including security patches and performance optimizations
Update frequency
Periodic revisions (v1.0 in June 2026, v1.1 in July 2026, v2.0 in October 2026)
Delivery formats
Web-based leaderboard and task walkthroughs
Task Count (v2.0)
132 tasks
Maintainer Effort per Task
40+ hours
Grading Criteria
Correctness, test quality, scope discipline, style, and codebase standards
Reasoning Effort Levels
Low, Medium, High, XHigh, Max
Evaluation Standard
Mergeability
Task Categories
Security patches, performance optimizations, bug fixes

How it works

  1. Task Creation: Open-source maintainers create realistic tasks and define the specific criteria for what makes a pull request mergeable in their repository.
  2. Model Evaluation: AI models attempt the tasks across different reasoning effort levels (low, medium, high, xhigh, max) while being monitored for unfair internet use.
  3. Multi-Stage Grading: An ensemble of unit tests, custom rubrics, and verifiers grades the output, followed by a manual review by a Cognition researcher.
  4. Leaderboard Publication: Results are aggregated into a weighted score and published on the live leaderboard, comparing model performance against cost and speed.

FAQ

How does FrontierCode differ from traditional coding benchmarks?

FrontierCode measures mergeability, which asks if a human maintainer would actually merge the code, rather than just checking if it passes basic unit tests. It uses an ensemble of unit tests, rubrics, and verifiers to assess correctness, style, and scope.

Who creates the tasks for FrontierCode?

Every task is crafted by the open-source maintainers of the repositories the tasks come from. These developers spend more than 40 hours per task defining what 'mergeable' means for their specific codebase.

How many tasks are included in the benchmark?

FrontierCode 2.0, released in October 2026, includes 132 tasks. This version expanded the suite to include more repositories and categories like security patches and performance optimizations.

Are models allowed to use the internet during the benchmark?

Models are allowed to use the internet as an engineer would, such as for documentation or error messages. However, a scanner and model judge zero out any runs that attempt to access reference solutions or original pull requests.

Added by AntSpace from public sources (cognition.com).

Related

More products by Cognition