The Epoch Capabilities Index (ECI) is a comprehensive metric created by Epoch AI to evaluate and track the general capabilities of artificial intelligence models over time. As individual evaluation tasks frequently suffer from benchmark saturation, the ECI addresses this by combining performance data from over 50 distinct benchmarks into a single, unified scale.

Methodology and Design

The foundational framework of the ECI is documented in the technical paper A Rosetta Stone for AI Benchmarks, developed in collaboration with researchers from Google DeepMind’s AGI Safety & Alignment team. While Google DeepMind provided funding and collaborative input, the ECI remains an independent product maintained by Epoch AI.

Rather than manually assigning difficulty ratings to specific tasks, the ECI uses a statistical model fit jointly across all evaluation data. It determines relative difficulty and task slope by analyzing instances where models are evaluated across multiple overlapping benchmarks. Models receive higher general ECI scores when they perform well on tasks determined to be statistically harder. To ensure stability and minimize noise, models must have at least four benchmark evaluations to be included in the general ECI.

Interpretation and Calibration

Similar to an IQ test or Elo rating, ECI scores represent broad underlying capabilities across a spectrum of tasks rather than performance on a single isolated skill. The values are linear and relative; a 10-point gain represents the same relative capability jump anywhere along the scale. However, absolute numerical values are arbitrary and cannot be interpreted without comparison.

To establish a clear baseline, raw capability scores are scaled against explicit anchor points: Claude 3.5 Sonnet is assigned an ECI score of 130, while GPT-5 is scaled to 150. The scale has no upper ceiling, permitting indefinite long-term progress tracking. Epoch notes that during initial development, a 5-point increase on the ECI scale roughly correlated with a doubling of the METR Time Horizon.

Domain-Specific and Cyber ECIs

In addition to general capability, Epoch provides Domain-Specific ECIs to measure proficiency in targeted disciplines such as software engineering (SWE) and mathematics:

• Domain-Specific ECIs: These metrics retain the benchmark difficulty and slope parameters established by the general ECI but recalculate model capability parameters strictly using a subset of benchmarks from that specific domain (requiring a minimum of two domain-specific benchmarks). Comparing a model's domain ECI to its general ECI indicates whether it overperforms or underperforms in that area.

• Cyber ECI: Unlike other domain indexes, the Cyber ECI incorporates specialized cybersecurity benchmarks not included in the primary ECI dataset. As a result, it follows a distinct refitting process and is updated less frequently.

Limitations and Bias Mitigation

Epoch addresses several technical challenges and limitations inherent in aggregated AI benchmarking:

• Developer Cherry-Picking: Model creators may selectively release scores where their systems perform best. Epoch mitigates this by running internal evaluations and incorporating independent third-party leaderboards alongside developer model cards.

• Benchmark Contamination and Optimization: Developers frequently tune models specifically for popular public benchmarks. Open-weight models appear to optimize against benchmark sets particularly aggressively, which may understate the true capability gap between open-weight and proprietary systems.

• Benchmark Compatibility: Only benchmarks with performance scores normalized between 0 and 1 are compatible; Elo-style relative comparisons (such as WebDev Arena) are excluded.