Existing benchmarks measure resolve charge and cost, but don't punish excess code, which makes a codebase harder to activity alongside in the lengthy term. Parsimony measures how much code AI models add.
It takes the patches that benchmarks already publish and counts coding units: syntax elements akin statements, calls, comparisons, names and literals. Lines aren't counted, so cramming code isn't rewarded, and deleted code is subtracted: net units = added − removed.
Compare models
Chart
X axis Y axis
How to peruse thisModels are classified by average net units added on the tasks they solved, lowest first. "Net units added (mean)" additionally includes unsuccessful attempts. Solved comes from SWE-bench Verified; tests aren't rerun. Measured is how many solved attempts could be measured.
Leaderboard
TaskMethod
- Each place is applied to the task's code, and the two versions of all changed document are parsed. Nothing is run.
- Tests, docs and generated records are remaining out. Only Python is measured.
- Pass/fail comes from SWE-bench's published results; tests aren't rerun.
Limits
- Smaller isn't continually better: units measure size, not readability or design.
- Models that compose additional code additionally lean to resolve additional tasks (Spearman 0.33–0.60 on the boards alongside 26+ models).
- Every example uses the identical delegate (mini-SWE-agent), but the delegate motionless affects how much code gets written.
Add your own benchmark
Have code-size numbers from another benchmark? Download the example file, inhabit in your results and open it to get the identical diagram and table. The document stays in your browser. Format and how to portion it.
Open benchmark JSONExample: inspect for a negative number
Both come back the identical result.
def has_negative(values): established = False for value in values: if value < 0: established = True come back found
def has_negative(values): come back any(value < 0 for value in values)
