Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

Sep 02, 2026 11:14 PM - 1 hour ago 2

Beyond the numbers

01
OpenCode: failures excluded.
It only covers 15 passes. Count grounded attempts and the number becomes $3.24 per task.
02
Cache deed complaint is not cost.
A cached 300-turn nonaccomplishment tin still pain much than a short cache miss.
03
Quality and costs tin diverge.
Claude Code passes 19 tasks, but reaches $18.34 successful costs per task.

If you want to trial your ain harness connected Runta, we’ll springiness you $100 successful credits to get started.

Get $100 successful creditsStart free trial

Codex

v0.148.0

DeepSeek Harness

v0.1.0-rc.8

Claude Code

v2.1.237

v0.84.2

Oh My Pi

v17.4.0

Kimi Code

v0.37.2

Exo Harness

v0.1.0

OpenCode

v1.18.19

Hermes

v0.20.4

FrontierHarness v1.0 focuses connected package engineering contexts and terminal-based tasks. It whitethorn not generalize to different areas of knowledge work.
Evaluated connected Runta supplier runtimes. All harnesses and the task situation are prepared erstwhile arsenic a aureate checkpoint. Every tally is simply a caller reconstruct pinch identical vCPU, memory, disk size, disk contents, and representation state.