[Submitted connected 18 Feb 2026 (v1), past revised 29 Jun 2026 (this version, v3)]
Authors:Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek Šuppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, Yiyang Nan, Jyoutir Raj, Yu Fan, Shubham Singh, Subramanyam Sahoo, Eliya Habba, Usman Gohar, Siddhesh Pawar, Robert Scholz, Arjun Subramonian, Jingwei Ni, Mykel Kochenderfer, Sanmi Koyejo, Mrinmaya Sachan, Stella Biderman, Zeerak Talat, Avijit Ghosh, Irene Solaiman
View PDF HTML (experimental)
Abstract:Artificial intelligence benchmarks are an important system for measuring exemplary advancement and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their semipermanent value. In this study, we specify benchmark saturation and analyse it crossed 60 connection exemplary benchmarks utilizing 14 properties that subordinate to saturation. We find that astir half of the our benchmarks grounds saturation, pinch rates expanding pinch age. Further, we find that resilience to saturation is impacted by expert-curation, not by nationalist trial data. Our results propose that creation choices tin widen benchmark longevity and pass much durable information approaches.Submission history
From: Mubashara Akhtar [view email]
[v1] Wed, 18 Feb 2026 16:51:37 UTC (222 KB)
[v2] Sat, 30 May 2026 16:41:50 UTC (640 KB)
[v3] Mon, 29 Jun 2026 17:01:58 UTC (636 KB)
English (US) · 
Indonesian (ID) ·