In an analysis of 24 state-of-the-art language-model benchmarks, Reuel et al. (2024) found that only 4 provided scripts for replicating results, and no more than 10 performed multiple evaluations or reported statistical significance.
Sources
- Can We Trust AI Benchmarks? An Interdisciplinary Review ... (arxiv.org)
- https://proceedings.neurips.cc/paper_files/paper/2024/file/26889e8359e7ef8a7f5d77457364ca55-Paper-Datasets_and_Benchmarks_Track.pdf (proceedings.neurips.cc)
- https://www.researchgate.net/publication/386014536_BetterBench_Assessing_AI_Benchmarks_Uncovering_Issues_and_Establishing_Best_Practices (researchgate.net)
More in Technology
- AI models evaluated against FrontierMath include Anthropic's Claude 3.5.
- The paper 'Harmonising Artificial Intelligence' was published by Oxford Information Labs in 2021.
- In MIT Technology Review's 2016 Breakthrough Technologies list, Helix offered genome sequencing for $80 and operated an app store where customers could share their genetic data with third parties for medical analysis or personalized merchandise.
- Interval Research Corporation was financed by Microsoft cofounder Paul Allen.
- Michael Marcus spent 25 years at the Federal Communications Commission, where he helped develop policies that made more than eight gigahertz of radio spectrum available for unlicensed use.
- Smartphone or tablet-based time use surveys can passively collect data such as geolocation for activities and periods when the device was in use.