A study (arXiv 2411.12990) found that commonly used AI benchmarks suffer from significant issues and exhibit large differences in quality.
Sources
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (arxiv.org)
- https://proceedings.neurips.cc//paper_files/paper/2024/hash/26889e8359e7ef8a7f5d77457364ca55-Abstract-Datasets_and_Benchmarks_Track.html (proceedings.neurips.cc)
- https://betterbench.stanford.edu/ (betterbench.stanford.edu)
- https://neurips.cc/virtual/2024/poster/97566 (neurips.cc)
More in Technology
- Google researchers labeled gender categories in their smile-detection model as 'Gender 1' and 'Gender 2' specifically to reduce the risk of introducing unconscious societal bias.
- When GPT-4 was tested in 2023 on easy Codeforces problems, Narayanan and Kapoor (2023b) found that it could regularly solve problems added before 5 September 2021 but got no question right among problems added afterward.
- A software defect in the Chinook helicopter was described as "positively dangerous" because deficiencies meant the pilot's full control of the engines "could not be assured."
- The UK has an AI Security Institute involved in testing AI models' cybersecurity behavior.
- Michael Marcus spent 25 years at the Federal Communications Commission, where he helped develop policies that made more than eight gigahertz of radio spectrum available for unlicensed use.
- AI melanoma-diagnosis systems perform better on white skin than black skin, largely because their training data contains more examples of white skin.