Google's Gemma-2 9b and OpenAI's GPT-4o performed poorly on a Stanford team's descriptive and normative bias benchmarks despite scoring near-perfectly on Anthropic's DiscrimEval.
Sources
- These new AI benchmarks could help make models less biased (technologyreview.com)
- https://arxiv.org/abs/2502.01926 (arxiv.org)
- https://arxiv.org/html/2502.01926v2 (arxiv.org)
- https://hai.stanford.edu/news/ais-fairness-problem-when-treating-everyone-the-same-is-the-wrong-approach (hai.stanford.edu)
- https://law.stanford.edu/stanford-lawyer/articles/ai-fairness-through-difference-awareness/ (law.stanford.edu)
More in Technology
- Google's Gemma-2 9b and OpenAI's GPT-4o achieved near-perfect scores on Anthropic's DiscrimEval bias benchmark, released in December 2023.
- The CIA's Aquiline drone program was initially modeled on birds' flight characteristics but never became operational.
- Nokia once manufactured rubber boots and toilet paper before becoming a mobile phone company.
- The paper arXiv 2411.12990 was accepted as a Spotlight Poster to NeurIPS 2024.
- Canyon SIGINT satellites were in use during the same period as Aquacade satellites with an apparently different set of capabilities.
- The paper 'Harmonising Artificial Intelligence' was published by Oxford Information Labs in 2021.