Hugging Face introduces FairStress benchmark to assess fairness in LLMs

Hugging Face introduced FairStress, a benchmark for measuring demographic fairness in language models beyond the final choice in high-impact decisions. The tool contains 300 realistic two-candidate decision scenarios across 29 areas, including hiring, healthcare, lending, education, housing and criminal justice, and is available for download. The company is also adding the benchmark to Olmo-eval, its open tool for running and comparing evaluations.

According to Hugging Face, tests across models including Olmo, Llama, Qwen, Gemma, Claude and the GPT family indicate that the same persuasive arguments can make models more or less likely to change a decision depending on which demographic group the argument favors. FairStress aims to measure this asymmetry, along with differences in explanations and internal model patterns, using a definition of fairness based on applying the same criteria equally—not an attempt to cover every conception of equity.