FairStress: A deeper evaluation of fairness in LLM decision-making

PUBLISHER ARTICLE PREVIEW

From the original article

Today we’re introducing FairStress, a benchmark for measuring demographic fairness in LLMs beyond the choice a model ultimately makes in decisions impacting people. FairStress is available for download, and we’re also building it into Olmo-eval, our open workbench for running and comparing model evaluations.

Models are increasingly being used in high-stakes settings where their outputs can affect people’s opportunities, resources, and care, from screening job applicants and approving loans to prioritizing patients for treatment and evaluating students for admissions. Standard demographic fairness benchmarks often test whether a model selects the same candidate in otherwise identical scenarios after the candidates’ demographic identities are swapped.

For example, if one job candidate has clearly stronger experience and qualifications, does the model still choose that candidate when the stronger candidate is Black in one version of the scenario and white in the other, with the qualifications held constant?

Arriving at the same answer in two versions of a scenario can hide meaningful differences in how readily a model changes that decision when challenged, however. Suppose Candidate A is clearly stronger and the model selects A whether A is Black or white. That looks fair at the output level, but if the model is then presented with a persuasive argument – for example, that Candidate B is preferred by senior leadership, or that people from B’s demographic group have historically been overlooked – it may switch away from the stronger candidate much more readily in one version of the scenario than the other.

Across open-weight and frontier models including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, and across multiple stages of training, we find that the same persuasive arguments – which we call adversarial prefixes – can make models more or less likely to change their decisions depending on which demographic group the argument favors. We also find differences in how models explain decisions involving demographic information – and in their internal activation patterns – across domains including hiring, lending, healthcare, education, and housing.

With FairStress, our target is symmetric fairness—when the evidence supports a decision, a model should apply the same criteria and be equally resistant to changing that decision regardless of which demographic group benefits. This reflects a particular definition of fairness centered on equality. All else equal, people of different identities should be equally likely to be selected; when one candidate is clearly the strongest choice, identity should not change that likelihood; and well-supported decisions should be equally robust to adversarial prefixes regardless of which group benefits.

FairStress does not attempt to capture every conception of fairness. An equity-based approach, for example, might treat historical discrimination or underrepresentation as relevant to a decision and therefore favor a minority candidate even when other factors are held constant.

FairStress contains 300 realistic two-candidate decision scenarios across 29 domains including hiring, healthcare, lending, education, housing, and criminal justice. For example, one scenario asks a model to choose between two candidates for a mechanical engineering job, while another asks which of two patients should receive the last unit of a scarce medication.

Some FairStress scenarios have a clearly stronger candidate whose profile better satisfies the stated criteria. That lets us test whether demographic information changes how often a model selects the candidate best supported by the evidence. Other scenarios are deliberately ambiguous—both candidates are strong in different, hard-to-compare ways so there’s no objectively better choice.

These cases let us examine which candidate a model favors when the evidence does not clearly point to one over the other.

FairStress also varies how explicitly demographic identity is presented, from no identity information to a demographically coded name, an indirectly implied identity, or an explicit demographic label. This allows us to test whether demographic differences in model behavior emerge only when identity is stated outright or also when it’s conveyed subtly.

Our benchmark goes further by testing how stable models’ decisions are when challenged by adversarial prefixes.