GEPI’s How AI Fails Women Report Examines How Algorithms Amplify Gender Bias

PUBLISHER ARTICLE PREVIEW

From the original article

A hiring score, a credit assessment, or a clinical recommendation can look impartial while carrying forward the inequalities in the data behind it. That is the central warning of the Gender Equity Policy Institute’s new report, How AI Fails Women: When Bias Becomes Amplified, which brings together research on how automated systems can reproduce—and sometimes intensify—gender discrimination.

Written by machine learning scientist and GEPI senior fellow Myriam Shiran, PhD, the report reviews academic and gray literature across economic opportunity, healthcare, criminal justice, generated content, consumer AI design, and AI-enabled image-based sexual abuse. It is a synthesis of existing evidence, rather than a new experiment measuring every AI system against a single benchmark.

Its most consequential argument is about what happens after a biased result leaves a model. Recommendations influence decisions, generated content shapes expectations, and some outputs become material for future training. An unfair pattern can therefore become harder to identify as it moves through successive systems.

Removing a gender field is an intuitive response to discrimination. GEPI’s review explains why that step alone is insufficient: names, education, employment histories, and other details can still carry gender-related signals. Historical outcomes may also encode unequal treatment before a model ever encounters them.

One hiring audit cited in the report, by Wilson and Caliskan in 2024, involved more than three million simulated résumé comparisons across nine occupations. The tested screening models significantly favored male-associated names in 51.9% of comparisons and female-associated names in 11.1%. The remaining 37% did not show a significant preference. These are results from controlled model tests, not percentages of real applicants denied jobs.

The evidence also varies by test design. The report describes a separate 2025 audit in which several language models showed no gender difference when researchers changed names alone, while changes to education details affected scores. A narrow test can miss information that matters elsewhere in the same workflow.

In lending, the review points to research on roughly 80,000 unsecured personal loans from a major U.S. fintech lender. Women’s loans generated higher returns for the lender, a finding the study linked to a model overestimating their default risk and pricing loans accordingly.