Loading...
In 2018, Reuters revealed that Amazon had built an AI recruiting tool that systematically penalized resumes containing the word 'women's' — as in 'women's chess club captain' or 'women's college.' The system had been trained on a decade of hiring data from Amazon's own workforce, which was overwhelmingly male in technical roles. The algorithm learned that successful hires (as defined by the training data) tended to be men, so it treated female-associated signals as negative predictors. Amazon's engineers tried to patch the bias by removing explicitly gendered terms, but the system found proxies — certain colleges, certain phrasings, certain extracurricular patterns that correlated with gender. Each fix led the model to discover subtler correlations. The project was eventually scrapped i...
Popular framing: A biased engineer wrote a sexist algorithm at Amazon.
Structural analysis: The system optimized for 'candidates who look like past successful hires,' but past success was itself shaped by biased hiring and promotion. Each patch on an explicit signal made the model find subtler correlates, and the veneer of mathematical objectivity made the bias harder to challenge than a human's. Feedback loops between training data and deployment freeze prior bias into future selection.
The gap matters because it misdirects intervention. If the problem is framed as bad data, the solution is better data collection — which keeps decision-making authority with algorithm vendors and employers. But if the problem is that the optimization target (past hiring success) encodes inequality, then no dataset curation can solve it — you must change what is being predicted. The popular framing also obscures information asymmetry: applicants cannot audit the model that judges them, so they cannot contest it, making the feedback loop self-sealing.
