History Isn't Neutral, and Neither Is the Data

Training data doesn't describe the world as it should be — it describes the world as it actually was, including whatever inequities existed at the time. A hiring-screening tool trained on ten years of a company's past hiring decisions doesn't just learn what the job requires. It learns who actually got hired, and if that history skewed toward a narrow set of backgrounds, the tool learns that pattern right along with it.

This is a different mechanism than simply having "not enough data." The data can be enormous, detailed, and perfectly accurate about what happened — and still bake in bias, because what happened wasn't fair to begin with. The model isn't misreading the data; it's reading it correctly and learning exactly the pattern that's there.

The catch is that the model has no way to tell the difference between a pattern that reflects a real job requirement and a pattern that reflects historical exclusion. Both look the same to it: a correlation between some input and the outcome "hired." Sorting out which correlations are legitimate and which are inherited unfairness is a human judgment call, not something the training process figures out on its own.

Why can a hiring-screening tool trained on accurate historical hiring data still end up biased?

A model trained on historically skewed hiring data learns who actually got ____ in the past, not just what the job genuinely requires.

Training data can be completely accurate about what happened in the past and still produce a biased model.