Why You Can't Grade Your Own Homework

Imagine a student who gets the exact practice questions and answer key ahead of time, memorizes every answer word for word, and then takes a test made of those same exact questions. A perfect score wouldn't prove they understood the subject — it would only prove they can memorize. Evaluating a machine learning model on its own training data has the exact same flaw.

If a model is tested using the very same labeled examples it was trained on, a model that simply memorized the quirks of those specific examples — instead of learning the actual underlying pattern — will score perfectly. That perfect score tells you nothing about how it handles a new input it has never seen before.

This isn't just a technical nitpick. A model can score well throughout development and still perform badly the moment it meets real, new data — if every check along the way only ever used data it had already memorized.

Why is it misleading to evaluate a model using the same data it was trained on?

A model that scores perfectly on its own training data but performs badly on new data has likely ____ the training examples rather than learning the general pattern.

A model that scores perfectly when evaluated on its own training data is guaranteed to perform just as well on brand-new data.