Splitting the Data: Train and Test Sets

The fix for grading your own homework is simple in concept: before training even starts, split the labeled data into two separate sets — a training set, which the model actually learns from, and a test set, which is set aside completely untouched until evaluation.

This only works if the separation is real: the test set has to represent questions the model genuinely never saw during training, not questions that snuck in some other way — otherwise you are right back to grading your own homework.

Picture 1,000 labeled photos of cats and dogs. A common approach is training on 800 of them and holding the other 200 back entirely — the model never sees those 200 photos, or their labels, at any point while it's learning. Once training is done, those 200 photos are used, for the first and only time, to check how well the model actually performs.

This rule is strict for a reason: even a small leak — like accidentally training on a handful of test examples — quietly inflates how good the model looks, without actually making it better. Keeping the test set untouched is one of the most basic disciplines in doing machine learning properly.

What is the purpose of holding out a separate test set that the model never trains on?

Before training begins, labeled data is typically split into a training set and a separate ____ that the model never learns from.

Accidentally including a few test examples in the training data has no real effect on how trustworthy the evaluation is.