Your model failed on new data. Check the split before changing it.

Workflow · 4 min read · By Bibek Howlader, Founder, Labrix

Your model looks excellent on the test set. You move it to another dataset, and the result falls apart.

The next experiment is tempting: a larger model, more tuning, a different architecture. Before spending that compute, write down what the first evaluation actually held out.

Was it new rows? New participants? A new institution? A future period?

Those are different tests. Which one matters depends on the research claim.

A published result can depend on the evaluation mistake

Kapoor and Narayanan's 2023 paper in Patterns investigated leakage in machine-learning-based science. Its civil-war prediction case study found that claimed advantages of complex machine-learning methods over logistic regression did not reproduce after leakage was addressed. That finding concerns the studies they examined; it does not establish that complex methods cannot improve prediction in other settings.

The research question remains useful. The claimed progress has to survive a valid comparison.

For a new project, that makes the evaluation design part of the contribution from the beginning.

Think about what “new” means

Here is an invented example, not a published benchmark or a Labrix result.

A team collects recordings of factory equipment. Each machine contributes multiple recordings, and each recording is divided into short overlapping windows. The goal is to detect faults on machines the team has never observed.

The team randomly assigns windows to training and test sets. Windows from the same recording can now appear in both. A good score may partly reflect familiar recording conditions or near-duplicate signals.

Holding out entire recordings would ask a harder question. Holding out entire machines would come closer to the stated use case. Holding out a factory could examine another boundary, if the intended claim includes transfer to new factories.

None of these splits is automatically right for every project. A split suitable for recognising another recording from a known machine could be unsuitable for a claim about unseen machines.

Scikit-learn's cross-validation documentation describes why grouped and time-ordered data need evaluation strategies that respect their structure. A random row split does not automatically make the evaluation units independent.

Check three possible paths for information to cross the boundary

Shared origins. Trace each row back to the thing it came from: a participant, recording, document, device, location, or collection batch. Ask whether sharing that origin between sets fits the intended use case. Check duplicates and derived samples before treating rows as separate observations.

Preprocessing. Check where scaling, imputation, feature selection, and other learned transformations were fitted. Training-only fitting must also happen inside each cross-validation fold. Scikit-learn's common-pitfalls guide demonstrates how preprocessing before separation can leak information and inflate evaluation scores. A pipeline can help enforce this boundary when used with an appropriate split.

Model selection. Record which results influenced architecture, features, thresholds, or tuning. A test set repeatedly consulted for those choices has become part of development. The cross-validation guide explains the role of validation and a held-out final evaluation in avoiding that feedback.

These checks do not explain every disappointing result. They help establish whether your comparison deserves the interpretation you gave it.

Write the evaluation contract before the next run

Use these questions to make a short project note:

What decision will the model support? Describe the intended use in one sentence.
When is the prediction made? List the information genuinely available at that moment.
What must be unfamiliar at evaluation? Identify the participant, machine, location, time period, or other boundary relevant to the claim.
What can change during development? Record which data can guide those choices.
What result would narrow the claim? Decide what you would report if performance drops beyond the original setting.

For our invented equipment project, the note might read:

We aim to classify faults in recordings from previously unseen machines of the studied type. We will develop the method using training and validation machines, then evaluate the selected procedure on separate machines. Recordings and their derived windows will remain together. Transfer to another factory remains a separate question.

This note is deliberately more specific than “we used an 80/20 split.” It tells a collaborator what the evaluation is meant to establish.

Keep the result that changed your mind

Suppose the revised evaluation produces a much lower score. Keep a record of both procedures, explain what changed, and report the defensible estimate. You have learned that the earlier evaluation answered an easier question than intended.

You can turn that finding into a useful investigation: which conditions account for the performance gap, and what additional evidence would separate the explanations?

A fall on new data can also reflect distribution shift, label differences, or other problems rather than leakage. Document those possibilities without treating one as proven by the drop alone. A clean split is necessary for some claims, but it is not a guarantee of useful performance in every setting.

In Labrix, keep the literature and manuscript discussion connected as you revise the claim. Checking the experiment itself requires inspecting the data and code; a convincing write-up cannot settle whether the split was valid.

Before changing the model, bring a diagram of your data's origins to a collaborator. Mark where training, validation, and test boundaries fall. Ask what information could travel across them.

What evaluation change most reduced a result in your own work: removing duplicates, separating groups, holding out a location, or respecting time? What did the lower score teach you?