Two papers disagree. What should your literature review say?

Discovery · 4 min read · By Bibek Howlader, Founder, Labrix

One paper says an intervention helped. Another says it did not. You have read both twice, and your literature review still ends with “the evidence is mixed.”

That sentence leaves the reader with the work you were trying to do: understand the disagreement.

Before choosing which result to believe, establish whether the papers asked the same question. Then ask whether their estimates differ enough to justify the story you are telling about them.

Two different labels can describe similar results

Consider an invented example from education research. These numbers illustrate the reasoning; they are not results from published studies.

Two randomised studies compare the same tutoring programme with usual teaching and measure the same examination score. Assume the estimates below come from comparable analyses. Positive values favour tutoring.

StudyEstimated difference95% confidence intervalAuthors' significance label
A+4 points+1 to +7Statistically significant
B+3 points−2 to +8Not statistically significant

If you record only the last column, the studies look like opposing findings. If you record the estimates, both point toward a similar improvement, with considerable uncertainty in Study B.

This table does not establish that the effects are equal. Overlapping confidence intervals are not a formal test of equality, either. It does show why “one worked and one failed” is an inadequate description.

For this specific numerical comparison, a more informative account would be:

Both studies estimated higher scores with tutoring. Study B was less precise, and its interval included both a small decrease and a substantial improvement. These results alone do not establish that the programme had different effects in the two studies.

Cochrane's guidance on evidence synthesis explains why counting statistically significant studies as successes and the others as failures can produce misleading conclusions. Keep the estimates and their uncertainty available to the reader.

Make the comparison before writing the explanation

Use a table like this while reading:

QuestionPaper APaper B
What population, dataset, or setting was studied?Record the relevant characteristics.Record the relevant characteristics.
What was the intervention, exposure, or method?Include implementation details.Include implementation details.
What was the comparison?Specify the baseline or control.Specify the baseline or control.
What outcome was measured, and when?Include units and follow-up.Include units and follow-up.
What design and analysis produced the estimate?Record adjustment and key assumptions.Record adjustment and key assumptions.
What was the estimate and uncertainty?Preserve the reported values.Preserve the reported values.
What could bias the result?Note the concern and evidence location.Note the concern and evidence location.

This is a reading aid, not a complete systematic-review method. For a formal review, use a suitable protocol, appraisal process, and synthesis method. PRISMA 2020 provides reporting guidance for systematic reviews; filling in a table does not establish compliance with it.

The comparison may reveal that one study tested immediate recall and the other measured retention after six months. Or that one compared a new method with a weak baseline while the other compared it with a strong one. Both findings could be accurate descriptions of different questions.

Name the kind of disagreement you have

After completing the table, try writing a single sentence that identifies the unresolved issue:

  • Different questions: “The papers concern different outcomes, so their results cannot directly settle the same claim.”
  • Different precision: “The estimated effects are similar, but one study leaves more uncertainty about their magnitude.”
  • Different conditions: “The findings differ between settings; the role of those settings remains to be tested.”
  • Different vulnerability to bias: “A design difference could help explain the discrepancy, but the comparison does not establish that explanation.”
  • Unresolved conflict: “The studies address a similar question and report materially different estimates that the available information does not explain.”

You may need more than one of these descriptions. The point is to state what you know and what remains open.

Treat an explanation as something to test

Suppose the tutoring effect is larger in a study with weekly teacher support. It is tempting to conclude that teacher support explains the difference.

But perhaps that study also used smaller classes, a different assessment, and different recruitment criteria. Comparing two papers cannot isolate which difference matters.

A useful next step might be to propose a direct comparison of support conditions. In a sufficiently rich evidence set, a planned analysis might investigate variation across studies. Such analyses still require care: Cochrane's meta-analysis guidance discusses the limitations of subgroup analyses and the risk of overinterpreting explanations for heterogeneity.

“Teacher support may explain the difference” is a hypothesis to investigate. Write it with enough detail that someone could design the investigation.

Give the reader the unresolved question

A useful synthesis paragraph can follow this order:

The studies examined [shared question]. They differed in [important conditions]. Their estimates were [results, including uncertainty]. The available evidence leaves [specific issue] unresolved. Addressing it would require [comparison, measurement, or study design].

You can keep the papers together in your Labrix library and use paper questions to locate relevant methods and results. Verify the extracted details against the originals before building your comparison around them.

For your next review meeting, bring two papers and the comparison table. Ask which row changes the interpretation most. That gives the discussion a concrete place to begin.

When papers in your field appear to conflict, what do you check first: the population, the measurement, the baseline, or the analysis? What made you learn to check it?