When medical studies disagree: how to build a clearer picture

First, write the two questions in full

Imagine two fictional studies of the same app. One tests whether it improves a symptom score over eight weeks in volunteers. The other asks whether offering it through routine clinics reduces hospital admissions over two years. A positive first result and a negative second result need not conflict.

Write each question as a sentence: in which people, does which intervention, compared with what, change which outcome over what time? Then underline the parts that differ. This small exercise prevents a broad product name from making distinct experiments look interchangeable.

The method also applies when comparing an event-prevention question, a weight-maintenance question and a diabetes-diagnosis question: each needs its own answer.

Design tells you how a comparison was created

Random assignment creates a comparison through chance allocation. Observational research records exposures and outcomes without assigning the intervention in that way. Both can be useful, but alternative explanations need to be considered in relation to the design. NIH’s research guide explains these starting distinctions.

For a hypothetical observational example, people who subscribe to an exercise app may already be more motivated or have more free time. A difference in their outcomes cannot automatically be assigned to the app. A randomized offer would ask a different causal question.

Avoid treating the design label as a quality guarantee. Follow-up, measurement and whether the intervention was actually delivered still matter. The label is the beginning of the appraisal, not its conclusion.

Follow participants through the study

A simple participant map is often more informative than a confident concluding paragraph. How many people were assessed, entered, received the intervention, reached the next stage and contributed outcome data? CONSORT’s reporting guidance includes a participant-flow diagram and reporting items that help readers reconstruct this path.

In an invented trial, 1,000 people might begin an initial treatment period and 600 proceed to randomization. A strong result among the 600 does not describe every aspect of starting the intervention among all 1,000. The reason people left is part of the interpretation.

Likewise, a result reported only among those who completed treatment may describe a selected group. Ask what analysis was planned and how missing outcomes or treatment changes were handled, rather than assuming that everyone initially enrolled appears in the final comparison.

A second paper may be another window on the same trial

A research programme can publish a main result, a subgroup analysis, a longer follow-up and a methods paper. These can each add information. They are not automatically four independent replications.

Our suggested reading card includes a trial identifier and a note on whether participants overlap with another source. In a hypothetical evidence map, label the original trial A, its follow-up A1 and its subgroup analysis A2. Label a newly recruited, separately randomized trial B. That makes it easier to see whether confidence rests on additional observations of the same group or on a genuinely separate test.

This matters especially when a news article says “several studies agree.” Ask whether that phrase means several independent experiments, several analyses or several publications discussing one experiment.

A pooled estimate still needs a coherent question

Meta-analysis combines estimates using statistical methods. Differences between studies—often called heterogeneity—must be examined; a pooled result does not erase them. Cochrane’s methods guidance describes how such variation affects interpretation.

Consider three invented studies: one measures symptoms at four weeks, one measures a laboratory value at six months, and one counts hospital admissions at two years. Averaging their headline percentage improvements would not create a meaningful measure of overall health benefit.

Even when the same outcome is measured, setting, treatment delivery and starting risk can differ. A useful review explains those differences, why pooling is appropriate and what the resulting estimate can and cannot represent. More participants can improve precision without fixing a mismatch between the combined evidence and the reader’s question.

Separate three sentences that often get blended

Try rewriting a research story into three labelled sentences.

Result: what the investigators actually measured. Interpretation: what that observation suggests when considered alongside the design and other evidence. Future hypothesis: what might be possible if further work supports the proposed explanation.

For an invented laboratory example, “a marker changed” is a result. “This pathway may be involved” is an interpretation. “Targeting it could eventually improve survival” is a future hypothesis. The final sentence must not be quietly presented as though it was the measured outcome.

FDA’s discussion of surrogate endpoints is relevant here: using a substitute for a direct clinical outcome requires evidence about what that substitute predicts. A plausible biological explanation can guide research without establishing that people will feel better or live longer.

Use a connection map instead of a winner’s podium

We suggest giving each related paper one role. It may directly test the same clinical question, add information about durability, investigate a possible mechanism, examine safety or show how delivery works in another setting. Write down the role before summarizing the finding.

This creates a more useful paragraph: one study establishes an effect under particular conditions; another raises a question about persistence; a third examines a different outcome. The reader can see both the connection and the boundary.

For example, a reading map could connect a prevention trial with a separate withdrawal study, or connect a clinical-outcome trial with implementation research. The first connection asks about persistence; the second asks about delivery. Neither connection alone establishes that one treatment is best for a particular person.

What would make you change your mind?

Before reading a new headline, state what kind of evidence would alter your understanding. A replication in a similar population might strengthen confidence. A carefully measured serious harm could change the benefit–burden balance. Longer follow-up could clarify durability. A well-designed study in a different setting could improve applicability.

Then distinguish “the result did not show the hoped-for effect” from “the question is settled.” An imprecise study can leave several possibilities open. Conversely, repeatedly moving the goalposts whenever a result disappoints makes a claim impossible to evaluate.

The hopeful future of medical reporting is a visible trail of learning: readers should be able to see what a new paper added, which conclusion changed and why. That is more useful than replacing yesterday’s certainty with today’s certainty.

A connection map for the next paper you read