When two things move together, what can we conclude?

The missing comparison

Suppose people who use a fitness app report better sleep. The app might help. But app users might also differ in work schedules, exercise habits or access to care. The association alone cannot tell us which explanation is correct. This is an illustrative scenario, not a reported study.

A causal question asks what would happen if comparable people followed one strategy rather than another. We cannot watch the same person both adopt and not adopt the app over the same four weeks. A study therefore needs a credible comparison group and explicit assumptions about how it represents that missing alternative.

A 36-point difference that disappears

Imagine 100 app users and 100 nonusers, with everyone answering the same question about good sleep after four weeks. The wording and numbers are invented, not a validated sleep measure. Among users, 80 have regular work schedules and 64 of those report good sleep. The remaining 20 have irregular schedules and four report good sleep. That makes 68 out of 100 overall.

Among nonusers, only 20 have regular schedules; 16 report good sleep. Of the 80 with irregular schedules, 16 report good sleep. Overall, that is 32 out of 100. The pooled difference is 36 percentage points. Yet within regular schedules both groups have 80% reporting good sleep, and within irregular schedules both have 20%.

Now give each schedule category a weight of one half in both groups: (0.5 × 80%) + (0.5 × 20%) = 50%. The standardized difference is zero. We changed the comparison's composition, not anyone's sleep. The table lets you check every count. In real data, this result would not prove the app has no effect: measurement error, chance and other differences could remain.

Confounding is a causal explanation, not a long list of variables

For the example, suppose work schedule is recorded before app adoption and influences both the opportunity to use the app and sleep. It is then a plausible common cause of exposure and outcome: a confounder. The two app groups have different schedule mixes, so the pooled comparison combines an app association with a schedule difference.

Knowing that a paper adjusted for age, income and ten other variables is insufficient. Were the important common causes measured well? Were the categories too crude? Did the model describe their relationships adequately? An unmeasured or poorly measured common cause can leave residual confounding after adjustment. A very large sample can make a biased estimate look precise without removing that bias.

Reverse causation: which came first?

Poor sleepers may be more likely to download a sleep-related app. An association between app use and worse sleep could then partly reflect the problem prompting use. Recording app use and sleep at the same visit cannot settle that sequence.

Measuring sleep before adoption and following people forward can help, but a prospective label does not rule out earlier symptoms influencing the decision. Look for a clear timeline, a meaningful starting point and an explanation of how pre-existing differences were handled. Timing narrows the possibilities; it does not remove every alternative cause.

Why random assignment helps

In a randomized trial, assignment to a group is determined by chance. This helps balance measured and unmeasured influences at the start. It does not automatically solve missing data, poor adherence, selective reporting or an unrepresentative sample.

Balance is an expectation over the randomization process, not a promise that every characteristic will match in a particular trial. Concealed allocation helps prevent recruiters from steering people toward a known next assignment. After assignment, differential loss of sleep reports could still distort the result. An analysis of assigned groups asks about assignment to the strategy; selecting only enthusiastic users can lose the protection that randomization provided.

Why adjusting for everything can backfire

The role of a variable depends on the question and the assumed causal structure. If the app changes bedtime habits and those habits change sleep, habits are a mediator. Adjusting for that pathway can remove part of the total effect we wanted to estimate; estimating a direct effect is a different question with additional assumptions.

Selection can also create associations. Suppose both app enthusiasm and sleep problems make someone more likely to answer an optional follow-up survey. Survey response is then a common effect of those two influences, sometimes called a collider. Restricting the analysis to respondents can connect the influences even if they were previously unrelated. Nonresponse is not automatically this particular mechanism: the argument depends on which factors actually affect response.

A causal diagram makes these assumptions discussable. It is a map of the researcher's assumptions, not proof that the arrows are true. More software, more variables or an AI-generated diagram cannot supply missing subject knowledge.

Observational evidence still matters

Observational studies can examine patterns, rare outcomes and real-world settings where randomization is difficult. Their interpretation depends on the design, measurement quality and how alternative explanations are addressed.

One useful approach is to describe the randomized trial researchers would ideally run, then ask how an observational dataset could approximate it. Who is eligible? What exactly counts as starting the app or continuing without it? When do eligibility, assignment and follow-up begin? What outcome is measured, over what period? Which comparison is the analysis trying to estimate?

This target-trial approach makes hidden choices visible. For example, defining users by whether they eventually complete four weeks of use can select people using information unavailable at the start. Aligning eligibility, treatment assignment and follow-up helps avoid such timing errors. It still requires defensible assumptions, adequate measurements and comparable alternatives; the label alone does not create randomization.

Connect studies by their weak points, not just their conclusions

A survey, a carefully designed cohort and a randomized trial can each contribute something different. A survey may reveal who uses an app. A cohort can examine timing and groups poorly represented elsewhere. A trial can estimate the effect of assignment under its own conditions. They may study different versions of use and different populations, so identical-looking headlines need not address the same question.

Before combining results, compare the exposure, outcome, follow-up and likely sources of bias. Agreement is more informative when plausible errors differ; five analyses of the same imperfect dataset are not five independent checks. Disagreement can identify a useful next study rather than a winner. Ask whether a new design can address the specific uncertainty the earlier work left behind.

Read the verb carefully

“Associated with” and “caused” make different claims. Check whether a headline has strengthened the language of the source. Ask what design was used, what adjustments were made, and what uncertainty the authors acknowledge.

For our fictional figures, a defensible description is: app users reported good sleep more often overall, but the difference disappeared when the two schedule categories received equal weight. Saying that the app improved sleep by 36 percentage points would turn a comparison into an unsupported intervention claim.

The most useful future study would address an identified gap: a well-defined start, a credible comparison, consistent outcome measurement or more complete follow-up. Better prediction from AI may help find patterns, but prediction alone does not establish what will happen if someone intervenes. Progress means reducing uncertainty about a specific causal question, not merely generating another striking association.

The fictional sleep data, before and after reweighting

All people, counts and outcomes are invented. Good sleep means answering the illustrative question at the four-week check; no clinical threshold is implied. The last row gives the two existing category-specific percentages equal weight in each group. No confidence intervals or significance claims have been calculated.