What statistical significance can tell you
A study may report that a result is “statistically significant.” Usually, this means the researchers used a statistical test and obtained a result that crossed a threshold they had set for judging whether the observed difference would be surprising under a particular assumption, such as no difference between groups. It does not mean the finding is important, certain, or ready to guide a decision.
The threshold helps researchers describe how their data compare with a statistical model. It does not tell you the probability that their explanation is true. Nor does crossing it establish that the study was well designed. A biased comparison, an unreliable measure, or an analysis chosen after seeing the results can still produce a statistically significant finding. Treat the phrase as an invitation to examine the result, not as a verdict.
Ask how large the difference is
To judge practical importance, start with the effect itself. What changed, by how much, and compared with what? Suppose a hypothetical study finds that a new scheduling method reduces the average time needed to complete a routine task from 60 minutes to 58 minutes. That difference might matter when the task is repeated frequently across a large operation. It might matter little to someone who performs it once a month. The statistical result alone cannot settle that question.
Look for the result in units you can understand: minutes saved, tasks completed, or the proportion of people reaching an outcome. If a report gives only a percentage change, ask what the starting level was. A large relative change can correspond to a small absolute difference when the outcome was uncommon to begin with. Conversely, a modest-looking average can conceal meaningful differences for particular groups. Check whether the report shows the range of individual outcomes rather than relying entirely on one summary number.
Read uncertainty alongside the estimate
A measured difference is an estimate, not an exact description of every setting in which the method might be used. Confidence intervals, when reported, provide one way to see how much uncertainty surrounds that estimate under the study’s assumptions. An interval spanning both a negligible benefit and a substantial benefit leaves an important practical question unresolved, even if the point estimate looks promising.
A result that does not cross a significance threshold also deserves careful reading. It might reflect little or no difference, but it might come from data too imprecise to distinguish among several plausible effects. Compare the estimate and its uncertainty with a difference that would actually matter for your decision. Ask whether the study could reasonably detect a difference of that size. Avoid turning “not statistically significant” into “proved ineffective,” just as you would avoid turning “statistically significant” into “proved worthwhile.”
Check whether the study supports the comparison
Effect size and uncertainty are only as useful as the evidence behind them. Identify who was studied, how the comparison was made, and what was measured. Were participants using the new scheduling method compared fairly with those using the usual one? Was completion time recorded consistently? Could changes in workload or task difficulty account for some of the difference? These questions matter even when the reported statistical analysis is sound.
Also consider whether the outcome matches the claim. Saving two minutes per task is not the same as improving work quality or reducing stress. If a headline makes a broad claim based on a narrow measure, restate the finding in the study’s own terms before deciding what it means. This keeps the evidence tied to what was actually observed.
Connect the finding to a real decision
Before calling a result useful, define the decision it would change. Who would need to adopt the method, what would it cost in time or effort, and what other outcomes might matter? A small benefit may justify a low-effort change; the same benefit may not justify a disruptive one. Write down the smallest difference that would make the change worthwhile before looking too closely at a persuasive headline.
A simple reading note can keep these judgments separate: record the estimated difference, its uncertainty, the quality of the comparison, and the practical threshold you care about. If sustained attention helps you work through those details, browse Focused Work · The Flow Handbook for further ideas on focused work. The goal is not to reduce a study to one label, but to explain clearly what its result can—and cannot—contribute to a decision.
