The problem with an incomplete set of studies
When evaluating evidence, it is tempting to treat published studies as the full record of what researchers found. But publication is a selection process. Some studies are written up, submitted and accepted; others never reach a journal or other accessible outlet. If the chance of publication depends on a study’s result, the available studies may give a misleading picture of the question they investigate. This is publication bias.
Imagine that several teams independently test whether a new study technique improves quiz performance. The teams that find a clear benefit publish their results, while teams that find little difference leave their work in a drawer. A reader searching the published literature would see a pattern of success, even though the complete set of tests tells a more mixed story. The published results are not necessarily false. The problem is that they may not be representative.
How results go missing
Studies can remain unpublished for many reasons. Researchers may decide that an inconclusive result is not worth preparing for submission. Editors and reviewers may judge it less interesting than a striking finding. A project may lose funding or staff before its results are reported. Not every missing study reflects a deliberate attempt to hide evidence, but the reason matters less to the reader than the pattern: are certain kinds of results harder to find?
Selection can also happen within a published study. A team might measure several outcomes but report only the ones that appear most compelling. It might try different analyses and emphasize one that produces a clearer effect. These practices are often discussed separately as selective outcome reporting or selective analysis, yet they create a related problem: readers see a filtered version of the work. A journal article can therefore be available while important parts of its evidence remain difficult to assess.
Why the distortion can spread
Publication bias matters especially when someone combines studies to answer a broader question. A review or meta-analysis can carefully summarize every study it finds and still reach a skewed conclusion if relevant studies are missing. If studies with weak or unfavorable results are less visible, the combined estimate may make a benefit appear larger or more certain than the complete evidence would support.
The effect is not limited to a numerical average. Missing work can narrow the apparent range of outcomes, making it harder to see when an approach fails or under what conditions it works. It can also shape later research: authors cite the visible studies, and new projects build on a record that seems more consistent than it is. However, an uneven collection of findings is not, by itself, proof of publication bias. Differences in study quality, participants, methods or settings may also explain why results vary.
What to check when reading a body of evidence
Start by asking how the studies were found. Did a review search only journal articles, or did it also consider dissertations, conference materials, reports and other relevant work? A wider search does not guarantee completeness, and unpublished material still needs critical appraisal. It can, however, reduce reliance on one route to publication.
Next, look for a record of what researchers planned before they knew the results, such as a study registration or a dated protocol. Compare planned outcomes with those eventually reported where possible. If a review has its own protocol, check whether its search and inclusion rules were set in advance. These records may be unavailable or incomplete, so their absence is a reason to note uncertainty, not to declare the findings invalid.
Finally, see whether reviewers discuss missing studies and test how sensitive their conclusion is to reasonable assumptions about them. Plots and statistical methods can sometimes flag patterns worth investigating, but they cannot reliably reconstruct every study that was never reported. Pay attention to whether the authors explain the limits of these checks rather than presenting a correction as a definitive answer.
Reach a conclusion that fits the visible record
Publication bias does not mean that every positive result should be rejected. It means that confidence in a whole body of evidence depends partly on how much of that body is visible. A useful assessment separates two questions: How credible are the studies we can examine, and how confident are we that they represent the work that was done?
When reporting findings, say what was searched, what might be missing and how that uncertainty affects the conclusion. For example, “The published studies generally report a benefit, but unpublished or unreported results could change its estimated size” is more informative than either “It works” or “The evidence is worthless.” That distinction helps readers use the available research without mistaking the available record for a complete one.
