# Improving Reproducibility by Incorporating Uncertainty

> **NIH NIH R01** · EMORY UNIVERSITY · 2022 · $270,507

## Abstract

In recent years, stakeholders in the scientific and lay communities have raised alarms about a lack of
reproducibility of scientific results. These stakeholders view the reproducibility crisis as a product of the
behavior of researchers and editors. While these behaviors likely have an impact on reproducibility, there are
credible reasons why studies of complex data should be expected to arrive at different estimates. First, each
study is differentially susceptible to systematic biases, including confounding, selection bias and measurement
error. These biases may be large drivers of the appearance of poor reproducibility. Second, many studies that
have been criticized for lack of replication are small, and therefore subject to substantial random variability. In
combination with selection forces emanating from significance testing, these small studies are likely to
overestimate effects, further contributing to the appearance of poor reproducibility. To date, proposed solutions
to the perceived reproducibility crisis have largely ignored these contributing factors. We propose to (a) use
simulation-based quantitative bias analysis techniques to adjust for the influence of systematic errors on
estimates of association and on summaries of an evidence base, and (b) use Bayesian statistical methods to
synthesize prior information with estimates of association and summaries of an evidence base to reduce
random variability. The premise of the proposed project is that the use of these informatics approaches will
reduce the potential for systematic and random error to misleadingly portray research as poorly reproducible
and will identify the most important limitations in an evidence base, which will optimize decisions regarding new
data collection. The proposed informatics will be extended and applied in the context of two high profile,
controversial topic areas with complex data, which will provide examples applicable to other topic areas. For
both topic areas, many sources of potential bias have been identified in the surrounding discourse—as have
powerful sources of prior information to temper uncertainty—but their influences on individual estimates of
association and summaries of the evidence base have not been fully quantified. Quantitative adjustments for
these errors using quantitative bias analysis and Bayesian methods—and for publication bias on meta-analytic
summaries—would improve reproducibility. We will then extend and apply web-enabled informatics tools to
implement the methods for any topic with a set of heterogeneous study results, allowing stakeholders without
advanced analytic skills to tailor the underlying assumptions and see for themselves the impact on the
summary results. By achieving our aims, this project will advance the use of research informatics to diminish
the reproducibility crisis, help to speed consensus-building for any research topic, and productively channel
research resources towards resolving the most influential source...

## Key facts

- **NIH application ID:** 10322751
- **Project number:** 5R01LM013049-04
- **Recipient organization:** EMORY UNIVERSITY
- **Principal Investigator:** Timothy L. Lash
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2022
- **Award amount:** $270,507
- **Award type:** 5
- **Project period:** 2019-02-06 → 2024-01-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10322751

## Citation

> US National Institutes of Health, RePORTER application 10322751, Improving Reproducibility by Incorporating Uncertainty (5R01LM013049-04). Retrieved via AI Analytics 2026-05-22 from https://api.ai-analytics.org/grant/nih/10322751. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*