Magnifying glass examining a field of scientific research papers, some brightly lit and others in shadow.

Is Your Research Underpowered? A New Way to Test the Strength of Scientific Studies

Rhea Morgan in Science & Nature August 2025 • 4 min read.

"Discover a groundbreaking method for evaluating the statistical power of research and ensuring reliable results, especially in economics and beyond."

In the world of scientific research, a critical question looms large: Are our studies strong enough to draw reliable conclusions? Every experiment, whether in economics, psychology, or medicine, faces the risk of producing a false negative – missing a real effect simply because the sample size wasn't large enough. This challenge is particularly relevant in fields where resources are limited, and researchers must carefully balance the cost of data collection with the need for statistical rigor.

A recent paper by Stefan Faridani at UC San Diego introduces a new method to tackle this very problem. Faridani's approach offers a way to estimate how many experimental studies might have reached different conclusions if they had been conducted with larger samples. This is crucial for understanding the robustness of existing research and for guiding future investigations.

Unlike traditional methods, this innovative technique doesn't rely on strict assumptions about the underlying distribution of true effects. Instead, it focuses on the continuity of these effects, making it a more flexible and practical tool for meta-analysts and research funding organizations alike. By adjusting for publication bias and analyzing t-scores, Faridani's method provides valuable insights into the statistical power of various research literatures.

Unveiling the Power Within: How the New Method Works

Magnifying glass examining a field of scientific research papers, some brightly lit and others in shadow.

At the heart of Faridani's method lies a clever statistical trick. Imagine you have a collection of experimental studies, each reporting a t-score – a measure of the strength of evidence against a null hypothesis. The goal is to estimate what would happen if you were to increase the sample size of each study by a certain factor, say, doubling it. Would the number of statistically significant results increase substantially, or would it remain relatively unchanged?

Faridani's approach allows you to estimate this "power gain" without actually conducting any new experiments. The method works by:

Smoothing the distribution of reported t-scores: This step helps to remove noise and reveal the underlying pattern of true effects.
Fitting a distribution of true intervention treatment effects: This step estimates the distribution of true effects that best matches the smoothed distribution of t-scores.
Adjusting for publication bias: This step corrects for the tendency of journals to preferentially publish statistically significant results.
Calculating the expected power gain: This step estimates the increase in statistically significant results that would occur if sample sizes were increased.

The beauty of this method is that it requires minimal assumptions. The only key assumption is that the distribution of true effects is continuous, meaning that there are no sudden jumps or breaks in the range of possible effect sizes. This assumption is far less restrictive than those required by other methods, making Faridani's approach more widely applicable.

The Bigger Picture: Implications for Research Funding and Practice

Faridani's research has important implications for how we conduct and fund scientific studies. The findings suggest that, at least in some fields like economics, randomized controlled trials (RCTs) may be less sensitive to sample size increases than previously thought. This challenges the conventional wisdom that bigger is always better, and it suggests that funding agencies might be better off supporting a larger number of smaller studies rather than a smaller number of large studies. By spreading resources across more investigations, we can increase the diversity of research questions being addressed and potentially uncover more unexpected discoveries.

About this Article -

This article was crafted using a human-AI hybrid and collaborative approach. AI assisted our team with initial drafting, research insights, identifying key questions, and image generation. Our human editors guided topic selection, defined the angle, structured the content, ensured factual accuracy and relevance, refined the tone, and conducted thorough editing to deliver helpful, high-quality information.See our About page for more information.

This article is based on research published under:

DOI-LINK: https://doi.org/10.48550/arXiv.2406.13122,

Title: Testing For Underpowered Literatures

Subject: econ.em

Authors: Stefan Faridani

Published: 18-06-2024

Everything You Need To Know

What is the core problem that Faridani's method addresses in scientific research?

Faridani's method tackles the issue of underpowered studies, where experiments might fail to detect a real effect due to insufficient sample sizes. This can lead to false negatives, especially in fields like economics, and impacts the reliability of research findings. The method helps in evaluating the strength of existing studies and guiding future investigations to ensure more robust and reliable results by providing a way to estimate the impact of larger sample sizes.

How does Faridani's method differ from traditional methods in assessing the strength of scientific studies?

Unlike traditional methods, Faridani's approach doesn't rely on strict assumptions about the underlying distribution of true effects. Instead, it focuses on the continuity of these effects. This makes it a more flexible and practical tool for meta-analysts and research funding organizations. Traditional methods often make strong assumptions about the data's distribution, which may not always hold true. Faridani's method's flexibility allows for a broader application across different fields and types of studies.

Can you explain the key steps involved in Faridani's method for evaluating statistical power?

Faridani's method involves several key steps. First, it smooths the distribution of reported t-scores to remove noise and reveal the underlying pattern of true effects. Second, it fits a distribution of true intervention treatment effects that best matches the smoothed distribution of t-scores. Third, it adjusts for publication bias, which accounts for the tendency of journals to publish statistically significant results more often. Finally, it calculates the expected power gain, estimating the increase in statistically significant results if sample sizes were increased. This process allows researchers to understand the impact of sample size on the study results.

What is the significance of the assumption of continuity in Faridani's method, and why is it important?

The assumption that the distribution of true effects is continuous is a cornerstone of Faridani's method. This means there are no sudden jumps or breaks in the range of possible effect sizes. This is a less restrictive assumption than those required by other methods, making Faridani's approach more widely applicable. It allows the method to be used in a broader range of scenarios, increasing its utility for meta-analysts and funding agencies who seek to assess the reliability and impact of studies across various disciplines, including economics.

How could funding agencies and researchers use the insights from Faridani's research to improve scientific studies?

Faridani's research suggests that, in some fields like economics, increasing sample sizes of randomized controlled trials (RCTs) may not always lead to a significant increase in statistically significant results. This challenges the conventional wisdom of always pursuing larger sample sizes. Funding agencies could benefit from potentially supporting a greater number of smaller studies rather than a smaller number of large ones. By spreading resources, they can encourage more diverse research questions, potentially uncovering more discoveries and improving the overall robustness and reliability of scientific findings. Researchers can use these insights to design more efficient studies, focusing on the optimal balance between sample size and the breadth of research questions addressed.