What is the Total Margin of Error (TME) and why is it important?

The Total Margin of Error (TME) is a comprehensive metric that goes beyond the traditional margin of sampling error. It accounts for both sampling error and non-sampling errors, such as nonresponse bias. TME is crucial because it provides a more accurate and realistic assessment of poll reliability. By incorporating both variance and bias, as measured by metrics like Mean Square Error (MSE), TME offers a holistic view of survey accuracy, helping to prevent overly optimistic interpretations of poll results.

Why is nonresponse a significant issue in election polls?

Nonresponse is a significant issue because it introduces bias into poll results. When only a small portion of the sampled individuals participate, the views of those who do respond may not accurately reflect the broader population. This non-random nonresponse leads to bias, which is a component of the Total Margin of Error (TME). Ignoring nonresponse can result in an underestimation of the true uncertainty in poll results, leading to misleading conclusions about public sentiment.

How does the focus on the margin of sampling error mislead the public?

Focusing solely on the margin of sampling error can create a misleading impression that polls are highly accurate reflections of public sentiment. While the margin of sampling error indicates statistical imprecision (variance), it doesn't account for non-sampling errors like nonresponse bias. This omission can result in an overestimation of poll accuracy because it overlooks potentially significant sources of error. Media outlets often contribute to this issue by emphasizing the margin of sampling error without adequately addressing the limitations imposed by non-sampling errors.

What are the key differences between variance and bias in the context of election polls?

In election polls, variance and bias are distinct components of error. Variance measures the statistical imprecision in an estimate, reflecting the degree to which results might vary from sample to sample. Bias, on the other hand, stems from non-sampling errors, particularly non-random nonresponse. Variance is relatively straightforward to calculate, whereas bias is often challenging to quantify because it involves assessing the impact of non-sampling errors. The Mean Square Error (MSE) combines both variance and bias to provide a holistic measure of survey accuracy, which is then used to calculate the Total Margin of Error (TME).

What steps can be taken to improve the accuracy and transparency of election polls?

To improve accuracy and transparency, polls should move beyond the traditional margin of sampling error and embrace the Total Margin of Error (TME) as a standard metric. This involves measuring and reporting both sampling error and non-sampling errors, especially nonresponse bias. The adoption of methodologies, such as those drawing on research, is essential. Furthermore, the principles of TME can be extended to address other potential sources of error, such as coverage issues. Embracing a comprehensive approach, with the use of the Mean Square Error (MSE), will provide a more accurate and realistic assessment of poll reliability, leading to better-informed public discourse and a more transparent understanding of election dynamics.

Distorted election poll with hidden errors.

Are Polls Really Telling the Truth? Unveiling the Total Margin of Error

Avery Sinclair in Politics & Policy September 2025 • 5 min read.

"Beyond Sampling: Discover how non-response and other biases affect election poll accuracy and what can be done about it."

For decades, political polling has strived to meet the rigorous standards of statistical survey research. Ideally, this involves pinpointing a specific population, drawing a random sample from it, and ensuring that those sampled respond truthfully and accurately. However, this ideal is rarely, if ever, achieved in practice, which leads to significant challenges in interpreting polling data.

Polling professionals are well aware of these challenges and have frequently voiced their concerns. For example, Prosser and Mellon noted in their 2018 review that the level of error consistently exceeds what stated margins of error imply. Two main issues persist: one internal to polling research and the other external. The internal problem is that pollsters often fail to quantify the implications of these issues when reporting results, focusing primarily on the margin of sampling error. The external problem is that media outlets often downplay or ignore these interpretative difficulties, further complicating the public’s understanding.

Survey statisticians and researchers have long identified various sources of total survey error, including sampling, nonresponse, measurement, and coverage errors. Among these, the issue of nonresponse—where only a small portion of the sampled individuals participate—is particularly troubling. This article delves into the often-overlooked impact of non-sampling error, especially nonresponse, on the reliability of election polls, offering insights on how to measure and account for these biases to achieve a more accurate representation of public opinion.

What's Wrong with Just Looking at the Margin of Sampling Error?

Distorted election poll with hidden errors.

Media outlets often focus on the margin of sampling error, and while it's valuable, it can give a misleading impression that polls are highly accurate reflections of public sentiment. Survey statisticians have long advocated for measuring the total survey error, using metrics like Mean Square Error (MSE), which accounts for both variance and bias. Here’s why:

Variance measures the statistical imprecision in an estimate, while bias stems from non-sampling errors, notably non-random nonresponse. Using the square root of maximum MSE provides what we can call the Total Margin of Error (TME). Measurement of TME encompasses both sampling error and all forms of non-sampling error.

Comprehensive Error Assessment: MSE combines variance and bias to provide a holistic measure of survey accuracy.
Focus on Variance Alone: Traditional focus on variance overlooks critical bias components, resulting in overly optimistic error assessments.
Limited Statistical Attention: Bias components are challenging to quantify, leading to less developed statistical tools for their assessment.

The fact that polling professionals do not quantify potential non-sampling error, including response bias, in the reporting of poll results would be understandable in the absence of appropriate methodology to perform the measurement. However, such methodology does exist, drawing on and extending our previous research in Dominitz and Manski (2017). In practice, focusing solely on variance can lead to an underestimation of the true uncertainty in poll results. While variance is relatively straightforward to calculate, bias often goes unmeasured due to the complexity in quantifying non-sampling errors.

The Road Ahead: Reassessing Polling Accuracy

It’s time to move beyond the traditional margin of sampling error and embrace the Total Margin of Error (TME) as a standard metric in polling. By accounting for non-sampling errors like nonresponse bias, the TME offers a more accurate and realistic assessment of poll reliability. This shift requires the development and adoption of new methodologies, but it's essential for maintaining public trust in polling data. Furthermore, the principles discussed can extend beyond nonresponse to address other potential sources of error, such as coverage issues related to defining likely voters. Embracing a comprehensive approach to measuring and reporting poll accuracy will lead to better-informed public discourse and a more transparent understanding of election dynamics.