How to protect the integrity of survey research: 12 recommendations

(Press-News.org) Science requires data, and survey research is one important means of gathering it. Surveys provide a scientific way of acquiring information that is used to inform policy decisions, guide political campaigns, clarify the needs of stakeholders, enhance customer service, help society understand itself, and improve the quality of life in the United States.

In recent years, concerns have been raised about growing rates of refusal to participate in surveys, as well as about inaccurate forecasts in high-profile elections, polls with contradictory findings, the declining trust in government and media institutions that fund such research, and skepticism fueled by political polarization. “Although polling is not irredeemably broken,” the authors of a new article write, “changes in technology and society create challenges that, if not addressed well, can threaten the quality of election polls and other important surveys on topics such as the economy.”

In this article, published in the journal PNAS Nexus, 20 experts from diverse fields – including academia, science, government, nonprofits, and the private sector – offer a dozen recommendations to improve the accuracy and trustworthiness of surveys.

Transparency, clarity, correcting the record The authors’ recommendations aim to better align the practices of survey research with three scientific norms: transparency, clarity, and correcting the record. Among the recommendations are that surveyors and survey researchers should:

Be transparent about their methods, including sampling design and modeling and weighting assumptions. Disclose the sources of respondent recruitment. Disclose the ways in which exposure to other surveys may affect the responses of members of panels. Disclose the known or expected consequences of attrition on panel surveys. Be more precise in the use of terms such as “representative sample” and spell out what that means. “If survey panelists’ responses have been potentially biased by responses to earlier surveys or questionnaires, researchers and readers need to know that,” said co-author Kathleen Hall Jamieson, director of the Annenberg Public Policy Center of the University of Pennsylvania.

The authors also recommend the creation of a publicly available, professionally curated archive of identified technical problems in polling tools and their remedies. And they recommend that professional organizations and universities develop and disseminate a guide to survey research for use in high school courses as well as a brief “users guide” designed for laypeople and journalists who do not have training in statistics or survey research.

“Around the world, survey research produces vital information,” said co-author Arthur Lupia, a distinguished professor at the University of Michigan. “These new recommendations have the potential to increase the quality of future surveys for generations to come.”

See the full list of recommendations online.

Origins and authors of the survey research recommendations The recommendations emerged from a virtual two-day retreat in November 2021 convened by Marcia McNutt, president of the National Academy of Sciences, and cohosted by The Annenberg Foundation Trust at Sunnylands and the Annenberg Public Policy Center (APPC). The sessions were coordinated by Lupia and Jamieson, who also is the Sunnylands program director. Among the participants were the authors of this paper, who include past and current editors of major academic journals, past presidents of the American Political Science Association and the American Association for Public Opinion Research (AAPOR), a past director of the U.S. Census Bureau, scholars who have led some of the nation’s largest university-based election surveys, and individuals responsible for the creation and maintenance of major government and nonprofit survey datasets.

about this retreat and other NAS-Sunnylands-APPC retreats on APPC's website. 

The authors of the PNAS Nexus article are:

Kathleen Hall Jamieson, Annenberg Public Policy Center, University of Pennsylvania Arthur Lupia, Department of Political Science, University of Michigan Ashley Amaya, Pew Research Center Henry E. Brady, Goldman School of Public Policy, University of California, Berkeley René Bautista, Methodology and Quantitative Social Science Department, NORC at the University of Chicago Joshua D. Clinton, Department of Political Science, Vanderbilt University Jill A. Dever, RTI International David Dutwin, NORC at the University of Chicago Daniel L. Goroff, Alfred P. Sloan Foundation D. Sunshine Hillygus, Department of Political Science, Duke University Courtney Kennedy, Pew Research Center Gary Langer, Langer Research Associates John S. Lapinski, Department of Political Science, University of Pennsylvania Michael Link, Ipsos Tasha Philpot, Department of Government, University of Texas at Austin Ken Prewitt, School of International and Public Affairs, Columbia University, and former director, U.S. Census Bureau Doug Rivers, Department of Political Science, Stanford University Lynn Vavreck, Department of Political Science, University of California, Los Angeles David C. Wilson, Goldman School of Public Policy, University of California, Berkeley Marcia K. McNutt, National Academy of Sciences “Protecting the integrity of survey research” was published online as an open access article in PNAS Nexus on March 28, 2023. DOI: 10.1093/pnasnexus/pgad049.

The Annenberg Public Policy Center was established in 1993 to educate the public and policy makers about communication’s role in advancing public understanding of political, science, and health issues at the local, state, and federal levels.

END