A new global study led by Kai Ruggeri, PhD, of the Columbia Mailman School of Public Health, and involving more than 80 collaborators from more than 30 countries, highlights the critical role of behavioural science in policy-making, while developing a new method for systematically evaluating experimental evidence on public policy. The findings have been published in the journal Nature.
In April 2020, a group of researchers published an influential paper on COVID-19 with 19 policy recommendations based on behavioural science. The paper was the result of an extensive collaboration of more than 40 experts, led by Jay Van Bavel of New York University and Robb Willer of Stanford, and has been cited thousands of times by governments, researchers and public figures. Its recommendations covered topics such as official communication on social distancing, how to get a vaccine once it is available, and the need to work within communities to make a real impact. Now, the new paper in Nature by Ruggeri et al. assesses the scientific evidence that has emerged since the first paper was published to support his claims and their applicability to policy.
"Governments around the world have formulated policy strategies for the pandemic explicitly based on the behavioural suggestions highlighted in the 2020 paper by Jay J. Van Bavel et al," says Ruggeri, professor of health policy and management at Columbia University's Mailman School of Public Health. "Given concerns about the public's lack of trust in science, particularly in the context of COVID-19, we felt it was important to evaluate the experimental evidence behind public policy recommendations to promote transparency and build trust."
Two independent teams of 72 experts - including both the authors of the 2020 paper and an independent team of reviewers - reviewed 747 pandemic-related research articles to assess the extent to which the claims in the original paper provided sound policy guidance.
"We systematically assigned a rating to each research paper. Higher ratings corresponded to research with stronger conclusions in terms of potential real-world impact, such as field studies. The ratings of each article were assigned independently and anonymously by several researchers," says Valerio Capraro, associate professor of psychology at the University of Milan-Bicocca and lead Italian author of the study.
Dr Alex Haslam, Professor of Psychology at the University of Queensland in Australia and co-author of the study, says: "There has been much discussion in recent years about the limitations of psychological and behavioural science, particularly in light of the so-called 'replicability crisis'. In contrast, this research has shown that there is a core of good theory in these fields that provides a sound basis for both scientific prediction and public policy. This theory may not always be flashy, but it is the foundation of good social science, and this study confirms that we can rely on it for guidance when we need it.
The study finds scientific evidence for 18 of the 19 statements in the 2020 document, including those on identity and community connectedness, leadership and trust, public health communication, social cohesion and disinformation. Of these 18, the 2020 paper correctly identified 16 behaviours that would be relevant during the pandemic, including some of the likely barriers to reducing the spread of the disease and the social challenges that policymakers would face. The researchers found no effect for two of the suggested strategies, both related to effective public communication (that messages should emphasise benefits to the recipient and focus on protecting others). In addition, the team found no evidence for a recommendation that has been much debated in public but not in the scientific community, which suggested that 'physical distancing' is preferable to 'social distancing'.
The most strongly supported statements related to the importance of interventions to counter misinformation and polarisation, which were found to be essential to ensure compliance with public health guidelines. The research also emphasised that, to be effective, communication must come from trusted leaders and reinforce positive social norms.
The public health interventions that received the most attention were not necessarily those with the best evidence. For example, handwashing was widely promoted as a strategy to stop the spread of COVID, but the effects found in the various studies were minimal or nil, particularly when compared with the use of face masks, isolation, distancing and vaccines.
With regard to face masks, early guidelines in some countries suggested that the practice would not reduce COVID-19, but later studies emphasised their effectiveness. Similarly, research has downgraded guidelines on the impact of school closures and surface disinfection.
"While there is understandable pressure to issue guidelines quickly during a crisis, making policy decisions without adequate evidence can be costly in many ways," says study co-author Katherine Baicker, PhD, Director General of the University of Chicago. "As new scientific evidence emerges over time, some people may view changing policy guidelines as a sign of unpreparedness - or even conspiracy - which weakens confidence in competence. Policymakers need to balance the need for speed with the need for solid evidence and credibility.
The new study also identifies several areas that are missing from the 2020 document. These include perceptions of threat and risk, the role of inequality and racism, scepticism about science, incentivising behaviour beyond simply describing benefits (e.g. providing financial rewards for vaccination) and the lack of clear leadership.
Finally, the research group makes recommendations to help researchers and policymakers respond to future pandemics and disasters. These include the need to study global populations, conduct more field trials and be more specific in formulating testable questions.
"The value of field-testing what really works to change health behaviours cannot be underestimated, and the strongest conclusions we were able to draw in this article often came from the partnerships researchers forged with local governments and health workers to carefully evaluate what really adds value in the midst of a crisis," says study co-author Katy Milkman, PhD, professor at the Wharton School of the University of Pennsylvania. The researchers also encourage scientists to forge more alliances with policy and decision makers - in local government, hospitals, schools, the media and beyond.
"This work has the potential to increase transparency and build trust in science and public health, as well as directly prepare the development of tools and knowledge for future pandemics or crises. Researchers can be a valuable source of policy advice in the context of a crisis, and our recommendations point to ways to further enhance this role of the social and behavioural sciences," said senior co-author Robb Willer, PhD, Professor of Sociology at Stanford University.
"This new paper rigorously evaluates our original team's policy recommendations to see if they are correct, using large amounts of data and a new team of independent reviewers from around the world. In addition to confirming the vast majority of our original claims, it sets a new standard for evaluating evidence when making policy decisions, especially urgent ones," said Dr Jay Van Bavel, Professor of Psychology at New York University, lead author of the pivotal 2020 paper and senior co-author of the new paper.
A full list of authors can be found in the article (DOI: 10.1038/s41586-023-06840-9).