24 Epilogue
Social science is undergoing a period of structural change. The credibility of decades of research findings has been questioned and many studies have been found to be flawed.
The first prong of the response to the upheaval was the so-called “credibility revolution,” which sought new research designs that could deliver credible evidence to answer social scientific questions. Randomized experiments rapidly rose in popularity in economics and political science. Observational causal inference methods such as regression discontinuities and difference-in-differences designs have become popular in sociology, political science, and economics. Sample sizes have increased and new populations outside university students have been explored in psychology and experimental economics.
The second prong was the so-called “open science movement.” Open science practices are motivated by the idea that even when a credible general design for drawing inferences is adopted, myriad small design decisions may influence the validity of the results. Sharing the plans, computer code, and materials used to implement the research as well as the data that result allows peer reviewers and readers to assess the large and small decisions the authors made and come to their own judgment about what was learned. Preregistration of plans before implementing research provides additional clarity: which of these decisions were made before seeing data and results and which were made after.
The two prongs are closely related to each other. Open science practices are meant to reinforce the work on credible designs: transparency of research methods incentivizes researchers to select credible research designs in the first place. Common to both is the idea that research design matters.
Strikingly, however, these advances have been made without a clear common understanding of what a design is or how to evaluate one. In this book, we provided a flexible approach to defining a design and a procedure for assessing its qualities. We identified four generic elements of a research design: the Model, the Inquiry, the Data Strategy, and the Answer strategy. “Declaring” these four elements makes it easier to communicate the most important analytic features of a design, enabling “diagnosis” of the credibility of claims that depend on them.
We hope our effort adds two new steps to the workflow promoted by credibility and transparency revolutionaries. First, we want scholars to develop designs by declaring them — in code, when possible, diagnosing their properties in terms of scientific, logistical, and ethical goals, and redesigning across feasible designs to select the final design. Second, we want scholars to share their designs so they can be more easily understood, more easily interrogated, and more easily built upon.
We see these steps as deeply complementary to the credibility revolution and the open science movement.
Declaring and diagnosing designs can make designs stronger. Many design choices can be made on the basis of analytic results, and these should be used when possible. But oftentimes analytic results provide incomplete answers. Sampling and eligibility procedures can interact with treatment allocation schemes, so causal identification results can be insufficient to assess the unbiasedness of the design for a population average treatment effect. Moreover, many theoretical results about research design are conditional on certain sample sizes, correlations between variables, or the correctness of functional forms. Assessing how designs perform based on the specific research setting and its sample size and empirical correlations between variables augments the general theoretical guidance. Of course, theory guides how to set up the design itself: identifying what kinds of problems can emerge in a model is an exercise shaped by theory.
Sharing research designs in code complements common open science practices in use today. By providing the design in code, the study can be replicated exactly in a new setting or at a later time period, reanalyzed with the realized data but new estimators, and the diagnosands reassessed on the authors’ original terms and under new conjectures about the model. Declarations also complement current practices in preregistration. Considerable debate surrounds what should be included in a preanalysis plan. Declarations in code provide an answer: you should declare sufficient information to enable someone to diagnose the design in terms of study-relevant diagnosands.
Better software tools will come along to declare, diagnose, and redesign studies in code. A body of domain-specific knowledge will develop about what models design must be assessed against to assure robustness. What we hope will remain is the idea that research designs can be thought of as objects that can be interrogated, defined by the specific steps in the procedures used to generate data and analyze it to provide answers to particular inquiries that are themselves well defined with respect to specified representations of the world. We hope that these ideas and tools will enable scholars to better respond to changed incentives in the social sciences to adopt credible research designs for the questions they are asking and to communicate that they have done so to reviewers and readers.