3 Research design principles
With the MIDA framework and the declare, diagnose, redesign algorithm in hand, we can articulate a set of six principles for research design.
This section offers succinct discussions of each principle. We will expand on the implications of these principles for specific design choices throughout the book.
Design principles
Principle 3.1 Design holistically
This is perhaps the most important of our principles. Designs are good not because they have good components but because the components work together to get a good result. Too often, researchers develop and evaluate parts of their designs in isolation: Is this a good question? Is this a good estimator? What’s the best way to sample? But if you design with a view to diagnosis you are forced to focus on how each part of the design fits together. An estimator might be appropriate if you use one assignment scheme but not another. The evaluation of data and answer strategies depends on whether your model and inquiry call for descriptive inference, causal inference, or generalization inference (or perhaps, all three at once).If we ask, “What’s your research design?” and you respond “It’s a regression discontinuity design,” we’ve learned something about what class your answer strategy might fall into, but we don’t have enough information to decide whether it’s a strong design until we learn about the model, inquiry, data strategy, and other parts of the answer strategy. Ultimately design evaluation comes not from assessment of the parts but from diagnosis of the full design.
When we consider whole designs rather than just thinking about one aspect at a time, we notice how designs that have “parallel” theoretical and empirical sides tend to be strong. We develop this idea in Section 9.3. If you want your estimate \(a_{d^*} = A(d^*)\) to be close to the estimand \(a_{m^*} = I(m^*)\), it’s often best to choose data strategies that parallel models and answer strategies that parallel inquiries, i.e., to make sure that this rough analogy holds: M:I::D:A.
Principle 3.2 Design agnostically
When we design a research study, we have in mind a model of how the world works. But a good design should work, and work well, even when the world is different from what we expect. One implication is that we should entertain many models, seeking not just to ensure the design produces good results for models that we think likely but trying to expand the set of possible models for which the design delivers good results. A second implication is that inquiries and answer strategies should still work when the world looks different from what we expect. Inquiries should have answers even when event generating processes are different to how you imagine them. In the same way, the ability to apply an answer strategy should depend as little as possible on strong expectations of how the data you will get will look.
A corollary to “Design agnostically” is that we should know for which models our design performs well and for which models it performs poorly. We want to diagnose over many models to find where designs break. All designs break under some models, so the fact that a design ever breaks is no criticism. As research designers, we just want to know which models pose problems and which do not.
Principle 3.3 Design for purpose
When we say a design is good we mean it is good for some specific purpose. That purpose should be captured by the diagnosands used to assess design quality and design decisions should then be taken with respect to the specified purpose. Too often, researchers focus on a narrow set of diagnosands, and consider them in isolation. Is the estimator unbiased? Do I have statistical power? The evaluation of a design nearly always requires balancing multiple criteria: scientific precision, logistical constraints, policy goals, as well as ethical considerations. And oftentimes these might come into conflict with each other. Thus one design might be best if the goal is to assessing whether a treatment has any effect, another if the goal is to assess the size of an effect. One design might be optimal if the goal is to contribute to general knowledge about how processes work, but another if the goal is to make a decision about whether to move forward with a policy in a given context.
In the MIDA framework, the goals of a design are not formally a part of a design. They enter at the diagnosis stage, and, of course, a single design might be assessed for performance for different purposes.
Principle 3.4 Design early
Designing an empirical project entails declaring, diagnosing, and redesigning the components of a research design: its model, inquiry, data strategy, and answer strategy. The design phase yields the biggest gains when we design early. By frontloading design decisions, we can learn about the properties of a design while there is still time to improve them. Once data strategies are implemented — units sampled, treatments assigned, and outcomes measured — there’s no going back. While applying the answer strategy to the revealed dataset, you might well wish you’d gathered data differently, or asked different questions. Post-hoc, we always wish our previous selves had planned ahead.
A reason deeper than regret for designing early is that the declaration, diagnosis, and redesign process inevitably changes designs, almost always for the better. Revealing how each of the four design elements are interconnected yields improvements to each. These choices are almost always better made before any data are collected or analyzed.
Principle 3.5 Design often
Designing early does not mean being inflexible. In practice, unforeseen circumstances may change the set of feasible data and answer strategies. Implementation failures due to nonresponse, noncompliance, spillovers, inability to link datasets, funding contractions, or logistical errors are common ways the set of feasible designs might contract. The set of feasible designs might expand if new data sources are discovered, additional funding is secured, or if you learn about a new piece of software. Whether the set expands or contracts, we benefit from declaring, diagnosing, and redesigning given the new realities.
In Part IV on the research design lifecycle, we push this principle to the limit, encouraging you to keep on designing even after research is completed, arguing that ex post design can help you assess the robustness of your claims and help decide how to respond to criticism of your work.