A research design starts from an idea, a kernel of a project. People arrive at their good ideas from every direction through idiosyncratic processes. Some people are inspired by their reading of academic literature. Others are sparked by a conversation with colleagues. Still others have their research questions thrust upon them by exigent circumstances. Ideas come from primary observation of social processes, reading secondary accounts by other authors, and our own past thoughts and lived experiences.
Whatever the process is that ignites the imagination, we can characterize what part or parts of the research design the idea is about. Ideas can be a piece of a model, a vague inquiry, a scrap of a data strategy, or an exciting answer strategy. An idea in the form of a model includes one or more nodes and one or more edges between them: for example, a treatment, a mediator, and an outcome. An inquiry is perhaps the most common kernel of a research project. It might be “does D cause Y.” A data strategy might be the discovery of a discontinuity in some administrative rule. A research project that starts with an answer strategy might be a new measurement technology that enables answering previously unanswerable – or unasked – questions.
The goal of brainstorming is to build a complete design from this kernel, regardless of where in the four elements M, I, D, or A the idea starts out. Sometimes a “theory-first” approach to developing a research project is unhelpfully contrasted with a “methods-first” approach. Better might be to describe the “theory-first” approach as starting with M or I and an “opportunity-first” approach as starting with D or A. We should seek to provide credible answers to important questions, and often progress in providing credible answers will start with new data and answer strategies. That said, unimportant questions aren’t worth the research investment, no matter the credibility of the answers – worse still are unreliable answers to the biggest questions in social science.
Brainstorming sessions with colleagues and mentors can help take a kernel of an idea to a set of feasible research designs. What do participants in the brainstorming session need to know about your idea so they can effectively help identify possible designs? We suggest using the following “problem statement.23” You need not have all the parts of the design worked out – that’s why you are having a brainstorming session! Systematizing what you do know is helpful. The more information you can provide, even if it is the range of possibilities rather than an exact specification.
Instructions: fill out any part you can!
- What is the population of units of interest?
- How many units are in the population?
- What are the important variables that describe each unit?
- Can you represent your theory as a DAG?
- What parts of the theory are you more or less confident of?
- How would an alternative model describe a similar process?
- What main question about the theoretical model will the design address?
- Why is this question important for scholarship, the public, or decision-makers?
- Are there auxiliary inquiries that could be used to check model assumptions?
- For which units is the inquiry defined?
- How will you select cases or sample units from the population defined in the model?
- If randomizing treatments:
- How many conditions will there be?
- At what level will you randomize?
- How many units can be assigned to each condition?
- Which procedure will you use to randomize?
- If not randomizing treatments:
- How do units come to receive different treatments?
- Are there known processes that approximate random assignment
- Regardless of how units come to be treated:
- How will you address the possibility of noncompliance?
- How will you address the possibility of spillovers?
- If there were no financial, logistical, or ethical constraints, what is the ideal experiment you would run?
- How will you measure outcomes?
- What survey instrument or measurement tool will you use?
- When will you measure outcomes? How many times?
- How will you minimize measurement error and attrition?
- How will you use the data that results from your proposed data strategy to produce answers to your inquiry?
- What subset of the data will you use analyze?
- What contrasts will you make to draw comparisons?
- Which outcomes will you analyze?
- What estimator will you use?
- How will you estimate uncertainty in your estimates?