Speaker
Description
While results from individual Randomized Controlled Trials (RCTs) often do
not hold beyond their particular setting, the accumulation of many RCTs can be
used to guide policy. But how many studies are required to confidently
generalize? Our paper examines construct validity, an often neglected yet
important element affecting generalizability. Construct validity deals with how
the operationalization of a treatment corresponds to the broader theoretical
construct it intends to speak to. The universe of potential operationalizations is
referred to as the design space. We use microfinance as an empirical example,
a literature that is exceptionally rich in RCTs. By systematically reviewing 38
microfinance RCTs, we demonstrate that even this deep experimental literature
only covers a tiny fraction of the design space and that small variations in the
treatment design matter for the observed treatment effects. Most papers
nevertheless generalize from the operationalized treatment to a broad
construct. We conclude that, to maintain construct validity, RCTs should
semantically limit their inference to the operationalized treatment under
evaluation – thereby trading relevance for rigor.
