As most people in any empirical or scientific field know, the gold standard for experimenting and establishing causality is the randomized controlled trial (RCT). In an RCT, subjects are randomly assigned to one of two conditions: an experimental group or a control group. The experimental group receives the intervention or drug and the control group receives standard care or a placebo (basically, the equivalent of the status quo). The idea behind the randomized controlled trial is to control the circumstances surrounding the experimental question as much as possible. This allows researchers to collect information on two groups that have essentially the same experience, except for the fact that one group received the intervention and the other did not. Because of this design, researchers can then measure outcomes at the end of the experiment and attribute any differences between the two groups to the intervention itself. In other words, RCTs help establish causality. In the absence of randomization, this type of control is almost impossible and it is much more difficult to conclude that the intervention is the cause of the outcome.
RCTs have been in use for a long time, mostly in clinical medicine. But are RCTs the best approach when it comes to dealing with complex systems such as healthcare or the economy? This is a question that has plagued global health and development experts for quite some time. In recent years, there has been more pressure particularly on development agencies to provide evidence that their interventions are effective. In some cases, the people calling for these outcome measures assert that RCTs are the only acceptable form of evidence. Some global health and development experts, such as Paul Farmer, have argued that this insistence is unhelpful and sometimes even unethical, since it is only ethical to conduct an RCT if you are not sure whether the intervention will be effective and in many cases, such as in the case of humanitarian assistance, it seems patently unethical to withhold interventions from a group of people just for the purposes of creating a control group. As Farmer argues, global health practitioners often “know the right thing to do” but struggle with figuring out the best way to deliver it. This circumstance does not call for an RCT, he claims, but rather resources and research on implementation.
Others, however, disagree with Farmer. One of the most enthusiastic champions of RCTs for health and development is Esther Duflo, a professor of economics at MIT and co-founder and co-director of the Abdul Latif Jameel Poverty Action Lab (J-PAL). In their book, Poor Economics, Duflo and co-author Abhijit Banerjee argue that we may not actually know that much about what works in global health and development. In their view, then, the quest to alleviate poverty worldwide does depend on conducting rigorous RCTs in order to understand in a more thorough and empirical manner what works and what doesn’t.
Duflo is not without her critics either. Many claim that, even putting aside the fact that RCTs are not always ethically or practically feasible and that they tend to be very expensive, the conditions in an RCT do not represent those of the real world and may not reflect how interventions will actually when rolled out in real-time on a larger scale. This is a problem of internal versus external validity, as public health experts have defined it. Internal validity refers to the extent to which you can actually assign the results of an experiment to the experimental intervention. On this parameter, RCTs do extremely well because they are specifically designed to avoid bias and confounding and often do lead to results that are valid in and of themselves. External validity, however, refers to the extent to which an experiment’s results are generalizable to the larger population. Since RCTs test only a small portion of the population in unusual and controlled conditions, they are less likely to have high external validity even if their results are internally sound. This is a serious concern in contexts such as global health and development, since these interventions interact intimately with large, complex systems and with other factors such as culture, belief, and geography.
RCTs certainly have their benefits. But they also have their drawbacks. Some global health experts are champions of RCTs and others are skeptical. The debate between these two camps will likely never be fully resolved. In the meantime, it is always a good idea to keep all these aspects in mind when designing an evaluation or experiment.
This is a great debate, mainly because it is happening among knowledgeable people. I have been in the “development world” for about 30 years now. My take is that we can only contribute to the changes we observe after our interventions, but we cannot attribute anything to us, especially in the development context. Therefore, the formula A causes B for which RCT is designed does not stand for me. However, RCT does provide a better idea about the extent to which an intervention could make a better impact than all of the other research designs. During my experience, I have always been curious to know what works and what does not and my gauge was always the use of RCT. Unfortunately never did I find perfect RCT results to claim attributions for anything that I did. My control group has always been contaminated, one manner or another despite good design. As a matter of fact, finding match-able control and treatment groups was a hurdle, which rendered the design very difficult and sometimes impossible. In the development arena, we face changes in cultural and political contexts, changes in participants behavior and random interventions made by other actors present in the field. All of these changes combined constitute powerful contamination agents for our control groups. It is hard, not to say impossible,to control for such unknowns in a design. An excellent example of contamination is in the agricultural sector. It is hard to impede the control group not to hear about, to see and even try what the treatment group is doing in terms of interventions I was the senior Monitoring and Evaluation officer for a project. The project’s goal was to increase farmers’ incomes. Based on the need assessment that informed the project, the activities consisted of the introduction of new technological packages to increase yield, among other things. The technological packages included: improve seeds, drip irrigation, new farm practices et cetera. So, we wanted to know what from the packages introduced was more conducive to higher yield i.e. was it all three combined, only one or two combined. We spent time and money to design the evaluation but we couldn’t until we quit. The reason was that we could not take the participants from different geographic areas because of climate, soil and even topographic factor differences. When we tried to consider the same geographic areas, the way people leave in these communities they know each other, they share information about their new farming systems including new seeds. sometimes they even share seeds especially if they are new ect…. As Paul farmer said, it is unethical to exclude somebody from applying something because we need him/her to form our control group. Furthermore, small scale farmers will try ANYTHING they seed promising results after one or two colleagues have done it.
Al final, if RCT may be more or less possible in the medical field, it is not for many others. Besides, if it makes sense in the developed countries, it is not always for the third world and even the developing countries. Finally looking for RCT to scale up a project does not seem cost-beneficial in my eyes.