12. Experiments

When you think of the word “experiment,” what comes to mind? Maybe you associate the term with images of white-coated scientists pouring chemicals into test tubes. And yes, without a doubt, experiments are central to scientific inquiry of all kinds. But it’s worth considering how various kinds of experiments pervade modern life. Marketing and social media companies run experiments all the time to figure out what products we are more likely to buy or what links we are more likely to click on. We watch television shows that use experimental methods to vet the effectiveness of different products or test whether urban legends are actually true. And we each “experiment,” in a broad sense, as we go about our daily lives. We try out a new energy drink or hot sauce, in search of something better. We compare different workout regimes or meditation techniques and take note of how we feel afterward.

This chapter is aimed at giving you a richer sense of how scientists use experimental methods to learn about the social world. Fundamentally, an experiment involves the researcher introducing a variable (the independent variable, also known as the experimental stimulus or treatment) and then observing what happens to another variable afterward (the dependent variable, also known as the outcome variable, outcome measure, or response variable). When we engage in experiments, we are typically pursuing a deductive approach (as discussed in Chapter 4: Research Questions), asking, “What effect does the independent variable have on the dependent variable?” Compared with other research methods, experiments excel at answering the “effect” part of this question. They tell us convincingly whether a cause-and-effect relationship really exists between two variables.

In judging cause and effect, experiments are often considered to be the “gold standard” of scientific research design, the yardstick against which all others are judged. When they are conducted with all the safeguards we describe in this chapter, we say that experiments have high internal validity (not to be confused with the validity of a measure, which we discussed in Chapter 7: Measuring the Social World). Internal validity is our study’s ability to determine if changes in the independent variable truly cause changes in the dependent variable. When we have high internal validity, we are able to rule out any confounders—variables other than stimulus or treatment that may be influencing the dependent variable.

In an experiment, we conduct research in a setting where we can manipulate the independent variable directly. Under properly controlled conditions, we can then see what happens to the dependent variable—all other things being equal—after the independent variable is changed. This approach produces results with higher validity than observational data, information gathered without the use of experimental procedures (i.e., where the researcher only “observes” and cannot alter the independent variable).

Most sociologists use ethnographic, survey, or interview data rather than experimental data, given the difficulties of studying social phenomena in a lab setting. Social psychologists are a major exception to this generalization: you can think of them as the perspicacious love child of two disciplines—employing the laboratory techniques foundational to psychology while studying the topics and issues foundational to sociology. Furthermore, as we will describe, sociologists can use observational data in quasi-experiments that approximate a true experimental design, improving their study’s internal validity beyond what they could do with merely an analysis of observational data.

Even if you do not wind up using experiments in your own research, it is essential that you understand why a well-done experiment excels at causal inference—its ability to infer, or conclude, whether a relationship is truly causal—and why other methods fall short. To fully appreciate the high internal validity of experimental approaches, let’s review the four criteria for causal theories that we mentioned way back in Chapter 3: The Role of Theory in Research:

1. Correlation: A change in one variable is associated with a change in the other. (This is the bare minimum requirement for causality—remember, “correlation is not causation”!)
2. Temporal precedence: The cause occurs before the effect. In this case, we’re interested in the sequence of events: the change in the independent variable has to take place before the change in the dependent variable, or the former cannot logically be considered a cause of the latter.
3. Rejection of alternative hypotheses: After testing whether other variables explain the phenomenon of interest, we conclude that these other possibilities are less convincing.
4. Causal mechanisms: We can describe the specific process by which changes in the independent variable lead to changes in the dependent variable.

Experiments can be designed to rigorously test whether an observed relationship meets the first three of these criteria. Their results can also shed light on the fourth criterion, causal mechanisms, which we typically describe by using theory—that is, applying logic and our knowledge of past research to understand how the variables of interest might be connected. (As we noted in earlier chapters, qualitative research excels at sketching out these causal processes.)

The first two criteria, correlation and temporal precedence, are a relatively low bar for a research study to meet. As we’ll describe in Chapter 14: Quantitative Data Analysis, we can even use statistical analysis to generate a number (e.g., a correlation coefficient) telling us how strongly associated our two variables are. However, that’s not enough for us to be able to conclude that a change in the independent variable causes a change in the dependent variable. We still need to show that the results we see are due solely to the influence of the independent variable and not to any other factor.

That’s the hardest of the three standards to meet—what we call nonspuriousness. You might remember that Chapter 3: The Role of Theory in Research discussed how we can easily be fooled by spurious relationships if we don’t account for all confounding variables, which are associated with both the independent and dependent variables of interest and therefore might be the real reason we detect a correlation. Experiments can help us rule out any confounders and avoid spuriousness. If we set up the experiment properly, we can eliminate all alternative explanations that might cause the outcomes we observe other than our independent variable. But that requires us to have all the necessary pieces of a rigorous experiment in place: pretests and posttests, experimental and control groups, random assignment to either group—that is, all the features of the ideal, or “true,” experiment that we will describe throughout this chapter. Any missing piece will open the door to other possible causal stories that explain our experiment’s results.

In addition to internal validity, we also often talk about external validity when we critique an experiment’s design. If internal validity is about causality, external validity is about generalizability—specifically, whether the results of an experiment conducted on a particular sample can be generalized to other situations and populations beyond the context of our research setting. While experiments tend to have high internal validity, they generally do worse with external validity. First of all, a laboratory experiment is typically done in an office, with researchers trying their best to emulate conditions in the real world. This artificial setting may not adequately match what actually happens in everyday social life. To improve upon the contrived nature of the lab setting, scientists may use field experiments, experiments conducted in real-world settings. Another issue is that we typically don’t have large random samples when we conduct experiments. Remember from our discussion in Chapter 6: Sampling that the standard method of obtaining a representative sample is to choose the cases in that sample randomly from the target population. The larger our sample is, the more precise our results are. That approach of generating large samples (a large n) is not typically possible or desirable with experiments.

We’ll have more to say about field experiments later, but we want to highlight here the second problem, which is harder to address. Randomly selecting participants for a social experiment is not always possible. Since we are ethical social scientists, we can’t force people to be part of our study. We must rely on volunteers, who can be difficult to recruit in large numbers. External validity is therefore a perennial problem with experimental designs, and we’ll go into some of the related issues in detail.

Opening chapter image credit: Martin Lopez, via Pexels. Adapted by Bizhan Khodabandeh.