1.4 Experimental Design and Ethics

Learning Objectives

By the end of this section, you will be able to:

Does aspirin reduce the risk of heart attacks? Is one brand of fertilizer better at growing roses than another? Is being tired behind the wheel as dangerous as being drunk behind the wheel? Questions like these get answered with randomized experiments. In this section you will learn what has to be built into an experiment before its results mean anything. A study that is designed badly does not produce weak data — it produces data you cannot use at all, which is worse, because the numbers still look like numbers.

1.4.1 The Anatomy of an Experiment

Definitions 1.4.1 and 1.4.2 — The explanatory variable is the one you set; the response is the one you read off.

Definitions 1.4.3 and 1.4.4 — The treatments are the values of one explanatory variable, handed out to the experimental units.

Notice how these four terms stack. The experimental units are who is in the study. The treatments are what you do to them. The explanatory variable is the general thing the treatments are versions of, and the response variable is the one number you measure at the end to see if any of it mattered. Naming all four for a study, before you look at a single result, is the fastest way to find out whether the study can answer the question it claims to answer.

The purpose of an experiment is to investigate the relationship between two variables. One variable is the one you push on; the other is the one you watch to see what happens.

1.4.2 Lurking Variables and Random Assignment

Definition 1.4.5 — In a self-selected comparison the gap has four candidate causes, and the study cannot tell them apart.

To prove that the explanatory variable is what is causing the change in the response variable, you have to isolate it. The researcher must design the experiment so that there is exactly one difference between the groups being compared: the treatment they were assigned. That is accomplished by random assignment.

Definition 1.4.6 — Random assignment spreads a hidden trait evenly, so the treatment is the only difference left.

Once random assignment has done its work, the only systematic difference left between the groups is the one the researcher imposed. So if the groups end up with different values of the response variable, that difference has to be a direct result of the different treatments. This is the whole reason experiments can prove a cause-and-effect connection while surveys and observational studies generally cannot.

Which raises an uncomfortable question: what happens when the explanatory variable is something you are not allowed to assign, or physically cannot assign? You cannot randomly assign someone a birth order, a hometown, or a gender. You cannot ethically assign someone to smoke for twenty years. In every one of those cases the experiment simply cannot be run, and the researcher is left comparing groups that formed themselves — which is exactly the situation random assignment was invented to escape. The next example works through one of these, and the reasoning it uses is the reasoning you should reach for any time a study reports a difference between groups the researcher merely found rather than made.

Suppose you want to investigate whether vitamin E prevents disease. You recruit a group of subjects and ask them if they regularly take vitamin E. You notice that the subjects who take vitamin E are healthier on average than those who do not. Does this prove that vitamin E works?

It does not. There are many differences between those two groups besides vitamin E. People who take vitamin E regularly often take other steps to improve their health: they exercise, they eat better, they take other supplements, they choose not to smoke. Any one of those could be the real reason the vitamin E group looks healthier. As described, this study proves nothing about vitamin E.

Figure 1.4.1 — Drawing the assignment in rāSHio: File → Random Numbers.

1.4.3 Blocking

Random assignment protects you from lurking variables you have not thought of. But when you already know a variable matters, leaving it to chance is a waste of the information you have.

Suppose we are comparing two versions of a tutoring programme, and we know that students who work more than twenty hours a week tend to score lower regardless of tutoring. With simple random assignment, we might by bad luck land most of the heavy-work students in one group. The comparison would then be partly a comparison of work schedules wearing a tutoring costume.

Blocking removes that risk by construction. Sort the students into two blocks — those working over twenty hours and those working under — then randomly assign half of each block to each version. Both treatment groups now contain the same mix of work schedules, guaranteed rather than hoped for.

There is a bonus that surprises people. Because each block is internally similar, the differences you see within a block are less noisy, so a blocked experiment can detect a smaller real effect with the same number of subjects. Blocking is not only insurance against bad luck; it is often the cheapest way to make a study more sensitive.

Definitions 1.4.7 and 1.4.8 — Sort on the variable you already know, then randomize separately inside each block.

1.4.4 The Power of Suggestion: Control Groups, Placebos, and Blinding

Definition 1.4.9 — The control group sets the baseline; only what rises above it is the treatment's effect.

Definition 1.4.10 — A placebo gives both groups the belief, so whatever is left over is the drug.

Definition 1.4.11 — Blinding hides which arm you are in; the assignment underneath never changes.

The double-blind design guards against a second, quieter problem. Even a scrupulously honest researcher who knows which subject got the real treatment will time them a little differently, prompt them a little differently, or read an ambiguous result a little more generously. Blinding the researcher removes that possibility entirely, which is why the double-blind, placebo-controlled randomized experiment is the design that medical journals treat as the gold standard. The next example is a case where full blinding is impossible — the subjects can tell instantly which treatment they are getting — and it shows you what a researcher does when they can only get half of it.

The expectation of the person being studied can be as powerful as the treatment itself. In one study of performance-enhancing drugs, researchers noted:

Results showed that believing one had taken the substance resulted in [performance] times almost as fast as those associated with consuming the drug itself. In contrast, taking the drug without knowledge yielded no significant performance increment.

Read that twice. Believing you took the drug worked nearly as well as taking it. Taking it without knowing did nothing measurable. When simply being in a study prompts a physical response, isolating the effect of the explanatory variable gets much harder — some of what you measure is the treatment, and some of it is the belief.

1.4.5 Ethics and the Cost of Fraud

The widespread misuse and misrepresentation of statistical information often gives the field a bad name. Some say that "numbers don't lie," but the people who use numbers to support their claims often do.

An investigation of the famous social psychologist Diederik Stapel led to the retraction of his articles from some of the world's top journals. Stapel is a former professor at Tilburg University in the Netherlands. An extensive investigation involving three universities where Stapel had worked concluded that the psychologist was guilty of fraud on a colossal scale. Falsified data taints over 55 papers he authored and 10 Ph.D. dissertations that he supervised.

Stapel did not deny that his deceit was driven by ambition. But it was more complicated than that, he told me. He insisted that he loved social psychology but had been frustrated by the messiness of experimental data, which rarely led to clear conclusions. His lifelong obsession with elegance and order, he said, led him to concoct sexy results that journals found attractive. "It was a quest for aesthetics, for beauty — instead of the truth," he said. He described his behavior as an addiction that drove him to carry out acts of increasingly daring fraud, like a junkie seeking a bigger and better high.

The committee investigating Stapel concluded that he was guilty of several practices, including:

• creating datasets that largely confirmed his prior expectations;
• altering data in existing datasets;
• changing measuring instruments without reporting the change; and
• misrepresenting the number of experimental subjects.

Clearly, it is never acceptable to falsify data the way this researcher did. Sometimes, however, violations of ethics are not as easy to spot.

Researchers have a responsibility to verify that proper methods are being followed. The report describing the investigation of Stapel's fraud states that "statistical flaws frequently revealed a lack of familiarity with elementary statistics." Many of Stapel's co-authors should have spotted irregularities in his data. Unfortunately, they did not know very much about statistical analysis, and they simply trusted that he was collecting and reporting data properly.

Many types of statistical fraud are difficult to spot. Some researchers simply stop collecting data once they have just enough to prove what they had hoped to prove. They don't want to take the chance that a more extensive study would complicate their lives by producing data that contradicts their hypothesis.

Professional organizations, like the American Statistical Association, clearly define expectations for researchers. There are even laws in the federal code about the use of research data.

It is important that students of statistics take time to consider the ethical questions that arise in statistical studies. How prevalent is fraud in statistical studies? You might be surprised — and disappointed. There is a website (http://www.retractionwatch.com) dedicated to cataloging retractions of study articles that have been proven fraudulent. A quick glance will show that the misuse of statistics is a bigger problem than most people realize.

Vigilance against fraud requires knowledge. Learning the basic theory of statistics will empower you to analyze statistical studies critically.

1.4.6 Protecting Human Subjects

Definitions 1.4.13 and 1.4.14 — A planned study reaches no one until it clears IRB review and then informed consent.

For this reason, research institutions establish oversight committees, and every planned study must be approved by one in advance. Key protections that are mandated by law include the following:

• Risks to participants must be minimized and reasonable with respect to projected benefits.
• Participants must give informed consent. This means that the risks of participation must be clearly explained to the subjects of the study. Subjects must consent in writing, and researchers are required to keep documentation of their consent.
• Data collected from individuals must be guarded carefully to protect their privacy.

These ideas may seem fundamental, but they can be very difficult to verify in practice. Is removing a participant's name from the data record sufficient to protect privacy? Perhaps the person's identity could be discovered from the data that remains. What happens if the study does not proceed as planned and risks arise that were not anticipated? When is informed consent really necessary? Suppose your doctor wants a blood sample to check your cholesterol level. Once the sample has been tested, you expect the lab to dispose of the remaining blood. At that point the blood becomes biological waste. Does a researcher have the right to take it for use in a study?

When a statistical study uses human participants, as in medical studies, both ethics and the law dictate that researchers should be mindful of the safety of their research subjects. The U.S. Department of Health and Human Services oversees federal regulations of research studies with the aim of protecting participants. When a university or other research institution engages in research, it must ensure the safety of all human subjects.

1.4.7 Associated and Independent Variables

Most statistical questions are really questions about whether two variables travel together. Before we can ask why they might, we need language for whether they do.

Hours of sleep and quiz score are plausibly associated: told that a student slept three hours, you would revise your guess about the score downward, even though you would sometimes be wrong. Shoe size and favourite music genre are plausibly independent: told a student wears a size 11, you have learned nothing useful about their playlist.

Association is also symmetric, which is worth saying out loud because everyday language hides it. If sleep tells you something about quiz scores, then quiz scores tell you something about sleep. The arrow you imagine between them is something you brought to the data, not something the association contains — which is precisely why association alone can never establish a direction, let alone a cause.

Definitions 1.4.15 and 1.4.16 — Knowing x narrows the values y can take, or it does not; that is the whole distinction.

Problem Set 1.4

Key Terms