Experimental studies are those in which researchers impose one or more treatments on a group of experimental units. Contrast with observational studies, in which researchers simply collect data without changing any conditions. - **Experimental unit**: the entities to which a treatment are applied - **Sampling unit**: the entities on which the response is measured - **Treatment**: something researchers apply to each experimental unit - **Response**: outcome of interest - **Randomization**: use of a chance process to assign treatment to experimental units - **Replication**: process of independently assigning the treatments to several experimental units ### experimental unit versus sampling unit Company X wants to test four advertisement campaigns. They assign each campaign to one of four social media platforms and measure the responses of 1000 internet users on each platform. The experimental unit in this case is the social media platform, the entity to which the treatment is applied. The sampling unit is the individual internet user. In this experiment, there is no replication because there are four treatments and four social media platforms. ## treatment design Treatment designs help researchers think clearly about experimental factors. - How many treatments? - How many treatment levels? - Relationship among treatments - Crossed or factorial? - Nested? - Fixed or random effects? - Continuous covariates? ## randomization design Concerns the assignment of treatments to experimental units. - Experimental unit? - Treatment assignment? - Completely Randomized Design (CRD)? - Randomized Control Block Design (RCBD)? - Sample units within experimental units? - Replication? - Repeated measurements? ## completely randomized design In a completely randomized design (CRD) each experimental unit has the same chance of receiving any treatment. CRD is flexible, easy to analyze with one-way ANOVA, has larger degrees of freedom, and allows for different numbers of experimental units within each treatment. CRD is not appropriate when it is difficult to find homogenous experimental units. CRD is best suited for a small number of treatments. ## randomized complete block design In a randomized complete block design, units are first divided into homogenous blocks before they are randomly assigned to a treatment group. This is beneficial when an experimenter is aware of specific differences among groups of units within an experimental group. Nuisance variables are blocked in the RCBD. In general, block what you can and randomize what you cannot. Use two-way ANOVA with a treatment factor and blocking factor. Two-way ANOVA can be used to test two pairs of hypotheses, first, whether neither the blocking factor ($\alpha_k$) nor treatment factor ($\tau_j$) is statistically significant: $\begin{align} H_0: Y_{ijk} = \mu + \epsilon_{ijk} && H_1: Y_{ijk} = \mu + \alpha_k + \epsilon_{ijk} \end{align}$ and second whether the treatment factor is statically significant given the blocking factor $\begin{align} H_0: Y_{ijk} = \mu + \alpha_k + \epsilon_{ijk} && H_1: Y_{ijk} = \mu + \tau_j + \alpha_k + \epsilon_{ijk} \end{align}$ ## one factor at a time The one-factor-at-a-time (OFAT) experimental design is a commonsense but flawed approach to experimental design. This is often incorrectly used in A/B testing or testing the efficacy of advertisements. In this design, all other factors are held constant while testing the effect of changing one factor at a time in multiple experiments. OFAT often requires more resources, are less precise, and cannot detect or estimate interactions than other experimental designs. ## factorial design The factorial design is a good option if there are two or treatment factors. It is an alternative to the one-factor-at-a-time (OFAT) approach. A fully factorial design is one in which experimental units take on all possible combinations of the levels across all factors. Note that a randomized complete block design differs in principle because the blocks are not considered a treatment.