50% Allocation: Why Online Experiment Interactions Rarely Matter

Online experiments often run concurrently, raising concerns about potential interactions between them. Interaction occurs when the effect of one experiment depends on the variant of another. For instance, if one experiment tests a new search model and another a recommendation model, the effectiveness of the search might be influenced by the recommendation algorithm. However, empirical evidence suggests that these interaction effects are typically small.

Theoretical analysis shows that for interactions to significantly impact decision-making, the interaction effect must be substantial and change the sign of the treatment effect. For example, if an experiment’s treatment effect is +$2 per unit under one condition, it would need to be less than -$2 per unit under another for interactions to matter. This scenario is rare, as the interaction would need to be extreme at typical 50-50 allocations.

Thus, while interactions do introduce bias, they rarely alter the decision on which variant to launch. This is because online experiments aim to make decisions, not to estimate perpetual treatment effects. Factors like seasonality or economic changes also introduce similar biases, yet decisions remain robust.

Source: towardsdatascience.com