4.9% Uplift: A New Method for Measuring Impact Without Experiments

Data scientists often face the challenge of answering “what if” questions without the benefit of controlled experiments. A new method has been developed to measure the impact of interventions, such as changes in pricing strategy or store organization, using historical data. This approach involves creating a counterfactual prediction based on past data, evaluating the model’s performance, and then comparing this prediction with actual post-intervention data to calculate the uplift.

The method uses data from similar, unaffected stores as a control group, employing machine learning techniques like random forest regressors to predict outcomes. In a test case, this method measured a 4.9% uplift in sales, close to the artificially introduced 3% increase. The confidence interval for this uplift was calculated to be around 4.5% over 84 days, making it suitable for many practical applications.

This approach contrasts with Google’s Causal Impact method, which, while similar, tends to overestimate effects and has a wider confidence interval, making it less effective for detecting small changes. The new method is particularly useful when the effect is assumed constant, control data is noisy, and there is no significant trend in the data.

Source: towardsdatascience.com