Multi-arm A/B Test for 3 Fast Food Promotions

Why This Project?

Retail data often comes in weekly panel format with uneven performance across locations. This project mirrors real-world business experimentation where you need to compare promotions while accounting for store-level differences and repeated measures. It gave me the opportunity to demonstrate proper handling of clustered data, time structure, and model robustness.

Methodology

Before estimating the effect of each promotion on weekly sales, I conducted a series of data checks to ensure the validity of the analysis.

Data

The dataset included four weeks of sales data per store, with each store randomly assigned to one of three promotional conditions. The unit of analysis was the store-week, resulting in repeated observations per store. To account for this, I used clustered standard errors at the store level in all regression models.

Randomization and Covariate Balance

To confirm that the promotion assignment was successfully randomized, I checked:

• Group sizes and their distribution across market sizes (via value_counts and pd.crosstab)
• Balance on store age using group means and standardized mean differences (SMDs) between treatment groups

All SMDs were below 0.2 except for Promo 3, which had a slightly older average store age, suggesting only mild imbalance.

Exploratory Analysis

I explored the relationship between potential covariates and weekly sales:

• Calculated Pearson correlation between store age and sales
• Created scatterplots and boxplots to assess distribution and skew
• Plotted sales over time by week and promotion group to assess time trends

Outlier Detection and Removal

Using the interquartile range (IQR) method, I identified and removed extreme sales values within each promotion group. This reduced the influence of outliers and improved the symmetry of the outcome distribution.

Outcome Transformation

To account for right-skew in sales, I applied a log transformation using log1p(SalesInThousands). This stabilized variance and reduced the impact of high-end values on model results.

Modeling Approach

I ran a series of OLS regression models, progressively adjusting for:

• Promotion group (categorical)
• Store age (continuous)
• Market size (categorical)
• Week (categorical, in exploratory models)

Standard errors were clustered by store to account for within-store correlation. I compared models with and without outliers, and with both raw and log-transformed outcomes, to assess the robustness of results.

Results

I tested the impact of three promotional strategies on weekly sales using a series of OLS regression models. Models included controls for market size and store age, and used clustered standard errors to account for repeated measures within each store. Results were evaluated with and without outliers, and using both raw and log-transformed versions of the outcome variable.

Primary Findings

• Promotion 2 consistently and significantly reduced sales compared to Promotion 1 across all model specifications.
• Promotion 3 showed no statistically significant difference from Promotion 1 in any model.
• Market size was a strong predictor of sales: stores in medium and small markets earned significantly less than those in large markets.
• Store age did not have a meaningful impact on weekly sales.

Effect Sizes

In the final model using log-transformed sales without outliers:

• Promotion 2 reduced weekly sales by approximately 23% compared to Promotion 1 (p < 0.001).
• Promotion 3 resulted in a non-significant change of ~2.6% lower sales.
• Stores in medium markets saw ~36% lower sales than those in large markets; small market stores earned ~10% less.

Robustness Checks

Results were consistent across:

• Raw sales and log-transformed sales
• Models with and without outlier removal
• Models including and excluding week fixed effects (no evidence of time trends)

Although residuals showed mild non-normality and some autocorrelation (Durbin-Watson < 1), the use of clustered standard errors addressed these issues.

Visualization

Plots of weekly sales over time confirmed that promotion effects were stable across the 4-week period. Histograms and boxplots showed a right-skewed distribution of sales, justifying the use of a log transformation. Outliers identified via the IQR method were primarily concentrated in smaller markets and higher sales values.