Model Quality Guidelines
Standards and Best Practices for High-Quality MMM Models
Overview
Model quality guidelines help you evaluate whether your marketing mix model is reliable, accurate, and suitable for business decisions. This page outlines standards across statistical fit, variable significance, diagnostics, and business validation.
Purpose: Ensure models meet minimum quality thresholds before use in optimization and forecasting
Quality Dimensions
1. Statistical Fit (R²)
Measures: How well model explains variation in KPI
Quality Thresholds:
80%+
Excellent
Rare in MMM, very strong fit
70-80%
Good
Target range for MMM
60-70%
Acceptable
Usable, room for improvement
50-60%
Marginal
Needs significant work
< 50%
Poor
Not ready for business use
Context matters:
Some KPIs harder to predict (brand awareness vs sales)
More variables generally increase R²
Adjusted R² accounts for variable count
2. Variable Significance
Measures: Statistical reliability of coefficients
Standards:
✅ All variables significant (p < 0.05)
T-statistic > 1.96
95% confidence coefficient ≠ 0
Gold standard
⚠️ Most variables significant
80%+ with p < 0.05
Some marginal (p < 0.10)
Acceptable if theoretically important
❌ Many non-significant
30% with p > 0.10
Model needs refinement
Remove weak variables
3. Coefficient Signs
Measures: Do relationships make business sense?
Requirements:
✅ Marketing variables: Positive
More spend → More sales
Fundamental expectation
✅ Price: Negative
Higher price → Lower demand
Standard economic relationship
✅ Competitor activity: Negative
More competition → Lower sales
Expected pattern
❌ Wrong signs indicate issues:
Data problems
Multicollinearity
Missing confounders
Model misspecification
4. Diagnostic Tests
Measures: Model assumptions met
Critical Tests:
Multicollinearity (VIF):
All VIF < 10 ✓
Most VIF < 5 (ideal)
Residual Normality:
P-value > 0.05 (Shapiro-Wilk)
Q-Q plot follows line
Autocorrelation:
Durbin-Watson: 1.5 - 2.5
No serial correlation
Heteroscedasticity:
P-value > 0.05 (Breusch-Pagan)
Constant variance
5. Business Validation
Measures: Does model match reality?
Validation Checks:
✅ Decomposition makes sense
Channel contributions align with expectations
Seasonal patterns match known trends
Major events show up in baseline
✅ ROI reasonable
Within industry benchmarks
Aligns with historical performance
Defendable to stakeholders
✅ Marginal returns declining
Saturation curves applied appropriately
Diminishing returns captured
Optimization feasible
Minimum Quality Standards
For Exploratory Models
Use for: Initial analysis, learning, experimentation
Minimum standards:
R² > 50%
Most variables p < 0.10
Correct signs on key variables
No perfect multicollinearity
Acceptable issues:
Some diagnostic failures
Non-significant controls
Moderate multicollinearity
For Production Models
Use for: Budget optimization, forecasting, strategic decisions
Minimum standards:
R² > 65% (preferably > 70%)
All marketing variables p < 0.05
All marketing variables positive
VIF < 10 for all variables
Durbin-Watson 1.5 - 2.5
Residuals approximately normal
Required:
Passes most diagnostic tests
Business validation complete
Stakeholder review
Documentation
For High-Stakes Decisions
Use for: Major budget shifts, executive presentations, board decisions
Gold standards:
R² > 75%
All variables p < 0.01
Perfect coefficient signs
VIF < 5 for all variables
Passes ALL diagnostic tests
Out-of-sample validation
Sensitivity analysis complete
Multiple model comparison
Quality Improvement Workflow
Phase 1: Initial Build (Target R² 50-60%)
Focus: Get basic model working
Actions:
Add core marketing variables
Include obvious controls (trend, seasonality)
Check coefficient signs
Achieve minimal fit
Quality check: R² > 50%, correct signs
Phase 2: Variable Optimization (Target R² 60-70%)
Focus: Improve variable selection
Actions:
Pre-test additional variables
Add significant variables
Remove non-significant variables
Apply adstock to media
Address multicollinearity
Quality check: R² > 60%, most p < 0.05, VIF < 10
Phase 3: Transformation (Target R² 70-80%)
Focus: Capture non-linearities
Actions:
Apply saturation curves to media
Test curve parameters
Create interaction terms if needed
Optimize transformations
Quality check: R² > 70%, all p < 0.05
Phase 4: Diagnostics (Production Ready)
Focus: Pass all tests
Actions:
Run full diagnostic suite
Address autocorrelation
Fix heteroscedasticity
Validate residuals
Check influential points
Quality check: Passes diagnostic tests
Phase 5: Validation (High Confidence)
Focus: Business validation
Actions:
Review decomposition with stakeholders
Validate ROI estimates
Test out-of-sample performance
Sensitivity analysis
Document thoroughly
Quality check: Business validation complete
Common Quality Issues
Issue: Low R² (< 60%)
Causes:
Missing important variables
Need transformations (adstock, curves)
Wrong model specification
High noise in KPI
Solutions:
Add more marketing variables
Apply adstock and saturation curves
Include control variables
Consider data quality
Issue: Non-Significant Variables
Causes:
Variable doesn't affect KPI
Multicollinearity
Insufficient variation
Wrong transformation
Solutions:
Remove if consistently non-significant
Check VIF for multicollinearity
Try different adstock rates
Verify data quality
Issue: Wrong Coefficient Signs
Causes:
Multicollinearity
Omitted variable bias
Data quality issues
Reverse causality
Solutions:
Check VIF
Add missing confounders
Validate data
Test Granger causality
Issue: Failed Diagnostics
Causes:
Autocorrelation in residuals
Non-constant variance
Non-normal residuals
Influential outliers
Solutions:
Add lagged dependent variable
Transform variables
Remove outliers
Add missing time effects
Quality Checklists
Pre-Presentation Checklist
Before showing model to stakeholders:
[ ] R² > 70%
[ ] All marketing variables significant (p < 0.05)
[ ] All marketing variables positive
[ ] VIF < 10 for all variables
[ ] Durbin-Watson 1.5 - 2.5
[ ] Residuals approximately normal
[ ] Decomposition reviewed
[ ] ROI calculations validated
[ ] Model documented
[ ] Results exportable
Pre-Optimization Checklist
Before using for budget allocation:
[ ] Saturation curves applied
[ ] Adstock optimized
[ ] Marginal returns declining
[ ] All diagnostics pass
[ ] Business validation complete
[ ] Sensitivity tested
[ ] Stakeholder buy-in
[ ] Documentation complete
Production Release Checklist
Before using for ongoing decisions:
[ ] Out-of-sample validation complete
[ ] Multiple model versions compared
[ ] Robustness tested
[ ] Update process defined
[ ] Monitoring plan in place
[ ] Documentation comprehensive
[ ] Training provided
[ ] Support plan established
Model Comparison Standards
Comparing Model Versions
Criteria for "better" model:
✅ Higher R² (adjusted for variables) ✅ More significant variables ✅ Better diagnostic performance ✅ More stable coefficients ✅ Superior business validation
Trade-offs:
Complexity vs interpretability
Fit vs parsimony
Statistical vs business criteria
Selecting Final Model
Weighted decision criteria:
40% Statistical fit
R² and adjusted R²
Variable significance
Diagnostic tests
30% Business validation
Decomposition credibility
ROI reasonableness
Stakeholder confidence
20% Robustness
Stability across specifications
Out-of-sample performance
Sensitivity to changes
10% Simplicity
Interpretability
Ease of explanation
Actionability
Documentation Standards
Minimum Documentation
Every model should document:
Model name and version
KPI and time period
Variables included (with transformations)
R² and key statistics
Major findings
Known limitations
Comprehensive Documentation
Production models require:
Full variable list with rationale
Transformation decisions and parameters
All diagnostic test results
Decomposition analysis
ROI calculations
Sensitivity analysis
Validation results
Update schedule
Contact information
Key Takeaways
Target R² > 70% for production MMM models
All marketing variables should be significant (p < 0.05) and positive
VIF < 10 for all variables to avoid multicollinearity
Pass diagnostic tests: normality, autocorrelation, heteroscedasticity
Validate with business logic: decomposition, ROI, seasonality
Build iteratively: basic model → optimization → transformation → diagnostics → validation
Document thoroughly before sharing or using for decisions
Compare multiple model versions to select best
Establish update and monitoring processes for production models
Last updated