6.1 Sensitivity Analysis
What Is Sensitivity Analysis?
Sensitivity analysis examines how your results change when you vary assumptions, model specifications, or data samples. It answers the question: "Are my results robust, or do they depend on arbitrary choices?"
Types of Sensitivity Analysis:
| Type | Description |
|---|
| Specification Sensitivity | Vary model specification (variables, functional form) |
| Sample Sensitivity | Exclude observations (jackknife, leave-one-out) |
| Parameter Sensitivity | Vary tuning parameters (Ξ» in Lasso) |
| Assumption Sensitivity | Test violations of assumptions |
Implementation in MORISTAT:
- Specification Sensitivity β Drop each variable one at a time
- Jackknife β Leave-one-out estimation
- Coefficient Stability Plot β Visualize coefficient variation
Interpretation Guidelines:
| Stability | Coefficient Range |
|---|
| High | CV < 10% β Results are robust |
| Moderate | 10% β€ CV < 25% β Some sensitivity |
| Low | CV β₯ 25% β Results are fragile |
6.2 Specification Testing
What Is Specification Testing?
Specification tests check whether your model is correctly specified. A model is correctly specified if all relevant variables are included, the functional form is appropriate, and the assumptions hold.
Common Specification Tests:
| Test | Purpose | Detects |
|---|
| RESET Test | Functional form | Nonlinearity, omitted variables |
| Hausman Test | Fixed vs Random Effects | Endogeneity of entity effects |
| Hausman-Wu Test | Endogeneity | Correlation between X and error |
| Sargan-Hansen J | Overidentification | Invalid instruments |
| Link Test | Model specification | Omitted variables, functional form |
RESET Test (Regression Specification Error Test)
Purpose: Test whether the model has the correct functional form.
Procedure:
- Estimate the model and obtain fitted values: ΕΆ
- Compute powers of ΕΆ: ΕΆΒ², ΕΆΒ³, ΕΆβ΄
- Regress Y on X, ΕΆΒ², ΕΆΒ³, ΕΆβ΄
- Test whether the coefficients on the powers are jointly zero
Null Hypothesis: Hβ: Model is correctly specified
Interpretation:
- p β₯ 0.10 β No evidence of misspecification
- p < 0.05 β Misspecification detected
Link Test (Pregibon's Link Test)
Purpose: Test for omitted variables or incorrect functional form.
Procedure:
- Estimate the model and obtain predicted values: ΕΆ
- Compute ΕΆΒ²
- Regress Y on ΕΆ and ΕΆΒ²
- Test whether the coefficient on ΕΆΒ² is zero
Interpretation:
- p β₯ 0.10 β No evidence of misspecification
- p < 0.05 β Omitted variables or incorrect form
6.3 Outlier Detection
What Are Outliers?
Outliers are observations that are unusual relative to the rest of the data. They can have a disproportionate influence on your results.
Types of Outliers:
| Type | Description | Impact |
|---|
| Regression Outlier | Unusual response given X | Affects coefficient estimates |
| Leverage Point | Unusual X values | Affects precision |
| Influential Point | Both unusual X and Y | Large effect on results |
Diagnostic Measures:
| Measure | Formula | Threshold |
|---|
| Leverage (h_i) | hα΅’ = Xα΅’(X'X)β»ΒΉXα΅’' | > 2k/n |
| Studentized Residual | rα΅’* = eα΅’ / (ΟΜβ(1-hα΅’)) | > 2 |
| Cook's Distance | Dα΅’ = (ΕΆ - ΕΆ_(i))'(ΕΆ - ΕΆ_(i)) / (kΟΜΒ²) | > 4/n |
| DFBETA | Change in Ξ² when i is removed | > 2/βn |
Leverage (hα΅’)
Purpose: Identify observations with unusual X values.
hα΅’ = Xα΅’(X'X)β»ΒΉXα΅’'
Interpretation:
- hα΅’ < 2k/n β Low leverage (OK)
- hα΅’ > 2k/n β High leverage (potentially influential)
Cook's Distance (Dα΅’)
Purpose: Measure the overall influence of observation i.
Dα΅’ = (ΕΆ - ΕΆ_(i))'(ΕΆ - ΕΆ_(i)) / (kΟΜΒ²)
Interpretation:
- Dα΅’ < 1 β Not influential
- Dα΅’ > 1 β Influential observation
- Dα΅’ > 4/n β Check observation
Studentized Residuals
Purpose: Identify observations with unusual Y values given X.
rα΅’* = eα΅’ / (ΟΜβ(1-hα΅’))
Interpretation:
- |rα΅’*| < 2 β Normal residual
- |rα΅’*| > 2 β Potential outlier
- |rα΅’*| > 3 β Clear outlier
6.4 Bootstrap Inference
What Is Bootstrap?
Bootstrap is a resampling method that estimates the sampling distribution of a statistic by repeatedly drawing samples with replacement from the original data.
When to Use Bootstrap:
| Situation | Why Bootstrap |
|---|
| Non-normal errors | Standard errors may be invalid |
| Small samples | Asymptotic approximations fail |
| Complex models | Analytical standard errors are difficult |
| Non-standard statistics | No known distribution |
Types of Bootstrap:
| Type | Description |
|---|
| Non-parametric | Resample observations |
| Parametric | Resample from fitted distribution |
| Residual Bootstrap | Resample residuals |
| Wild Bootstrap | Resample with heteroskedasticity |
Bootstrap Procedure:
- Estimate the model on the original sample β Ξ²Μ
- For b = 1 to B:
- Draw a bootstrap sample (with replacement) of size n
- Estimate the model on the bootstrap sample β Ξ²Μα΅
- Compute bootstrap statistics:
- Standard error: SE = SD(Ξ²Μα΅)
- Confidence interval: [2Ξ²Μ - Ξ²Μ_(0.975), 2Ξ²Μ - Ξ²Μ_(0.025)]
- Bias-corrected interval
Bootstrap Confidence Intervals:
| Type | Formula | When to Use |
|---|
| Normal | Ξ²Μ Β± zΒ·SE* | Symmetric distribution |
| Percentile | [Ξ²Μ*(Ξ±/2), Ξ²Μ*(1-Ξ±/2)] | General use |
| BCa | Bias-corrected and accelerated | Skewed distributions |
6.5 Model Averaging
What Is Model Averaging?
Model averaging combines predictions from multiple models to improve forecast accuracy and account for model uncertainty.
Why Model Averaging?
| Problem | Solution |
|---|
| Model uncertainty | Weighted average of models |
| Overfitting | Shrinkage toward simpler models |
| Prediction accuracy | Improved out-of-sample performance |
| Robustness | Less sensitive to model choice |
Methods of Model Averaging:
| Method | Weights |
|---|
| BMA | Posterior probabilities |
| AIC Weights | exp(-0.5ΞAIC) |
| BIC Weights | exp(-0.5ΞBIC) |
| Stacking | Cross-validated weights |
| Equal Weights | 1/M |
AIC Weights:
wα΅’ = exp(-0.5Β·ΞAICα΅’) / Ξ£β±Ό exp(-0.5Β·ΞAICβ±Ό)
Where ΞAICα΅’ = AICα΅’ - min(AIC)
BMA (Bayesian Model Averaging):
Posterior Model Probability:
P(Mα΅’ | data) = P(data | Mα΅’)P(Mα΅’) / Ξ£β±Ό P(data | Mβ±Ό)P(Mβ±Ό)
Model-Averaged Coefficients:
E(Ξ² | data) = Ξ£α΅’ P(Mα΅’ | data) Β· Ξ²Μα΅’
6.6 Publication-Ready Output
Reporting Standards:
| Style | Reference |
|---|
| APA | American Psychological Association (7th Edition) |
| AEA | American Economic Association Style Guide |
| Econometrica | Econometric Society Style |
Essential Reporting Elements:
- Model Specification β Variables, functional form
- Sample Description β N, time period, data source
- Estimation Method β OLS, IV, FE, etc.
- Coefficient Estimates β With standard errors and significance
- Goodness of Fit β RΒ², AIC, BIC, Log-Likelihood
- Diagnostic Tests β VIF, BP, DW, JB, Hausman
- Robustness Checks β Sensitivity, alternative specifications
Coefficient Table Format (AEA Style)
Table 1: Determinants of Earnings
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
(1) (2) (3)
OLS 2SLS FE
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Education 0.089*** 0.132** 0.095***
(0.012) (0.018) (0.014)
Experience 0.026*** 0.024*** 0.025***
(0.008) (0.009) (0.008)
Ability (unobserved) β β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
N 1,000 1,000 1,000
RΒ² 0.456 0.423 0.389
F-statistic 45.23 38.12 52.67
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Notes: *** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses.
Column (1) uses OLS. Column (2) uses IV with QOB instruments.
Column (3) uses Fixed Effects.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Diagnostic Summary Table
Table 2: Diagnostic Tests
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Test Statistic p-value Conclusion
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
VIF (Education) 2.34 β No multicollinearity
VIF (Experience) 1.87 β No multicollinearity
Breusch-Pagan 18.45 0.009 Heteroskedasticity
Durbin-Watson 1.89 β No autocorrelation
Jarque-Bera 2.34 0.567 Normality
Hausman-Wu 6.78 0.009 Endogeneity
Sargan-Hansen J 2.34 0.311 Instruments valid
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
6.7 Implementation Checklist
Before Reporting Results:
- Run sensitivity analysis β Check coefficient stability
- Perform specification tests β RESET, Link, Hausman
- Check outliers β Leverage, Cook's D, DFBETA
- Compute bootstrap standard errors β If assumptions are violated
- Consider model averaging β If model uncertainty is high
- Format publication-ready tables β APA/AEA style
- Include all diagnostic test results β VIF, BP, DW, JB
6.8 Chapter Summary
Key Takeaways
- Sensitivity Analysis β Test robustness to model specification, sample, and assumptions
- Specification Testing β Use RESET, Link, and Hausman tests to check model validity
- Outlier Detection β Identify influential observations using leverage, Cook's D, and DFBETA
- Bootstrap Inference β Use resampling for valid standard errors when assumptions are violated
- Model Averaging β Combine models to account for model uncertainty
- Publication-Ready Output β Present results professionally with all necessary diagnostics
Checklist Before Publishing:
- All model assumptions checked and reported
- Sensitivity analysis performed
- Outliers identified and addressed
- Robust standard errors used (if needed)
- Complete diagnostic tests reported
- Results are reproducible
6.9 What's Next?
π Chapter 7: Reporting and Visualization
We will cover:
- Result Presentation β Professional tables and figures
- Publication-Ready Output β APA, AEA formatting
- Interactive Dashboards β Dynamic data exploration
- Automated Reporting β Reproducible research