Model Confidence Dashboard
The Model Confidence Dashboard provides an executive-level view of PulseGrid's PG-MIM algorithm reliability. It computes the Model Confidence Index (MCI), a composite score from 0 to 100 that aggregates four dimensions of model performance.
MCI Formula
`
MCI = 0.35 × ValidationAccuracy + 0.20 × RegimeConfidence + 0.25 × ScoringConsistency + 0.20 × StatisticalSignificance
`
Component Scores
| Component | Weight | Description | Scoring Method |
|---|---|---|---|
| Validation Accuracy | 35% | Cross-asset divergence prediction accuracy | Normalizes accuracy from 50% baseline to 100% scale |
| Regime Confidence | 20% | Market regime detection reliability | Regime detector classification certainty × 100 |
| Scoring Consistency | 25% | Accuracy uniformity across instruments | 100 − (coefficient of variation × 100), clamped to [0, 100] |
| Statistical Significance | 20% | p-value, effect size, sample adequacy | Weighted combination of p-value score, Cohen's h, and sample size |
Grade Scale
| Grade | MCI Range |
|---|---|
| A+ | ≥ 90 |
| A | ≥ 80 |
| B+ | ≥ 70 |
| B | ≥ 60 |
| C+ | ≥ 50 |
| C | ≥ 40 |
| D | ≥ 30 |
| F | < 30 |
Dashboard Sections
1. MCI Gauge: Circular gauge displaying the composite score with letter grade
2. Component Score Cards: Four cards showing individual component scores with progress bars
3. Component Radar Chart: Visual comparison of the four MCI dimensions
4. Resolution Methods Breakdown: Pie chart showing how divergences were resolved (exact match, partial move, directional, staleness, time decay, pending)
5. Per-Asset-Class Confidence: Horizontal bar chart with accuracy and confidence per asset class
6. Key Metrics Grid: Six metric cards (accuracy, instruments, divergences, resolved, prophetic, info ratio)
7. Statistical Detail: p-value, z-score, effect size, and confidence interval with significance badges
8. MCI Trend: Weekly MCI history from archived snapshots
9. Confidence by Regime: Regime-dependent MCI analysis panel (see below)
10. Strengths / Weaknesses / Recommendations: Auto-generated narrative assessment
Confidence by Regime
The Confidence by Regime panel shows how the Model Confidence Index varies across different market regimes. This analysis supports the thesis that regime-aware scoring provides differentiated predictive power.
Regime Definitions and Horizon Multipliers:
| Regime | Horizon Multiplier | Confidence Adjustment | Characteristics |
|---|---|---|---|
| Steady State | 1.0x (baseline) | +5 | Low volatility, mean-reverting, normal correlations |
| High Volatility | 0.8x (compressed) | -5 | Elevated VIX, wider spreads, faster signal decay |
| Crisis | 0.6x (severely compressed) | -15 | Extreme correlations, liquidity stress, rapid regime shifts |
| Recovery | 1.3x (extended) | +10 | Declining volatility, trend restoration, normalizing correlations |
How Regime MCI is Computed:
The current regime uses actual observed accuracy data. For other regimes, the MCI is simulated by:
1. Adjusting accuracy by +/-15% based on the horizon multiplier's deviation from baseline
2. Applying a regime-specific confidence adjustment to the composite score
3. Estimating the resolved divergence count based on the horizon effect
Interpretation:
- A narrow MCI spread across regimes (less than or equal to 10 points) suggests the model is robust across market conditions
- A moderate spread (11-20 points) indicates the model is moderately sensitive to regime changes
- A wide spread (greater than 20 points) suggests the model is significantly regime-dependent and may require regime-specific calibration
Academic References:
- Ang, A. & Timmermann, A. (2012). "Regime Changes and Financial Markets."
- Hamilton, J.D. (1989). "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle."
Data Appendix Export
The "Data Appendix" button generates a publication-ready HTML document suitable for inclusion in academic manuscripts or book appendices. It includes:
- Full MCI breakdown with component scores
- Confidence by Regime analysis with horizon multipliers and formulas
- Per-asset-class confidence table
- Statistical methodology disclosure
- Divergence resolution methodology breakdown
- All relevant academic references
Methodology References
- Grinold, R.C. & Kahn, R.N. (2000). Active Portfolio Management. 2nd ed. McGraw-Hill.
- Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum.
- Ang, A. & Timmermann, A. (2012). Regime Changes and Financial Markets. Annual Review of Financial Economics.
- Hamilton, J.D. (1989). A New Approach to the Economic Analysis of Nonstationary Time Series. Econometrica.
Limitations
- The MCI is a composite metric and may mask individual component weaknesses
- Component weights are fixed and may not be optimal for all market conditions
- The MCI is expected to improve over time as more divergence events are observed
- Historical MCI values from snapshots reflect the data available at that time, not retroactive recalculation
- Regime simulations are estimates based on horizon multipliers, not actual observed data under those regimes
- The regime classification itself has uncertainty, which propagates into regime-specific MCI estimates