FAMILY_REGIME_SWITCHING_VOLATILITY: Experiment Log

Overview

Testing regime-switching volatility models for Dutch potato price forecasting through Markov-switching volatility patterns, weather-coupled regime detection, and multi-scale regime decomposition. This hypothesis builds on FAMILY_PRICE_VOLATILITY_CLUSTERING (INCONCLUSIVE with 8.2% QLIKE improvement) by advancing to sophisticated regime-switching methodologies and FAMILY_SPRING_VOL (CONDITIONALLY SUPPORTED with 84x volatility regime differences) by implementing systematic regime-switching models instead of seasonal volatility analysis.

Hypothesis Origins

• FAMILY_PRICE_VOLATILITY_CLUSTERING: INCONCLUSIVE with 8.2% QLIKE improvement meeting SESOI but lacking statistical significance, demonstrating volatility prediction potential needing regime-switching enhancement
• FAMILY_SPRING_VOL: CONDITIONALLY SUPPORTED with 84x volatility regime differences, validating volatility clustering and establishing foundation for regime-switching models
• FAMILY_WEEKLY_VOLATILITY: REFUTED/INCONCLUSIVE due to implementation issues but validated volatility modeling approach
• FAMILY_WEATHER_ACCUMULATION: SUPPORTED with 95.5% improvement demonstrating weather-price coupling that can trigger volatility regimes
• Industry catalyst: 2018 drought, COVID-19 (2020), energy crisis (2022), storage crisis (2024) created distinct volatility regimes requiring sophisticated modeling
• Academic basis: Hamilton (1989) Markov-switching models, weather-coupled regime switching, multi-scale regime decomposition theory

Experiment Design

• Method: Rolling-origin cross-validation
• Initial window: 365 days (1 year minimum)
• Step size: 7 days (weekly)
• Test windows: 60 days maximum
• Baselines: MANDATORY STANDARD BASELINES - persistent, seasonal_naive, ar2, historical_mean (from get_standard_baselines())
• REAL DATA ONLY: Boerderij.nl API prices, Open-Meteo weather data for regime triggers

Data Sources (REAL DATA ONLY)

• Prices: Boerderij.nl API (NL.157.2086) - Dutch consumption potatoes for regime detection - git:current
• Weather: Open-Meteo API (52.55°N, 5.55°E) - temperature, precipitation, soil moisture for regime triggers - git:current
• Volatility regimes: Calculated from REAL price data using Markov-switching models - git:current
• Seasonal patterns: Calculated from REAL price data using seasonal decomposition - git:current

Experiment Runs

Variant A: Markov-Switching Volatility Regimes

Status: Not started - Model: Hamilton methodology Markov-switching volatility with 3 regimes (Low, Medium, High volatility) - Features: volatility_regime_probabilities, regime_transition_signals, volatility_state_dependent, regime_persistence_indicators, switching_probability_dynamics, regime_dependent_mean_returns, volatility_clustering_regime, regime_stability_measures, hamilton_filter_probabilities, price_lag_1w - Horizons: 1-month, 2-month - Mechanism: Markov-switching volatility captures regime-dependent price dynamics with superior volatility forecasting through regime identification - Expected: >16% improvement through regime-switching volatility modeling

Variant B: Weather-Coupled Regime Detection

Status: Not started - Model: Weather-coupled regime switching with 4 regimes (Normal, Drought stress, Heat stress, Combined stress) - Features: weather_stress_regime_indicators, drought_stress_volatility_regime, heat_stress_volatility_regime, combined_stress_regime, weather_regime_transition_prob, meteorological_volatility_driver, seasonal_weather_regime_patterns, weather_persistence_regime_signals, climate_regime_coupling_strength, extreme_weather_regime_activation, price_lag_1w - Horizons: 1-month, 2-month - Mechanism: Weather stress conditions trigger volatility regime switches creating predictable weather-volatility coupling patterns - Expected: >18% improvement through weather-coupled regime switching

Variant C: Multi-Scale Regime Decomposition

Status: Not started - Model: Multi-scale regime decomposition with seasonal (4), weather (3), and market (2) regimes operating simultaneously - Features: seasonal_volatility_regimes, weather_volatility_regimes, market_volatility_regimes, multi_scale_regime_interaction, hierarchical_regime_probabilities, scale_specific_regime_persistence, regime_decomposition_components, cross_scale_regime_alignment, regime_scale_dominance_indicators, temporal_regime_coherence, scale_dependent_volatility_clustering, price_lag_1w - Horizons: 1-month, 2-month - Mechanism: Multi-scale regime decomposition captures simultaneous seasonal, weather, and market regime dynamics with superior temporal pattern recognition - Expected: >20% improvement through multi-scale regime decomposition

Statistical Tests

• Diebold-Mariano test with Harvey-Leybourne-Newbold correction vs ALL 4 standard baselines
• TOST equivalence test with SESOI = 16% improvement (regime switching shows strong volatility effects)
• Bai-Perron structural break test for regime periods vs normal periods
• Markov regime tests with 2-4 regimes and switching variance
• FDR correction for multiple comparisons across variants
• Directional accuracy threshold = 60%

Regime Analysis Framework

• Markov-Switching Models: Hamilton filter with EM estimation, time-varying transition probabilities, regime-dependent variance components
• Weather Coupling: Drought index + heat stress index + precipitation anomaly triggers, percentile-based weather extreme detection
• Multi-Scale Decomposition: Seasonal (52-week memory), weather (12-week memory), market (4-week memory) with hierarchical coupling
• Regime Validation: Known volatility periods (2018 drought, 2020 COVID, 2022 energy crisis, 2024 storage crisis) used for regime identification validation

Regime Validation Events

• 2018 Drought Regime: Q2/Q3 high volatility regime validation through drought stress indicators
• 2020 COVID Shock: Q2 extreme volatility regime testing with sudden regime transition
• 2022 Energy Crisis: Q1/Q2 high volatility regime switching testing during energy cost volatility
• 2024 Storage Crisis: Q1 medium-high volatility regime identification with storage constraints

Expected Regime Effects

• Markov-Switching: Distinct volatility regimes with different statistical properties and transition probabilities
• Weather Coupling: Weather stress triggers regime switches through threshold activation mechanisms
• Multi-Scale: Simultaneous seasonal, weather, and market regimes with cross-scale interactions
• Regime Persistence: Regime duration dependence and memory effects improving volatility forecasting

Verdicts

(Experiments not yet run)

HE Notes

• Created 2025-08-19 advancing beyond FAMILY_PRICE_VOLATILITY_CLUSTERING (8.2% QLIKE) and building on FAMILY_SPRING_VOL (84x volatility regime validation)
• Superior to simple volatility: uses sophisticated regime-switching models vs basic GARCH volatility clustering
• SESOI set to 16% due to strong regime-switching literature evidence for volatility forecasting
• All variants use ONLY REAL DATA from verified Boerderij.nl API and Open-Meteo interfaces
• Focus on 30/60-day horizons where regime effects strongest
• Validates using 2018 drought, 2020 COVID, 2022 energy crisis, 2024 storage crisis natural experiments

Decision Log

(To be updated after experiment completion)

Codex Validation — 2025-11-10

Files Reviewed

• run_experiment.py
• experiment.md
• hypothesis.yml

Findings

1. Real weather data now used. load_real_weather_data pulls daily Open-Meteo observations (max/min/mean temperature plus precipitation) and builds a soil-moisture proxy instead of the previous random simulation.
2. Experiments still pending. experiment.md has no runs yet, so none of the regime-switching models have been trained or evaluated.
3. Baseline comparison absent. With no run, there is still no evidence that the volatility-regime features beat price-only baselines.

Verdict

NOT VALIDATED – The synthetic weather issue is resolved, but the modeling/verification stage remains undone; results must still demonstrate statistically significant improvements over the baselines.