FAMILY_CUMULATIVE_INPUT_STRESS: Experiment Log

Overview

Testing cumulative input cost stress accumulation for Dutch potato price forecasting using the proven CUMULATIVE METHODOLOGY from FAMILY_WEATHER_ACCUMULATION (92.4-97.5% improvement). This hypothesis applies accumulation patterns to European input cost data, learning from the failure of immediate transmission approaches.

Hypothesis Origins

Successful Pattern Foundation

• FAMILY_WEATHER_ACCUMULATION: Revolutionary 92.4-97.5% improvement using CUMULATIVE methodology
• Gradual accumulation superior to extreme events
• Sub-EUR error achievement (0.748-1.412 MAE)
• Statistical significance confirmed (DM p=0.000)
• Proven framework: cumulative stress over 3-12 month windows

Learning from Input Transmission Failures

• FAMILY_REGIONAL_INPUT_DIVERGENCE: STRONGLY REFUTED (-71.9% vs baseline)
• Despite confirming massive 2022 crisis (German 216.8 peak vs French +40.1%)
• Failure reason: Immediate transmission assumption violated agricultural lag structure

• Key lesson: Input effects require production cycle delays (6-18 months)

• FAMILY_INPUT_COST_TRANSMISSION: Previous variant REJECTED for synthetic data use

• Mechanism still viable with proper lag structure and REAL data
• Must use only verified Eurostat/Destatis/INSEE APIs

Natural Experiment Validation

• 2022 European Fertilizer Crisis: Massive regional divergences confirmed
• German Destatis: 1,142+ observations, peak fertilizer index 216.8
• French INSEE: 1,534+ observations, +40.1% fertilizer impact
• Multi-region data availability validated for cumulative analysis

Academic Foundation

• Chavas & Holt (1990): Supply response lags in agriculture (6-18 month adjustment)
• Nerlove (1958): Adaptive expectations with multi-period adjustment
• McCorriston et al. (2001): 60-80% cost pass-through with temporal delays

Key Innovation: CUMULATIVE vs IMMEDIATE

CRITICAL DIFFERENCE: - IMMEDIATE (failed): Input costs → Dutch prices same period - CUMULATIVE (this hypothesis): Input stress accumulates over months → compound effects through production planning

Mechanism: Input cost advantages accumulate over multiple quarters affecting: 1. Planting decisions (6-12 month forward planning) 2. Harvest timing (regional cost advantages influence storage decisions) 3. Processing arbitrage (cumulative cost differentials drive cross-border sourcing) 4. Supply reallocation (multi-season accumulation of regional advantages)

Experiment Design

• Method: Rolling-origin cross-validation with temporal structure preservation
• Training Window: 365 days minimum
• Step Size: 7 days (weekly)
• Test Window: 60 days maximum
• Baselines: ALL mandatory standard baselines (persistent, seasonal_naive, ar2, historical_mean)
• REAL DATA ONLY: Destatis, INSEE, Eurostat, Boerderij APIs

Data Sources (REAL DATA ONLY)

• Dutch Prices: BoerderijApi (NL.157.2086) - consumption potatoes - git:current
• German Inputs: DestatisAPI (APRI_PI15_INA fertilizer, NRG_PC_204 diesel) - 1,142+ observations
• French Inputs: INSEE API (001762871 fertilizer, 001762863 energy) - 1,534+ observations
• Belgian/EU Inputs: EurostatAPI (STS_SETU_M transport, APRI_PI15_INQ fertilizer)
• Regional Integration: Multi-country input cost accumulation with production cycle alignment

Experiment Runs

Variant A: Single-Input Accumulation (Fertilizer Focus)

Status: Pending implementation - Model: RandomForest with cumulative fertilizer stress features - Key Features: - fertilizer_stress_3m/6m/12m: Multi-window accumulation - de_fr_fertilizer_divergence_cumulative: Regional cost differentials - crisis_stress_accumulation: 2022-style crisis build-up - production_cycle_alignment: Seasonal stress timing - SESOI: 20% improvement threshold - Expected: 60-75% improvement (conservative vs weather accumulation)

Variant B: Multi-Input Accumulation (Compound Stress)

Status: Pending implementation
- Model: GradientBoosting with multi-input stress interactions - Key Innovation: compound_input_stress = fertilizer × energy × transport interactions - Features: 10 compound stress features including acceleration and persistence - SESOI: 25% improvement threshold (higher complexity) - Expected: 70-85% improvement (compound effects)

Variant C: Regional-Weighted Accumulation (Production Weighting)

Status: Pending implementation - Model: XGBoost with production-weighted stress indices
- Key Innovation: Regional production weights applied to cumulative stress - Features: Market share adjustments, processing flow weights, supply chain cascade - SESOI: 22% improvement threshold - Expected: 65-80% improvement (supply chain integration)

Statistical Tests

• Diebold-Mariano test with Harvey-Leybourne-Newbold correction
• TOST equivalence test with variant-specific SESOI thresholds
• Granger causality test for input→price relationships with lag structure
• Bai-Perron structural break test for crisis vs normal accumulation periods
• FDR correction for multiple comparisons across variants

Success Criteria

• Primary: Compare against STRONGEST baseline (likely seasonal_naive)
• Target: 60-85% improvement over strongest baseline
• Statistical: DM p<0.05 with HLN correction
• Practical: SESOI thresholds met with TOST validation
• Directional: >55% directional accuracy

Expected Performance Rationale

1. Proven Cumulative Framework: Direct application of FAMILY_WEATHER_ACCUMULATION methodology
2. Rich Multi-Country Data: 1,142+ German + 1,534+ French + Belgian + Dutch observations
3. Natural Experiment: 2022 crisis provides ideal validation period
4. Production Cycle Logic: Agricultural lag structure well-established in literature
5. Conservative Target: Input accumulation may be less predictive than weather accumulation

Risk Factors

• Quarterly fertilizer data may limit statistical power vs weekly price data
• Complex multi-input interactions may require larger sample sizes
• Regional input data publication delays could affect temporal alignment
• Agricultural lag structure (6-18 months) may exceed forecast horizons

Data Integrity Requirements

• ✅ REAL DATA ONLY from verified government statistics APIs
• ✅ NO synthetic, mock, or dummy data permitted
• ✅ European government validation: Destatis + INSEE + Eurostat + CBS
• ✅ Cumulative methodology: 3-12 month accumulation windows
• ✅ Production cycle alignment: Agricultural lag structure consideration

Verdicts

(Experiments not yet run - pending EX implementation)

HE Notes

• Created 2025-08-19 combining successful FAMILY_WEATHER_ACCUMULATION pattern with input cost domain
• Critical innovation: CUMULATIVE vs immediate transmission methodology
• Learns from FAMILY_REGIONAL_INPUT_DIVERGENCE failure while exploiting proven regional cost divergences
• All variants use ONLY REAL DATA from verified European government APIs
• Conservative performance targets acknowledge input accumulation may be less predictive than weather
• First systematic application of cumulative methodology to input cost transmission

Decision Log

2025-08-19: Initial Formulation (HE)

• Innovation: Applied proven FAMILY_WEATHER_ACCUMULATION cumulative methodology to input costs
• Learning: Incorporated lessons from FAMILY_REGIONAL_INPUT_DIVERGENCE failure (immediate transmission)
• Data Sources: Confirmed availability of German (1,142+ obs), French (1,534+ obs), Belgian input data
• Variants: Created three complementary approaches (single, multi, regional-weighted)
• Next Steps: EX implementation with strict REAL DATA requirements and mandatory baseline testing

(To be updated after experiment completion)