Hyperparameter Selection under Dependence

Choosing the best setting and reporting its score on the same folds favors noise. The notebooks place parameter search inside an outer evaluation and preserve the temporal protections needed by overlapping labels.

The central relation

Within outer split $m$ , the mean inner score for setting $\theta$ is

\overline s_m(\theta) =\frac{1}{K_{\mathrm{in}}}\sum_{j=1}^{K_{\mathrm{in}}}s_{m,j}(\theta).

$K_{\mathrm{in}}$ is the number of inner folds, $s_{m,j}(\theta)$ is the held-out inner score, and $\overline s_m(\theta)$ is used to select a setting without looking at the outer test observations.

Testing model choices through time

The nested-score plot separates inner selection from outer evaluation. The expanding-history result shows whether the preferred setting changes as more dates become available.

Inner selection scores and outer held-out scores for several parameter settings. — The setting is chosen from inner folds, while the outer fold measures its result on observations that did not enter the search.

Selected hyperparameters across expanding chronological training histories. — The expanding-history check records whether the selected setting persists or changes as the available past grows.

Nested evaluation measures the complete selection procedure. It does not guarantee that a setting chosen from past data remains best in a different market period.

Notebooks


1. How nested evaluation controls selection bias
2. Search with Purged Validation
3. Mapping stability across the parameter surface