Hyper-Parameter Tuning with Cross-Validation

This chapter explains how to perform hyper-parameter tuning for financial machine learning, emphasizing that standard cross-validation (CV) methods will fail and lead to overfitting. The key is to use the Purged k-fold CV (from Chapter 7) as the core mechanism for any tuning method.

Tuning Methods

1. Grid Search Cross-Validation

• Concept: An exhaustive search that tries every possible combination of hyper-parameters from a predefined grid.
• Implementation: The clfHyperFit function is introduced, which integrates GridSearchCV (from scikit-learn) with the custom PurgedKFold class. This ensures that the search for the best parameters is not contaminated by information leakage between the train and test folds.

2. Randomized Search Cross-Validation

• Concept: A more efficient method for models with many parameters. Instead of trying every combination, it samples a fixed number of parameter combinations from a specified distribution (e.g., "sample 100 combinations").
• Benefits:
  1. Provides control over the computational budget.
  2. Is not slowed down by irrelevant parameters, unlike grid search.
• Log-Uniform Distribution: For parameters that are non-negative and non-linear (like C or gamma in an SVM), sampling from a standard uniform distribution (e.g., U[0, 100]) is inefficient. The author proposes a "log-uniform distribution" where the logarithm of the value is sampled uniformly. This ensures that the search is balanced between small (e.g., 0.01-1) and large (e.g., 1-100) scales.
  • PDF:
    $$f[x]=\left\{\begin{array}{cl} \frac{1}{x \log [b / a]} & \text { for } a \leq x \leq b \\ 0 & \text { otherwise } \end{array}\right.$$

Scoring Metrics: Why 'Accuracy' is Wrong

Choosing the right scoring function for tuning is critical. The author argues that 'accuracy' is a poor metric for financial models.

• Preferred Score: 'neg_log_loss' (Negative Log Loss)

  • Why: Log loss (or cross-entropy) heavily penalizes confident, wrong predictions. Accuracy, by contrast, treats a 90% confident "miss" the same as a 51% confident "miss."
  • Equation:
    $$L[Y, P]=-N^{-1} \sum_{n=0}^{N-1} \sum_{k=0}^{K-1} y_{n, k} \log \left(p_{n, k}\right)$$
  • Relevance: An investment strategy's PnL is highly sensitive to confident, wrong bets. Since neg_log_loss accounts for the probability of the prediction (position size) and can be weighted by the return of the observation (sample weight), it is a much better proxy for PnL than simple accuracy.

• Score for Meta-Labeling: 'f1'

  • Why: In meta-labeling, the model predicts a binary outcome (bet / no-bet), which is often highly imbalanced (mostly '0's). A model that just predicts "no-bet" all the time would have high accuracy but is useless.
  • Solution: The 'f1' score, which is the harmonic mean of precision and recall, correctly measures the model's ability to find the rare, positive 'bet' cases.

API reference

RiskLabAI implements these in Python and Julia (signatures auto-generated from the package source):

def clf_hyper_fit(
    feature_data: pd.DataFrame,
    label: pd.Series,
    times: pd.Series,
    pipe_clf: Pipeline,
    param_grid: dict[str, Any],
    validator_type: str = "purgedkfold",
    validator_params: Optional[dict[str, Any]] = None,
    bagging: Optional[list[Union[int, float]]] = None,
    rnd_search_iter: int = 0,
    n_jobs: int = -1,
    **fit_params,
) -> Union[GridSearchCV, RandomizedSearchCV, Pipeline]:

function grid_search_cv(
    cv,
    x::AbstractMatrix{<:Real},
    y::AbstractVector,
    param_grid::AbstractDict;
    scoring::Symbol = :accuracy,
    random_state::Integer = 0,
)

def clf_hyper_fit(
    feature_data: pd.DataFrame,
    label: pd.Series,
    times: pd.Series,
    pipe_clf: Pipeline,
    param_grid: dict[str, Any],
    validator_type: str = "purgedkfold",
    validator_params: Optional[dict[str, Any]] = None,
    bagging: Optional[list[Union[int, float]]] = None,
    rnd_search_iter: int = 0,
    n_jobs: int = -1,
    **fit_params,
) -> Union[GridSearchCV, RandomizedSearchCV, Pipeline]:

function random_search_cv(
    cv,
    x::AbstractMatrix{<:Real},
    y::AbstractVector,
    param_grid::AbstractDict;
    n_iter::Integer = 10,
    scoring::Symbol = :accuracy,
    random_state::Integer = 0,
)

Full source: Python · Julia