ML Asset Allocation (HRP)

This chapter introduces Hierarchical Risk Parity (HRP), a machine learning-based asset allocation method designed to overcome the critical flaws of traditional quadratic optimizers, such as Markowitz's Critical Line Algorithm (CLA).

The Flaw of Traditional Optimizers: Markowitz's Curse

Traditional mean-variance optimization (like CLA) suffers from three major problems: instability, concentration, and underperformance out-of-sample.

These flaws are a result of Markowitz's Curse:

• The optimizer relies on inverting the covariance matrix.
• When assets are highly correlated (which is precisely when diversification is most needed), the covariance matrix becomes ill-conditioned (its "condition number," the ratio of max to min eigenvalues, is high).
• Inverting an ill-conditioned matrix is numerically unstable, meaning tiny changes in the input (e.g., a single correlation estimate) can lead to dramatically different and unstable portfolio allocations.
• This instability is why naïve 1/N (equally-weighted) portfolios often outperform sophisticated optimizers out-of-sample.

The HRP Solution: From Geometry to Hierarchy

HRP's core innovation is that it does not require matrix inversion. It restructures the problem using graph theory and machine learning:

• Traditional optimizers view the portfolio as a "complete graph," where every asset is a potential substitute for every other asset. This is unstable.
• HRP first converts the portfolio into a "tree structure" (a hierarchy). This is more stable and intuitive, as allocations are distributed top-down among related clusters of assets (e.g., assets in the same sector).

The HRP Algorithm (3 Stages)

HRP is built in three stages:

1. Tree Clustering

This stage uses machine learning to build the hierarchy (the "tree") of assets.

• It computes a correlation-based distance matrix $D$ where the distance $d_{i,j}$ between two assets $i$ and $j$ is:
  $$d_{i, j}=\sqrt{\frac{1}{2}\left(1-\rho_{i, j}\right)}$$
• It then applies a hierarchical clustering algorithm (like scipy.cluster.hierarchy.linkage) to this distance matrix, grouping similar assets together into branches.

2. Quasi-Diagonalization

This stage reorders the covariance matrix so that similar assets (as defined by the clusters) are placed next to each other.

• The result is a "quasi-diagonal" or block-diagonal matrix.
• Crucially, this is just a re-indexing; it does not change the basis (like PCA) and does not require matrix inversion.

3. Recursive Bisection

This is the top-down allocation step.

1. Start with the full portfolio (100% weight).
2. Recursively bisect (split) the portfolio into two sub-clusters based on the hierarchy from Step 1.
3. Calculate the total variance of each sub-cluster (using inverse-variance weighting within the cluster, $\tilde{V}^{(j)}$).
4. Distribute the total weight between the two sub-clusters in inverse proportion to their respective variances. The split factor $\alpha_i$ for the first cluster is:
   $$\alpha_{i}=1-\frac{\tilde{V}_{i}^{(1)}}{\tilde{V}_{i}^{(1)}+\tilde{V}_{i}^{(2)}}$$
   (The second cluster gets $1 - \alpha_i$).
5. Repeat this process down the tree until all assets have been allocated a weight.

Out-of-Sample Performance

Monte Carlo simulations show that HRP is significantly more robust than traditional methods.

• In-Sample: CLA produces the "optimal" minimum variance portfolio (by definition).
• Out-of-Sample:
  • CLA performs the worst, exhibiting the highest variance. It overfits the in-sample data.
  • Inverse-Variance Portfolio (IVP) performs better.
  • HRP delivers the lowest out-of-sample variance, successfully providing a stable and robust portfolio.

API reference

RiskLabAI implements these in Python and Julia (signatures auto-generated from the package source):

def inverse_variance_weights(covariance_matrix: pd.DataFrame) -> np.ndarray:

function inverse_variance_weights(covariance_matrix::AbstractMatrix{<:Real})

def cluster_variance(
    covariance_matrix: pd.DataFrame, clustered_items: list[str]
) -> float:

function cluster_variance(
    covariance_matrix::AbstractMatrix{<:Real},
    clustered_items::AbstractVector{<:Integer},
)

def quasi_diagonal(linkage_matrix: np.ndarray) -> list[int]:

function quasi_diagonal(linkage_matrix::AbstractMatrix{<:Real})

def recursive_bisection(
    covariance_matrix: pd.DataFrame, sorted_items: list[str]
) -> pd.Series:

function recursive_bisection(
    covariance_matrix::AbstractMatrix{<:Real},
    sorted_items::AbstractVector{<:Integer},
)

def pca_weights(
    cov: np.ndarray,
    risk_distribution: Optional[np.ndarray] = None,
    risk_target: float = 1.0,
) -> np.ndarray:

function pca_weights(
    cov::AbstractMatrix{<:Real};
    risk_distribution::Union{Nothing,AbstractVector{<:Real}} = nothing,
    risk_target::Real = 1.0,
)

def hrp(cov: pd.DataFrame, corr: pd.DataFrame) -> pd.Series:

function hrp(
    covariance::AbstractMatrix{<:Real},
    correlation::AbstractMatrix{<:Real},
)

def get_optimal_portfolio_weights(
    covariance: np.ndarray, mu: Optional[np.ndarray] = None
) -> np.ndarray:

function get_optimal_portfolio_weights(
    covariance::AbstractMatrix{<:Real};
    mu::Union{Nothing,AbstractVecOrMat{<:Real}} = nothing,
)

def get_optimal_portfolio_weights_nco(
    covariance: np.ndarray,
    mu: Optional[np.ndarray] = None,
    number_clusters: Optional[int] = None,
) -> np.ndarray:

function get_optimal_portfolio_weights_nco(
    covariance::AbstractMatrix{<:Real};
    mu::Union{Nothing,AbstractVector{<:Real}} = nothing,
    number_clusters::Union{Nothing,Integer} = nothing,
)

Full source: Python · Julia