Clustering Predictors for Ensemble Learning

Adding predictors or models helps less when their errors move together. The notebooks begin with a voting calculation, construct clusters from dependence, and test whether those clusters persist across samples.

The central relation

For an odd number $N$ of independent voters with accuracy $p$ , majority accuracy is

A_N(p)=\Pr\!\left(K>\frac{N}{2}\right) =\sum_{k=(N+1)/2}^{N}{N\choose k}p^k(1-p)^{N-k}, \qquad K\sim\operatorname{Binomial}(N,p).

$K$ is the number of correct votes, $N$ is the ensemble size, and $p$ is each voter’s accuracy. The expression gives the independent-error benchmark before dependence is introduced.

How dependence limits ensemble gains

The controlled result measures how shared errors change ensemble accuracy. The public stability plot shows how often predictor pairs remain in the same cluster.

Ensemble accuracy as predictor error dependence changes in a controlled experiment. — The gain from adding voters falls as their errors become more dependent, even when individual accuracy is unchanged.

Stability of public predictor clusters across repeated samples or windows. — The stability result records how often the public predictor grouping is recovered rather than treating one dendrogram as certain.

A cluster is a summary of measured dependence under a chosen distance and linkage rule. It does not establish that predictors share an economic cause.

Notebooks


1. How predictor dependence limits bagging
2. Choosing the number of clusters from correlations
3. Testing nested clusters for stability