Feature Importance under Dependence

Feature importance can be misleading when predictors substitute for one another. The notebooks compare standard measures with cluster and orthogonalized alternatives under controlled dependence and public data.

The central relation

Mean decrease in impurity for feature $j$ can be written as

\operatorname{MDI}_j =\frac{1}{B}\sum_{b=1}^{B} \sum_{t\in T_b:\,v(t)=j}p(t)\,\Delta i(t).

$B$ is the number of trees, $T_b$ is tree $b$ , and $v(t)=j$ means that node $t$ splits on feature $j$ . The node sample share is $p(t)$ and $\Delta i(t)$ is the impurity reduction from that split.

How correlated predictors divide importance

A controlled experiment shows how standard scores divide credit among substitutes. The public result reports importance after correlated predictors are grouped.

Impurity and permutation importance for controlled predictors with known signal and dependence. — When predictors share signal, standard importance can divide or redirect credit even though predictive information remains present.

Cluster-level feature importance for groups of correlated public predictors. — Cluster importance measures the contribution of a related group before assigning meaning to individual members of that group.

Importance is conditional on the fitted model, feature set, scoring rule, and dependence structure. It is not a causal effect.

Notebooks


1. Comparing feature importance measures
2. Measuring importance among correlated predictors
3. Debiasing importance rankings through orthogonalization