Ensemble Methods: Combining Weak Learners for Robust Alpha

Most writing about trading ensembles argues that more models are better. That is the wrong axis. The error of an ensemble decomposes into three terms, and in trading only one of them is ever the binding constraint.

This article is about that one term, and about two consequences of it that the rest of this blog does not cover: stacking as an explicit signal-combination layer, and whether combining many signals actually nets down turnover before the trades reach the market.

The Term That Actually Binds: Bias-Variance-Covariance

For a regression ensemble of $N$ models, the expected squared error of the average prediction $\bar{f}$ decomposes as:

$E\left[\left(\bar{f} - y\right)^2\right] = \overline{\text{bias}^2} + \frac{1}{N}\overline{\text{variance}} + \left(1 - \frac{1}{N}\right)\overline{\text{covariance}}$

Three terms, three levers:

• Bagging attacks the variance term by averaging over bootstrap samples.
• Boosting attacks the bias term by iteratively correcting residuals.
• Stacking attacks the covariance term by learning combination weights instead of assuming equal ones.
• Increasing $N$ shrinks only the variance contribution. It does nothing at all to covariance — the $(1 - 1/N)$ coefficient is already 0.99 at $N = 100$.

That last point is the whole argument. In trading, models are trained on the same instruments, over the same history, against targets that are themselves near-identical transformations of the same price series. Their errors are not close to independent. So the covariance term is large and the variance term is already exhausted, which means model count buys you almost nothing and structural difference buys you everything.

The concrete, falsifiable form of the claim: adding the 101st gradient-boosted tree should move the ensemble metric less than adding one structurally different model — an LSTM, a linear model on a disjoint feature domain, anything with a different inductive bias. The marginal tree is nearly collinear with the trees already in the book; the structurally different model is not.

Correlation, not model count, sets the effective breadth of an ensemble — the derivation, the crypto-measured correlation factors by asset class, and the simulation against real pair data are in Signal Correlation: How Many Pairs to Monitor.

Do not reduce ensemble diversity to a single number. The Deflated Sharpe Ratio computes five effective-$N$ estimators (average correlation, participation ratio, PCA-95%, Kaiser, Cheverud-Nyholt) on a real 640-cell grid, shows they span $N_{\text{eff}}$ from 1.6 to 370, and argues that the band — not any single point estimate — is the deliverable. Report the band.

Stacking as a Signal-Combination Layer

Bagging and boosting are well covered elsewhere. Spread Modeling with Machine Learning ships the gradient-boosting setup, the loss selection for skewed targets, the SHAP importance findings, the feature taxonomy, and a purged walk-forward split with the forward-window overlap that motivates the gap. Start there; this section assumes it.

What is not covered is the level above: a meta-learner that takes base-model predictions as its input features and learns how to combine them.

• Level 0 (base learners): diverse models producing predictions from raw features.
• Level 1 (meta-learner): a model that learns which base learners to trust, and when.

The mechanism that makes this safe is out-of-fold meta-features. The meta-learner must never see a base-model prediction that was made on data the base model was trained on — those predictions are optimistically biased in a way that does not survive to live trading, and the meta-learner will weight them accordingly.

import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.model_selection import TimeSeriesSplit


class TradingStackingEnsemble:
    """
    Two-level stacking ensemble for return prediction.

    Level 0: base learners, each on its own feature domain
    Level 1: meta-learner trained on out-of-fold base predictions
    """

    def __init__(self, base_models: list, meta_model, n_splits: int = 5):
        self.base_models = base_models  # list of (name, model, feature_cols)
        self.meta_model = meta_model
        self.n_splits = n_splits
        self.fitted_base_models_ = {}

    def _generate_meta_features(
        self,
        X: pd.DataFrame,
        y: pd.Series
    ) -> pd.DataFrame:
        """Out-of-fold predictions from each base model."""
        tscv = TimeSeriesSplit(n_splits=self.n_splits)
        meta_features = pd.DataFrame(index=X.index)

        for name, model, feature_cols in self.base_models:
            oof_preds = pd.Series(np.nan, index=X.index)

            for train_idx, val_idx in tscv.split(X):
                X_train = X.iloc[train_idx][feature_cols]
                y_train = y.iloc[train_idx]
                X_val = X.iloc[val_idx][feature_cols]

                fold_model = clone(model)
                fold_model.fit(X_train, y_train)

                if hasattr(fold_model, 'predict_proba'):
                    oof_preds.iloc[val_idx] = fold_model.predict_proba(X_val)[:, 1]
                else:
                    oof_preds.iloc[val_idx] = fold_model.predict(X_val)

            meta_features[name] = oof_preds

        return meta_features.dropna()

    def fit(self, X: pd.DataFrame, y: pd.Series):
        meta_features = self._generate_meta_features(X, y)
        y_meta = y.loc[meta_features.index]

        self.meta_model.fit(meta_features, y_meta)

        for name, model, feature_cols in self.base_models:
            model.fit(X[feature_cols], y)
            self.fitted_base_models_[name] = model

        return self

    def predict(self, X: pd.DataFrame) -> np.ndarray:
        meta_features = pd.DataFrame()

        for name, model, feature_cols in self.base_models:
            fitted = self.fitted_base_models_[name]
            if hasattr(fitted, 'predict_proba'):
                meta_features[name] = fitted.predict_proba(X[feature_cols])[:, 1]
            else:
                meta_features[name] = fitted.predict(X[feature_cols])

        return self.meta_model.predict(meta_features)

The splitting discipline here is not optional and not novel — use a purged, embargoed walk-forward, and understand what each class of leak is worth in inflated Sharpe before trusting any of these numbers. Both are already measured: the look-ahead bias taxonomy quantifies execution, indicator and normalization leakage in a controlled simulator, and walk-forward optimization covers anchored vs rolling windows, CPCV, purging and the WFER degradation diagnostics.

Three design decisions are specific to stacking and worth stating.

Meta-learner simplicity. Use a linear model — ridge, lasso, logistic regression. The base learners handle nonlinearity; the meta-learner handles combination only. A complex meta-learner on top of complex base learners is second-order overfitting, and L1 regularization at the combination layer has the useful side effect of zeroing out base models that contribute only noise.

Probability outputs over hard labels. predict_proba carries confidence information that a class label discards, and lets the meta-learner weight confident base predictions more heavily.

Feature-domain partitioning. This is the highest-leverage diversity lever, and it follows directly from the covariance argument: models that see the same features produce correlated errors regardless of algorithm.

FEATURE_GROUPS = {
    'momentum': ['ret_5d', 'ret_10d', 'ret_21d', 'ret_63d'],
    'volatility': ['vol_21d', 'vol_ratio', 'vol_skew', 'garch_vol'],
    'volume': ['volume_ratio', 'obv_slope', 'vwap_dist', 'adv_ratio'],
    'mean_reversion': ['dist_from_ma21', 'dist_from_ma63', 'rsi_14', 'bb_pct'],
    'microstructure': ['bid_ask_spread', 'kyle_lambda', 'amihud_illiq'],
    'fundamental': ['ep_ratio', 'bp_ratio', 'roe', 'accruals'],
}

base_models = []
for domain, features in FEATURE_GROUPS.items():
    model = GradientBoostingClassifier(
        n_estimators=200, max_depth=3,
        learning_rate=0.05, subsample=0.7
    )
    base_models.append((domain, model, features))

Each base model becomes a domain expert: the momentum model learns momentum patterns without interference from volume noise, and the meta-learner learns which expert to trust under which conditions.

Two caveats on that last clause. First, regime-conditional weighting is a solved and published problem — regime detection with HMMs derives regimes properly via Forward/Viterbi/Baum-Welch and feeds them as features to a downstream ensemble, and dynamically combining strategies covers regime-weighted mean-reversion vs momentum with reported results. Do not substitute an ad-hoc momentum-sign proxy for either. Second, selection on in-sample performance manufactures winners that do not survive; the generic mechanism and the CSCV defense are measured in the probability of backtest overfitting. Stacking is a narrower instance of exactly that problem.

For weighting return streams rather than model predictions, do not re-derive mean-variance. 12 Portfolio Optimization Algorithms, Compared races HRP, HERC, GHRP, MHRP, Black-Litterman, NCO, MVO, risk parity and equal weight with measured results and open-source code; Markowitz portfolio theory for crypto has the SLSQP mean-variance implementation; and the 1/N-is-hard-to-beat result is already tested against HRP and CVaR in the HRP/CVaR portfolio pipeline.

Turnover Netting: The Claim Worth Testing

Here is the part of the alpha-library argument that this blog's execution-cost cluster does not currently cover.

When Alpha #4,721 says "buy MSFT" and Alpha #8,392 says "sell MSFT" in the same rebalance, those trades cross internally. Only the net position reaches the market. The claim is that with enough signals a large fraction of gross intended turnover cancels before incurring spread, fees, or market impact — which would mean an ensemble is cheaper to run than the sum of its components, not merely more stable.

This is a distinct mechanism from what cascade strategy orchestration already handles. Cascade resolves conflicts at the slot level: same pair, opposite direction is prohibited and settled by score, cross-pair eviction runs through a priority queue, and the article proves empirically that per-strategy PnL cannot simply be summed because of time overlap and capital constraints. Netting is about gross-to-net position arithmetic reducing the traded quantity itself.

It is also cheap to falsify: compute gross vs net turnover on real fills when the component signals are combined, then price both through the existing cost machinery — maker-taker fees and rebates, implementation shortfall and TCA, and slippage cost models.

A note on scale, since the alpha-library framing invites it. WorldQuant reports having produced a very large alpha library, and Kakushadze's 101 Formulaic Alphas documents the shape of the individual signals. But mining alphas at industrial scale is also the largest multiple-testing machine imaginable, and this blog's position on that is settled: a search of size $K$ must be deflated against the expected best-by-luck of a search that size. The Deflated Sharpe Ratio shows a 640-cell grid already breaking naive significance testing, and The Honest Negative shows a ~37,000-trial search producing a +16.35% out-of-sample winner that deflates to DSR 0.00. Library size is not evidence of edge. It is the denominator you have to pay.

Putting It Together

Raw Data
  |
  v
Feature Engineering
  |
  v
Feature Subsets (partitioned by domain -> decorrelated errors)
  |
  +---> Base Model 1: RF on momentum features
  +---> Base Model 2: GBM on volatility features
  +---> Base Model 3: LSTM on price sequences
  +---> Base Model 4: Lasso on fundamental features
  +---> Base Model 5: XGBoost on microstructure features
  |
  v
Out-of-Fold Predictions (purged walk-forward)
  |
  v
Meta-Learner (Ridge)
  |
  v
Combined Signal -> Net Position -> Execution
  |
  v
DSR >= 0.95 and PBO <= 0.2 gate, or it does not ship

The last box is not decoration. Stacking over $K$ base models is a selection procedure of size $K$, and an ensemble article that does not deflate is indefensible.

Takeaways

1. Structural difference beats model count. The covariance term does not shrink with $N$. Adding a sixth correlated tree is not diversification; adding a different model class or a disjoint feature domain is.
2. Report the effective-$N$ band, not a point. Five estimators, five answers, sometimes two orders of magnitude apart.
3. Keep the meta-learner linear. Complexity at the combination layer is almost always overfitting, and L1 there does useful automatic model selection.
4. Out-of-fold or nothing. A meta-learner trained on in-sample base predictions learns the wrong weights with high confidence.
5. Deflate the ensemble itself. Stacking is a search. Price it like one.

Further reading:

• Kakushadze, Z. (2016). 101 Formulaic Alphas
• Ensemble Learning in Investment — CFA Institute Research Foundation, 2025
• Stock Return Prediction: Stacking a Variety of Models — Journal of Financial Economics
• Jansen, S. Machine Learning for Algorithmic Trading, 2nd edition