Knowledge Distillation: Compressing Trading Models for Low-Latency Deployment

The accuracy-vs-latency tension in ML-driven trading already has a published answer on this blog. Spread modeling with machine learning recommends a two-stage split: a fast gradient-boosting model does the latency-critical real-time quoting, while a deep model runs asynchronously and feeds it a secondary signal or adjusts its parameters. Two models, two clocks, one system.

Knowledge distillation is a different answer to the same tension. Instead of running the slow model alongside the fast one, you use it once, offline, to train the fast one — the student learns the teacher's full probability distribution over outcomes, not just the hard labels, and the teacher then leaves the hot path entirely. One model at inference time, no asynchronous coupling, no staleness window.

Which answer wins is empirical, and this article does not yet answer it. What follows is the machinery, plus an explicit statement of the measurements that would decide it. Nothing here is a benchmark result; where a number would normally go, there is a marker saying what has to be run.

One framing correction up front, from DeepLOB and deep learning on the order book: high classification accuracy does not automatically translate into profit — the predicted move must clear the bid-ask spread. "Preserve the teacher's directional accuracy" is therefore the wrong thing to optimize a distillation setup against.

The Teacher-Student Framework

The original formulation by Hinton, Vinyals, and Dean (2015) is straightforward. You have a teacher model $T$ (large, slow, accurate) and a student model $S$ (small, fast, to be trained). The student learns from two signals simultaneously:

1. Hard targets: the ground-truth labels $y$ (e.g., price went up or down)
2. Soft targets: the teacher's output probability distribution $q_T$ over all classes

The student's loss function combines both:

$\mathcal{L} = \alpha \cdot \mathcal{L}_{\text{CE}}(y, \sigma(z_S)) + (1 - \alpha) \cdot T^2 \cdot D_{\text{KL}}\left(\sigma\left(\frac{z_T}{T}\right) \| \sigma\left(\frac{z_S}{T}\right)\right)$

where $z_T$ and $z_S$ are the teacher and student logits, $\sigma$ is the softmax function, $T$ is the temperature parameter, and $\alpha$ controls the balance between the two loss components.

Why Soft Targets Matter for Trading

The three-class up/stationary/down mid-price formulation, the $\pm\alpha$ thresholding, and why the resulting imbalance means you report weighted F1 rather than accuracy are all set up in DeepLOB — assume that label scheme here. The distillation-specific point is what the teacher emits before the argmax: a hard "up" carries one bit, while 0.72/0.21/0.07 also says the move may stall and will almost certainly not reverse. That structure across classes is the extra training signal, and it is why a soft-target student can generalize better than the same student trained on labels alone.

A warning on what that confidence is not. Softmax output is not calibrated uncertainty, and treating 0.55 vs. 0.85 as a position-sizing input is the shortcut that conformal prediction for trading exists to refuse — it derives sizing from interval width, an edge ratio, and a no-trade filter when the interval straddles zero, none of which a raw softmax gives you. Earning the sizing claim here means measuring the student's calibration against the teacher's (reliability diagram, ECE) and showing distillation preserves it. That result is not in this article yet.

Temperature and Soft Targets

The temperature parameter $T$ controls the "softness" of the probability distribution. Given logits $z_i$, the softmax with temperature is:

$\sigma(z_i; T) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

When $T = 1$ (standard softmax), the distribution is peaky — the dominant class gets most of the probability mass. As $T$ increases, the distribution flattens, revealing the relative magnitudes of the logits more clearly.

There is a plausible argument that trading models want a moderate temperature: financial predictions are far less confident than image classification, so a teacher may output 0.55/0.30/0.15 rather than 0.99/0.005/0.005, leaving less peakiness to soften before the signal washes out. That is an argument, not a finding — the range has to come from a sweep on real data, scored by weighted F1, and may differ by regime.

The $T^2$ factor in the KL divergence term compensates for the reduced gradient magnitudes at higher temperatures. Without it, the distillation loss would become negligibly small as $T$ increases.

Choosing Temperature via Grid Search

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
from sklearn.metrics import f1_score

def distillation_loss(
    student_logits: torch.Tensor,
    teacher_logits: torch.Tensor,
    labels: torch.Tensor,
    temperature: float,
    alpha: float,
) -> torch.Tensor:
    """Combined hard-target + soft-target distillation loss."""
    hard_loss = F.cross_entropy(student_logits, labels)

    soft_teacher = F.log_softmax(teacher_logits / temperature, dim=-1)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)

    soft_loss = F.kl_div(
        soft_student,
        soft_teacher,
        log_target=True,
        reduction="batchmean",
    )

    return alpha * hard_loss + (1.0 - alpha) * (temperature ** 2) * soft_loss


def search_temperature(
    teacher: nn.Module,
    student_factory,       # callable returning a fresh student
    train_loader: DataLoader,
    val_loader: DataLoader,
    temperatures: list[float] = [1, 2, 3, 5, 8, 12],
    alpha: float = 0.3,
    epochs: int = 30,
    lr: float = 1e-3,
    device: str = "cuda",
):
    """Grid search over temperature, scored by weighted F1 (not accuracy:
    the up/flat/down label scheme is heavily imbalanced toward flat)."""
    best_f1, best_T, best_student = 0.0, 1.0, None

    for T in temperatures:
        student = student_factory().to(device)
        optimizer = torch.optim.AdamW(student.parameters(), lr=lr)

        for epoch in range(epochs):
            student.train()
            for X, y in train_loader:
                X, y = X.to(device), y.to(device)
                with torch.no_grad():
                    teacher_logits = teacher(X)
                student_logits = student(X)

                loss = distillation_loss(
                    student_logits, teacher_logits, y, T, alpha
                )
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

        student.eval()
        preds, targets = [], []
        with torch.no_grad():
            for X, y in val_loader:
                preds.append(student(X.to(device)).argmax(dim=-1).cpu())
                targets.append(y)

        f1 = f1_score(
            torch.cat(targets), torch.cat(preds), average="weighted"
        )
        print(f"T={T:>4.1f}  val_weighted_f1={f1:.4f}")
        if f1 > best_f1:
            best_f1, best_T, best_student = f1, T, student

    print(f"\nBest temperature: T={best_T}, val_weighted_f1={best_f1:.4f}")
    return best_T, best_student

Distilling Ensembles into a Single Model

A quant ensemble mixes inductive biases: a gradient-boosted tree on order-book features, a 1D-CNN over recent ticks, a transformer over multi-timeframe windows, a linear model on macro factors. Averaging is more stable than any member alone, and running all four multiplies latency and cost — the situation the two-stage split from spread modeling with machine learning handles by demoting slow members to an asynchronous side channel. Distillation instead collapses all four into one student in the hot path.

The ensemble teacher's output is the average of its members' softmax outputs:

$q_{\text{ensemble}}(x) = \frac{1}{K} \sum_{k=1}^{K} \sigma(z_k(x) / T)$

where $K$ is the number of ensemble members. The student is trained against this averaged distribution.

class EnsembleTeacher(nn.Module):
    """Wraps K models, returns averaged logits for distillation."""

    def __init__(self, models: list[nn.Module]):
        super().__init__()
        self.models = nn.ModuleList(models)

    @torch.no_grad()
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        logits = torch.stack([m(x) for m in self.models], dim=0)
        return logits.mean(dim=0)   # average logits, not softmax


class TradingStudent(nn.Module):
    """Lightweight MLP for sub-millisecond inference."""

    def __init__(self, input_dim: int, hidden: int = 64, n_classes: int = 3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden),
            nn.ReLU(),
            nn.BatchNorm1d(hidden),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.BatchNorm1d(hidden),
            nn.Linear(hidden, n_classes),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x)

The parameter-count asymmetry is the whole point: a two-layer MLP with 64 hidden units is on the order of 8,000 parameters for a 60-feature, 3-class task, against an ensemble whose combined count runs into the millions.

What the Student Retains and What It Loses

This is the load-bearing empirical question and the article does not answer it. The intuition is that the student tracks the ensemble in-distribution and falls off in stressed regimes, where the ensemble's diversity is doing the work — but a retention figure only means something measured on real order-book data, split by regime, and reported as weighted F1. A student that holds up on calm days and collapses during a liquidation cascade is a different product from one that degrades gracefully, and an aggregate number cannot tell them apart.

Three mitigations are worth testing against that measurement rather than asserting in advance:

1. Include stressed periods in the distillation set, so the student sees the regimes where the gap is expected to open.
2. Feature-based distillation — match intermediate representations, not only final outputs.
3. Auxiliary regime head on the student, forcing regime-aware features into the shared trunk.

Self-Distillation: When the Student Becomes the Teacher

Self-distillation is a technique where a model distills knowledge from itself.

Born-Again Networks (BANs)

Train a student with architecture identical to the teacher. The "born-again" student often outperforms the original, and the process iterates:

$M_0 \xrightarrow{\text{distill}} M_1 \xrightarrow{\text{distill}} M_2 \xrightarrow{\text{distill}} \cdots$

Each generation trains on soft targets from the previous one, with gains typically saturating after a few generations. For trading models this costs nothing architecturally — no new features, no new data, just a different training procedure — which also means it is cheap to test and there is no excuse for reporting it untested.

Depth-Wise Self-Distillation

Attach auxiliary classifiers at intermediate layers. The deepest exit serves as teacher for the shallower ones. At inference you choose an exit: shallow for lower latency, deep for maximum accuracy.

This is the idea here with the best fit to a trading system, because exit depth becomes a runtime latency knob: one trained network covers a range of budgets instead of committing to a single architecture at training time. When the book is moving fast you take the shallow exit and accept a worse posterior; when it is quiet you pay for full depth. Both the accuracy-per-exit and latency-per-exit curves are measurable, and their crossover decides whether the knob is worth having.

class SelfDistillingNet(nn.Module):
    """Network with early-exit classifiers for variable-latency inference."""

    def __init__(self, input_dim: int, n_classes: int = 3):
        super().__init__()
        self.block1 = nn.Sequential(
            nn.Linear(input_dim, 128), nn.ReLU(), nn.BatchNorm1d(128)
        )
        self.block2 = nn.Sequential(
            nn.Linear(128, 64), nn.ReLU(), nn.BatchNorm1d(64)
        )
        self.block3 = nn.Sequential(
            nn.Linear(64, 32), nn.ReLU(), nn.BatchNorm1d(32)
        )

        self.exit1 = nn.Linear(128, n_classes)
        self.exit2 = nn.Linear(64, n_classes)
        self.exit3 = nn.Linear(32, n_classes)  # final exit

    def forward(
        self, x: torch.Tensor, exit_layer: int = 3
    ) -> torch.Tensor:
        h1 = self.block1(x)
        if exit_layer == 1:
            return self.exit1(h1)

        h2 = self.block2(h1)
        if exit_layer == 2:
            return self.exit2(h2)

        h3 = self.block3(h2)
        return self.exit3(h3)

    def forward_all_exits(self, x: torch.Tensor):
        """Return logits from all exits (for self-distillation training)."""
        h1 = self.block1(x)
        h2 = self.block2(h1)
        h3 = self.block3(h2)
        return self.exit1(h1), self.exit2(h2), self.exit3(h3)


def self_distillation_step(
    model: SelfDistillingNet,
    x: torch.Tensor,
    y: torch.Tensor,
    temperature: float = 4.0,
    alpha: float = 0.5,
) -> torch.Tensor:
    """One training step with self-distillation from deepest exit."""
    logits_1, logits_2, logits_3 = model.forward_all_exits(x)

    loss_hard = F.cross_entropy(logits_3, y)

    loss_distill_1 = distillation_loss(
        logits_1, logits_3.detach(), y, temperature, alpha
    )
    loss_distill_2 = distillation_loss(
        logits_2, logits_3.detach(), y, temperature, alpha
    )

    return loss_hard + 0.5 * loss_distill_1 + 0.5 * loss_distill_2

Where the Inference Budget Comes From

Distillation only matters if inference has to sit inside a hard budget, and the full tick-to-trade ladder — NIC-to-userspace, kernel bypass, the sub-100 µs total, and the sub-10 µs tier that forces FPGA and shared memory — is already laid out in data and communication in algorithmic trading. The row that ladder leaves open is model inference, and that is the row distillation is trying to fill.

Resist filling the other rows with a model-class latency table. Spread modeling with machine learning already publishes the GBM-vs-deep-learning comparison plus the caveat that matters more than the numbers: latency is implementation-dependent, and the same LightGBM model takes tens of microseconds per row from Python but a few microseconds from a compiled predictor. Any latency claim here has to name framework, core, and batch size, or it is noise.

On GPUs specifically: fixed per-launch overhead has to be amortized before a device helps at all, and single-row inference sits far to the left of the roofline ridge where it never is. When the GPU pays off measures that amortization curve properly with a batch sweep, including how a discrete PCIe card pushes the ridge further right — read that rather than trusting a constant quoted from memory.

Quantization After Distillation

A distilled student compresses further: INT8 weights (roughly 2x on CPU with AVX-512 VNNI), binary/ternary weights that turn multiplies into adds, and pruning to skip near-zero computation.

The tempting claim is that distillation-then-quantization preserves more accuracy than quantization alone, since the student already learned a compact representation. Do not ship on it. The GPU precision trap is the blog's standing position on reduced numeric precision: it silently returned plausible-looking garbage, and what made the fast path shippable was a quantified equivalence gate — fills shifted, PnL delta in bps — not an assertion. An INT8 student is a different model until that gate is measured against the FP32 student.

import torch.quantization as quant

def quantize_student(student: nn.Module, calibration_loader: DataLoader):
    """Post-training static quantization for CPU deployment."""
    student.cpu()
    student.eval()
    student.qconfig = quant.get_default_qconfig("x86")

    student_prepared = quant.prepare(student)

    with torch.no_grad():
        for X, _ in calibration_loader:
            student_prepared(X)

    student_quantized = quant.convert(student_prepared)
    return student_quantized

FPGA Deployment: The Distill-to-Bitstream Pipeline

FPGAs are the sub-10 µs tier in the latency ladder, and the Tbricks/Broadridge review covers them in production alongside kernel-bypass NICs — deterministic latency, no OS jitter, co-located with the network stack. What is not covered anywhere on this blog is how a distilled model gets onto one.

DeepLOB's production notes list ONNX/TensorRT, INT8 quantization, and FPGA deployment as three options and stop there. This is what the third expands into:

1. Train ensemble teacher (offline, GPU cluster, hours/days)
       |
2. Distill to small MLP student (offline, single GPU, minutes)
       |
3. Quantize student to INT8 / fixed-point (offline, CPU)
       |
4. Convert to HLS (High-Level Synthesis) or RTL
       |
5. Synthesize FPGA bitstream (offline, hours)
       |
6. Deploy to FPGA card in production server
       |
7. Inference: market data -> FPGA -> trading signal

The binding constraint is that the model must fit in the device's logic elements — LUTs, DSP slices, block RAM. As an order-of-magnitude budget rather than a measurement: a 2-layer MLP with 64 hidden units and INT8 weights is on the order of 8,000 multiply-accumulates per inference and ~16 KB of weights, a small fraction of a mid-range part. This is where distillation earns its keep — the ensemble teacher does not fit at any budget; the student is nowhere near the limit.

Tools that automate PyTorch/ONNX to synthesizable hardware include AMD/Xilinx Vitis AI, hls4ml (from CERN), and FINN (from Xilinx Research).

Example: hls4ml Conversion

import hls4ml
import onnx

dummy_input = torch.randn(1, 60)  # 60 input features
torch.onnx.export(student, dummy_input, "student.onnx", opset_version=13)

hls_config = hls4ml.utils.config_from_onnx_model(
    onnx.load("student.onnx"),
    granularity="name",
    default_precision="ap_fixed<16,8>",
    default_reuse_factor=1,          # full parallelism
)

hls_model = hls4ml.converters.convert_from_onnx_model(
    "student.onnx",
    hls_config=hls_config,
    output_dir="hls_student",
    backend="VivadoAccelerator",
    board="alveo-u250",
)

hls_model.compile()
hls_model.build(csim=True, synth=True)

hls_model.report()

hls_model.report() is the only credible source of resource and latency numbers for a given model, board, precision, and reuse factor — the figures move substantially with default_reuse_factor alone. Quoting a "typical" synthesis table without running it is guessing.

Practical Considerations

Precomputing Teacher Logits

Distillation needs teacher predictions over the whole training set — a one-time offline cost worth paying deliberately: run the ensemble once, persist the logits, train students against the cache. Temperature sweeps and architecture searches then cost nothing extra in teacher forward passes, which is what makes the sweeps above practical at all.

The One Distillation-Specific Monitor

Feature-pipeline hygiene, rolling normalization because z-score parameters drift, input distribution-shift monitoring, and regime-triggered retraining are all covered in DeepLOB's production section and apply unchanged here.

The monitor specific to distillation is teacher-student KL divergence on live data. The teacher still exists offline; run it on a sample of live inputs and compare distributions. Rising KL means the student's approximation is degrading in regimes it was not distilled on — and it fires before accuracy does, because it does not wait for labels. The retraining threshold has to be calibrated against observed KL in known-good and known-degraded periods; picked a priori it is arbitrary.

When Not to Distill

• The teacher is already small (a linear model, a shallow GBM): distillation adds a pipeline stage for no compression.
• Latency is not a constraint (daily rebalancing, end-of-day signals): deploy the teacher.
• Interpretability outranks speed: a distilled network is harder to explain than the tree ensemble it replaced.
• The two-stage split already works: if the asynchronous slow model in the spread-modeling architecture is delivering, distillation has to beat it on a measured comparison before it justifies replacing a working system.

Summary

Distillation is a coherent alternative to the two-stage fast/slow split: train the best teacher you can afford offline, transfer its soft-target structure into a student small enough for the hot path, quantize, deploy on CPU or FPGA. The depth-wise variant goes further and makes latency a runtime choice rather than a training-time one.

What this article deliberately does not claim is that any of it beats what the blog already publishes. That verdict needs three measurements on real order-book data: the student-vs-ensemble weighted F1 retention curve split by regime, the temperature sweep, and an INT8 parity gate in the style of the GPU precision trap. Until those exist, this is a description of a technique, not a recommendation to deploy it.

References

1. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531

2. Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018). Born-Again Neural Networks. ICML. arXiv:1805.04770

3. Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019). Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. ICCV. arXiv:1905.08094

4. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. ICLR. arXiv:1412.6550

5. Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge Distillation: A Survey. International Journal of Computer Vision, 129, 1789-1819. arXiv:2006.05525

6. Duarte, J., et al. (2018). Fast Inference of Deep Neural Networks in FPGAs for Particle Physics (hls4ml). Journal of Instrumentation, 13, P07027. arXiv:1804.06913

7. Umuroglu, Y., et al. (2017). FINN: A Framework for Fast, Scalable Binarized Neural Network Inference. FPGA '17. arXiv:1612.07119

8. Zhang, Z., Zohren, S., & Roberts, S. (2019). DeepLOB: Deep Convolutional Neural Networks for Limit Order Books. IEEE Transactions on Signal Processing, 67(11), 3001-3012. arXiv:1808.03668