The Framework Tax: When Your Backtest Library Is Slower Than a Naive Pandas Loop

Part of the "Backtests Without Illusions" series.

There is a comfortable assumption at the bottom of most algo-trading projects: that a mature, well-starred backtesting library is fast. It has years of contributions, a real event loop, a broker model, a commission scheme. Surely it beats the hacky pandas loop you would have written yourself. So you reach for it, wire up your strategy, and start a parameter sweep — a few thousand configurations, an overnight job. You come back in the morning and it is still running.

We benchmarked eight backtest engines on one identical parameter sweep and found something that should change how you pick a tool for search. Two of the most popular open-source event-driven frameworks — backtrader and bt — ran the sweep slower than a naive pandas loop we wrote as the throwaway baseline. Not a little slower. backtrader took roughly 2.5× as long as the pandas baseline; bt, about 4.7×. Meanwhile a vectorized/compiled engine on the same work ran about 13,000× faster than bt.

This is the framework tax. Popular backtesters are built for one honest run — one strategy, one dataset, careful fills, a broker that behaves like a broker. That is exactly what you want for a final validation or a live-parity check. It is exactly the wrong tool for the thing algo research actually spends its time on: running the same strategy ten thousand times with slightly different knobs. This article measures the tax, explains the mechanism, and gives a decision rule for when a "real" backtesting framework is the wrong choice.

Every number here comes from one reproducible harness (benchmarks/bench_oss_engines.py, commit 250dbb5) on an identical, parity-locked workload. Where we did not run an engine ourselves, we say so and put it in a separate honesty section rather than inventing a figure.

One run versus ten thousand

The defining fact of parameter search is that the engine runs thousands of times but the analysis happens once. Whatever fixed cost you pay to set up a backtest — build the event loop, instantiate a broker, allocate an object per bar — you pay it on every single combination. A cost that is invisible in one run ("who cares about 6 seconds?") becomes the entire bill across a sweep ("6 seconds × 10,000 = 16.6 hours").

Backtest engines fall into three paradigms, and the paradigm is destiny for sweep performance:

1. Event-driven — the engine walks bar by bar, emitting events, calling your next()/onBars() callback, routing orders through a broker object. This is the architecture of backtrader, backtesting.py, PyAlgoTrade, zipline, and nautilus_trader. It mirrors how live trading actually works, which is precisely why it is trusted for realism — and precisely why it is slow: the per-bar Python overhead is paid 150,000 times per combo.
2. Weights/rebalance — bt sits here. You hand it a target-weight matrix and it rebalances on the dates you specify. No per-bar callback, but still a per-event object graph (a tree of algos, a transaction ledger) evaluated in Python.
3. Vectorized / compiled — the whole strategy is expressed as array operations (vectorbt), a JIT-compiled kernel (numba), or native code (Rust, an MLX GPU kernel). There is no per-bar Python at all. The loop, if there is one, runs at machine speed.

The rest of this article is the empirical consequence of that taxonomy. We built one workload, made every engine do provably the same work, and timed it.

The workload: one strategy, eighty knobs, identical for everyone

A benchmark is only honest if every engine is doing the same job. Ours is deliberately plain — a workload where the only thing that differs between engines is the engine itself.

• Data. A single synthetic geometric-Brownian-motion close series: 150,000 bars, seed=42, per-bar volatility sigma=0.0008, x0=30000. Deterministic, so anyone can reproduce it bit-for-bit. Close-only by construction — OHLC is set to the close on every leg, because the strategy is a close-cross.
• Strategy. A Hull Moving Average cross: an HMA of length L against a faster HMA-of-a-third variant (HMA vs HMA3). Always in the market, flip long/short on each cross. This is a real, non-trivial indicator — two nested weighted moving averages plus a square-root-window smoother — not a toy SMA, so the per-bar work is representative.
• The sweep. 80 HMA lengths spanning 6..200. That is the "ten thousand runs" made small enough to measure directly: 80 independent combos, each a full backtest over 150k bars.
• Costs. A round-trip fee of 0.09%, split per side for engines that model side commissions. Same-bar fill at close[i] — the signal on bar i is executed at that bar's close, the convention our production engine uses.

The per-combo timer wraps exactly two things: the numpy HMA precompute and the engine run. Setup that is genuinely one-time (loading data, building the bar objects) sits outside the timer. There is one warmup run, then best-of-N repeats, and — because the event-driven engines are slow enough that 80 full combos would take many minutes to over an hour — we time a uniform sample of the grid and extrapolate linearly. Combos are independent, so linear extrapolation is exact in expectation; the same convention is applied to the pandas baseline, so no engine is advantaged by it.

Parity: proving everyone does the same work

Here is the trap a naive engine comparison falls into: a "fast" engine might just be doing less. If backtrader books 2,700 trades and your vectorized engine books 40, the vectorized engine isn't faster — it's wrong, and the comparison is meaningless.

So we lock the comparison with a trade-count parity check. At L=104, the numpy reference produces exactly 2,707 closed trades. Every engine must reproduce that within a tolerance of ±1, or the run aborts with a work-parity FAILED assertion. The tolerance exists only because engines disagree on bookkeeping conventions — whether the final open position is force-closed and counted, whether the initial entry is a "trade" — not on the trades themselves:

Every engine lands on 2707 ± 1. Whatever the speed differences turn out to be, they are not an artifact of one engine quietly skipping work. This is the discipline that lets us put an event-driven framework and a GPU kernel in the same table and mean it.

The results

Here is the whole table, sorted fastest to slowest. combos/s is the throughput; the last column is how long the full 80-combo sweep takes. The baseline row is M0 — the naive pandas engine, a for loop over bars with scalar bookkeeping, the thing you'd write in an afternoon and throw away. Everything slower than that baseline is in bold.

Read the table top to bottom and the paradigm sorts itself: the top five are all vectorized or compiled, the bottom five are all event-driven or object-graph based — with the naive pandas loop sitting above two mature, popular frameworks. The spread from top to bottom is four orders of magnitude. On the exact same 2,707-trade workload, the MLX kernel finishes the sweep in a tenth of a second; bt needs twenty-two and a half minutes. That is a factor of roughly 13,000×.

The scandal in the middle of the table

The eye-catching numbers are at the extremes, but the instructive result is in the middle: backtrader (0.11 combos/s) and bt (0.06 combos/s) are both slower than the naive pandas baseline (0.28 combos/s).

This deserves to sink in. M0 is not a clever engine. It is a Python for loop that indexes into a DataFrame, tracks a position and cash in plain scalars, and appends trades to a list — the deliberately unoptimized "control" we included to have something obviously bad to beat. pandas' per-row access is notoriously slow, and we leaned into it. And yet two of the most-recommended backtesting libraries in the ecosystem lose to it: backtrader by 2.5×, bt by 4.7×.

The nuance that keeps this honest: not every event-driven engine is slower than pandas. backtesting.py (1.42 combos/s) beats the baseline by 5×, because it is a lean, numpy-backed event loop that keeps per-bar object creation to a minimum. PyAlgoTrade (0.51) edges ahead of the baseline too. So "event-driven" is not automatically a death sentence — but the heavier the per-bar machinery, the worse it gets, and backtrader and bt carry the heaviest machinery here. The paradigm sets the ceiling; the implementation decides where under that ceiling you land.

The point is not that these are bad libraries. backtrader's broker model and bt's tree-of-algos design exist to buy you correctness and expressiveness — realistic order handling, portfolio rebalancing, analyzers. Those features have a runtime cost, and that cost is invisible when you run once. Across a sweep it is the whole story.

Why event-driven engines pay the tax

The mechanism is not mysterious. An event-driven backtest, per bar, does something like this:

1. Advance the clock, slice the next bar off the data feed, and materialize it as an object (a Bar, a Line, a dict).
2. Fire a callback into user code (next(), onBars()), which is a Python function call with its own frame.
3. Inside the callback, query the broker/position state, again through method calls and attribute lookups.
4. If an order is created, route it through the broker: validate it, check margin/cash, schedule a fill, mutate a portfolio object, append to a transaction ledger.
5. Update analyzers, observers, and any bookkeeping the framework maintains.

Now multiply by 150,000 bars, then by 80 combos: twelve million bar-iterations per sweep, each one a fistful of Python-level object allocations and dynamic dispatches. Python's per-operation overhead — dozens to hundreds of nanoseconds for an attribute lookup or a small allocation — is trivial once and ruinous twelve million times. The bt case is a variant of the same disease: even though it rebalances only on trade dates rather than every bar, each rebalance evaluates a tree of algo objects and touches a pandas-backed portfolio ledger, and there are 2,707 of those per combo, times 80.

The naive pandas loop beats backtrader and bt for a blunt reason: it does less per bar. It skips the broker, the event objects, the analyzer stack, the order-routing state machine. It pays pandas' ugly per-row tax, but that single ugly tax is still cheaper than the framework's tidy, feature-complete, object-per-event tax. When you strip a backtest down to "position × next-return − fees," most of what a framework does per bar is overhead you are not using during a search.

And this is the trap: the overhead is the reason you chose the framework. You wanted the realistic broker. You wanted the analyzers. During final validation, you want all of it. During a 10,000-combo search where you only read one scalar objective out the other end, you are paying for a limousine to run laps.

The other end of the table: vectorized and compiled

The top of the table is what happens when you delete the per-bar Python entirely.

• vectorbt (56.9 combos/s) expresses the whole strategy as numpy/numba array operations. There is no bar loop in Python — the signal, the position, the PnL are all array-level. It runs the sweep in 1.4 seconds versus bt's 22.5 minutes: about 950× faster on identical work. (We cover vectorbt's design in more depth in the vectorbt overview and the broader pandas-vs-polars comparison.)
• numba (39.7) is the pandas loop, unchanged in shape, JIT-compiled to native code. Same algorithm as M0, but @njit turns 0.28 combos/s into 39.7 — a ~140× speedup from a decorator, because the interpreter overhead that dominated the scalar loop simply evaporates.
• mp + numba (246) runs the compiled kernel across CPU cores. Combos are embarrassingly parallel — each is independent — so multiprocessing scales nearly linearly on top of the JIT.
• native Rust (~350) removes the last of the Python glue: the whole sweep is native.
• MLX GPU (779) maps the sweep onto an Apple-silicon GPU kernel. 80 combos become 80 parallel lanes of arithmetic; the sweep finishes before you release the enter key.

Two things are worth naming precisely. First, numba proves the paradigm matters more than the language. M0 and numba run the same algorithm — the difference is purely that one is interpreted per-bar Python and the other is compiled. That is the entire framework tax in a controlled A/B: ~140× for deleting the interpreter from the inner loop. Second, the jump from numba (39.7) to mp+numba (246) to MLX (779) is no longer about the engine at all — it is about orchestration and hardware. Once the per-bar tax is gone, speed becomes a question of how many combos you run in parallel and on what silicon. We walk that full progression in the backtest engine speed ladder, and the reason the last mile is dominated by process/serialization cost in the IPC-tax article.

What we did not measure (and why we are telling you)

A benchmark's credibility lives in what it refuses to fake. We ran eight engines end-to-end under parity. Several well-known frameworks we did not put a number on, and we would rather list them honestly than extrapolate a figure we did not measure:

• zipline / zipline-reloaded — event-driven, Quantopian lineage. Heavy setup (a full trading calendar and data bundle), which makes an apples-to-apples per-combo timing fiddly. Architecturally it sits with backtrader in the event-driven camp; we would expect it near that end of the table, but we have not proven it.
• nautilus_trader — event-driven with a Rust/Cython core, explicitly designed for live-parity. Its core is compiled, so it is the most likely event-driven engine to not pay the full Python tax — a genuinely interesting measurement we have not yet run.
• QuantConnect Lean — C#-based, a different runtime entirely; not directly comparable in a Python harness.
• Jesse — event-driven, crypto-focused; we reviewed its design in a separate note but did not benchmark it here.
• QSTrader — event-driven, portfolio-oriented; same paradigm caveats.
• fastquant — we tried; the install/API was broken in our environment, so there is no number. We are not going to guess one.

Two honest caveats about the numbers we do report. The vectorbt, numba, mp+numba, native-Rust and MLX figures come from our own engine ladder on the identical workload, not from the OSS harness that produced the four event-driven rows — they are the same workload but a different measurement rig, and the native-Rust figure is an approximate ~350, not a tight measurement. And absolute combos/s is hardware-specific; what travels is the ordering and the ratios, which are large enough (13,000× top-to-bottom, 2.5–4.7× for the pandas-vs-framework inversion) that no reasonable hardware difference flips them.

In defense of event-driven engines

It would be easy to read this as "event-driven backtesters are bad." That is the wrong lesson, and an unfair one.

Event-driven engines are built for a different job, and they are good at it. The per-bar broker, the order lifecycle, the fill logic, the analyzers — that machinery exists to make a backtest resemble live trading as closely as possible. When your goal is a single, trustworthy dress rehearsal of a strategy you are about to deploy, you want the engine to sweat every fill, model partial executions, respect margin, and refuse to let you trade at prices you could not have gotten. That fidelity is the product. Its runtime cost is the price of realism, and for one run the price is negligible.

The failure is not the engine; it is using it for the wrong phase. Research has two distinct phases with opposite requirements:

• Search wants throughput. You are exploring a landscape, most of which is garbage, and you need to evaluate thousands of points to find the few worth a second look. Fidelity per point barely matters — you are ranking, not deploying. Here the framework tax is pure waste.
• Validation wants fidelity. You have a handful of candidates and you need to know, as precisely as possible, whether they survive realistic execution, fees, slippage, and the look-ahead traps that inflate paper returns. Here the event-driven engine earns its cost.

The mistake the framework tax punishes is running your search on your validation engine — paying limousine rates to explore a landscape you are going to throw 99% of away.

The decision rule

The practical takeaway compresses to a single decision:

Search on a vectorized/compiled engine. Validate the survivors on an event-driven one.

Concretely:

1. Build or borrow a fast kernel for the sweep. vectorbt if you want it off the shelf; a numba-compiled loop if your strategy does not vectorize cleanly (the @njit decorator alone bought ~140× here). Run the full parameter space through it.
2. Never run a large sweep on backtrader, bt, zipline, or any heavy event-driven framework. If a sweep on one of those is your bottleneck, the fix is not a bigger machine — it is the wrong engine. Even the naive pandas loop would beat two of them.
3. Promote a short-list to the event-driven engine for a fidelity check. Take the handful of survivors and re-run them on the realistic engine, where the broker model and fill logic can expose problems the fast kernel abstracted away.
4. Enforce parity between the two. The fast engine and the fidelity engine must agree on trade counts and PnL on a fixed configuration (our ±1-trade check at L=104), or the search and the validation are measuring different strategies and the whole pipeline is a lie.

This is the same two-speed architecture that shows up whenever search and validation have opposite cost profiles, and it is why our own stack keeps a fast vectorized/compiled path for the parameter search and reserves the heavy machinery for the final objective evaluation and plateau checks.

Takeaways

1. Popular ≠ fast for sweeps. On one identical, parity-locked workload (150k bars, 80 HMA-cross combos, 2,707 trades), backtrader (0.11 combos/s) and bt (0.06) both ran slower than a naive pandas loop (0.28). A mature, well-starred framework is not automatically the fast choice.
2. The framework tax is per-bar, and a sweep multiplies it. Twelve million bar-iterations per sweep, each carrying an event object, a callback, and a broker round-trip. A cost that is invisible in one run is the entire bill across ten thousand.
3. Paradigm sets the ceiling. Vectorized/compiled engines (vectorbt 56.9, numba 39.7, mp+numba 246, native Rust ~350, MLX 779) beat the event-driven engines by two to four orders of magnitude — up to ~13,000× top-to-bottom. The same algorithm, merely JIT-compiled, went 140× faster.
4. Not all event-driven engines are equal. backtesting.py (1.42) and PyAlgoTrade (0.51) still beat the naive baseline; the tax scales with how heavy the per-bar machinery is. Implementation decides where under the ceiling you land.
5. Two engines, two phases. Search on a vectorized/compiled kernel; validate the survivors on the realistic event-driven engine. Enforce trade-count/PnL parity between them so both are measuring the same strategy.
6. Be honest about what you measured. We benchmarked eight engines under parity and listed the ones we did not (zipline, nautilus_trader, Lean, Jesse, QSTrader, and the fastquant install that would not run) rather than invent numbers for them.

The uncomfortable summary: if a parameter sweep is your bottleneck, the problem is probably not your machine and not your strategy. It is that you are running a search on an engine that was built for a single honest run — and you would be faster with the pandas loop you were too embarrassed to keep.

The full harness, the parity assertions, and the raw per-engine JSON results live in benchmarks/bench_oss_engines.py and benchmarks/results_oss/ at commit 250dbb5. For the compiled/GPU end of the ladder, see the backtest engine speed ladder and the IPC-tax analysis.