Does a Full-Day Context Beat a Ten-Minute One? Flash Attention and the Sequence-Length Question

Here is the question this article exists to answer: if a transformer could attend to an entire trading day at one-second resolution instead of a ten-minute window, would it predict better?

Until recently you could not even ask. Standard attention needs $\mathcal{O}(N^2)$ memory, so a 23,400-step day at 12 heads in float16 wants roughly 12.9 GB for the score matrix alone — more than the model parameters, and more than most cards will give you. The question was closed by arithmetic before anyone got to test it.

Flash Attention (Dao et al., 2022) opens it. Not by approximating attention — it computes the exact same result — but by restructuring the computation to be IO-aware, minimizing traffic between GPU memory levels. That is the genuinely interesting content here, and most of this article is spent on how it works: tiling, the online-softmax recurrence, the $\Theta(N^2 d^2 / M)$ IO bound, and backward-pass recomputation.

But the mechanism is the enabler, not the claim. "Longer context is better" is an empirical statement about markets, and this blog's standing position — from Temporal Fusion Transformers, which found that vanilla transformers on financial series overfit and short-lookback recurrent models stay competitive at high frequency — cuts the other way. So the article closes on the measurement, not the mechanism.

Why attention is memory-bound

Attention computes $\text{softmax}(\mathbf{Q}\mathbf{K}^\top / \sqrt{d_k})\mathbf{V}$ — the primitive itself, in a trading context, is covered in Temporal Fusion Transformers for Multi-Horizon Portfolio Forecasting. The whole problem is one line of that: the intermediate score matrix $\mathbf{S} = \mathbf{Q}\mathbf{K}^\top/\sqrt{d_k}$ is $N \times N$, it is written to memory, read back for the softmax, written again, and read again for the final matmul — and it must be kept for backpropagation.

Attention's arithmetic intensity is $\approx \min(d, N)$, so about 64 FLOP/byte at $d = 64$ — well left of the A100 ridge point. It sits on the sloped bandwidth ceiling, not the flat compute ceiling: the GPU spends more time moving $\mathbf{S}$ around than multiplying anything. The roofline framework this uses — ridge point, sloped versus flat ceiling, and why the same reasoning decides whether a GPU is worth buying at all — is built with measured numbers in When the GPU Pays Off.

The memory hierarchy the algorithm exploits

SRAM is roughly 10x the bandwidth and 100x lower latency, at a thousandth of the capacity. Everything Flash Attention does follows from that trade: give up capacity, buy bandwidth and latency. The same "restructure the algorithm rather than buy hardware" move, measured on a CPU backtest, is the backtest speed ladder.

The Flash Attention algorithm

Flash Attention processes attention in tiles sized to fit in SRAM, and never materializes the full $N \times N$ matrix in HBM at all.

Partition $\mathbf{Q}$ into $T_r = \lceil N/B_r \rceil$ row blocks and $\mathbf{K}, \mathbf{V}$ into $T_c = \lceil N/B_c \rceil$ column blocks, with $B_r, B_c$ chosen so a tile plus its accumulators fit on chip. For each query block, iterate over all key-value blocks:

For each query block Q_i:
    Initialize: O_i = 0, m_i = -inf, l_i = 0    # output, running max, running sum
    For each KV block (K_j, V_j):
        1. Load Q_i, K_j, V_j from HBM to SRAM
        2. Compute S_ij = Q_i @ K_j^T / sqrt(d)  # in SRAM
        3. Compute local max: m_ij = rowmax(S_ij)
        4. Compute P_ij = exp(S_ij - m_ij)        # in SRAM
        5. Compute local sum: l_ij = rowsum(P_ij)
        6. Update running statistics:
           m_new = max(m_i, m_ij)
           l_new = l_i * exp(m_i - m_new) + l_ij * exp(m_ij - m_new)
           O_i = O_i * (l_i * exp(m_i - m_new) / l_new)
                + P_ij @ V_j * (exp(m_ij - m_new) / l_new)
           m_i = m_new, l_i = l_new
    Write O_i to HBM

The online softmax recurrence

The trick that makes tiling possible is online softmax. A naive softmax needs two passes over the row: one to find the max (for numerical stability), one to exponentiate and normalize. Two passes over a row you refuse to store is a contradiction — so Flash Attention keeps running statistics and rescales as it goes.

After blocks $1, \ldots, j$:

and the output accumulator is corrected by the same factor:

Every time a new block raises the running max, previously accumulated output is retroactively rescaled by $e^{m^{(j-1)} - m^{(j)}}$ — as if the new max had been known from the start. The result is algebraically identical to the two-pass softmax. In exact arithmetic this is not an approximation; it is a reassociation. (In finite precision it is a different rounding path, which matters — see the exactness check below.)

IO complexity

This is the formal statement of the win. Flash Attention performs

HBM accesses, where $M$ is SRAM size, against $\Theta(Nd + N^2)$ for the standard implementation. Note that $M$ appears in the denominator: the bigger the on-chip scratchpad, the fewer round-trips, which is why the algorithm is stated in terms of the memory hierarchy rather than the FLOP count. For typical $d = 64$ and $M \approx 100$ KB, the ratio favors Flash Attention by roughly 5-10x fewer accesses.

Backward pass: recompute instead of store

Backpropagation through attention normally needs the $\mathbf{P}$ matrix that the forward pass just refused to keep. Flash Attention recomputes the tiles from $\mathbf{Q}, \mathbf{K}$ during the backward pass, storing only the output $\mathbf{O}$ and the softmax statistics $(m, \ell)$ — both $\mathcal{O}(N)$, not $\mathcal{O}(N^2)$. It trades a modest amount of redundant arithmetic for the memory term that was the entire problem. This is the same bargain as gradient checkpointing, applied at tile granularity inside a single operator.

FA2: parallelism

Flash Attention 2 (Dao, 2023) kept the algorithm and fixed the scheduling:

1. Fewer non-matmul FLOPs. FA1 spent real time on rescaling, max-finding, and exponentiation — operations that run on CUDA cores, not tensor cores. FA2 defers rescaling to the end of the inner loop.
2. Parallelism over sequence length. FA1 parallelizes over batch and heads only. FA2 also parallelizes over query blocks. This matters specifically for the trading case, where you often have one very long sequence per asset and a batch size of 1-4 — exactly the regime where batch-and-head parallelism starves the GPU.
3. Warp work partitioning. Each warp takes a different subset of query blocks rather than splitting a score computation and reducing across warps, removing a cross-warp reduction.

Reported result: ~70% of theoretical peak FLOPs on A100 versus ~35% for FA1.

FA3: Hopper mechanics

Flash Attention 3 (Dao, Shah, 2024) is architecture-specific to H100:

1. Asynchronous warp specialization. Hopper's Tensor Memory Accelerator (TMA) moves HBM→SRAM asynchronously. FA3 splits warps into producers issuing TMA loads for the next KV block and consumers computing on the current one, so data movement hides behind arithmetic.
2. Interleaved matmul and softmax. $\mathbf{Q}_i\mathbf{K}_j^\top$ for one block runs on tensor cores while the softmax for the previous block runs on CUDA cores — two different hardware units, genuinely concurrent rather than time-sliced.
3. FP8 with incoherent processing. H100 does FP8 at 2x FP16 throughput. Naive FP8 attention is wrecked by outliers; FA3 randomly rotates vectors before block-wise quantization to spread outlier magnitude across coordinates, reported at 2.6x lower numerical error than naive FP8.

Causal masking is where trading gets a discount

Causal masking is mandatory for time series — the model must not attend to the future — and under tiling it is not an added cost but a saved one. Any tile whose keys are entirely in the future relative to its queries is skipped outright, never loaded and never computed, cutting roughly half the work. In PyTorch this is is_causal=True; nothing else is required.

The integration is eight lines

Almost none of the code you need is about Flash Attention. Swap the explicit score-matrix path for the fused kernel:

scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(d_head)
scores = scores.masked_fill(causal_mask, float("-inf"))
out = torch.matmul(torch.softmax(scores, dim=-1), v)

out = torch.nn.functional.scaled_dot_product_attention(
    q, k, v,
    dropout_p=self.dropout if self.training else 0.0,
    is_causal=True,
)

That is the whole change. q, k, v are shaped (batch, heads, seq, head_dim); the causal mask is gone because the kernel builds it. For a complete annotated PyTorch trading model to drop this into — input projection, blocks, and a 3-class up/flat/down head — use DeepLOB, and for a full training pipeline see Temporal Fusion Transformers. Building a fourth copy of that scaffolding here would teach nothing.

Requirements: compute capability >= 8.0 (A100, H100, RTX 3090+), half-precision inputs, PyTorch >= 2.0. Verify the fast path actually engaged with torch.backends.cuda.sdp_kernel diagnostics and torch.cuda.max_memory_allocated() — SDPA silently falls back to the math kernel if any precondition fails, and a silent fallback looks exactly like a working model that is merely slow.

The measurement: does the longer context pay?

Everything above says a 32K or 128K context is now affordable. It says nothing about whether it is useful. The honest experiment:

Train the same architecture at $N \in \{512, 4096, 32768\}$ with SDPA-Flash on the BTC series used elsewhere in this series, holding parameters, optimizer, and target fixed so sequence length is the only variable. Report two things:

1. Cost. Measured wall-clock per epoch and torch.cuda.max_memory_allocated() at each $N$.
2. Benefit. Out-of-sample predictive performance versus $N$, on a walk-forward split.

An earlier draft of this article carried a table of memory figures per sequence length derived analytically from the $\mathcal{O}(N)$ activation formula. Those rows are removed: they were never measured, and they disagreed with the article's own memory-budget arithmetic. A derived number presented in a results table is a fabricated result, and this blog does not ship those.

The interesting property of this experiment is that it is publishable in either direction. If out-of-sample performance rises monotonically with $N$, that justifies the whole long-context program. If it plateaus at a few thousand steps, or degrades, that is a stronger piece — a companion to the honest negative — and it would mean the memory wall was never the binding constraint on trading transformers.

More context is more capacity, therefore more overfitting surface

There is a specific reason to expect the flat or negative outcome. Temporal Fusion Transformers already documents that vanilla transformers applied naively to financial series overfit — they lack temporal inductive biases, and short-lookback recurrent models stay competitive at high frequency. Extending the context from 512 to 32,768 steps does not add information proportional to the length; the marginal 32,000th lag of a near-efficient price series carries very little. What it reliably adds is parameters' worth of things to fit.

So the sweep over $N$ must be treated as what it is: a model-selection search, with the same machinery this blog applies to every other search. Three sequence lengths times whatever else varies is a trial count, and the winner must clear a Deflated Sharpe Ratio computed against that trial count and a PBO gate, not merely beat its neighbors. Otherwise "long context wins" is indistinguishable from picking the best of three noisy runs.

An exactness check, because "exact" is doing a lot of work

Flash Attention is exact in exact arithmetic. The recommendation attached to it — run in fp16 or bf16, and on H100 consider FP8 — is not. Those are separate claims and the second one dominates in practice: reassociating a sum and dropping to half precision are both perturbations, and the article that introduced the ordering guarantee should not then hand-wave the precision one.

The blog already has the right instrument. The GPU Precision Trap establishes the standard: low precision does not warn you, it returns plausible garbage, and you prove correctness with a parity oracle on a downstream discrete quantity — trade counts — not by eyeballing curves. Applied here:

• Compute attention with SDPA-Flash in bf16 and with an fp64 reference implementation on identical inputs; report max relative error on the output tensor.
• Push it through to the decision: for a model that emits an up/flat/down label, report how many labels flip between the two paths, as a fraction of total decisions.

Small, bounded, explainable disagreement is the signature of a correct fast path. An unbounded one means the FP8 recommendation was never safe for this model. Neither number is known until it is run.

When to reach for it

Compressed to the decision, which has the same shape as the GPU decision guide:

• Above ~2K timesteps on a CUDA GPU: yes, unconditionally. It is a one-line change producing exact output, and the win grows with $N$. There is no scenario where you want the materialized $N \times N$ path instead.
• Below ~512 timesteps, on CPU, or with non-attention architectures (CNNs, SSMs like Mamba): irrelevant. Left of the ridge, the fixed overhead is the whole cost and attention was never your bottleneck.
• The thresholds above are folklore, not measurement — they come from the general literature, and the crossover on your own model and card is a ten-line benchmark. Run it rather than trusting the round numbers.

Conclusion

Flash Attention is a clean and genuinely important result: by respecting the memory hierarchy and reassociating the softmax, it computes exact attention with $\mathcal{O}(N)$ memory instead of $\mathcal{O}(N^2)$, and the $\Theta(N^2 d^2 / M)$ IO bound explains precisely why. Adopting it in a trading transformer is a one-line change with no accuracy cost in exact arithmetic and a large memory win.

What it does not do is answer the question at the top. It converts "a full-day context is impossible" into "a full-day context is cheap," which is a change in the cost of the experiment, not its result. The memory wall coming down is an invitation to measure, and the measurement is what turns this from a paper summary into a finding.

References

1. Dao, T., Fu, D.Y., Ermon, S., Rudra, A., Re, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS (2022). arXiv:2205.14135
2. Dao, T. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." ICLR (2024). arXiv:2307.08691
3. Dao, T., Shah, J. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision." NeurIPS (2024). arXiv:2407.08608
4. Vaswani, A., et al. "Attention Is All You Need." NeurIPS (2017).
5. Milakov, M., Gimelshein, N. "Online normalizer calculation for softmax." arXiv:1805.02867 (2018).