The “Quadratic Tax”: Why Financial Foundation Models Struggle to Scale

The context hunger of financial data

In the world of Natural Language Processing (NLP), a context window of 8k or 32k tokens is often plenty. But in High-Frequency Trading (HFT) and Capital Markets, we are drowning in data. With the NYSE now processing over 1.2 trillion messages daily, the “context hunger” for Time Series Foundation Models (TSFMs) is unprecedented.

To capture a regime shift or a volatility spike, a model might need to “look back” at 64,000+ ticks.

The bottleneck: the O(n²) wall

The standard Transformer architecture relies on Dense Attention. Mathematically, this is a quadratic operation: O(n²).

This means that if you double your input sequence, your computational cost and memory requirements don’t just double—they quadruple. For financial institutions, this “Quadratic Tax” creates a hard ceiling on model performance and a massive spike in cloud/GPU infrastructure costs.

Moving toward near-linear complexity

To solve the 1.2 trillion message problem, we have to move past O(n²). In my current research and development of the ma-transformer project, I’ve been focusing on Sub-Quadratic Attention through hardware-aware sparsity.

By utilizing custom CUDA kernels, we can implement patterns that provide the “long memory” finance requires without the quadratic penalty:

• Sliding Window Attention: Capturing local micro-structures and high-frequency patterns with O(n·w) complexity.
• Dilated (Strided) Attention: Allowing the model to “skip” across time to find macro-trends and long-term dependencies.
• Hardware-Aware Sparsity: This isn’t just a mathematical trick. It requires deep-level optimization in CUDA—focusing on memory coalescing and fused kernels—to ensure that the theoretical “Near-Linear” scaling actually translates to real-world throughput on NVIDIA Hopper and Blackwell architectures.

The bottom line for capital markets

The goal isn’t just to make models “faster.” The goal is to make them economically scalable.

When we move from O(n²) to near-linear complexity, we allow firms to process more data, with higher precision, at a fraction of the hardware footprint. In an industry where latency is measured in microseconds and efficiency is measured in basis points, breaking the quadratic wall is no longer optional.

I am currently documenting these benchmarks and open-sourcing the reference kernels in my project, ma-transformer. If your team is hitting a scaling wall with long-sequence financial data, I’d love to trade notes.