September 2026

Truth In 30 Million Token Context Windows

Patrick S. McConlogue · Craig B. Weisman

Abstract

In English. Everybody else bought a bigger window and then hoped the fact was sitting in it. Hope is not a plan. We built a detector. If the needle is in the thirty million tokens, we find it and we can prove it. If it is not in there, we say so — on purpose — instead of making something up and calling it an answer. The other systems scored about zero on the needles. We scored one. That is not a rounding error. That is the whole sport. Why? Because guessing is what you do when you do not have a receipt. We brought a receipt. The rest of this section is the adult version.

Technical abstract. Large language models fail on long-context retrieval: needle-in-a-haystack misses, and hallucination when the fact is not in the window. SUPERCTX is a retrieval architecture, not a larger attention window. It combines an S4 spine (HiPPO-LegS), a MemoryKernel, and a TurboQuant HiPPO bank. The active window is 30 million tokens with linear scaling. On a 50,811-entry synthetic corpus the query face narrows candidates by a median 44.8×. Needle retrieval on a 30M-token corpus is 1.000 (n=300) against vanilla RAG families at 0.000–0.003. The system can emit a certified negative: a recomputation-checkable claim that the indexed memory does not contain the fact. That claim is the product, not a threshold on a similarity score.

1. Introduction

Transformer attention is quadratic in sequence length. RAG retrieves snippets and then hopes the generator stays inside them. Neither path certifies presence or absence. SUPERCTX stores history in a HiPPO bank and answers with a certificate. Every claim that a fact is in the bank, or is not, is checkable by recomputation. The numbers below are from the plannertuner measurements, not from a vendor context-window slide.

2. SUPERCTX Architecture

Three parts: S4 multiscale recurrence (HiPPO-LegS), MemoryKernel with SegAttaches, TurboQuant HiPPO bank. The 9B-class decoder conditions on a short fused prefix; the 30M-token history lives in the bank, not in the KV cache.

2.1 S4 multiscale recurrence (HiPPO-LegS)

The S4 spine is the continuous memory lane. HiPPO-LegS projects the input onto a polynomial basis so the recurrence is a legal SSM, linear in length, without washout at the horizons we measure. Symbols do not enter this lane.

2.2 MemoryKernel and SegAttaches

MemoryKernel (thor_train/memory_kernel.py) is the attach API after the recurrence:

• HiPPO SegAttach — continuous S4 states.
• Titans SegAttach — trainable neural memory, O(1) per token at the memory module.
• Symbol SegAttach — symbolic side channel, post-recurrence only.

2.3 TurboQuant HiPPO bank

TurboQuant (thor_train/turboquant_codec.py) stores HiPPO states with Hadamard rotation and Lloyd-Max scalar quantization. Measured reconstruction: 5.31e-09 MSE at 4-bit. At 30M tokens the 4-bit bank is 7.5 MB.

2.4 Four architectural bounds

1. NTM/DNC/LongMem/Titans: external trainable memory, not KV-cache scaling.
2. HiPPO theory: the continuous lane stays a legal SSM recurrence.
3. TurboQuant codec: Hadamard + Lloyd-Max. Valid modes: none | turboquant | supersniff.
4. Symbols attach post-recurrence. Never as HiPPO input.

3. Measured Results

3.1 Corpus and task

v6 sealed corpus, 188 MB across 10 shards, 50,811 entries (5,000 fictional companies, 2,100 instruments, 20,000 trades, CEO remarks). Thirty tasks V01–V30. Serve table: scientistking/predbank_v6/serve_table_merged_v6.db.

3.2 Candidate-set narrowing

• Median narrowing: 44.8× (median 1,133 entries per task).
• Zero-narrowing: 10 of 30 tasks. No extractable handle, full-corpus open.
• Reachable entries: 5,684 / 50,811 (11.2%).

The 205× narrowing figure is a clause-join property of this synthetic corpus and does not survive any r < 1 as a serve-narrowing claim. The serve-time filter built to realize it measured as a no-op.

3.3 Comparison to RAG baselines

Head-to-head on a 30M-token corpus, identical queries, four vanilla RAG families (mean-pool, SIF, BM25, token-MaxSim):

The difference is detection with a certificate. SUPERCTX can emit the empty set with a guarantee. RAG cannot.

3.4 Position against published long-context systems

This section uses arithmetic from a sourced model config and published claims that resolve to the stated arXiv ids. It does not treat vendor window sizes as effective context.

Qwen3-8B config.json (Hugging Face, Qwen/Qwen3-8B): 36 layers, 8 KV heads, head_dim 128, hidden_size 4096. KV bytes/token = 2 × 36 × 8 × 128 × 2. Prefill attention FLOPs ≈ 4·L·N²·d = 5.31e20 at N=3e7; at 50% MFU on an H100 BF16 peak (989.4 TFLOPS) that is ~12.4 GPU-days. Dense 2·N·P at P=8e9 is 4.8e17 FLOPs, ~16 minutes. SUPERCTX does not run that attention; the table is the cost of keeping 30M tokens in KV versus in the bank.

Gemini 1.5 reports near-perfect retrieval up to 10M tokens on a needle-style task (Reid et al., arXiv:2403.05530). That is a published claim on a different task, a different stack, and a datacenter decode. SUPERCTX reports 1.000 on n=300 needles at 30M tokens on the synthetic corpus above, plus certified absence, which that report does not. The tasks are not a head-to-head. The storage and abstention columns are the ones that do not exist on the attention side.

Prior art for explaining missing query results sits in databases (Why / WhyNot), not in RAG leaderboards. Vectara-style hallucination rates measure generator faithfulness, not a certificate that the index lacks the fact.

4. Discussion

4.1 Certified absence

In finance, law, and health, “we found no evidence” is a claim that has to survive audit. Matched-filter verification turns that shrug into a defensible negative. The empty set is a first-class output.

4.2 Linear scaling

S4 is O(N). Attention at 30M is dominated by the N² term (section 3.4). The bank is 7.5 MB. That is the economic fact, independent of GPU price tables.

4.3 Limits

The corpus is synthetic. Ten of thirty tasks do not narrow. The serve-time filter that was supposed to realize clause-join narrowing measured as a no-op. Real-world corpora, a working serve filter, and a stable query API remain open.

5. Conclusion

SUPERCTX is certified retrieval with provable abstention: S4 + MemoryKernel + TurboQuant. Needle retrieval 1.000 versus RAG 0.000–0.003; joint 0.940 versus 0.102; 7.5 MB versus 4.42 TB to hold 30M tokens. The next measurement is the same protocol on a customer corpus, not a larger claimed window.

References

• Gu, Dao, Ré, Sohl-Dickstein (2020). HiPPO: Recurrent Memory with Optimal Polynomial Projections. arXiv:2008.07669.
• Gu, Goel, Ré (2021). Efficiently Modeling Long Sequences with Structured State Spaces (S4). arXiv:2111.00396.
• Behrouz, Zhong, Mirrokni (2024). Titans: Learning to Memorize at Test Time. arXiv:2501.00663.
• Dettmers, Lewis, Belkada, Zettlemoyer (2022). LLM.int8(). arXiv:2208.07339.
• Lin, Tang, Tang, Yang, Dang, Gan, Han (2023). AWQ. arXiv:2306.00978.
• Dettmers, Svirschevski, Egiazarian, Kuznedelev, Frantar, Ashkboos, Borzunov, Hoefler, Alistarh (2023). SpQR. arXiv:2306.03078.
• Tseng, Chee, Sun, Kuleshov, De Sa (2024). QuIP#. arXiv:2402.04396.
• Fu, Dao, Saab, Thomas, Rudra, Ré (2022). Hungry Hungry Hippos (H3). arXiv:2212.14052.
• Li, Gururangan, Dettmers, Lewis, Althoff, Smith, Zettlemoyer (2022). Branch-Train-Merge. arXiv:2208.03306.
• Gu, Dao (2023). Mamba. arXiv:2312.00752.
• Wu, Rabe, Hutchins, Szegedy (2022). Memorizing Transformers. arXiv:2203.08913.
• Borgeaud et al. (2022). RETRO. arXiv:2112.04426.
• Wang, Dong, Cheng, Min, Huang, Zhang, Chen, Wei (2023). LongMem. arXiv:2306.07174.
• Mohtashami, Jaggi (2023). Landmark Attention. arXiv:2305.16300.
• Graves, Wayne, Danihelka (2014). Neural Turing Machines. arXiv:1410.5401.
• Graves et al. (2016). Hybrid computing using a neural network with dynamic external memory (DNC). Nature 538:471–476.
• Rae, Potapenko, Jayakumar, Lillicrap (2020). Compressive Transformers. arXiv:1911.05507.
• Reid et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530.