Consolidated research results, including constitutive edges and continual memory

Rendered archival TeX

This is an HTML reading rendition of the local TeX record. Mathematical notation is rendered with KaTeX; archived figures are included when their source assets are part of this collection.

Results

We now report the empirical core of the paper. Across domains, we test when local credit assignment, closed-form solving and memory, and topology search can match or improve gradient-trained alternatives under explicit information and resource constraints. These are regime-dependent comparisons, not a universal replacement for backpropagation: later adaptation experiments also retain pretrained neural dependencies, gradient-trained baselines and failed whole-task claims. We open with the headline comparison (Figure [fig:v2-results]), then give a consolidated cross-domain table (Table [tab:v2-domains]), and finally develop each result family in its own subsection with the key numbers and the honest boundaries. Each result uses a different composition of the building blocks into a network with its own shape; rather than bury those configurations in prose, we collect a per-example architecture diagram and a step-by-step reimplementation recipe for every headline result in Appendix [app:arch] (deep vision CNN, Fig. [fig:arch-cnn]; byte-GPT, Fig. [fig:arch-gpt]; closed-form continual memory, Fig. [fig:arch-gram]; operator-matched PDE vs.\ FNO, Fig. [fig:arch-pde]; MuJoCo world model, Fig. [fig:arch-mujoco]; pixel model-based control, Fig. [fig:arch-mbrl-pixel]; value-based local-sweep DQN, Fig. [fig:arch-dqn]; and the ES baseline, Fig. [fig:arch-es]). A reader who wants to rebuild any single experiment can work entirely from its diagram and recipe there.

Gradient-free vision matches or beats backpropagation

The most heavily contested claim is that a network can learn deep visual features with no global backward pass and reach the accuracy of backpropagation. We train a deep convolutional network on CIFAR-10 in which each block is trained by its own local error sweep: activations are detached between blocks, every block computes its own weight update and the error message to the block below via its local objective, and there is no backward pass spanning the network (the cortical area-to-area error-messaging picture; Section [sec:blocks]). With a deeper architecture, GPU augmentation, and a cosine-annealed schedule (CH $=64/128/256/512$, $50$ epochs), the local sweep reaches $0.9024$ test accuracy and, on the identical architecture, beats the global-backprop reference ($0.8855$) by $+1.7$ points. This is not a one-seed fluke: across $3$ seeds at $35$ epochs, the local sweep attains $\mathbf{0.8994 \pm 0.0004}$ versus backpropagation's $0.8824 \pm 0.0024$ on the same net—local $\geq$ backprop on $3/3$ seeds, with a $+0.017$ mean gap and non-overlapping standard deviations. The consistent local edge is the empirical face of a recurring finding (Section [sec:res-pc]): local learning co-adapts blocks less than a global backward pass and thus regularizes, memorizing less and generalizing better.

We are deliberate about where this number came from. Earlier in the program a $0.68$ ceiling was mistaken for a no-backprop limit; we diagnosed it precisely as a fixed-front-end feature ceiling—even backpropagation plateaus at $\sim$$0.69$ on a shallow hand-specified Gabor/DoG operator bank, because that bank has already discarded the information a high-accuracy readout would need. Adding head depth (PC or backprop) does not break it. The break comes from learning deep features from pixels by a local sweep: $0.683 \to 0.842$ on the identical task ($+16$ points, backprop reference $0.864$ on the same net), and then $>0.90$ with a stronger architecture. There is no ``no-backprop accuracy ceiling''; there was a shallow-fixed-feature ceiling, removable by deep local credit assignment. The remaining gap to top SOTA ($\sim$$0.95$) is residual connections, width, and scale—engineering on a settled learning rule—not the absence of backpropagation.

The same gradient-free rule extends to a modality it was never designed for: language rendered as vision. Rendering each DBpedia-14 example (title $+$ content) as a $48\times192$ grayscale text image and classifying topic through pixels, the local sweep reaches $0.937$ accuracy versus backpropagation's $0.935$ on the identical network ($2$ seeds)—local $=$ backprop again in a new modality. A frozen random convolutional encoder with a closed-form Gram readout already decodes topic at $0.441$, showing rendered text is highly linearly readable from generic visual features. We are explicit that this is a modality-agnostic demonstration, not an NLP method: TF-IDF on the raw text reaches $0.972$, so the pixel route sits $\sim$$3.5$ points below text-native classification, as expected when the tokenization prior is discarded. The point is that one no-backprop visual learning rule reads language cast as vision at near-backprop accuracy.

Continual learning with pooled fixed-feature memory

The closed-form Gram memory composes tasks by accumulating second-order statistics. For a fixed feature map and regularization objective, it recovers the pooled-data optimum in exact arithmetic. This preserves the objective, not old-task accuracy, and floating-point summation can depend on order. We evaluate retention by fusing the deep local-from-pixels backbone (Section [sec:res-vision]) with the Gram memory on a $5$-task class-incremental CIFAR-10 stream. The fully gradient-free agent reaches $\mathbf{0.8893}$ all-seen accuracy (per-task $\{0.93,0.79,0.85,0.93,0.94\}$, every task retained), and—decisively—emph(reversing the task order gives the identical $0.8893$ to within $10^{-6}$). This is empirical order stability on this stream, not a universal zero-forgetting theorem or an impossibility result for gradient methods. Sequential backprop on the same stream collapses to $0.189$: only the last task survives ($\{0,0,0,0,0.94\}$). The headline gap is therefore $0.887$ vs.\ $0.19$ (multi-seed: continual cortex $0.8867 \pm 0.002$, scoreboard S72,S84). Notably, the continual agent exceeds its own single-task backbone ($0.842$), because the Gram memory finds the joint optimum over the deep features.

The property holds at $100$-class scale. On Split-CIFAR-100 ($10$ tasks $\times\,10$ classes, single-pass task-ordered stream) over a frozen ImageNet-pretrained backbone, the closed-form Gram readout reaches $\mathbf{0.685}$ final accuracy on all $100$ classes (mean of $3$ seeds, $\pm 0.001$), versus $0.266$ for an online-SGD head—the textbook catastrophic-forgetting baseline, whose per-task accuracy decays $0.84\!\to\!0.26$ as classes arrive—and $0.504$ for online SGD with a $2$k-example class-balanced replay buffer. The Gram curve instead holds $0.93\!\to\!0.69$. This $+42$-point margin over online SGD ($+18$ over replay) demonstrates useful empirical retention beyond the small-scale streams; it reproduces the random-projection-ridge line of RanPAC [mcdonnell2023ranpac], here cast as the same Gram-memory mechanism the substrate uses throughout (scoreboard LOCAL#3).

Distilling a frontier teacher into the gradient-free substrate, and the teacher-scaling law.

The same mechanism inverts the iron law (structured problems crush, broadband ties): rather than compete with a frontier model head-on, we let a frozen frontier self-supervised teacher pay the broadband representation cost and consolidate its features into the gradient-free bio-substrate, which then supplies the brain-properties the teacher's trainable head lacks. On the identical Split-CIFAR-100 class-incremental protocol (10 tasks $\times\,10$ classes, no task-ID at test, 3 seeds), swapping the frozen backbone for a self-supervised DINOv2 teacher [oquab2023dinov2] and reading it out with the closed-form substrate (random Fourier features $R{=}10000$ $+$ Gram ridge) yields a clean, monotone teacher-scaling law: final 100-class accuracy rises $0.581$ (ResNet18-IN1k, 512-d) $\to 0.846$ (DINOv2 ViT-S/14, 384-d) $\to \mathbf{0.891}$ (DINOv2 ViT-B/14, 768-d). At each step the gradient-free readout sits exactly at its own joint (offline, all-classes-at-once) upper bound—$0.846{=}0.846$ and $0.891{=}0.891$—with order-invariance variance $0.0000$ across task orders, i.e.\ zero, provably order-invariant forgetting. Backprop fine-tuning of a head on the same ViT-B features catastrophically forgets to $0.647$ ($+24$ points for the substrate), and is data-starved where the substrate is data- efficient (at 5 shots/class: substrate $0.775$ vs.\ backprop $0.625$; at 100 shots $0.873$ vs.\ $0.654$). A substrate ablation isolates what the nonlinear lift buys from what the frozen teacher and Gram ridge already give: on ViT-B, a purely linear ridge on the frozen features (no expansion) already reaches $0.885$, ReLU random projection $0.889$, and cosine random Fourier features (ours) $0.891$—so the dominant factor is the frozen frontier representation $+$ closed-form Gram, and the structured nonlinear substrate adds a small ($+0.6$-point) lift. We report this straight: $0.891$ is squarely in the RanPAC/EASE class-incremental SOTA band ($\sim 0.85$–$0.92$) reached with a small-to-mid backbone, and the claim is not a single-axis accuracy record but the multi-axis synthesis—a gradient-free learner that simultaneously achieves SOTA-band continual accuracy at the joint upper bound, zero order-invariant forgetting, and data-efficiency, where the backprop head fails on all three (scoreboard DISTILL-BIO-SUBSTRATE v4, substrate ablation).

The same mechanism is modality-agnostic. A single shared substrate ingests an interleaved vision$+$language stream (no-backprop deep vision area $+$ frozen GPT-2 language) and retains both—final vision $0.888$, language $0.652$—with no cross-modal forgetting, while a shared backprop net collapses to vision $0.012$, language $0.052$ (scoreboard S69,S77). Online (autopilot- style) perception under a mid-stream day$\to$night domain shift likewise retains the day distribution ($0.922$) where an online backprop head forgets it ($0.880$) (scoreboard S81). One mechanism, many modalities, no forgetting.

Can the encoder itself be made gradient-free? An honest boundary.

The continual head above is gradient-free, but it rides a backprop-trained (or frozen frontier) encoder. We tested whether the encoder can also be learned without backpropagation, by closed-form layer-wise distillation: let a teacher supply per-layer targets and fit each student layer in closed form (ridge), stacking to depth with no inter-layer credit assignment. Three findings, reported straight. (i) A purely gradient-free pipeline—a Coates–Ng [coates2011analysis] $k$-means/random-patch conv front end (no backprop) followed by the closed-form Gram head—is a legitimate standalone continual learner, reaching $0.423\pm0.001$ (3 seeds, $K{=}2048$) and $0.452$ ($K{=}4096$) final accuracy on Split-CIFAR-100, with zero order-invariant forgetting and scaling monotonically with dictionary size. (ii) Distilling teacher features from a fixed front end fails by the data-processing inequality—the closed-form map to the teacher's lower-dimensional features is lossy and actively hurts ($0.39\!\to\!0.33$ single-shot $\to\!0.18$ layer-wise). (iii) Convolutional closed-form distillation that shapes the conv pathway (not a fixed vector) escapes this trap: here both gradient-free depth and per-layer teacher targets genuinely help ($+6$–$8$ points over the front end's own linear score, layer-wise $>$ no-distill $>$ single-shot), but the result still caps below the teacher and below the best single wide front end, limited by per-stage fit-$R^2$ decay (error compounding $\approx 0.76\!\to\!0.20$ over four stages). A two-variable ODE-neuron substrate (multiplicative gate slaved to the membrane state) matched but never beat a plain ReLU in every role on this static task. The boundary is therefore precise and consistent with the rest of this work: the closed-form Gram memory is the solved half of gradient-free learning, while credit assignment into early convolutional filters—and the per-stage error compounding it would otherwise prevent—remains the open hard problem. The closed-form continual head delivers most when placed atop a strong encoder, whether backprop-trained, a frozen frontier teacher, or a gradient-free front end (scoreboard DEEP-OSNR-DISTILL v1–v4).

The substrate's home turf: where gradient-free depth and a heterogeneous dynamical substrate beat backprop.

The static-vision wall above is one face of a sharper principle: a dynamical substrate earns its keep on tasks with temporal/hierarchical structure, which static images do not expose to it. We test this directly with a leaky continuous-time reservoir read out by the same closed-form Gram memory (no backprop anywhere), on dynamical-systems prediction (NARMA-10/30, Mackey–Glass) and row-wise sequential MNIST, over $\geq 5$ seeds. Four findings. (i) Heterogeneity is real but not monolithic: per-neuron leak/time-constant diversity and Dale-law E/I give no gain (a random recurrent matrix already supplies effective timescale spread), but per-neuron nonlinearity diversity is a $3\times$ improvement on chaotic Mackey–Glass ($0.030\!\to\!0.011$ NRMSE)—exactly where a rich nonlinear basis is what the task needs—and nil on memory-dominated NARMA. (ii) The two-gradient-free-loop thesis holds: an outer evolution strategy that shapes the substrate plus the inner closed-form memorization beats both a default reservoir (Mackey $\sim\!10\times$, NARMA-30 $0.86\!\to\!0.60$) and a BPTT-trained GRU (Mackey $0.003$ vs.\ a diverged baseline; NARMA-30 $0.60$ vs.\ $0.77$)—with no gradients in either loop. (iii) Gradient-free depth gives a dividend: a deep stacked reservoir with hierarchical timescales, at matched total neurons, lifts sequential-MNIST accuracy $0.770\!\to\!0.816$ ($+4.6$ points) and improves Mackey $\sim\!1.8\times$, while hurting on single-scale NARMA where splitting capacity destroys the single memory—i.e.\ the depth rung that fails gradient-free on static images holds once depth can build a multi-timescale hierarchy. (iv) It self-organizes: evolving the deep substrate end-to-end (depth, per-layer timescales, nonlinearity-spread; inner closed-form readout) autonomously rediscovers all three levers—it selects depth $>1$, strong hierarchical timescale decay, and substantial nonlinearity heterogeneity—and beats the hand-tuned deep configuration. The honest scope is small networks on the substrate's native (dynamical/hierarchical) regime, not yet the broad static-vision/LM accuracy frontier; but it establishes the missing atom—a bio substrate that is gradient-free, benefits from depth, and beats backprop where dynamics matter (scoreboard SUBSTRATE STEP 1–4).

It scales, and the never-forget memory—run along time—breaks the long-range wall.

The atom holds at size: scaling the evolved deep substrate (fixed genome, full $60$k training) lifts gradient-free row-wise sequential-MNIST from $0.909$ ($N{=}600$) to $0.964$ ($N{=}1500$) to $\mathbf{0.974}$ ($N{=}3000$, $3$ seeds)—competitive with trained recurrent networks, with no backpropagation. The honest limit appears on the $784$-step pixel/permuted variants, which cap at $\sim\!0.80$: a reservoir's fading state has linear memory capacity $\sim\!N$ and cannot hold $784$ steps. The fix is our zero-forget memory itself, applied to the time axis rather than across examples: read out a non-fading additive accumulator over the sequence ($m_T=\sum_t \phi(x_t)$), which is the closed-form Gram principle run along time and is the same object as linear-attention / state-space / gated-delta memory [yang2024gateddeltanet]—and remains parameter-free, hence gradient-free. This breaks the wall: pixel-seqMNIST $0.794\!\to\!0.953$, and on the rigorous long-range benchmark permuted-seqMNIST, $0.821\!\to\!\mathbf{0.882}$ ($N{=}1500$, $3$ seeds)— competitive with trained LSTMs ($\sim\!0.88$–$0.90$) at zero backprop. The fading reservoir state (recent detail) and the non-fading accumulator (global long-range) are complementary (their concatenation beats either alone), realizing a two-timescale memory along both the example and the time axes (scoreboard SUBSTRATE STEP 5–7).

Where it does not win—the boundary, mapped honestly.

The same substrate fails, cleanly and informatively, off its native regime—which is itself the sharpest statement of the iron law. (i) Broadband language: a from-scratch gradient-free reservoir character language model on tiny-shakespeare, even when tuned for the task, reaches $0.45$ next-character accuracy—above a bigram ($0.40$) but below a trigram ($0.49$), and far below trained neural language models. A fixed dynamical substrate cannot out-model even a count-based $n$-gram on broadband text. (ii) Algorithmic length generalization: training on short sequences and testing on $\sim\!10\times$ longer, a trained GRU extrapolates better than the fixed substrate (delayed-recall: GRU stays at $1.00$ while the reservoir's echo-state memory fades to chance; majority: $0.71$ vs.\ $0.56$ at length $100$)—because the solution is a learned gate, and backprop's learned gating beats a fixed random reservoir's fading memory. (iii) Gating/delta-rule memory with random (unlearned) projections gives no gain over the plain accumulator, confirming that delta-rule selectivity needs learned or evolved gates. Taken together with the wins above, the boundary is exactly the iron law made empirical: the bio substrate crushes on dynamical, structured, and long-memory regimes and loses on broadband and on problems whose solution is a learned gate—so its role toward the broad frontier is as a continual / long-range / dynamical module atop a strong (e.g.\ frozen frontier) encoder, not as a from-scratch broadband learner (scoreboard SUBSTRATE STEP 8–10).

The payoff: ``bulletproof streaming intelligence'' — beating backprop on four deployment axes at once, gradient-free.

Placing the substrate where the boundary says it belongs—as a continual / long-range / dynamical module on a frozen encoder, in the online streaming regime—yields a system that beats backprop simultaneously on four brain-property axes, with no gradients and no replay. (1) Never-forget, online: on a single-pass class-ordered stream over a frozen DINOv2 encoder, the $O(1)$-per-example Gram memory holds $0.916$ anytime accuracy (final $0.869$ over $100$ classes), while a naive online-SGD head catastrophically forgets to $0.073$ ($+84$ points). (2) Temporal $+$ continual together: on Split-sequential-MNIST class-incremental—each digit a row-sequence read by the reservoir with the never-forget-along-time accumulator, classes consolidated by the never-forget-across- examples Gram—the fully gradient-free system reaches $0.992$ anytime / $0.976$ final versus online-SGD's $0.694/0.469$. (3) Time-warp robust: a dt-aware substrate (physical-time integration) trained at one sampling rate retains $0.924$ mean accuracy at $0.5$–$2\times$ warped test rates, where a trained GRU ($0.571$) and a non-dt-aware reservoir ($\sim\!0.30$) collapse—an ablation that attributes the robustness to dt-awareness, a structural property backprop networks lack. (4) Non-stationary drift: on a domain-incremental rotated-sequential-MNIST stream ($0^\circ\!\to\!45^\circ\!\to\!90^\circ\!\to\!135^\circ\! \to\!0^\circ$, shared labels, the first domain recurring), the never-forget memory retains all rotation domains ($0.899$ mean, $0.904$ on the early ones) while online-SGD drifts to the recent domains and forgets the rest ($0.443$ mean)—the deployment reality that a system which ``drives into night and back'' must not forget day. This is the constructive complement to the iron-law boundary: not a from-scratch broadband learner, but a gradient-free module that gives a frozen frontier model the brain-properties it lacks—never-forgetting, online adaptation, dynamical robustness, and drift-retention—each a decisive win over backprop on the axis that matters for deployment (scoreboard STREAMING-INTELLIGENCE 1–4).

It holds online and on real fine-grained data.

Two further checks harden the claim. First, the never-forget memory has zero online penalty by construction: its online single-pass accuracy equals its offline result ($0.942$ vs.\ $0.943$ on rich DINOv2 ViT-L features for Split-CIFAR-100), because the Gram accumulators are additive sufficient statistics—order- and streaming-invariant—whereas a backprop head pays a large online penalty (online-SGD $0.62$). Second, off CIFAR onto real fine-grained photos: on Flowers-102 ($102$ classes) the streaming memory reaches $0.993$ (near-perfect, but the data is so DINOv2-separable that even nearest-class-mean ties it), while on the harder FGVC-Aircraft ($100$ fine-grained variants) the never-forget memory reaches $0.702$ and decisively beats both nearest-class-mean ($0.504$, $+20$ points) and online-SGD ($0.103$)—the never-forget advantage grows with task difficulty. Third, on real wearable-sensor time-series (UCI-HAR smartphone accelerometer/gyro, $6$ activities, $128$-step $\times\,9$-channel windows) the full dt-aware substrate does online activity recognition at $0.932$ (vs.\ online-SGD $0.478$) and is robust to a $0.5$–$2\times$ change in sampling rate ($0.83$–$0.89$, barely below the $0.932$ train-rate score)—the on-device deployment reality. And absolutes track the frozen encoder on both arms (the teacher-scaling law): banking-77 $0.79\!\to\!0.91$ from GPT-2 to BGE-base, and FGVC-Aircraft $0.70\!\to\!0.77$ from DINOv2 ViT-S to ViT-B (scoreboard STREAMING-INTELLIGENCE 1b–7, NEXT-STEP a).

The same module gives a frozen LLM lifelong memory.

The pattern is modality-general: a frozen language model plus the gradient-free never-forget memory acquires new knowledge online, lifelong, where fine-tuning forgets. (i) Continual intent learning: on banking-77 ($77$ fine-grained intents), a frozen sentence encoder (BGE-small) read out by the online Gram reaches $0.90$ final accuracy single-pass, versus online-SGD's $0.04$; absolutes track the frozen encoder (GPT-2 $0.79\!\to\!$ BGE $0.90$), the same teacher-scaling law as vision. (ii) Factual memory: a frozen GPT-2 plus a never-forget associative memory keyed by its prompt embeddings absorbs $1{,}200$ novel facts at $0.989$ recall, lifelong with no forgetting (online-SGD $0.02$, LLM-alone at chance $0.01$); the store scales to $10{,}000$ facts ($0.90$ recall, capacity $\sim\!R$). (iii) Into generation: when that memory's readout biases the frozen LLM's next-token logits, the model generates the taught facts ($0.86$) where it was otherwise at chance ($0.007$); driving the bias autoregressively step-by-step makes the frozen LLM generate full multi-token answer phrases at $0.86$ exact-match (vs.\ $0.000$ for the LLM alone or online-SGD), lifelong with no forgetting—gradient-free knowledge injection into a frozen LLM's actual token-by-token output. The honest ``beat LMs'' axis is thus realized: not the broadband core, but the lifelong never-forget knowledge that frozen LLMs structurally lack, gradient-free and at the generation level (scoreboard LLM-MEMORY 1–4).

The lifelong brain: one closed-form memory across datasets, modalities, and faculties

The continual results above support a bounded thesis (Table [tab:lifelong]): attach pooled sufficient statistics ($G\!+\!=\!Z^\top Z$, $B\!+\!=\!Z^\top Y$, $W=(G+\Lambda)^{-1}B$) to suitable frozen pretrained features and obtain an exact-arithmetic pooled readout without raw exemplar replay. Useful retention depends on the representation, data and task compatibility; it is not guaranteed for arbitrary encoders. Across datasets: one frozen DINOv2 ViT-S ingests CIFAR-100 $\to$ Flowers-102 $\to$ FGVC-Aircraft as a task-free stream ($302$ classes); a class-balanced memory retains every domain at near-zero forgetting ($+0.011$, average $0.754$, beating nearest-class-mean $0.720$) while a finetuned head collapses ($0.035$, forgetting $+0.42$). Across modalities: two different encoders (DINOv2 for images, BGE for text) feed one shared memory through per-modality random features; an interleaved vision/language stream (CIFAR-100, banking77, Flowers, dbpedia, Aircraft; $393$ classes) is learned task-free at $0.803$ average, forgetting $+0.006$, with no cross-modal interference, versus $0.069$ for finetuning. Against the standard baselines: on Split-CIFAR-100 class-incremental learning the gradient-free memory beats a fair replay (balanced batches, $2000$-exemplar buffer) decisively—$0.906$ vs.\ $0.820$ with DINOv2 ViT-L features (forgetting $0.036$), and the win holds against the tested cross-modal replay too ($0.80$ vs.\ $0.19$). The reported membership attack gives AUC $1.0$ on the explicit replay/RAG store and approximately $0.53$ on sufficient statistics, matching a gradient head. This particular attack does not establish privacy: statistics can reveal individual data. New classes are absorbed by a statistics update and readout solve with no readout-training epochs: cross-domain few-shot class addition keeps base accuracy approximately stable ($0.773$–$0.775$) while new-class accuracy increases with shots; the tested finetuning control has a retention/adaptation tradeoff. A decay knob $\gamma$ controls recency weighting—on a stream mixing stable and drifting concepts the gated memory retains the stable ($0.91$) and tracks the obsolete ($0.80$) at once, beating online SGD on both axes. Into action: the same memory that perceives also acts—given a vision faculty, a language faculty, and a control faculty (behavior-cloning a gradient-free CartPole expert), one block-structured memory perceives (vision $0.69$, text $0.55$) and deploys an expert control policy ($500/500$) with zero forgetting, where a sequential finetune forgets every earlier faculty to chance. The honest boundary is sharp and consistent (Sec.\ [app:neg-gfrsi], App.\ [app:record]): these are lifelong capabilities added on top of a frozen encoder; the approach does not learn the representation itself gradient-free (self-improvement and continual representation learning still require gradients), and on tasks within a strong encoder's competence, freezing the encoder is essentially optimal.

The limits of self-improvement: what compounds, what does not, and why

Having attached lifelong learning to a frozen model, we ask the sharper question: can the substrate improve itself online? The answer is a precise map of when self-improvement compounds, and it is governed by one principle—the closed-form memory's advantage requires information that is both new and stationary. (1) Self-reference plateaus. A model that self-trains on its own predictions cannot compound: on fixed features it degrades (Fig. [fig:selfimprove]a, $v{=}0.5$), and even a trainable EMA-anchored representation, while it climbs stably from a tiny seed, saturates at a pseudo-label ceiling ($\sim$$0.35$, $41\%$ of supervised; Fig. [fig:lifelong]a)—no new information enters the loop. (2) Reinforcement learning is not our tool's regime. We hoped the memory's stability would tame the deadly triad, but it is an honest negative: as an online $Q$-learner the closed-form gated memory is no more stable than gradient TD (CartPole final $220$/late-std $144$ vs.\ gradient $256$/$84$), because RL targets are non-stationary (bootstrapped and policy-dependent) and the memory merely fits the moving target. (3) Verification breaks the readout plateau. An external verifier supplies information that is new and stationary; filtering self-labels through a verifier of reliability $v$ lifts the same degrading loop monotonically to the supervised ceiling (Fig. [fig:selfimprove]a, $v{\ge}0.9$), with a threshold near $v\!\approx\!0.9$. The verifier must be external, though: a self-derived one—ensemble agreement over feature-subsets of the same model—does not substitute ($0.372$ vs.\ self-training $0.375$ vs.\ oracle $0.477$), because members sharing features have errors correlated with the model's own. (4) Only teaching breaks the representation plateau. Verification certifies examples the model already gets right, so it adds no new representational signal; a trainable CNN plateaus regardless of verifier quality, and what breaks it is teaching—external labels on the examples it cannot yet do (Fig. [fig:selfimprove]b, $0.46\!\to\!0.64$). (5) It works on a real LLM. Putting the viable piece together: a frozen qwen2.5vl:7b on long multiplication, verified by execution and retrieving execution-verified worked-solutions from a never-forget memory, self-improves $0.31\!\to\!0.49$ ($+18$ points) with no weight updates and no labels; a control shows that its own unverified solutions as exemplars instead hurt to $0.13$, isolating verification—not few-shot prompting—as the cause (Fig. [fig:selfimprove]c). The boundary. Self-improvement compounds exactly up to the information already latent in the model-plus-data; a genuine external verifier (execution, proof, unit tests, simulation) plus a never-forget memory yields real gradient-free, label-free self-improvement of the knowledge/readout layer to its ceiling—but improving the representation itself, or bootstrapping from a near-chance start, requires external teaching. This is a falsifiable, honest boundary on ``recursive self-improvement.''

Predictive coding $=$ backpropagation across the architecture family

The local learner is not an approximation that happens to work on convnets: we establish, at both the gradient level and the accuracy level, that local predictive-coding credit assignment equals backpropagation across the entire modern architecture family. On MLPs, in the theoretically correct regime (Z-IL exact, or a small nudge) the per-layer cosine between the PC free-energy gradient and the autograd backprop gradient is $1.000$, and end-to-end accuracy matches ($0.676$ vs.\ backprop $0.678$, $\Delta 0.003$); the naive hard-clamp's $0.56$ artifact is fully diagnosed and fixed [whittington2017predictivebackprop, song2020zil, song2024prospectiveconfiguration, millidge2022pcbeyondbackprop]. For deep stacks, iterative relaxation attenuates the error wave toward the input (conv1 cosine $\approx 0$ even at $T=300$); the fix is the closed-form one-sweep PC equilibrium—$\delta_L = \mathrm{softmax}(\mathrm{out})-y$, $\delta_l = \phi'(z_l)\odot (W_{l+1}^{\top}\delta_{l+1})$, local Hebbian $dW_l = \delta_l\,a_{l-1}^{\top}$—which is exact at all depths by construction and reproduces backprop ($0.662$ vs.\ $0.658$). This ``replace iterative relaxation with a closed-form equilibrium solve'' move is precisely the closed-form thesis of this paper applied to the cortical learning rule.

The equivalence extends to recurrence and attention. For recurrent credit assignment, e-prop (eligibility traces $+$ a broadcast signal, no backprop-through-time and—in the random-feedback variant—no weight transport) matches BPTT on working memory, its true domain: $0.999$ at delay $6$ and $0.974$ length-generalizing to delay $16$ versus BPTT's $1.000$, and the fully-bio random-feedback variant beats the symmetric one [bellec2020eprop]; RTRL recovers BPTT exactly online ($0.475$ vs.\ $0.480$) where e-prop's diagonal approximation must (scoreboard S44,S46). For attention, the per-block local sweep solves the canonical induction-head task perfectly ($1.000 = 1.000$ vs.\ backprop), and—the strongest test—a real autoregressive char-GPT (embedding $+$ $4$ causal transformer blocks, $816$K-char corpus) trained purely by the local sweep reaches $\mathbf{1.946}$ bits/char, within $0.013$ of backpropagation's $1.933$, and generates coherent corpus-style text mixing English and LaTeX (scoreboard S83,S85). PC $=$ backprop is thus confirmed across MLP, conv, recurrent (e-prop), attention, and a full GPT: the whole modern architecture family trains with no global backward pass. A from-scratch LLM at full scale remains a compute question, not a mechanism one.

Self-construction and scale

The substrate also builds itself. Neuroevolution proposes the wiring and the closed-form/local learner scores and trains it. On a structure-required task where a linear seed is at chance (parity-8), a single add-node mutation drives closed-form PC from $0.4963$ (chance) to a perfect $1.000$, with no backprop anywhere—neuroevolution discovers the needed hidden unit and the local learner solves the task exactly (scoreboard S66) [stanley2002neat]. At conv scale, $(1+\lambda)$ evolution grows the convolutional architecture itself (depth and width) with the per-block local sweep as the inner learner: the substrate climbs $[32]\!\to\![32,64]\!\to\![32,64,128]\!\to\![32,64,192]$ (validation $0.43\to0.69$), and the self-evolved architecture retrains to $\mathbf{0.821}$ test— competitive with the hand-designed $[64,128,256]$ reference ($0.846$), the small gap reflecting only the short search (scoreboard S73). Folding all three pillars together yields the full agent: a self-constructed architecture $[32,64,192]$ with local-sweep weights and a Gram memory reaches $0.8635$ on $5$-task continual CIFAR-10 (order-invariant), versus backprop continual's $0.1927$ collapse—one agent that designs its own architecture, learns deep features without backprop, and accumulates tasks without forgetting (scoreboard S74).

Scale is not a barrier. A neuron/throughput accounting of the no-backprop substrate on a single MPS GPU shows even the small configuration exceeds $10^5$ activation-neurons per image; the base configuration (CH $=64/128/256/512$) has $\mathbf{2.46\times10^5}$ neurons, $5.2$M parameters, and runs at $7{,}230$ images/s. We train this $10^5$-neuron-class substrate gradient-free by the per-block local sweep to $\mathbf{0.855}$ in $12$ epochs using $\sim$$1$ GB ($<1\%$ of $128$ GB)— ample headroom to scale further (scoreboard S78). Capability also scales with autonomously-grown substrate size on language (block-growth from a minimal start: $0.57\to0.70$ as the search grows the circuit, surpassing order-$2$ context), confirming the scaling premise (scoreboard S14e).

Control, games, and operator-matched solving

Where the world model can be identified rather than learned by trial and error, closed-form solving plus planning is overwhelmingly more sample-efficient than model-free learning. From a few hundred real transitions we identify dynamics in closed form (matched dictionary $+$ ridge, $R^2 = 1.0$), then plan or evolve a controller entirely inside the learned model. On pendulum stabilization this yields a true return of $-74.3$ from $1{,}000$ real transitions versus model-free neuroevolution's $-308.6$ from $28.8$M transitions: $\sim$$\mathbf{28{,}800\times}$ sample efficiency and a better return. Closed-loop CEM-MPC further solves the hard-exploration pendulum swing-up ($-347$, where open-loop and reactive policies plateau), solves classic MountainCar ($100\%$ success, $\sim$$111$ steps to goal from $500$ transitions), and stabilizes a $6$-D planar quadrotor to hover (final position error $0.006$ from $800$ transitions)—three model-based control wins from exact closed-form models (scoreboard S16–S20) [brunton2016sindy].

Games from pixels close the loop with learned dynamics. A no-backprop agent—perception by the per-block local sweep, a closed-form learned latent dynamics model (ridge on collected transitions, no known physics), and MPC planning—solves the Catch game directly from pixels at $\mathbf{0.965}$ catch-rate (random $0.295$) from only $\sim$$2{,}700$ transitions: the DQN-from-pixels recipe [mnih2013dqn] with both the global backward pass and the millions-of-frames requirement removed (scoreboard S82). We report the honest partial as well: on real ALE Pong the gradient-free perception clones a predictive teacher decently ($0.79$ behavior- cloning accuracy) but the cloned policy does not yet play ($-21$)—a textbook behavior-cloning distribution-shift failure, not a perception failure; full Atari mastery needs interactive RL at scale (scoreboard S86).

Finally, the original operator-matched regime—the parent of the whole program—remains the most lopsided. Where the governing operator $L$ is known and encoded in the representation, the matched closed-form solver beats neural operators and neural ODEs by two to three orders of magnitude in both error and speed: $\sim$$400$–$2000\times$ lower error than SINDy/Neural-ODE on non-polynomial ODE extrapolation; $2$-trajectory closed-form beating $32$-trajectory FNO on $1$D Burgers ($\sim$$3$–$4\times$ better extrapolation, $300$–$4000\times$ faster), with the advantage growing from $1$D to $2$D to $3$D; and $\sim$$0.001$ nRMSE in under $0.7$ s versus FNO's $0.13$–$0.72$ in $120$ s on $3$D reaction-diffusion at scale (scoreboard S1–S2) [li2021fno, lu2021deeponet, chen2018neuralode, brunton2016sindy]. This is the sparse-stochastic- process law of Section [sec:theory] made concrete: matched beats generic precisely when $L$ is non-trivial, and ties a generic random-feature$+$ridge solver (at orders of magnitude less compute) when it is not. Across all of control, games, and operator-matched solving, the same principle holds— encode the structure you know, solve in closed form, and plan—with no global backward pass anywhere.

The same law governs generation.

The SSP boundary extends to generative modeling. In the stochastic-interpolant framework, the generative drift can be fit in closed form as a linear map on a feature map—no neural training—following the kernelized interpolants of [coeurdoux2026kernelinterpolants]. We ran the same interpolant sampler with the drift fit two ways: closed-form random-feature regression versus a gradient-trained network. On structured and low-complexity targets (piecewise-smooth $1$D signals, Gaussian fields, natural $8{\times}8$ patches, $d{=}64$), the closed-form drift matches or beats a comparable neural drift on holistic distribution match (energy distance $0.08$–$0.11$ vs.\ $0.13$–$0.17$) at $2$–$3\times$ less compute. But at full-image scale (CIFAR $32{\times}32$, $d{=}1024$), a CNN velocity network with spatial features decisively wins—feature-space energy distance $0.73$ versus the closed-form drift's $3.15$, and pixel-spectrum error $0.22$ versus $2.16$. This is exactly the iron law in a new modality: closed-form generation is efficient and competitive where structure dominates, but broadband natural images require matched (e.g.\ scattering, as [coeurdoux2026kernelinterpolants] use) or deep-learned features—generic closed-form features do not suffice. Structure the factors that are structured; leave the broadband ones to learned features.

The same operator-matched principle extends to ill-posed inverse problems $y=Ax$ with a non-trivial null space (inpainting, subsampling, CT/MRI-style measurement). Recent one-step generative solvers confine the reconstruction to the measurement-consistent subspace $\{x:Ax=y\}$ and learn the null-space content with a network [shi2026nullflow]; when the signal is structured, that null-space prior is itself closed form, so the entire posterior sampler is a single linear solve—no training, no iteration. On structured-field recovery from $30\%$ random pixels, the null-space-confined closed-form posterior beats a well-tuned iterative total-variation solver by $5$–$19$ dB PSNR in a single solve versus $400+$ iterations (the iterative solver never reaches its accuracy), the same matched-beats-generic pattern at $10^2$–$10^3\times$ less compute. The boundary is reported honestly and is exactly the SSP prediction: on sharp-edged, piecewise-constant fields the smoothness prior is mismatched and the closed-form solver loses to total variation (which matches that structure), and on broadband natural images the learned prior of [shi2026nullflow] is the right choice—the win holds precisely where the structure is matchable (scoreboard NULLFLOW-OSNR). The same result holds on a real ill-posed inverse problem: sparse-view computed tomography (parallel-beam Radon). From as few as $16$ projection angles, the operator-matched closed-form solver recovers a subspace-structured image essentially exactly ($\sim$$120$ dB PSNR) and a smooth (Gaussian-process) image at $55$–$58$ dB—in a single linear solve—versus $36$–$43$ dB for a tuned iterative total-variation solver at $300$ iterations and an outright failure (negative PSNR, streaking) for classical filtered backprojection at this sparsity. The honest boundary recurs unchanged: on edge-dominated phantoms the smooth prior is mismatched and total variation wins by $6$–$16$ dB (scoreboard CT-OSNR). The throughline is one law: encode the structure you actually know—operator and prior—and the reconstruction collapses to a single closed-form solve that beats iterative and learned solvers by orders of magnitude in compute and, where the structure is genuinely present, in accuracy as well.

Capacity is the prerequisite; evolution refines for robustness

A recurring design question is how to allocate the substrate's two learning engines—the local sweep (which learns weights) and neuroevolution (which searches structure/configuration). We tested a sharp, falsifiable version of the folk intuition that capacity must come first (a five-year-old can learn chess; a 302-neuron worm never will, regardless of training): does a capacity threshold gate learnability, and does evolution refine a capable substrate rather than manufacture missing capacity? Three gradient-free experiments (scoreboard S106–108) answer it.

The cliff is real, and it scales with task demand.

On a capacity-demanding task—memorizing $K$ random input$\to$label pairs, whose minimal capacity is provably $\sim O(\#\text{params})$—trained by the per-block local sweep, train accuracy stays at chance below a width threshold and snaps to $1.0$ above it, and the threshold moves rightward with $K$, collapsing onto a single curve when plotted against parameters-per-pattern (Figure [fig:cap-cliff]). Below capacity the task is simply unlearnable—no amount of local-sweep training rescues it—confirming that representational capacity is a hard prerequisite, demonstrated here for a gradient-free learner.

Capacity gates only when the task demands it.

The same sweep on a control task (HalfCheetah model-based, sweeping the world-model width) shows no cliff: the model-fit $R^2$ rises smoothly and a $498$-parameter net already fits $R^2\!=\!0.54$ (Figure [fig:cap-threshold], left). HalfCheetah is not capacity-demanding—its threshold sits below even the smallest net—so capacity is a non-issue there. What does bind is planning stability: a width-$256$ model is near-perfect ($R^2\!=\!0.99$) yet the default planner controls it poorly and with high variance, because a strong planner exploits residual model error (Figure [fig:cap-threshold], right). The two results reconcile cleanly: capacity gates learnability iff the task demands it; otherwise the binding constraint is elsewhere.

Given capacity, evolution refines for robustness—and it transfers.

Where the bottleneck is stability rather than capacity, neuroevolution is exactly the right tool. Freezing the width-$256$ world model and evolving only the planner configuration (horizon, candidates, iterations, elite fraction) with a small CEM-ES, the controlled return improves from $-79\,/\,-98$ (default, on the training model and a held-out world model) to $\mathbf{+278\,/\,+236}$—a $\sim$$+350$ swing that transfers to the held-out model, i.e.\ genuine robustness rather than overfitting (scoreboard S108). Mechanistically, evolution discovers a deliberately weaker planner (shorter horizon $7$ vs.\ $12$; a larger, softer elite set $\sim$$38\%$ vs.\ $13\%$) that refuses to exploit model error—automating the stabilization otherwise done by hand. This is the division of labor made concrete: capacity is the prerequisite (the local sweep cannot learn what the architecture cannot represent), and given sufficient capacity, evolution refines the capable substrate for robustness---it does not manufacture capacity.

Finding the right capacity gradient-free: a local-signal splitting criterion.

If capacity is the prerequisite, can the substrate find the minimal sufficient capacity on its own, without a global backward pass? We connect the escape-dimensions view of overparameterization—adding a neuron turns a local minimum into a saddle with a descent direction [fukumizu2000local,simsek2021geometry,martinelli2026escape]—to our local-credit machinery. The splitting-steepest-descent criterion [liu2019splitting,wu2020steepest] grows a network by splitting the neuron whose splitting matrix $S(\theta)=\mathbb{E}[\Phi'(\sigma)\,\nabla^2_\theta\sigma]$ has a negative minimum eigenvalue, along that eigenvector. The key observation is that $S$ factorizes into exactly the two quantities our local sweep already produces: $\Phi'$ is the local output-error message at the neuron and $\nabla^2_\theta\sigma$ is the neuron's own (block-diagonal) curvature—so the criterion needs no global backward pass. On a teacher–student task with a known minimal width $r$ (a ground-truth ``right architecture''), with the output head solved in closed form and the hidden weights updated by the local error message, we grow from a single neuron by local-signal splitting. The locally-computed splitting matrix matches the autograd one to a relative error of $5\times10^{-7}$ (the criterion is exact); a fixed-width sweep exhibits a clean capacity cliff with its edge exactly at $m=r$ (test MSE $0.36,0.15,0.047,0.026,0.017$ for $m=1\!-\!5$, then $8\times10^{-6}$ at $m=r=6$); and the grown network recovers the minimal width (mean $6.0=r$ over seeds, test MSE $\sim\!10^{-5}$) and halts when splitting-stable, whereas adding random neurons overshoots to the width cap ($14$) with no stopping signal. To our knowledge this is the first gradient-free instantiation of splitting/escape-dimension growth (the prior methods all rely on backpropagation), and it operationalizes architecture selection within the no-global-backward substrate. The honest scope: the criterion must be evaluated at a genuine parametric minimum (under-converged rounds over-split), the two-layer case makes $\Phi'$ fully local while deeper networks require the local sweep to deliver it inward, and the target here is realizable and noiseless (scoreboard LOCALSPLIT).

Finding the winner the way biology does: local grow, local prune, under a budget.

The developing brain does overproduce and then prune (synaptic overproduction and elimination; activity-dependent neuronal death; neural Darwinism), but it does not find the winner by a global, post-hoc search over an overparameterized network: selection is local (activity-dependent competition for limited trophic resources) and budgeted. We mechanize this gradient-free on the teacher–student task above: grow by the local splitting index, prune by a local death saliency (a neuron's output-power contribution $a_\ell^2\,\mathbb{E}[\sigma_\ell^2]$, a local Optimal-Brain-Damage signal), under a width budget, with the output head solved in closed form (the surviving units re-share the load). Across seeds at minimal width $r{=}6$, local on-demand growth with a disciplined criterion reaches the minimal capacity at roughly $2.7\times$ less total compute (neuron-epochs) than the overparameterize-then-prune caricature, and pruning genuinely reclaims over-provisioned capacity (a loose grower's width $18$ is pulled back to $\sim\!8$). The honest qualifier is that uncontrolled overproduction-plus-pruning churns (it is more expensive than a good growth signal alone), which is exactly why biology bounds overproduction with a structured prior and a metabolic budget rather than overproducing without limit; exact recovery of $r$ additionally needs consolidation (merging redundant near-duplicate units), which a naive per-round merge does not provide (scoreboard NEURAL-DARWINISM).

Conditional computation amortizes capacity (compartments as a gradient-free mixture of experts).

The other half of the brain's efficiency is that it does not run one monolithic network: it holds a modular repertoire and routes inputs to a sparse subset, so capacity is amortized and the ``winner'' is selected per input. This is a mixture of experts, and it is the computational form of our cortical-compartments thesis. We test it gradient-free on a $K$-regime task (each input cluster has its own teacher): $K$ small experts, each with a closed-form Gram readout over locally-trained features, plus a cheap fixed router (unsupervised $k$-means centroid, no trained gate). Across seeds the routed mixture reaches test MSE $0.0030$ at the active-compute of a single expert, versus $0.0198$ for a capacity-matched monolith that pays full compute on every input ($6.6\times$ lower error at $4\times$ less per-input compute) and $0.0348$ for a compute-matched monolith ($11.6\times$ lower). The cheap fixed router matches an oracle router, and—tellingly—it need not recover the true regimes (its agreement with them is only $\sim\!0.63$): the experts specialize to whatever consistent partition the router defines, exactly as cortical areas specialize to the inputs that consistently reach them (scoreboard GF-MOE).

Continual RL and fast online adaptation

The reinforcement-learning examples below preserve an older policy by freezing its trunk and head, rather than by applying a universal Gram-memory theorem to changing Bellman targets. A separate gated error-correcting readout demonstrates online adaptation; its delta write is a gradient update, as noted below.

Continual multi-game RL with zero forgetting and plasticity.

Training a gradient-free local-sweep DQN on Pong and then Breakout, a baseline that fine-tunes the whole network forgets Pong catastrophically (+10.2 $\to$ $-21.0$, the worst score) while learning Breakout (14.5)—the standard continual-RL failure. Freezing a Pong-only conv trunk and adding a Breakout head removes the forgetting (Pong preserved at +10.2) but the single-game features do not transfer, so Breakout barely learns (2.6, near random)—zero forgetting without plasticity. The resolution follows the capacity finding (S[sec:res-capacity]): make the shared trunk general by joint-training it on both games. A single shared substrate then plays both (Pong $19.4$, Breakout $15.4$), and—freezing that general trunk—a fresh Breakout head reaches $18.4$ while Pong is preserved exactly at $19.4$. So a general/over-provisioned shared substrate preserves the older policy while learning a new head on these two already represented games. Both games influenced the jointly trained trunk, so this is not unseen-game feature transfer. Freezing old modules can also be used with gradient-trained systems; this is not a capability exclusive to the present update rule (scoreboard #109–110).

A gated-delta recurrent memory and online concept drift.

Our closed-form Gram memory accumulates outer-product statistics; generalizing it with a forgetting gate $g$ and an error-correcting (delta) write gives a fixed-size recurrent state $S_t = g\,S_{t-1} + \beta\,(v_t - g\,S_{t-1}k_t)\,k_t^\top$ (the gated-delta rule, inspired by recent linear-attention work and the closed-form continuous-time / liquid-network family, whose gated leaky-integrator cell $h' = g\odot(1-\sigma_\tau) + h_{\text{cand}}\odot\sigma_\tau$ is the same primitive arrived at as the analytic solution of a time-constant ODE [hasani2020ltc,hasani2022cfc,li2026cfc]), updated by a local, forward, $O(1)$ recurrence with no backprop-through-time. On associative recall the delta correction adds exactly what plain accumulation lacks—state tracking: when memory slots are repeatedly overwritten, the gated-delta state returns the latest value ($1.0$ at $512$ overwrites) where plain accumulation buries it under the superposition of all past writes ($0.37$); the gate gives controllable recency (scoreboard #111). Used as an online classifier on a CIFAR-100/ViT feature stream whose label mapping drifts every $1500$ steps (Figure [fig:online-drift]), the gated-delta learner reaches $0.948$ overall and—the key result—adapts fastest after each drift (post-drift recovery $0.477$), beating both vanilla online accumulation ($0.314$, which cannot unlearn) and online SGD ($0.809$, which adapts too slowly, post-drift $0.114$). To be precise: all three are online rules on a single linear readout over frozen features—no deep network and no backprop-through-depth in any of them—and the gated-delta write is itself an LMS/delta gradient step, so this is a comparison of online update rules, not gradient-free versus backprop. The point is that the delta rule's one-shot, closed-form error-correction (plus the forgetting gate) overwrites a stale concept in a few examples where softmax-SGD on the same readout crawls and plain accumulation cannot unlearn at all—fast online adaptation to non-stationarity, the foundation for online video/stream understanding (scoreboard #112). (The framework's no-global-backward claim proper is the deep local-sweep feature learning of S[sec:res-vision]/S[sec:res-pc], not this frozen-feature readout test.)

The honest frontier: non-stationary event streams, where the win is adaptation, not in-distribution accuracy.

A disciplined statement of where this substrate is and is not preferable is itself a result. On an event-driven stream (sparse $\mathrm{d}I/\mathrm{d}t$ change-spikes feeding a fixed leaky-memory feature map), we predict a target whose input$\to$output mapping switches mid-stream (regime $A\!\to\!B\!\to\!A$, a $90^\circ$ remap), comparing the gated-delta readout against a classical constant-velocity Kalman filter, online (N)LMS, and an un-decaying closed-form accumulator ($T{=}6000$, $3$ seeds). We are explicit about the boundary our own analysis predicts: in-distribution the fixed Kalman filter is best ($0.0001$ MSE, $\sim\!12\times$ better than ours)—an optimal fixed model is unbeatable when its model is correct, and we do not claim otherwise. But under the mid-stream switch the picture inverts: the Kalman filter breaks ($0.338$, the wrong fixed model), the un-decaying accumulator cannot unlearn, online LMS plateaus $\sim\!17\times$ worse, while the gated-delta readout adapts in $\sim\!14$ steps and retains the original regime on return (per-phase MSE $0.0012$ throughout, retention ratio $1.0$), all gradient-free and $O(1)$ per step. The contribution is the precise delineation: the substrate's genuine niche is online non-stationary streaming—fast local plasticity together with zero-decay retention—rather than beating specialized or optimal models in the stationary regime (scoreboard EVENTSTREAM).

Gated-delta recurrence as load-bearing memory for non-Markovian control.

The same gated-delta state, used as the recurrent node of a control policy and trained entirely without gradients, supplies the memory a partially observed task requires. On velocity-free CartPole—the agent observes only cart position and pole angle, with both velocities hidden, so the problem is non-Markovian and a memoryless policy cannot recover the missing state—a $(1+\lambda)$ neuroevolution loop (scalar return as the only signal; no backpropagation and no backpropagation-through-time anywhere) evolves a policy whose single recurrent register follows $S_t = g\,S_{t-1} + \beta\,(v_t - g\,S_{t-1}k_t)\,k_t^\top$. It reaches a perfect $\mathbf{500/500}$ return on all three seeds, matching a feedforward policy that sees the full observation ($500/500$), while an identically evolved memoryless feedforward policy on the partial observation stalls at the partial-observability ceiling of $41.6/500$—a $12\times$ gap with zero cross-seed variance (scoreboard MOONSHOT4). The recurrent state reconstructs the hidden velocities from the observation history; here the gated-delta recurrence is not an online-readout convenience but the load-bearing computation, and it is acquired with no gradient signal of any kind. The honest scope: CartPole is a classic, short-horizon control task and the topology is a fixed register with evolved weights, gate, and write strength rather than fully mutated recurrent edges; longer observation delays and explicit recurrent-edge growth are the natural next stress tests.

Online control adaptation: when the game changes.

The same online principle transfers from perception to control. A CartPole agent is run with a closed-form world model (ridge on state/action features) refit online with a forgetting gate, plus short-horizon CEM-MPC—all gradient-free; it balances the pole (mean episode length $300$). Mid-stream we reverse the controls (flip the force sign): the world has changed. A frozen model collapses (episode length $7.8$, below the random $\sim$$22$), but the online-refitting model re-learns the reversed dynamics and recovers to $274$—$91\%$ of pre-reversal balance (scoreboard #113). Two ingredients are required, and we state them plainly: fast-enough forgetting (a $\sim$$100$-step memory; a $\sim$$1000$-step memory lingers on the stale dynamics) and a brief burst of exploration after the change (a wrong model makes episodes collapse instantly, starving the refit). Together with the perception result above, the substrate adapts online to both changing inputs and changing dynamics, gradient-free—the foundation for online game-playing and autopilot-style adaptation. The control result is reproducible: across four seeds the online-refit model recovers to $284.8\pm24$ versus the static model's $\sim$$7.9$ (scoreboard #113b). A third leg closes the trilogy at the level of the world model itself: on a linear system whose dynamics regime drifts every $1000$ steps, plain accumulation diverges (the forgetting gate is necessary), and the gated-delta model re-adapts fastest after each regime change while linear least-squares is more precise at steady state—an adaptation-versus-precision tradeoff (scoreboard #114). Across inputs, policy, and dynamics, then, the substrate adapts online to a non-stationary world without a global backward pass.

Online adaptation on a real pixel game.

The control result above is on a near-linear toy (CartPole); we next reproduce it on a real Atari game. A gradient-free local-sweep DQN (Double, $n$-step$=3$, the stable configuration of S[sec:blocks-valuerl]) learns Pong to $+15.9$; we then flip the controls mid-stream (swap up/down before the action reaches the emulator—the game changes). A frozen snapshot of the learned agent collapses to $-21.0$ (its policy is now exactly wrong), while the same agent continuing to local-sweep online re-adapts to $+20.0$, back to near-ceiling under the reversed controls (scoreboard #115). Honestly, re-adaptation is not instant: it is online RL re-learning a reversed policy, crossing from collapse to ceiling at $\sim$$250$k flipped-control steps (noisy around the crossover), not zero-shot transfer. But it is the pixel-game analog of the CartPole result—an agent that plays a real game and re-adapts online when the game changes, where a frozen agent dies. The effect is not Pong-specific: on Breakout (a structurally different paddle-and-bricks game with fire-to-start and multiple lives), flipping the paddle's left/right controls collapses a frozen agent from $8.2$ to $0.1$ (per-life score) while the same agent re-adapting online recovers to $8.8$, back to its pre-flip level (scoreboard #117).

A fully gradient-free online stack.

The perception-drift result (#112) used frozen pretrained ViT features; we close that caveat by replacing them with our own. A local-sweep CNN encoder (no backprop, no pretraining; CIFAR-10 test accuracy $0.86$) is frozen, and its features feed the gated-delta online classifier on the same label-permutation drift stream. The #112 pattern reproduces end-to-end gradient-free: gated-delta dominates post-drift recovery ($0.627$) over online SGD ($0.167$, precise at steady state but slow) and plain accumulation ($0.129$, which collapses without a forgetting gate); overall $0.817$ versus the ViT's $0.948$, the gap reflecting our weaker encoder ($0.86$ versus the pretrained transformer) but with the relative adaptation story identical (scoreboard #116). Nothing in this stack—encoder or readout—used a global backward pass or pretraining.

The same online story in language.

The adaptation principle is not specific to vision or control. We apply the gated-delta memory to a streaming language task in its native domain: enwik8 bytes are predicted one token at a time from a hashed trigram context, and every $100$k bytes the byte identities are re-permuted (the language analog of the label-permutation drift), so the context-to-next-byte map must be re-learned. The finding matches the dynamics result (#114) exactly. First, the forgetting gate is necessary: plain accumulation collapses to near-uniform after each drift ($0.006$ next-byte accuracy, versus $0.0039$ uniform) because it cannot unlearn the stale mapping. Second, among the adapting rules it is an adaptation-versus-precision tradeoff, and we state it without spin: gated-delta re-adapts fastest immediately after each drift (first-$2$k recovery $0.309$ versus single-layer SGD's $0.164$), while SGD reaches higher steady-state accuracy ($0.357$ versus $0.271$). The gate buys fast re-adaptation, not uniformly higher accuracy (scoreboard #118). This is complementary to the static gradient-free char-GPT on enwik8 (Appendix [app:arch-gpt], scoreboard #85/#91): that showed the transformer family trains by local credit assignment; this shows the memory primitive adapting online to a non-stationary text stream. The online-adaptation pattern—gate necessary, gated-delta fastest to re-adapt, single-layer SGD more precise at steady state—now holds consistently across four domains: perception (#112, #116), control (#113), dynamics (#114), and language (#118).

Crushing real 3D first-person games (ViZDoom)

The Atari results (S[sec:res-control]) are 2D arcade games; we close the loop on a genuinely 3D, first-person setting: ViZDoom, the standard 3D-game RL benchmark, learned from the raw $84{\times}84$ first-person screen. We test both OSNR learning mechanisms as the agent's value function, to separate the credit-assignment rule from the value representation:

• Track A --- local-sweep credit assignment. A convolutional value net trained by the per-block local sweep (no global backward pass), the same Double/$n$-step DQN recipe used on Atari.

• Track B --- closed-form/ODE memory. The value function is the gated-delta leaky-integrator memory (the Euler discretization of $\dot S = -\lambda S + \beta (y - S\phi)\phi^{\top}$), trained online by temporal-difference delta writes over features from a frozen, local-sweep-pretrained encoder.

Both are gradient-free (no global backward pass) and OSNR-native; the comparison isolates which OSNR facet does the learning. Results across three scenarios of increasing difficulty (returns, mean of $10$ greedy episodes):

Three findings. (1) Both OSNR facets crush a real 3D game: on basic the local-sweep DQN reaches $+81$ and the ODE value core matches it at $+80$ (scoreboard #120/#124)—the leaky-integrator memory is a viable, stable value function, not just an associative store. (2) The local-sweep DQN solves the hardest standard scenario, deadly\_corridor (traverse a corridor of enemies to an armor vest), reaching $+1524.7$ versus random $-40.4$ with no reward-shaping tricks beyond the scenario's built-in distance reward and no curriculum (scoreboard #125). (3) Stability is the honest caveat for the ODE core: a naive online TD delta rule collapses after peaking (the deadly triad compounded by the forgetting gate decaying a converged memory, scoreboard #122); minibatch-replay delta updates plus a slower gate fully stabilize it on basic (#124) but only partially on the harder defend\_the\_center (peak parity at $10.6$, final drift to $3.2$, #126). The discrete local-sweep value net is the more robust of the two facets; the ODE value core is the more biologically/ODE-grounded and reaches peak parity. Across the three scenarios the ODE core degrades gracefully with difficulty—full parity on basic ($+80$), peak parity on defend\_the\_center ($10.6$), and partial-but-clear progress on deadly\_corridor ($+186$ vs.\ random $-40$, navigating the corridor without completing it, #130)—whereas the deep local-sweep value net fully solves all three. Net: a single gradient-free substrate, in either of its two OSNR forms, learns real first-person 3D games from pixels; the deep value net is the more capable, the ODE memory the more biologically grounded.

Online re-adaptation in a 3D game.

Finally we combine the two themes—3D games and online adaptation. After the local-sweep DQN learns basic ($+83.5$), we flip its controls mid-stream (swap textsc(move_left)/textsc(right) so the agent moves away from the monster). A frozen snapshot collapses to $-253$ (near random), while the agent that keeps local-sweeping online re-adapts to $+84$—full recovery, reached within the first $25$k flipped-control steps and held thereafter (scoreboard #131). With this, gradient-free online re-adaptation to a changed world is demonstrated across every modality in the paper: perception, control, dynamics, and language (#112–118), 2D games (#115/#117), 3D physics (#119/#120), and now a 3D first-person game (#131).

Vision as RL: classification as a contextual bandit.

If games are solved by value learning over pixels, the labeling/vision problem can be cast the same way: state $=$ image, action $=$ predicted class, reward $=1$ if correct else $0$ (the label becomes a reward). We train one local-sweep CNN two ways differing only in feedback: supervised cross-entropy (a dense, directional gradient that knows the answer) versus a one-step bandit (a sparse scalar that only reveals whether the agent's own guess was right, eps-greedy class choice, DQN update on the chosen action). On CIFAR-10 (random $0.10$): supervised reaches $0.900$ and the bandit reaches $0.789$—a gap of only $\approx0.11$ (scoreboard #128). So image classification is genuinely solvable as RL by the same machinery that plays Doom, and the cost of replacing labels with reward-only feedback is modest, not catastrophic, despite the far sparser signal. Vision is, in effect, a one-step RL episode whose action is the class; when only outcome feedback is available, the RL framing is the natural one and it works. Three honest refinements. (i) Recipe, not capacity: doubling network width gives no supervised gain ($0.895$), whereas adding standard augmentation does ($0.900\!\to\!0.913$); the local sweep is not the limiter (it matches backprop) and the path to $\sim$$95\%$ is the training recipe, not more neurons (#128b/c). (ii) An augmentation\,$\times$\,signal-density interaction: the same augmentation that helps the supervised arm ($+1.3$) hurts the bandit ($-3.2$, widening the gap to $0.156$)—each augmented view is a fresh hard one-shot problem for a sparse reward but useful extra signal for a dense gradient. (iii) Reward defines the representation: features learned from a different reward (Doom game score) do not transfer to recognition—a frozen game-reward encoder linear-probes below a random encoder on CIFAR-10 ($0.389$ vs.\ $0.452$, #129). RL solves the vision you reward it for; it does not yield general vision for free.

Training efficiency: where gradient-free is faster (and where it is not)

A first-principles question: is the gradient-free substrate faster to train than backpropagation? We answer it honestly, with controlled measurements on CIFAR-10 (identical architecture, optimizer, data; ms/step is warmup-ed and device-synchronized).

The exact local sweep equals backprop.

The per-block local sweep computes backprop's exact gradient (cosine $\approx 1$), so it reaches identical accuracy ($0.904$ vs.\ $0.902$) at near-parity speed ($14.9$ vs.\ $14.3$ ms/step)—and its best-possible implementation is backprop. It therefore costs essentially nothing but cannot, by construction, beat backprop on a single device (scoreboard S1). Speed must come from elsewhere.

Decoupled local rules: a parallel-hardware advantage.

Rules that break the sequential backward chain (direct feedback alignment, greedy local losses) compute only weight-gradients from a local/projected error—no input-gradient propagation, and the blocks become independent. On a single device this buys little ($\sim$$1.1\times$, from skipping the input-gradient) and costs $9$–$12$ accuracy points. But the critical path collapses: backprop's backward is a sequential sum over blocks, while decoupled updates are a max over blocks. Measured on $N$-device-ideal hardware this is a $2.1\times$ ($4$ blocks), $2.74\times$ ($8$), $3.06\times$ ($16$)—growing with depth, because backprop cannot parallelize its backward lock (scoreboard S2/S3). A hybrid (cheap/parallel early, exact late) recovers most of the accuracy ($0.877$ vs.\ $0.902$); realized as wall-clock only on parallel hardware.

Doing less work: freezing converged blocks.

Once an early block's features converge, freeze it and stop the backward at that boundary. This is a single-device, near-accuracy-preserving speedup: a consistent $\sim$$0.8$-point cost (multi-seed) for $11\%$ less wall-clock at depth $3$, growing to $27\%$ at depth $6$ (scoreboard S4/S4b/S4c). Per-step cost falls as blocks freeze ($14.5\!\to\!5$ ms). The win grows with depth—the regime that matters.

The OSNR-native win: closed-form where the sub-problem is least-squares.

A linear readout is a least-squares problem; the closed-form ridge solve reaches the exact regularized optimum in one solve. With the regularizer tuned it matches cross-entropy SGD accuracy ($0.4389$ vs.\ $0.4396$) at $25$–$30\times$ less wall-clock (a single solve is $\sim$$150\times$), and the incremental version is a rank update with no re-epoching (scoreboard S5). And when the operator itself is structured the win compounds: for a banded (e.g.\ pentadiagonal) Gram the exact solve is $O(d)$ rather than dense $O(d^3)$—bit-identical solution (max difference $\sim$$10^{-13}$), $9796\times$ at $d{=}8192$ and unbounded beyond (dense is infeasible past $d{\sim}8\mathrm{k}$ while the banded solve handles $d{=}262\mathrm{k}$ in $5$ ms, scoreboard S6). This is the one place we are faster without any accuracy trade, and it is exactly ``value scales with specifiable structure'' expressed as wall-clock—the largest, cleanest speedups arrive precisely on OSNR's home turf.

Synthesis.

Gradient-free training is not faster from the exact sweep alone (that equals backprop). Real speed-without-accuracy-loss comes from (i) closed-form solving where structure is specifiable ($25$–$30\times$ dense, $10^4\times$ and unbounded when the operator is banded—the largest and OSNR-native), and on parallel hardware (ii) decoupled/hybrid local rules ($2$–$3\times$, growing with depth); freezing converged blocks adds a single-device $11$–$27\%$ at a small ($\sim$$0.8$pt) cost. The advantages scale with the two things the substrate is built around—specifiable structure and decouplable locality—and with problem size and network depth. The exact local sweep on a generic deep net, by contrast, is simply backpropagation and offers no single-device speedup; the gradient-free substrate is faster exactly where its structure is, not in general.

Bio-inspired generalization: invariance, modularity, and the limits of memorization

The substrate's properties suggest a route to human-like generalization—recognizing objects from any viewpoint from few examples, composing novel combinations, and resisting noise—without the millions of examples a monolithic backprop net needs. We frame and test this from first principles. The frame: backpropagation is the adjoint-state method (Pontryagin) solving approximate dynamic programming with Bellman objectives to first order; our per-block local sweep computes that adjoint locally, and our closed-form/Gram memory is the exact Bellman/DP solve in the linear case. So the substrate is local-Pontryagin (sweep) $+$ exact-Bellman (closed-form memory) $+$ evolution. Three predictions follow, each confirmed (rotating/colored-MNIST, frozen encoder $+$ closed-form ridge probe).

Invariance from temporal continuity beats memorization (D1).

An encoder trained by temporal contrast (positives $=$ adjacent frames of a viewpoint sweep, no labels) learns viewpoint invariance. In the decisive test —labels at one canonical angle, test at all angles—it reaches $0.539$ versus a supervised-on-few encoder's $0.335$ ($+20$ points, $K{=}10$) and nearly matches hard-coded rotation augmentation ($0.589$). Viewpoint generalization from few examples is invariance (learnable unsupervised from temporal continuity), not memorized views.

Independently-trained modules compose to novel combinations (D3).

On colored-MNIST trained over a subset of (digit, color) combinations, two independently-trained invariant modules (a color-invariant digit module and a digit-invariant color module) reach $0.462$ joint accuracy on unseen combinations versus a jointly-trained monolith's $0.159$ ($\sim$$3\times$). The monolith entangles the factors and exploits the colour$\to$digit shortcut, so its digit accuracy on novel combinations collapses to chance ($0.165$); the invariant modules ignore the shortcut and compose.

Decoupled local learning memorizes less (D5).

On a learnable teacher-student task, backprop fits fully random labels to $1.000$ (perfect rote memorization) while decoupled local learning reaches only $0.903$; under $40\%$ label noise, decoupled generalizes better on the clean test ($0.507$ vs.\ $0.464$) while memorizing the noise less. Backprop's apparent edge is partly rote capacity.

Integrated agent and honest limits (APEX).

Combining the levers—an invariance-pretrained encoder plus the closed-form zero-forgetting memory—into one gradient-free agent on a harder two-nuisance benchmark (rotated $+$ colored MNIST), it beats a backprop monolith by $\sim$$2.2\times$ on few-shot ($0.272$ vs.\ $0.124$, $K{=}10$) and $\sim$$2.6\times$ on continual class-incremental learning ($0.475$ vs.\ $0.183$). Honestly, it loses the noise axis here ($0.439$ vs.\ $0.802$): on easy MNIST with ample data backprop is already noise-robust, so the D5 less-memorization advantage—which is task-difficulty-dependent—does not surface, and a frozen encoder with a ridge readout underperforms end-to-end supervision under noise. The synthesis: invariance and order-invariant memory deliver human-like sample-efficiency and lifelong learning in one gradient-free agent; the reduced-memorization benefit needs the harder-task regime. Across all of it the pattern is one thesis—structure and locality generalize; monolithic memorization does not—the same conclusion the efficiency results (S[sec:res-speed]) reach on the speed axis.

Exact systematic generalization where broad models collapse

The iron law of this program—value scales with specifiable structure—has a sharp corollary for competing with broad foundation models. One cannot out-interpolate them on broadband tasks; their home is correlational interpolation at scale and our edge vanishes there. But the law lifts one level, from operators to rules: where a task is generated by an exact per-step rule, a structure-capturing model recovers that rule from a few short examples and extrapolates it to unbounded inputs, whereas a correlational model interpolates within its trained support and collapses beyond it. This is precisely the documented Achilles' heel of transformers and LLMs—systematic/length generalization—turned into a home-axis win.

We test the cleanest instance: multi-digit addition (the exact carry rule). All models train (or are prompted) on length $\le N{=}8$ digits and are tested out to $40$. We compare two structure-capturers—textsc(ours-rnn), a GRU whose recurrence is a length-invariant per-step rule, and textsc(ours-symb), which recovers the finite transition table $f(a,b,\text{carry})\!\to\!\text{digit}$ from short examples and applies it recurrently (a closed-form discrete analogue of operator recovery)—against a from-scratch transformer (learned positions) and, as the broad-SOTA baseline, a real instruction-tuned LLM (llama3.3:70b) prompted few-shot at length $8$. The transformer is a fair, non-strawman baseline: it reaches $0.64$–$0.67$ exact-match within its trained support.

noindent Both structure-capturers hit $1.00$ exact-match at every length out to $40$ ($5\times$ beyond the trained $N{=}8$, $3$ seeds), while both correlational models—the from-scratch transformer and the $70$-billion-parameter LLM—hold near their few-shot support and then collapse toward $0$ as inputs lengthen. The win is exactly scoped: this is systematic generalization of an algorithmic rule, not broad knowledge. The LLM retains everything it knows; it simply cannot execute exact long-carry arithmetic, which is the documented weakness and exactly where a rule-capturer wins by construction. This is our OSNR/known-operator thesis promoted from signals and operators to rules and programs, and it is the bio-inspired core of productivity and systematicity: capture the generative mechanism from few examples, then compose and extrapolate it without bound.

A natural objection is that a reasoning model with chain-of-thought might execute the carry rule step by step. We test this with deepseek-r1:70b (same few-shot prompt, greedy): exact-match falls from $0.80$ at length $8$ to $0.60$ at length $16$, and at length $24$ single chain-of-thought generations exceed a $20$-minute per-query budget and the evaluation becomes intractable on our hardware. Chain-of-thought therefore does not escape the failure mode—it trades the hard collapse for an exploding inference cost while accuracy still degrades—whereas the rule-capturers are exact, instant, and linear in length at every length tested. Capturing the generative rule, not scaling correlational compute, is what yields exact extrapolation.

The effect is not specific to addition.

To show this is a property of per-step rules and not a single cherry-picked task, we repeat the protocol (train length $\le 8$, test to $40$, $3$ seeds) on four algorithmic tasks, each an exact finite-state rule read left to right: textsc(copy) ($y_t{=}x_t$), textsc(sum-mod-10) (running sum mod $10$), textsc(parity) (running XOR), and textsc(running-max). A generic GRU recurrence (same architecture for all tasks, no task knowledge) is compared to the from-scratch transformer.

noindent The transformer collapses past its trained support on all four tasks; the recurrence holds $\approx\!1.00$ at every length out to $40$ on three of the four. The honest exception is textsc(sum-mod-10), where the GRU also degrades ($1.00$ at length $8$ to $0.49$ at length $40$): maintaining an exact ten-state modular accumulator over many steps stresses a continuous hidden state, which drifts. It still vastly outperforms the transformer (which never learns the accumulator out-of-distribution), and the failure is precisely diagnostic—an explicit finite-state/symbolic capturer (as in the addition carry table above) is exact at any length on all four tasks, since each is a finite-state machine. A generic learned recurrence captures most per-step rules and extrapolates them; exact long-range discrete state is the regime where the explicit symbolic form is strictly stronger.

Augment, don't replace: a frozen LLM plus an exact module.

The practical lesson is not to compete with a broad model but to repair its systematic-generalization failure while keeping its strengths—the same pattern as the conv-backbone-plus-structured-head and learned-prior-plus-closed-form-operator results elsewhere in this paper. We pair a frozen real LLM (llama3.3:70b, the ``cortex'': language understanding) with an exact structured arithmetic module (the ``executor''), on natural-language arithmetic word problems that require both understanding diverse phrasings (six wordings each for $+$, $-$, $\times$) and exact arithmetic on long operands. We compare the bare LLM (reads the problem, emits the number), a no-LLM keyword parser feeding the exact module, and the hybrid (the LLM classifies only the operation—a short, length-invariant output—and a deterministic extractor plus the exact module compute the result).

noindent The hybrid reaches $1.00$ end-to-end at every operand length and beats both baselines because it needs and uses both parts: the LLM classifies the operation perfectly across all six wordings ($1.00$, and length-invariant since the output is a single token), and the module performs the arithmetic exactly. Neither alone suffices—the bare LLM collapses as operands lengthen ($0.40\!\to\!0.00$), and the no-LLM parser stays mediocre at all lengths ($0.50$–$0.65$) because it misclassifies indirect phrasings (``reduced by'', ``scaled by a factor of''). This is the ``augment, don't replace'' thesis made concrete: keep a frozen broad model's semantics, attach a structured module for the exact rule it cannot execute, and its systematic-generalization failure is fixed without retraining. The task is deliberately controlled (canonical operand order so extraction is unambiguous; the LLM's role is operation classification) to isolate the principle—route the structured subproblem to an exact module—rather than to claim a turnkey solver.

The operator is the lever: matched and inferred state-transitions in a state-space layer

State-space models (S4, Mamba) and continuous-time ``liquid'' networks (LTC, CfC) are the leading efficient alternative to attention, and their power is known to sit in the state-transition operator: S4's headline ablation raises sequential-MNIST from $60\%$ to $98\%$ purely by initialising the state matrix to the prescribed HiPPO operator, the paper noting the gain ``comes primarily from the initialisation, not the learned parameterisation.'' This is our iron law inside the state-space family—a known structured operator carries the performance. We ask the natural OSNR question: in a state-space layer, can a structure-matched or closed-form-inferred operator beat both a freely-learned operator and the generic HiPPO prior? Inferring the operator is the substrate analogue of our system-identification results (S[sec:res-control])—identifying the dynamics from data rather than learning them by gradient descent.

We isolate the operator in a diagonal complex state-space layer: every arm shares the same fixed random input projection and the same closed-form ridge (Gram) readout; only the pole vector (the operator $A$) differs. The arms are textsc(learned) (poles trained by Adam), textsc(hippo) (the S4D-Lin generic prior), textsc(matched) (the true modes—an oracle ceiling), and textsc(inferred) (poles estimated by closed-form Hankel–DMD system identification, with no oracle and no gradient, stabilised to the unit disk). The task has known ground truth: a single-input single-output linear dynamical system (a bank of six damped oscillators) whose response to white input must be predicted; out-of-distribution is the same system run for twice the sequence length, and we sweep the broadband (white-noise) fraction of the signal. Normalised MSE, three seeds:

noindent Five findings. (i) In-distribution, a freely-learned operator fits as well as the oracle—learning the operator is easy when train and test lengths match. (ii) Out-of-distribution length is the discriminator: the learned operator overfits the training horizon (it parks poles near the stability boundary to maximise in-distribution fit) and fails to extrapolate, degrading to $0.57$ and, as broadband noise grows, blowing up to $8.3$, $10$, $174$; the matched and inferred operators stay at the in-distribution floor, length-invariant by construction. (iii) Inferring the operator works: the closed-form-DMD operator—no oracle, no gradient—tracks the oracle (OOD $0.046$ vs $0.003$) and beats the learned operator out-of-distribution by one to three orders of magnitude, recovering the true modes to mean distance $0.045$–$0.12$. (iv) It beats the generic HiPPO prior in the structured regime ($0.003$ vs $0.22$ in-distribution); HiPPO is stable out-of-distribution (its poles are properly damped) but mediocre (wrong modes), whereas matched/inferred obtain both accuracy and stability. (v) The iron law governs the boundary: as the broadband fraction grows the arms converge ($0.003\!\to\!0.90$ in-distribution) and the advantage vanishes—matched structure cannot beat broadband noise, exactly the sparse-stochastic-process prediction. The scope is honest—a controlled linear single-input system, with a bare-Adam learned arm (production S4/Mamba add the structured initialisation and stability parameterisation that place them nearer our textsc(hippo) arm). The lesson generalises: a freely-learned state-transition overfits the horizon, while a structure-matched or closed-form-inferred operator is length-invariant and beats the generic prior wherever the dynamics are identifiable—the operator, not the surrounding learned machinery, is the lever.

Nonlinear systems: where the operator helps, and where it does not.

Moving off the linear toy onto nonlinear dynamics maps the boundary honestly. On a Wiener system (linear dynamics followed by a static output nonlinearity) a gradient-free hybrid—the closed-form-inferred linear operator with a closed-form nonlinear (random-Fourier-feature) readout—roughly halves the error of the linear-readout arms at every nonlinearity level, in and out of distribution (for example normalised MSE $0.225$ vs.\ $0.310$ at strong saturation): the inferred operator captures the dynamics and the closed-form nonlinear readout inverts the static nonlinearity, with no gradient anywhere. The advantage is regime-specific, not universal. On a Duffing oscillator, where the nonlinearity sits in the state feedback, the operator-inference advantage vanishes—a freely-learned operator already nails the two modes and is stable out-of-distribution, and a static-readout hybrid cannot represent feedback nonlinearity—and on a biophysical Morris–Lecar neuron with a hidden recovery variable, all arms are weak (the partial observability defeats single-output identification and calls for latent augmentation). The operator-stability win of the previous paragraph is likewise strongest for rich, multi-mode, unsaturated dynamics and shrinks when a single learned mode pair suffices. The consistent thread is the iron law one level deeper: a structure-matched or inferred operator, optionally with a closed-form nonlinear readout, wins exactly to the extent the structure is present and identifiable, and ties or loses where it is not.

The inferred operator as a sparse router.

A separate controlled test asks whether operator identification can reduce computation rather than only improve prediction. On noisy damped oscillators with off-grid impulses, a robust two-state flow fit followed by a fixed unlabelled median/MAD residual threshold reaches event F1 $0.9994$ at 30–40 dB while activating only the approximately $5.5\%$ event intervals (12 seeds). A finite-difference gate reaches only about $0.20$ F1. The value/velocity jet also localizes events to $0.0425$ and $0.1249$ of a sample at 40 and 30 dB, respectively. The boundary is sharp: at 20 dB detection remains useful (F1 $0.8553$), but timing error is $0.2830$ samples, worse than midpoint guessing. On an independently integrated 1D advection–diffusion field, correcting only the largest $1\%$ of an inferred-operator residual gives nRMSE $0.0011$, versus $0.0813$ for a sparsified raw temporal difference and $0.0036$ for an unconstrained learned spectral transition (eight seeds). Strong cubic reaction degrades the linear operator to $0.0045$; a tiny closed-form local mismatch model restores $0.0017$. Thus the useful primitive is not a particular network family but an analyze--annihilate--route--reconstruct factorization: preserve identifiable dynamics and allocate flexible computation to sparse innovations. In the closed-loop extension, the receiver feeds its reconstruction into the next prediction for 139 transitions. At about $0.21\%$ of a dense float32 field per transition, one off-grid cardinal packet gives trajectory nRMSE $0.00965$, versus $0.01713$ for a point packet and $0.03971$ for a learned-spectral point codec; the spline wins all eight paired seeds. Under strong cubic feedback the pure operator is a negative ($0.12662$), while the local mismatch model plus cardinal packet restores $0.02286$.

The public-data test freezes the released PDEBench Test-17 FNO predictions and encodes their residuals on 3,900 held-out forecast fields. Eight multiscale cardinal packets use fewer bits than nine DCT packets but reach field/Frobenius nRMSE $0.001460/0.000390$, versus $0.001941/0.000500$ for DCT. The paired sample-error reduction is $10.1\%$ (bootstrap $95\%$ interval $4.3$–$15.6\%$). One-scale cubic splines are weak: hierarchy is essential. Batched FFT correlations and Gram solves reproduce reference OMP accuracy $3.5\times$ faster, although they remain slower than DCT. The public result is not a one-dataset effect: with no retuning on PDEBench Test 19, the same eight packets reach field/Frobenius nRMSE $0.012304/0.004138$, versus $0.018944/0.006492$ for nine DCT packets, a $32.5\%$ paired reduction (bootstrap $95\%$ interval $30.3$–$34.8\%$) with $100/100$ sample wins. These public results are target-time residual coding/assimilation, not blind forecast improvement; the mechanical timing claim still assumes a full value/velocity jet.

Stronger controls and a packet-channel audit delimit the effect. An exact orthonormal Haar basis and a channel-specific PCA/KLT basis learned only on samples 0–899 are frozen before the 100-sample test split. With eight spline versus nine control packets, paired reductions on Test 17 are $14.0\%$ versus Haar and $6.8\%$ versus PCA; on Test 19 they are $17.9\%$ and $32.3\%$. Giving every control ten packets, adding an explicit 16-bit block scale, and quantizing amplitudes to four bits makes payload exactly equal ($1.660\%$): the spline reduces paired error versus PCA by $7.4\%$ (95% interval $3.0$–$11.5\%$) on Test 17 and $30.3\%$ ($28.2$–$32.5\%$) on Test 19. Test 19 remains positive at two bits ($13.4\%$), 10-dB coefficient SNR ($23.6\%$), and 20% packet loss ($18.7\%$), winning all 100 paired samples. The honest boundary is Test 17: its clean and eight-bit intervals against the stronger ten-packet PCA cross zero, as do all transform comparisons at two bits. The evidence is therefore for regime-dependent matched residual geometry, not universal codec superiority; predicted rather than target-time residuals remain the next gate.

The predicted-residual gate is mixed and spline-negative. A causal follow-up uses only previous residuals and selects a cross-channel spectral AR order and ridge on samples 800–899. Test 17 selects order four: delayed spline packets improve paired sample error by $18.25\%$ over no correction, but DCT and PCA are slightly better ($0.51\%$ and $0.64\%$). Test 19 selects order one; spline packets worsen no correction by $0.30\%$ (95% interval $-0.52$–$-0.08\%$) and lose to all three transforms. Thus spatial residual compressibility does not by itself confer temporal predictability; the public spline advantage remains an assimilation result, not a forecast result.

The public 2D extension gives a matched win and a transfer boundary. On $128\times128$ Test-26 FNO residuals, eight tensor-product multiscale cardinal packets use $0.0748\%$ dense payload and reach sample nRMSE $0.001850$, versus $0.001889$ for nine training-only separable-KLT packets at $0.0790\%$: a $2.08\%$ paired reduction (95% interval $0.55$–$3.60\%$). Spline also beats point, 2D-DCT, exact 2D-Haar, and one-scale cardinal controls on all 20 samples. Without retuning on Test 27, it still beats the fixed transforms but loses to learned KLT ($0.001562$ versus $0.001462$). Hence multiscale cardinal geometry can beat a learned covariance basis in a matched 2D regime, but it is not universally optimal.