Operator-native activation compression: prototype documentation

# Operator-Native Neural Activation Compression (Prototype)

This repo implements a falsifiable benchmark for the hypothesis that transformer KV cache trajectories can be compressed as low-dimensional operator trajectories instead of dense tensors.

## Hypothesis

For a temporal activation component, if

`x[t+K] + h[K-1]x[t+K-1] + ... + h[0]x[t] = 0`

then

`x[t] = sum_k c_k z_k^t`.

The roots `z_k` are operator poles. Compression is possible when `K << T`.

For vector activations, we use a low-rank spatial basis plus operator temporal basis:

`X ≈ A(t) B^T`,

where temporal columns of `A(t)` are modeled by exponentials (Prony/DMD-style).

## What is tested

Given `K[layer, head, token, dim]` and `V[layer, head, token, dim]` from `gpt2`, we compress along token dimension `T` and evaluate:

1. Reconstruction error `||X - X_hat||_F / ||X||_F`
2. Attention-output error `||Attn(Q,K,V)-Attn(Q,K_hat,V_hat)||_F / ||Attn(Q,K,V)||_F`
3. Attention-output cosine similarity
4. Memory estimate and compression ratio

This is empirical falsification. It does not assume global transformer linearity.

## Methods

- `fp16` dense baseline (memory reference)
- `int8` / `int4` uniform quantization
- truncated `svd`
- exact `dmd` over token trajectory (full or sliding-window via `--window-size`)
- `hybrid` (SVD rank-`r` + Prony temporal modeling with `K` poles)
- `poly` (low-rank + polynomial temporal basis)
- `exp_poly_fixed` (low-rank + fixed stable exponential-polynomial basis)
- `piecewise_poly` / `piecewise_exp_poly_fixed` (sliding-window temporal models)
- `learned_exp_poly` (experimental fallback to stabilized exp-poly path)

## Experiment 2: Higher-order E-spline / Exponential-Polynomial Temporal Models

Experiment 1 tested pure modal trajectories (`sum_k c_k z_k^t`) and found global DMD too narrow/unstable on raw KV trajectories.

Experiment 2 tests higher-order operator null-space structure:

`x[t] = sum_k sum_{m=0}^{M_k-1} c_{k,m} t^m z_k^t`

Repeated roots create `t^m z_k^t` terms. Polynomial trends are the special case `z=1` with repeated roots. This is closer to E-spline / FRI theory than pure first-order modal fitting, and piecewise windows test local operator structure instead of a single global recurrence.

## Experiment 3: Graph Spectral Geometry Compression

Pivot hypothesis: dominant structure may be geometric (attention-graph spectral) instead of purely temporal.

For each layer/head, define graph from attention probabilities (`tokens=nodes`, `attention=edges`), build Laplacian, compute eigenvectors, project `X` to graph Fourier basis, and compress in spectral coordinates.

Methods:
- `graph_spectral_topk` (retain low-frequency or high-energy spectral modes)
- `graph_spectral_lowrank` (spectral basis + low-rank coefficients)
- `graph_heat_diffusion` (heat-kernel filtered spectral representation)
- `graph_wavelet` (simple spectral wavelet kernel)
- `graph_local_spectral` (sparse local-neighbor attention graph + spectral top-k)

Key CLI options:
- `--graph-type directed|symmetric`
- `--laplacian-type combinatorial|normalized`
- `--max-eigenmodes`
- `--spectral-mode-selection lowfreq|energy`
- `--heat-times`
- `--local-topk-neighbors`

## Install

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## Run

```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 256 \
  --methods int8,int4,svd,poly,exp_poly_fixed \
  --ranks 2,4,8,16,32 \
  --poly-degrees 1,2,3,4,6 \
  --num-poles 2,4,8 \
  --poly-orders 0,1,2 \
  --factor-dtype fp16 \
  --output results/gpt2_kv.csv
```

Piecewise:

```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 512 \
  --methods svd,poly,piecewise_poly,exp_poly_fixed,piecewise_exp_poly_fixed,int8,int4 \
  --ranks 4,8,16,32 \
  --window-size 64 \
  --window-overlap 0.5 \
  --poly-degrees 1,2,3,4 \
  --num-poles 2,4,8 \
  --poly-orders 0,1,2 \
  --factor-dtype fp16 \
  --output results/gpt2_kv_512_spline_window64.csv
```

Graph spectral compression:

```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 256 \
  --methods int8,int4,svd,graph_spectral_topk,graph_spectral_lowrank \
  --graph-type symmetric \
  --laplacian-type normalized \
  --max-eigenmodes 64 \
  --spectral-mode-selection lowfreq \
  --ranks 4,8,16,32 \
  --local-files-only \
  --output results/gpt2_graph_spectral.csv
```

Local graph spectral:

```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 512 \
  --methods graph_local_spectral,graph_spectral_lowrank,svd,int8 \
  --window-size 64 \
  --max-eigenmodes 32 \
  --graph-type symmetric \
  --laplacian-type normalized \
  --local-files-only \
  --output results/gpt2_graph_local.csv
```

## Experiment 4: Rich Attention Operators (Multihead, Directed, Diffusion)

New operator probes:
- `graph_multihead_lowrank` with `--multihead-mode mean,entropy_weighted,learned_oracle`
- `graph_directed_transport` with `--directed-basis eig,schur` and `--transport-operator P,Lrw`
- `graph_heat_multiscale`
- `graph_diffusion_wavelet`

Fairness controls:
- `--basis-storage counted|free` (default `counted`)
- `--factor-dtype fp16|fp32`
- complex transport modes counted as two reals (`--count-complex-as-two-reals true`)

Run 1: multi-head geometry
```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 256 \
  --methods svd,int8,int4,graph_spectral_lowrank,graph_multihead_lowrank \
  --multihead-mode mean,entropy_weighted \
  --graph-type symmetric \
  --laplacian-type normalized \
  --max-eigenmodes 64 \
  --spectral-mode-selection lowfreq,energy,mixed \
  --ranks 4,8,16,32 \
  --factor-dtype fp16 \
  --basis-storage counted \
  --local-files-only \
  --output results/gpt2_graph_multihead.csv
```

Run 2: directed transport
```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 256 \
  --methods svd,int8,graph_directed_transport \
  --directed-basis schur,eig \
  --transport-operator P,Lrw \
  --spectral-mode-selection lowfreq,energy,mixed \
  --max-eigenmodes 16,32,64 \
  --ranks 4,8,16,32 \
  --factor-dtype fp16 \
  --basis-storage counted \
  --local-files-only \
  --output results/gpt2_graph_directed.csv
```

Run 3: heat/diffusion wavelets
```bash
python src/run_benchmark.py \
  --model gpt2 \
  --max-tokens 256 \
  --methods svd,int8,graph_heat_multiscale,graph_diffusion_wavelet \
  --heat-times 0.01,0.03,0.1,0.3,1.0,3.0 \
  --max-eigenmodes 16,32,64 \
  --ranks 4,8,16,32 \
  --factor-dtype fp16 \
  --basis-storage counted \
  --local-files-only \
  --output results/gpt2_graph_heat.csv
```

Layerwise analysis:
```bash
python src/analyze_layer_geometry.py --input results/gpt2_graph_multihead.csv --output-dir results/layer_geometry
```

Head clustering:
```bash
python src/cluster_heads.py --input results/gpt2_graph_multihead.csv --output-dir results/head_clusters
```

Plot:

```bash
python src/plot_results.py --input results/gpt2_kv.csv --outdir results/figures
```

## Interpretation

Primary graph: compression ratio vs attention-output error.

- Weak signal: hybrid beats SVD at same memory on some layers/heads.
- Strong signal: >10x compression with <5% attention error.
- Very strong signal: >50x with negligible attention error or preserved generation metrics.

Potential falsification: DMD/hybrid do not improve over SVD/quantization at matched memory across most layers/heads and prompt regimes.

## References

1. Vetterli, Marziliano, Blu (FRI, 2002): https://ieeexplore.ieee.org/document/1003065
2. Unser & Blu publications: https://bigwww.epfl.ch/publications/
3. Generalized Sampling (2005): https://bigwww.epfl.ch/publications/unser0501.pdf
4. DMD (Schmid, 2010): https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/dynamic-mode-decomposition-of-numerical-and-experimental-data/AA4C763B525515AD4521A6CC5E10DBD4
5. Koopman/DMD overview: https://hal.science/hal-04568027/document
6. Palu (KV low-rank): https://arxiv.org/abs/2407.21118
7. KIVI (2-bit KV quant): https://arxiv.org/abs/2402.02750