Model architecture
Build a patch transformer—and make every pixel count
A TensorFlow/Keras classifier turns a narrow chart image into tokens. Padding, residual paths and head dimensions determine the model you actually built.
ImplementationA new branch of Research & Algorithms
Machine learning.
Down to the
implementation.
The architecture, the tooling, the custom code—and the detail that makes it work.
Hands-on engineering from my project archive. From patch transformers and evolutionary policies to streaming sensors and compact model memory.
THE IMPLEMENTATION STACK
01 / Representation
What reaches the model?
02 / Architecture
Where does information flow?
03 / Learning system
Who owns the update?
04 / Evidence
What did the implementation establish?
61 source-grounded case studies
Original code excerpts · Inspectable module maps
Start here
Inside the implementation
Training systems
Take control of PyTorch’s reverse pass, one block at a time
A CNN DQN explicitly propagates vector–Jacobian products across detached blocks. This is custom gradient orchestration—not an autograd replacement.
Implementation
Representation learning
Add classes by accumulating statistics over frozen features
A pretrained representation, random Fourier features and a ridge solve form a simple continual readout. Its guarantees stop at the fixed feature map.
Implementation
Sensing & deployment
Keep the audio callback small; estimate the motion elsewhere
An acoustic prototype combines a high-frequency carrier, queued microphone blocks and a three-sample sinusoidal identity. It is not a validated breathing monitor.
Implementation prototype
Evolution & control
Evolve a population of networks with one vectorized evaluator
JAX handles the population axis, Flax defines the policy and EvoJAX updates a search distribution. The interfaces—not just the optimizer—make the experiment work.
Implementation
Compression
Four-bit arithmetic is not yet a four-bit representation
A tiny quantizer makes the distinction between simulated distortion, resident array storage and a real packed payload impossible to miss.
Implementation auditConnected ideas, practical routes
Choose a reading path
Build and port neural architectures
From image patches to relational attention—and the tensor contracts between them.
Own the learning loop
Population axes, replay buffers, gradient routing and reproducible checkpoints.
Make representations earn their keep
Contrastive learning, distillation and fixed-feature memory, with the controls that changed the interpretation.
Take models into a system
Causal data, bounded memory, live sensing and deliberately small services.
- When does an online bar actually become observable?
- Idempotent event handling starts with the right identity
- A compressed memory needs an answer budget as well as a byte budget
- Keep the audio callback small; estimate the motion elsewhere
- A ring buffer is a concurrency protocol, not just an array
- Make a read-only service small by construction
The complete engineering library
Find your next implementation.
61 articles
Model architecture
Build a patch transformer—and make every pixel count
A TensorFlow/Keras classifier turns a narrow chart image into tokens. Padding, residual paths and head dimensions determine the model you actually built.
Implementation
Model architecture
Porting a transformer is more than translating layer names
The PyTorch version makes the training loop explicit—and exposes three architectural differences that a line-by-line port can hide.
Implementation
Model architecture
One encoder, many entities: building pairwise attention
A shared CNN, identity embeddings and an explicit pair constructor turn a set of images into relation tokens. The interesting engineering decision is where the quadratic expansion happens.
Implementation
Model architecture
Building graph attention from node and edge signals
Separate temporal encoders feed a custom graph-attention layer. A fourteen-line excerpt shows why sparse topology must survive the softmax.
Implementation audit
Model architecture
A padded sequence needs more than zero-valued inputs
A three-layer LSTM classifier illustrates the difference between masking a measurement and masking a recurrent state transition.
Implementation audit
Evaluation practice
Give a classifier an abstain option—and account for it
A probability-to-decision layer separates model confidence from action. Its denominator is as important as its threshold.
Evaluation
Evaluation practice
A reusable ML pipeline starts before the estimator
Scaling and PCA are learned components too. This archive example gets the fit/transform boundary right while showing why a date filter is not automatically a forward test.
Implementation
Evaluation practice
When your test set quietly becomes a validation set
A compact model-selection loop makes an important boundary visible: the data that chooses the winner cannot also provide its untouched final score.
Evaluation audit
Data systems
Turning a time series into an image is a modeling decision
Before a CNN sees a chart, plotting code has chosen the context, coordinate system and information budget. Treat that renderer as part of the model.
ImplementationEvolution & control
Evolve a population of networks with one vectorized evaluator
JAX handles the population axis, Flax defines the policy and EvoJAX updates a search distribution. The interfaces—not just the optimizer—make the experiment work.
Implementation
Data systems
Build a foveal time-series state without borrowing the future
A multi-resolution feature builder combines fast detail and slower context. Its aggregation boundary reveals a subtle source of look-ahead.
Implementation audit
Evolution & control
Keep an evolving network and its feature schema in sync
A NEAT training notebook shows why a genome checkpoint is only one part of a reproducible policy.
Implementation
Evaluation practice
An OOD notebook needs an explicit distribution boundary
Loading a trained genome is straightforward. Establishing that its evaluation is genuinely out of distribution takes a separate evidence trail.
Evaluation
Evolution & control
Add prioritized replay without changing what a transition means
A recurrent DQN prototype adds priority sampling and importance weights. The replay buffer becomes part of the learning algorithm, not just storage.
Implementation
Evolution & control
Separate the three clocks in a recurrent DQN
Environment steps, replay updates and target synchronization are different events. A small PyTorch agent makes their roles visible.
Implementation
Evolution & control
Tensorized neuroevolution starts with an upstream dependency map
TensorNEAT is third-party software, not an original model from this archive. Its package metadata provides a useful map of the stack a local adaptation must respect.
Upstream reference
Evolution & control
Represent variable neural graphs with fixed-size tensors
A PyTorch adaptation stores nodes and connections in bounded arrays. The central challenge is preserving graph semantics while making population evaluation regular.
Implementation prototype
Evolution & control
Bridge an evolutionary optimizer and a Flax policy
A nine-line flatten/unflatten adapter is the hinge between a structured neural network and a population optimizer.
Implementation
Data systems
Keep the policy snapshot separate from the future tape
An episode dataset has two time directions: observations describe what was known; the future tape measures what happened next.
Implementation
Training systems
Put the state machine inside Numba, not the experiment design
A compiled simulation kernel can accelerate repeated evaluations. It cannot turn an in-sample grid search into walk-forward validation.
Implementation
Evaluation practice
Aggregate a strategy evaluation without losing the quiet days
Session-level accounting, zero-action days and regime labels belong in the result—not just the winning events.
Evaluation
Evaluation practice
Ten thousand bootstrap draws are still twenty observed days
A small resampling notebook illustrates the difference between Monte Carlo precision and the amount of independent evidence.
Evaluation audit
Training systems
Parallel hyperparameter search needs one experiment ledger
Optuna workers can share a study, but they must also share a frozen objective, data identity and resource budget.
Implementation
Data systems
A data callback is not a durable dataset
A scheduled market-data recorder separates callbacks, buffers and file writes. Each boundary has a different failure mode.
Implementation
Data systems
When does an online bar actually become observable?
A timestamp-bucket aggregator closes a bar when the next bucket arrives—not merely when a clock crosses its nominal boundary.
Implementation
Data systems
Idempotent event handling starts with the right identity
Typed deduplication keys prevent repeated messages from becoming repeated state changes. A bounded cache is useful—but it is not exactly-once delivery.
Implementation
Representation learning
Learn an invariant encoder from neighboring views
A PyTorch contrastive experiment asks which changes an embedding should ignore before fitting a small supervised readout.
Implementation
Evaluation practice
Evaluate invariance where the labeled examples are scarce
The support distribution and test transformations matter as much as the encoder. A saved few-shot study makes that distinction concrete.
Evaluation
Representation learning
Train one convolutional block at a time
Greedy local contrastive learning freezes earlier blocks and gives the current block its own projection head. That is different from splitting an ordinary backward pass.
Implementation
Evaluation practice
A local learning rule still has to beat random features
A saved CIFAR-10 comparison turns a disappointing result into a useful engineering decision: isolate the representation before expanding the training recipe.
Negative result
Representation learning
Distill a representation, not just a class prediction
A compact CNN learns against teacher features with an optional classification loss. The feature interface is the real contract between teacher and student.
Implementation
Evaluation practice
A smaller student is not automatically a compressed teacher
Parameter accounting and capability retention are separate tests. A two-run distillation record makes both visible.
Negative resultRepresentation learning
Add classes by accumulating statistics over frozen features
A pretrained representation, random Fourier features and a ridge solve form a simple continual readout. Its guarantees stop at the fixed feature map.
Implementation
Evaluation practice
Match the joint readout without claiming universal zero forgetting
The saved DINOv2-feature experiment supports a precise result: additive statistics recover the same fixed-feature fit. It does not freeze every earlier prediction.
EvaluationTraining systems
Take control of PyTorch’s reverse pass, one block at a time
A CNN DQN explicitly propagates vector–Jacobian products across detached blocks. This is custom gradient orchestration—not an autograd replacement.
Implementation
Evaluation practice
A nine-second RL run can validate plumbing, not competence
A Pong smoke test completes on MPS but does not learn a useful policy. Its action distribution is more informative than its runtime headline.
Negative result
Evolution & control
Put a learned actuator model between policy and physics
A SAC experiment separates the controller, actuator law and simulated world. That modularity makes model transfer testable.
Implementation
Training systems
Make each checkpoint an experiment receipt
Weights, validation traces, resource measurements and an immutable-world check turn a checkpoint into something another engineer can inspect.
Implementation
Evaluation practice
Prediction gains do not automatically become control gains
A multi-arm locomotion gate tests whether an improved model enables a useful policy—not just whether a fitted curve looks better.
Negative result
Scientific ML
Infer a state-space model’s poles before fitting its readout
Delay embeddings and a reduced eigensystem provide one route from observed dynamics to a compact recurrent feature bank.
Implementation
Evaluation practice
Stress a dynamical model along two independent axes
Noise robustness and length generalization are different questions. A saved state-space panel keeps both visible.
Evaluation
Scientific ML
Build a sinusoidal coordinate network with the right initialization
A SIREN baseline maps coordinates to a field. Its frequency scale and weight initialization are part of the architecture—not incidental optimizer settings.
Implementation
Scientific ML
Differentiate a field four times without losing the graph
A classical PINN baseline exposes the cost of high-order residuals and the distinction between an analytical derivative and an automatically differentiated network.
Implementation
Scientific ML
Package an operator-aware basis as a PyTorch layer
Local trigonometric atoms and global null-space functions can share one feature interface. Their derivatives do not share one universal shortcut.
Implementation audit
Training systems
A research project needs a current state, not just a long history
A project-state ledger connects code, evidence and closed decisions so an old experimental queue does not silently become today’s plan.
Engineering practice
Evaluation practice
Turn an overbroad memory claim into a regression test
Retaining sufficient statistics is not the same as preserving old predictions. An eleven-line counterexample keeps that distinction executable.
Engineering practice
Compression
A compressed memory needs an answer budget as well as a byte budget
A bounded streaming capsule keeps a small state, but its numerical assurance can fail before memory runs out.
Evaluation
Compression
Collect a transformer’s internal state at the correct boundary
A KV-cache experiment needs more than hidden-state tensors. Queries, keys and values must come from the same actual attention computation.
Implementation audit
Compression
Design a KV-compression benchmark before designing a codec
Reconstruction error, attention error and memory accounting test different promises. A benchmark should retain all three.
Engineering practiceCompression
Four-bit arithmetic is not yet a four-bit representation
A tiny quantizer makes the distinction between simulated distortion, resident array storage and a real packed payload impossible to miss.
Implementation audit
Compression
Evaluate compression at the operation that consumes it
Attention-output error is more task-connected than tensor error—but only if the proxy matches the attention semantics you intend to preserve.
Implementation audit
Evaluation practice
Nineteen thousand metric rows are not nineteen thousand experiments
A head-by-head compression table needs hierarchy-aware aggregation before it can support a model-level claim.
EvaluationSensing & deployment
Keep the audio callback small; estimate the motion elsewhere
An acoustic prototype combines a high-frequency carrier, queued microphone blocks and a three-sample sinusoidal identity. It is not a validated breathing monitor.
Implementation prototype
Evaluation practice
A hundred updates per second is not a hundred independent measurements
A sensing report becomes more useful when it separates sampling rate, window length, hop size, queue delay and physiological validation.
Engineering practice
Sensing & deployment
Design a sensor data contract before training a sensor model
The hardware roadmap is a plan, not a completed multimodal experiment. Its concrete starting point is a timestamped observation packet.
Engineering practice
Sensing & deployment
A ring buffer is a concurrency protocol, not just an array
An ESP32 CSI prototype reveals three low-level issues that can quietly invalidate a downstream ML dataset: wraparound, ownership and clock width.
Implementation audit
Sensing & deployment
Carry the filter state across speech frames
A source–filter speech experiment shows why continuity is an engineering property of the whole decoder, not just a good fit inside each frame.
Implementation prototype
Sensing & deployment
Verify a resource limit where the process actually runs
A container operations lesson generalizes to ML services: configuring a CPU limit and observing that limit are separate tasks.
Engineering practice
Sensing & deployment
Make a read-only service small by construction
A minimal Python image copies only the bridge implementation. Dependency boundaries can reduce operational capability as well as image size.
Implementation
Representation learning
What happens when a text classifier reads pixels instead of tokens?
A language-as-image experiment turns rendering into a lossy front end and compares two ways of training the resulting CNN.
Implementation
Evaluation practice
Before improving the network, test the representation
A saved text-as-image comparison shows close visual training results—but the straightforward raw-text baseline is stronger.
EvaluationNo matching articles. Try a framework name or clear the filters.