Represent variable neural graphs with fixed-size tensors

The architecture in context

The system we are building

An evolving topology does not naturally fit a dense batch dimension. The prototype reserves node and connection capacity, marks unused entries and computes a valid node order. This trades flexible allocation for regular tensor storage. The model can then evaluate multiple observations against one encoded graph, with later classes exploring population-oriented execution.

Who does what in the stack

PyTorch tensors: Store fixed-capacity genomes and batched node values.
Custom topological scheduler: Orders valid feed-forward computation.
Population adapters: Explore batching without changing genotype semantics.

The local file adapts tensorized-neuroevolution ideas to PyTorch, including topological scheduling, validity masks and population containers. It should be presented as a local adaptation alongside upstream attribution, not as the invention of TensorNEAT. Different class variants in the same file are not interchangeable benchmark results.

Framework responsibility map. Each row maps a library or custom component to its job; rows are not a sequential inference graph. Open full-size SVG ↗

Open up the implementation

Evaluate a padded evolving graph

A concrete operation-level view of this implementation; no unobserved neural architecture is implied. Open full-size SVG ↗

The PyTorch port encodes absent genes using NaNs and uses masks to select enabled incoming connections. A topological schedule supplies dependencies that a generic dense matrix multiplication does not capture. The forward function allocates node values for the maximum capacity even when the active genome is small.

The mathematical contract

v_j=\sigma_j(b_j+\sum_i m_{ij}w_{ij}v_i)

Padding is friendly to fixed-shape computation but can waste work on sparse genomes. Dynamic topology and batch-wide tensorization must agree on node identity. The exception handler returns zero outputs on setup failure; those outputs must be flagged invalid rather than scored as a legitimate low-performing policy.

Implementation and resource card

Capacity / budget: Allocated arrays use max_nodes / connection capacity; effective capacity uses valid nodes and enabled edges. Both counts matter.
Execution evidence: This revision inspects and explains the archived implementation. It does not rerun the original workload. No unrecorded convergence time, throughput or accelerator result is supplied.
Current reproduction context: Current workstation, supplied by the author: Apple M4, 128 GB unified RAM, 40 GPU cores and 16 CPU cores. This is context for prospective reproduction, not attribution of every archived run. Python and framework versions are not fully locked for these historical sources; declarations, when available, are identified separately.

From explanation to a reproducible check

Use a hand-built acyclic graph with one disabled edge and one unused slot. Compare a dictionary-based evaluator with the tensorized implementation. Deliberately introduce a cycle or malformed node index and require an error record; silently returned zeros are not a passing correctness check.

Preserve input identities, configuration and failure records with the result. A successful numerical check only establishes the operation it exercises: it does not certify an entire dataset, model or deployed system. Reproduce the interface on a small deterministic input before optimizing throughput or increasing workload size.

A closer look at the implementation

The code that carries the idea

The excerpt skips input and invalid nodes, gathers incoming weights and combines non-NaN and enabled flags into a validity mask. Padded capacity therefore remains part of the representation but should contribute no signal. Topological ordering is what ensures a feed-forward node reads already computed predecessors.

Python · file · lines 492–514

        # Process nodes in topological order (only valid ones)
        for node_idx in node_order:
            # Skip input nodes
            if node_idx < inputs.shape[1]:
                continue
                
            # Skip invalid nodes (shouldn't happen with new _compute_node_order)
            if not valid_nodes[node_idx]:
                continue
                
            # Get incoming connections
            incoming_weights = expanded_conns[:, node_idx, 0]
            incoming_enabled = expanded_conns[:, node_idx, 1]
            
            # Mask for valid connections
            valid_mask = (~torch.isnan(incoming_weights) & 
                        ~torch.isnan(incoming_enabled) & 
                        incoming_enabled.bool())
            
            if not valid_mask.any():
                continue
            
            # Compute weighted sum of inputs

Verbatim archive excerpt from tensorneat_pt.py. Context-dependent historical code, not a standalone runnable program. Comments retain their original wording; the article distinguishes implemented behavior from stale or overbroad comments.

The boundary that matters

The surrounding forward method catches setup errors and returns zeros. That is convenient for a long-running search but can silently turn an invalid graph into an apparently valid low-fitness candidate. Cycles, capacity overflow and malformed indices need explicit accounting.

Represent variable neural graphs with fixed-size tensors

From input to outcome