Tensor Parallelism

Splitting an individual model layer's weight matrices across multiple GPUs or nodes so a single forward pass executes in parallel, letting models too large for one device run across a cluster.

grounded in: spark.sh's recurring '2-node GB10 telemetry refresh' commits (a multi-node inference cluster) plus the HN trend of meshing consumer devices to run large models locally.

Connected concepts

Distributed inference, GPU interconnect fabric, Decentralized mesh inference, Mixture-of-Experts