Context parallelism

A model-parallelism strategy that shards the sequence/context dimension across multiple devices so attention over million-token windows scales beyond a single accelerator's memory.

grounded in: The 'Flash-MSA … accelerate million-token training runs' trend theme of scaling attention to million-token sequences, which requires sharding the sequence dimension across devices.