All-reduce — summing across GPUs
A collective operation in which every GPU contributes a tensor and every GPU receives the sum (or average) of all of them.
Data parallelism uses it to average gradients; tensor parallelism uses it inside every layer (sections 11.2 and 11.4).