Arithmetic intensity
How much arithmetic you do per byte you move. Below the hardware's compute-to- bandwidth ratio you are memory-bound and the tensor cores sit idle, which is the normal state of single-stream token generation.
- Intensity, in FLOPs per byte.
- Floating-point operations performed.
- Traffic between memory and the compute units.