NVIDIA kernel benchmarks
One portable FP32 kernel, unmodified, measured on NVIDIA T4, L4 and A100 — GEMM and saxpy with bit-exact validation; percentages are against spec-sheet peaks.
This page measures one portable FP32 kernel, unmodified, on NVIDIA T4, L4 and A100 — the portable GPU kernel layer whose run-verification is in the GPU support matrix. Correctness is asserted before any timing: worst-case oracle delta of 0.0 on GEMM including zero-padded ragged tiles, and saxpy exact at n=65537, a size that is not a multiple of the kernel's block size. Percentages are ratios against each chip's own spec-sheet peak.
Results
FP32 GEMM (square, sizes 512-4096) and saxpy (64M elements) on T4, L4 and A100. Each GEMM cell is time / achieved GFLOPS. Median of 3 timed runs after warmup; the timing wraps kernel enqueue + synchronize only. The same 32x32 tile / 2x2-register kernel runs on all three GPUs; nothing is retuned per chip.
A100-SXM4-40GB (sm_80)
19.5 TF / 1555 GB/s
0.756 ms / 354.8 GF
4.753 ms / 451.9 GF
34.898 ms / 492.3 GF
211.56 ms / 649.6 GF
3.3%
1341.8
86.3%
L4 (sm_89)
30.3 TF / 300 GB/s
1.036 ms / 259.0 GF
7.176 ms / 299.2 GF
55.943 ms / 307.1 GF
454.48 ms / 302.4 GF
1.0%
234.1
78.0%
T4 (sm_75)
8.1 TF / 320 GB/s
3.434 ms / 78.2 GF
25.183 ms / 85.3 GF
93.079 ms / 184.6 GF
747.56 ms / 183.9 GF
2.3%
256.4
80.1%
Peak values are the published spec-sheet figures (FP32 TFLOPS and memory bandwidth), so every %peak column is a ratio against the chip's own ceiling, not against another GPU.
GEMM orders A100 > L4 > T4 at every size. Bandwidth ratios measured close to theory: A100/T4 5.3x measured vs 4.9x theoretical, A100/L4 5.7x measured vs 5.2x theoretical.
saxpy orders A100 > T4 > L4, following the spec sheet: T4's 320 GB/s GDDR6 (256-bit) exceeds L4's 300 GB/s (192-bit). Both deliver 79-86% of their own bandwidth peak.
T4 large-size numbers are at its power-limited steady clock: 1215-1230 MHz of a 1590 MHz maximum, at 69 W of a 70 W board limit. L4 and A100 ran at max clocks under load. Clocks were sampled during the timed regions.
Implementation notes
The kernel is the portable validation tile (32x32, 2x2 register block), compiled once per chip from the same source with no per-architecture tuning. Percent-of-peak reflects the validation kernel; optimized NVIDIA kernels are measured against this baseline.
GEMM is bit-exact against the CPU oracle: worst-case delta 0.0, on both the 256^3 cube and the zero-padded ragged 100x140x84 shape (the tile-edge case a padded destination produces).
saxpy is exact at n=65537. That size is not a multiple of the block size, so the measurement exercises the tail path.
Methodology
Timed region: kernel enqueue + synchronize only — no host-side conversion, no allocation, no framework. Median of 3 timed runs after warmup.
Correctness: the oracle diffs above run before any timing at each shape; no timing is reported from a kernel that failed its oracle diff.
Environment: NVIDIA driver 595.71.05 on all three machines, CUDA 13.2 reported, Ubuntu 22.04 GCP spot VMs (spot capacity us-central1), mojo 1.0.0 / MAX 26.5.0 from the repo lock.
Scope
Percent-of-peak reflects the validation kernel, not a tuned-kernel ceiling.
These are kernel-layer results. LLM-serving numbers are on NVIDIA LLM throughput; lane-by-lane scope is on the GPU support matrix.
V100 (sm_70) is rejected at compile time by the pinned toolchain; the CUDA floor is Turing — see Unsupported architectures.
Reproduce
The CUDA engine compile is reproducible with no GPU, exactly as CI runs it (inside an ubuntu:22.04 amd64 container with the repository mounted at /work):
The measurement needs a CUDA host (driver >= 580, per the GPU support requirements) and a runtime built with the CUDA knobs — MOJO_BUILD_CUDA_GPU=1 and MOJO_BUILD_CUDA_TARGET set to the target architecture (sm_75 / sm_80 / sm_89). The harness drives the portable GEMM and saxpy kernels through the same launch seam with the protocol above: oracle diff first, then warmup, then median-of-3 timings wrapping enqueue + synchronize only. The cross-GPU sweep uses the same kernel, protocol and knobs; a packaged sweep script ships separately.
Related
GPU support — the verification matrix these numbers cover, lane by lane.
NVIDIA LLM throughput — the CUDA-native LLM session on A100/L4.
Compute kernels — the Metal-side kernel measurements.
Prefill (token input) throughput — the Metal prefill path end to end.
Batched decode throughput — batched serving on Metal.
Benchmarks — methodology.
Last updated
Was this helpful?