> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/benchmarks/nvidia-kernel-benchmarks.md).

# NVIDIA kernel benchmarks

One portable FP32 kernel, unmodified, measured on NVIDIA T4, L4 and A100 — GEMM and saxpy with bit-exact validation; percentages are against spec-sheet peaks.

This page measures one portable FP32 kernel, unmodified, on NVIDIA T4, L4 and A100 — the portable GPU kernel layer whose run-verification is in the [GPU support matrix](/gpu-and-llm-inference/gpu-support.md). Correctness is asserted before any timing: worst-case oracle delta of 0.0 on GEMM including zero-padded ragged tiles, and saxpy exact at n=65537, a size that is not a multiple of the kernel's block size. Percentages are ratios against each chip's own spec-sheet peak.

## Results

FP32 GEMM (square, sizes 512-4096) and saxpy (64M elements) on T4, L4 and A100. Each GEMM cell is time / achieved GFLOPS. Median of 3 timed runs after warmup; the timing wraps kernel enqueue + synchronize only. The same 32x32 tile / 2x2-register kernel runs on all three GPUs; nothing is retuned per chip.

| GPU (arch)              | FP32 peak / mem BW  | GEMM 512            | GEMM 1024           | GEMM 2048            | GEMM 4096            | %peak @4096 | saxpy 64M GB/s | %BW peak |
| ----------------------- | ------------------- | ------------------- | ------------------- | -------------------- | -------------------- | ----------- | -------------- | -------- |
| A100-SXM4-40GB (sm\_80) | 19.5 TF / 1555 GB/s | 0.756 ms / 354.8 GF | 4.753 ms / 451.9 GF | 34.898 ms / 492.3 GF | 211.56 ms / 649.6 GF | 3.3%        | 1341.8         | 86.3%    |
| L4 (sm\_89)             | 30.3 TF / 300 GB/s  | 1.036 ms / 259.0 GF | 7.176 ms / 299.2 GF | 55.943 ms / 307.1 GF | 454.48 ms / 302.4 GF | 1.0%        | 234.1          | 78.0%    |
| T4 (sm\_75)             | 8.1 TF / 320 GB/s   | 3.434 ms / 78.2 GF  | 25.183 ms / 85.3 GF | 93.079 ms / 184.6 GF | 747.56 ms / 183.9 GF | 2.3%        | 256.4          | 80.1%    |

* Peak values are the published spec-sheet figures (FP32 TFLOPS and memory bandwidth), so every %peak column is a ratio against the chip's own ceiling, not against another GPU.
* GEMM orders A100 > L4 > T4 at every size. Bandwidth ratios measured close to theory: A100/T4 5.3x measured vs 4.9x theoretical, A100/L4 5.7x measured vs 5.2x theoretical.
* saxpy orders A100 > T4 > L4, following the spec sheet: T4's 320 GB/s GDDR6 (256-bit) exceeds L4's 300 GB/s (192-bit). Both deliver 79-86% of their own bandwidth peak.
* T4 large-size numbers are at its power-limited steady clock: 1215-1230 MHz of a 1590 MHz maximum, at 69 W of a 70 W board limit. L4 and A100 ran at max clocks under load. Clocks were sampled during the timed regions.

## Implementation notes

* The kernel is the portable validation tile (32x32, 2x2 register block), compiled once per chip from the same source with no per-architecture tuning. Percent-of-peak reflects the validation kernel; optimized NVIDIA kernels are measured against this baseline.
* GEMM is bit-exact against the CPU oracle: worst-case delta 0.0, on both the 256^3 cube and the zero-padded ragged 100x140x84 shape (the tile-edge case a padded destination produces).
* saxpy is exact at n=65537. That size is not a multiple of the block size, so the measurement exercises the tail path.

## Methodology

* Timed region: kernel enqueue + synchronize only — no host-side conversion, no allocation, no framework. Median of 3 timed runs after warmup.
* Correctness: the oracle diffs above run before any timing at each shape; no timing is reported from a kernel that failed its oracle diff.
* Environment: NVIDIA driver 595.71.05 on all three machines, CUDA 13.2 reported, Ubuntu 22.04 GCP spot VMs (spot capacity us-central1), mojo 1.0.0 / MAX 26.5.0 from the repo lock.

## Scope

* Percent-of-peak reflects the validation kernel, not a tuned-kernel ceiling.
* These are kernel-layer results. LLM-serving numbers are on [NVIDIA LLM throughput](/benchmarks/nvidia-llm-throughput.md); lane-by-lane scope is on the [GPU support matrix](/gpu-and-llm-inference/gpu-support.md).
* V100 (sm\_70) is rejected at compile time by the pinned toolchain; the CUDA floor is Turing — see [Unsupported architectures](/gpu-and-llm-inference/gpu-support.md#unsupported-architectures).

## Reproduce

The CUDA engine compile is reproducible with no GPU, exactly as CI runs it (inside an `ubuntu:22.04` amd64 container with the repository mounted at `/work`):

```bash
bash .github/scripts/build_cuda_engine_linux_amd64.sh
```

The measurement needs a CUDA host (driver >= 580, per the [GPU support requirements](/gpu-and-llm-inference/gpu-support.md#requirements-proven-on-ubuntu-2204-gcp)) and a runtime built with the CUDA knobs — `MOJO_BUILD_CUDA_GPU=1` and `MOJO_BUILD_CUDA_TARGET` set to the target architecture (`sm_75` / `sm_80` / `sm_89`). The harness drives the portable GEMM and saxpy kernels through the same launch seam with the protocol above: oracle diff first, then warmup, then median-of-3 timings wrapping enqueue + synchronize only. The cross-GPU sweep uses the same kernel, protocol and knobs; a packaged sweep script ships separately.

## Related

* [GPU support](/gpu-and-llm-inference/gpu-support.md) — the verification matrix these numbers cover, lane by lane.
* [NVIDIA LLM throughput](/benchmarks/nvidia-llm-throughput.md) — the CUDA-native LLM session on A100/L4.
* [Compute kernels](/benchmarks/compute-kernels.md) — the Metal-side kernel measurements.
* [Prefill (token input) throughput](/benchmarks/llm-prefill-throughput.md) — the Metal prefill path end to end.
* [Batched decode throughput](/benchmarks/batched-decode-throughput.md) — batched serving on Metal.
* [Benchmarks](/benchmarks/overview.md) — methodology.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/benchmarks/nvidia-kernel-benchmarks.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.