> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/gpu-and-llm-inference/gpu-support.md).

# GPU support

GPU backend support matrix — Metal (default serve path), NVIDIA CUDA (hardware-verified kernel layer and LLM session), ROCm status, and per-lane requirements.

Metal is the default serve path on Apple Silicon: `pip install algenta` installs the runtime with Metal on macOS 26 or newer (see [Metal](#metal-apple-silicon)). NVIDIA CUDA runs the portable kernel layer and the resident LLM session, hardware-verified on the GPUs tabled below; the Linux x86-64 runtime wheel ships its engines for T4, A100, L4 and H100/H200. AMD ROCm is not verified. Install, requirements, entry points and the verified-shape table for the NVIDIA lane are on [CUDA backend](/gpu-and-llm-inference/cuda-backend.md).

## Support matrix

| Backend               | Status                                                                                                    | Verification                                                                                                                             |
| --------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Metal (Apple Silicon) | Run-verified; default serve path on Apple Silicon. Ships in the macOS 26+ runtime wheel                   | Full serving path on M-series hardware: resident kernels, batched dispatch, prefill, decode                                              |
| NVIDIA CUDA           | Supported. Hardware-verified kernel layer and LLM session; engines ship in the Linux x86-64 runtime wheel | Kernels bit-exact on T4 (sm\_75), L4 (sm\_89), A100 (sm\_80); LLM session measured on A100 (sm\_80), L4 (sm\_89), H100 and H200 (sm\_90) |
| AMD ROCm              | Not verified                                                                                              | Compile gate in progress                                                                                                                 |
| V100 / Volta (sm\_70) | Not supported                                                                                             | Compile-time constraint of the pinned toolchain                                                                                          |

## MAX hardware compatibility

The engine runs on the Modular MAX runtime; MAX's hardware compatibility matrix (vendor documentation, max.modular.com/packages) lists, for NVIDIA, driver >= 580 with B200 tested for serving and B300, B100, DGX Spark, H200, H100, L4, L40, A100, A10, A1000, RTX 50XX, RTX 40XX, RTX 30XX, Jetson Orin/Thor, T4 and RTX 20XX known compatible for development; for AMD, ROCm >= 6.3.3 (MI355X requires ROCm >= 7.0) with MI355X, MI300X and MI325X tested for serving and MI250X plus the Radeon RX/RDNA families known compatible for development.

Algenta's own verification within that matrix: LLM session measured on A100, L4, H100 and H200 (this page); kernel layer bit-exact on T4, L4 and A100; B200 (sm\_100) and the AMD compile gate are being exercised and land here as they clear.

## Metal (Apple Silicon)

Metal runs the full serving path: resident kernels, batched dispatch, prefill and decode. `pip install algenta` installs the runtime with Metal on macOS 26 or newer; on macOS 13 to 15 the installed runtime is CPU-only. Measured throughput:

* [Prefill (token input) throughput](/benchmarks/llm-prefill-throughput.md)
* [Decode throughput](/benchmarks/decode-throughput.md)
* [Batched decode throughput](/benchmarks/batched-decode-throughput.md)
* [Metal LLM decode](/gpu-and-llm-inference/qwen3-metal-decode.md) — the SDK entry point

## NVIDIA CUDA

Install, hardware requirements, the callable entry points, the determinism gates and the per-checkpoint verified-shape table are on [CUDA backend](/gpu-and-llm-inference/cuda-backend.md).

### Kernel layer

The portable GPU kernels compile to PTX via the NVPTX backend and run bit-exact against the CPU oracle on T4, L4 and A100 (worst-case GEMM delta 0.0 including zero-padded ragged tiles; saxpy exact at n=65537). Measured numbers and methodology: [NVIDIA kernel benchmarks](/benchmarks/nvidia-kernel-benchmarks.md).

Build flags: `-D ALGENTA_GPU` compiles the portable GPU kernels; `-D ALGENTA_METAL` additionally compiles the Metal serving path. A CUDA build contains the portable kernels without the Metal session layer.

### LLM session

The CUDA-native resident session (v2 prefill GEMMs, CUDA-graph decode with split-K flash-decoding) is measured against llama.cpp on A100, L4 and H100: prefill 3,260 vs 2,527 tok/s (F16); decode 427.1 / 408.7 vs 356.9 / 366.6 (F16) on the A100 and 656.1 / 622.3 vs 468.3 / 525.0 (F16) on the H100 with packed-int8 weights. The same session family carries the larger checkpoints: Qwen2.5-7B-Instruct prefill, decode and server TTFT on A100, and Gemma-3-27B-it prefill on H200 with its serving cells on H100. Numbers and methodology: [NVIDIA LLM throughput](/benchmarks/nvidia-llm-throughput.md) and [NVIDIA LLM serving](/benchmarks/nvidia-llm-serving.md). Per-checkpoint shapes and hardware: [CUDA backend](/gpu-and-llm-inference/cuda-backend.md#verified-shapes).

### Compile gate in CI

The linux-x86\_64 runtime build cross-compiles the CUDA engine for sm\_75, sm\_80, sm\_89 and sm\_90a on every change, without a GPU on the builder, and ships all four in the linux-x86\_64 runtime wheel. Hardware verification runs on the GPUs in the support matrix above. For a local worker build the default target is `sm_90`; override `MOJO_BUILD_CUDA_TARGET`:

| Fleet GPU  | `MOJO_BUILD_CUDA_TARGET` |
| ---------- | ------------------------ |
| T4         | `sm_75`                  |
| A100       | `sm_80`                  |
| L4         | `sm_89`                  |
| H100, H200 | `sm_90`                  |

### Requirements

* NVIDIA driver >= 580 (validated at 580.173.02 and 595.71.05).
* clang for the external link step.
* The repo's pinned Mojo toolchain (mojo 1.0.0 / MAX 26.5.0), resolved from the lockfile.
* `ubuntu-drivers install --gpgpu` installs the headless driver without `nvidia-smi`; install the matching `nvidia-utils-<ver>-server` package for it.
* Direct `mojo` invocations outside the managed environment need `MODULAR_HOME=<env>/share/max`.

## Unsupported architectures

The pinned toolchain (mojo 1.0.0 / MAX 26.5.0) rejects sm\_70 at compile time:

```
oss/modular/mojo/stdlib/std/gpu/host/info.mojo:2145:5: note: constraint failed: GPU architecture 'sm_70' is not supported.
```

The constraint is enforced in the Mojo stdlib before the PTX assembler runs; `MODULAR_NVPTX_COMPILER_PATH` swaps only the downstream `ptxas` binary and does not change the outcome (verified). Supported range: sm\_52, 60/61, 75, 80, 86, 87, 89, 90(a), 100(a), 110, 120(a), 121. This matches Modular's baseline (Turing and newer).

## AMD ROCm

MAX bundles `_miopen`/`_rocblas`; the compile gate is in progress. Nothing has run on ROCm hardware.

## Notes

* Metal is the default serve path on Apple Silicon, in the runtime `pip install algenta` installs on macOS 26 or newer. The CUDA backend is hardware-verified on the GPUs above and its engines ship in the Linux x86-64 runtime wheel; CI compiles and links the CUDA lane on a builder with no GPU.
* Install, requirements, entry points and the verified-shape table for the NVIDIA lane: [CUDA backend](/gpu-and-llm-inference/cuda-backend.md).

## Related

* [CUDA backend](/gpu-and-llm-inference/cuda-backend.md)
* [NVIDIA kernel benchmarks](/benchmarks/nvidia-kernel-benchmarks.md)
* [NVIDIA LLM throughput](/benchmarks/nvidia-llm-throughput.md)
* [NVIDIA LLM serving](/benchmarks/nvidia-llm-serving.md)
* [Metal LLM decode](/gpu-and-llm-inference/qwen3-metal-decode.md)
* [Prefill (token input) throughput](/benchmarks/llm-prefill-throughput.md)
* [Batched decode throughput](/benchmarks/batched-decode-throughput.md)
* [Runtime libraries](/engine-runtime/runtime-libraries.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/gpu-and-llm-inference/gpu-support.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.