> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/benchmarks/llm-inference-overview.md).

# LLM inference overview

LLM inference on Apple silicon — prefill, single-stream decode, and batched decode for Qwen3-0.6B-Base, measured against llama.cpp and mlx-lm.

Algenta ships a compiled Metal kernel path that runs Qwen3-0.6B-Base end to end on Apple silicon — prompt in, tokens out, no server or sidecar — in the runtime `pip install algenta` installs on macOS 26 or newer ([GPU support](/gpu-and-llm-inference/gpu-support.md)). Inference has three workloads, each measured separately below.

## Results

| Workload                             | What it is                                | Bottleneck                            | Result                                                    | Page                                                                  |
| ------------------------------------ | ----------------------------------------- | ------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------- |
| Prefill (token input)                | Read the whole prompt, build the KV cache | Compute (batched matmul)              | \~10,700 tok/s, 1.049x llama.cpp                          | [Prefill throughput](/benchmarks/llm-prefill-throughput.md)           |
| Decode (single-stream)               | Generate one token at a time, one request | Memory bandwidth (weight sweep/token) | Ahead of llama.cpp and mlx-lm at generation lengths >= 16 | [Decode throughput](/benchmarks/decode-throughput.md)                 |
| Batched decode (continuous batching) | Generate for B requests at once           | Weight sweep amortized over B         | 2.1-2.6x llama.cpp concurrent at B=8-32                   | [Batched decode throughput](/benchmarks/batched-decode-throughput.md) |

Measured on Apple M4 Max: prefill 10,676-10,734 tok/s (llama.cpp 10,199); single-stream decode ahead at generation lengths >= 16; batched decode 2.1-2.6x llama.cpp at B=8-32, with the batched path sweeping the weights once per step for all B sequences.

## Workload characteristics

* Prefill processes every prompt token together, so each projection is one wide `[m_rows, d_model] @ weights` matmul — compute-bound, scaling with GEMM tile quality.
* Single-stream decode generates one token per step and streams the full model weights for each. Every engine sits near the same memory-bandwidth floor; Algenta leads at generation lengths

  > \= 16.
* Batched decode steps B sequences per step. The weights are swept once but produce B tokens, so the per-token weight cost falls with B.

## Notes

* Any token count. The prefill path re-grids per call (padded-destination contract), so prompts and batches of any size work, not just the measured 256 / B values. Context auto-raises past the historical 512 default up to a 65,536 ceiling; deep soaks verified at 2,048 new tokens (0.6B) and 1,024 (1.7B), every token reference-checked.
* Model-agnostic within the Qwen3 family. Buffer and grid shapes are derived from the checkpoint's own dims. Verified end-to-end on Qwen3-0.6B-Base and Qwen3-1.7B with identical greedy output (first token matches an independent `transformers` CPU reference), runnable through the SDK's model-agnostic path — see [Model-agnostic LLM inference](/guides/run-qwen3-family-inference.md).
* Cached weight loading. The one-shot SDK shape persists the exact post-quantization device bytes next to each converted checkpoint; second-and-later calls load \~8x fewer bytes and skip quantization entirely (3.10x / 1.93x whole-call wall on 0.6B / 1.7B), token-identical to the uncached path.
* Correctness gating. Every path checks generated tokens against a reference before any number is reported.

## Related

* [Prefill throughput](/benchmarks/llm-prefill-throughput.md) — token-input processing.
* [Decode throughput](/benchmarks/decode-throughput.md) — single-stream generation vs llama.cpp and mlx-lm.
* [Batched decode throughput](/benchmarks/batched-decode-throughput.md) — continuous batching.
* [Metal LLM decode](/gpu-and-llm-inference/qwen3-metal-decode.md) — call it from Python.
* [GPU support](/gpu-and-llm-inference/gpu-support.md) — which GPU lanes are run-verified, and what that covers.
* [NVIDIA LLM throughput](/benchmarks/nvidia-llm-throughput.md) — CUDA-native session numbers on A100/L4.
* [NVIDIA LLM serving](/benchmarks/nvidia-llm-serving.md) — serving concurrency on A100/L4 vs llama.cpp and vLLM.
* [Run local LLM decode](/guides/run-qwen3-decode.md) — step-by-step guide.
* [Model-agnostic LLM inference](/guides/run-qwen3-family-inference.md) — the model-agnostic SDK path (0.6B, 1.7B, and other supported shapes).
* [Benchmarks](/benchmarks/overview.md) — methodology.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/benchmarks/llm-inference-overview.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.