> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/guides/run-qwen3-decode.md).

# Run local LLM decode

Install Algenta, get a supported checkpoint, and run greedy decode on your own Apple Silicon GPU — requirements, limitations, and a complete working example. Currently tested on Qwen3 models.

This guide installs Algenta, downloads the Qwen3-0.6B-Base checkpoint, tokenizes a prompt, and runs a greedy decode through the compiled int8 Metal GPU kernel path on an Apple Silicon machine. Everything runs inside the runtime that ships with the `algenta` package; there is nothing separate to start or manage.

Scope: Qwen3-0.6B-Base on Apple Silicon with Metal. Read [Requirements](#requirements) and [Limitations](#limitations) before starting.

## Requirements

* Apple Silicon (M-series) Mac with Metal. There is no CPU fallback and no support for other GPU vendors. The LLM serving path runs on Metal only. NVIDIA CUDA is run-verified for the portable GPU kernel layer on T4, L4 and A100 hardware, and a CUDA-native LLM session is benchmarked on A100/L4, but the shipped serving path here is Metal-only; AMD ROCm is not verified. See [GPU support](/gpu-and-llm-inference/gpu-support.md).
* macOS 26 or newer and Python 3.12 or newer. `pip install algenta` installs the native runtime with Metal on macOS 26 and newer; on macOS 13 to 15 it installs the CPU-only runtime, which cannot run this Metal path.
* The checkpoint this kernel path was compiled for: [`Qwen/Qwen3-0.6B-Base`](https://huggingface.co/Qwen/Qwen3-0.6B-Base) (Apache-2.0, ungated — no Hugging Face account or token needed). No other Qwen3 size and no other model family.
* `numpy`, to convert the checkpoint's published BF16 weights. Install it with the `benchmark` extra: `pip install "algenta[benchmark]"`.
* A tokenizer for the checkpoint. This capability takes token ids in and token ids out, not raw text — the example below uses `transformers`' `AutoTokenizer` against the same checkpoint directory.

## Limitations

* One model, one exact shape. Qwen3-0.6B-Base's exact architecture only (`hidden_size=1024`, 28 layers, 16 query heads, 8 key/value heads, `head_dim=128`, `vocab_size=151936`, tied embeddings). A checkpoint of any other shape raises `UnsupportedCheckpointArchitectureError` naming every mismatched field before any conversion or GPU work starts. Other Qwen3 sizes (for example Qwen3-1.7B) run through the model-agnostic path — see [Model-agnostic LLM inference](/guides/run-qwen3-family-inference.md).
* One hardware family. Apple Silicon with Metal. A host without a Metal accelerator raises a named error. CUDA runs the portable kernel layer and has a benchmarked native LLM session, but not this shipped session path; ROCm is not verified — [GPU support](/gpu-and-llm-inference/gpu-support.md).
* Greedy decode only. Every generated token is the argmax at that step. There is no sampling, no temperature, no top-p/top-k, and no beam search.
* One call, start to finish. A single call prefills the prompt and decodes to `max_new_tokens` or an end-of-text token. A step-by-step session is available through the Hub kernel; see [Kernels on Hugging Face](/guides/kernels-on-hugging-face.md).
* Every call converts the checkpoint first. The checkpoint's BF16 weights are converted into the kernel's own flat format on every call by default, which takes time before decoding starts.
* Token ids in, not raw text. Tokenize with the checkpoint's own tokenizer — either `algenta.load_tokenizer()` (built into the SDK, no dependencies) or `transformers`' `AutoTokenizer`; the example below shows the latter.

## Install Algenta

{% stepper %}
{% step %}

### Install the package

The `algenta` package installs the matching compiled native runtime automatically — there is no separate Mojo/Metal toolchain to install.

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install "algenta[benchmark]"
```

{% endstep %}

{% step %}

### Verify the runtime is healthy

```bash
algenta runtime doctor
```

Expect a report that ends in `READY`, with the bundled native runtime verified and no fallback. Its `runtime wheel` line names the installed wheel; this guide needs the `macosx_26_0_arm64` one.
{% endstep %}
{% endstepper %}

## Get the checkpoint

Algenta does not download the checkpoint; get a copy of `Qwen/Qwen3-0.6B-Base` onto disk first:

```bash
pip install huggingface_hub
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='Qwen/Qwen3-0.6B-Base',
    local_dir='./qwen3-0.6b-base',
    allow_patterns=['*.json', '*.safetensors', 'tokenizer*', 'vocab*', 'merges*'],
)
"
```

## Run a decode

{% stepper %}
{% step %}

### Tokenize your prompt

`qwen3_i8_allsimd7_decode()` takes a list of integer token ids, not a string. Tokenize with the checkpoint's own tokenizer:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("./qwen3-0.6b-base")
prompt_token_ids = tokenizer("The capital of France is", return_tensors=None)["input_ids"]
```

{% endstep %}

{% step %}

### Call the decode function

`qwen3_i8_allsimd7_decode()` takes a `MojoRuntime`, the checkpoint directory, your prompt token ids, and how many tokens to generate. It converts the checkpoint automatically the first time you call it.

```python
from algenta import MojoRuntime, qwen3_i8_allsimd7_decode

runtime = MojoRuntime(mode="local")

result = qwen3_i8_allsimd7_decode(
    runtime,
    checkpoint_dir="./qwen3-0.6b-base",
    prompt_token_ids=prompt_token_ids,
    max_new_tokens=16,
)
```

{% endstep %}

{% step %}

### Read the result

`result` is a `Qwen3DecodeResult`: `token_ids` is your prompt followed by every generated token, in order.

```python
print(result.token_ids)
print(result.decode_tokens_per_sec)

generated_only = result.token_ids[result.n_prompt :]
print(tokenizer.decode(generated_only))
```

{% endstep %}
{% endstepper %}

Expected result: `result.token_ids` is a non-empty list of integers starting with your prompt's own token ids, followed by up to `max_new_tokens` generated ids (fewer if the model produced an end-of-text token first). Decoding `generated_only` with the tokenizer reads as ordinary continuation text, and `result.decode_tokens_per_sec` is a positive number.

## Troubleshooting

Detected mismatches raise named errors:

| What you did                                                                       | What you get                                                                                                       |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Pointed `checkpoint_dir` at a checkpoint that is not Qwen3-0.6B-Base's exact shape | `UnsupportedCheckpointArchitectureError`, naming every mismatched field, before any conversion or GPU work starts. |
| Pointed `checkpoint_dir` at a path with no `config.json` / `model.safetensors`     | `Qwen3DecodeError` with code `checkpoint_not_found`.                                                               |
| Ran on a host, or a runtime build, without working Metal GPU support               | `Qwen3DecodeError` with code `metaldispatch_shader_toolchain_absent`.                                              |

Every one of these is a `Qwen3DecodeError` (or its subclass `UnsupportedCheckpointArchitectureError`), and every instance carries a stable `.code` string:

```python
from algenta import Qwen3DecodeError

try:
    result = qwen3_i8_allsimd7_decode(
        runtime,
        checkpoint_dir="./qwen3-0.6b-base",
        prompt_token_ids=prompt_token_ids,
        max_new_tokens=16,
    )
except Qwen3DecodeError as error:
    print(error.code, str(error))
```

## Next steps

{% content-ref url="/pages/kn3LAnWtPCV6grecFXGD" %}
[Model-agnostic LLM inference](/guides/run-qwen3-family-inference.md)
{% endcontent-ref %}

{% content-ref url="/pages/u8Uwo1LQEZTcaYVscvop" %}
[Metal LLM decode](/gpu-and-llm-inference/qwen3-metal-decode.md)
{% endcontent-ref %}

{% content-ref url="/pages/mSGCsrqc462rjkTu7gGN" %}
[Decode throughput](/benchmarks/decode-throughput.md)
{% endcontent-ref %}

{% content-ref url="/pages/dC5uSbQw5pwjOTPP9AWU" %}
[Install](/getting-started/install.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/guides/run-qwen3-decode.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.