> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/benchmarks/normalization.md).

# Normalization

Normalization kernel results — RMSNorm and LayerNorm forward and backward cost, the price of the numerically safe variance, and gradient checks against finite differences.

Results for the normalization kernels: RMSNorm and LayerNorm, forward and backward. A norm is the cheapest layer in a transformer and the one best placed to quietly damage a model, so most of this page is about correctness with the timings alongside.

## Hardware

| Property           | Value                            |
| ------------------ | -------------------------------- |
| Chip               | Apple silicon, 16 physical cores |
| Precision measured | 64-bit floating point            |

Results are reported for this configuration only. Other chip generations and memory configurations are published separately as they are measured, never averaged into a single figure.

## Cost per element

| Row width | Rows | Cost per element |
| --------- | ---- | ---------------- |
| 512       | 64   | 8.09 ns          |
| 2,048     | 64   | 3.11 ns          |
| 8,192     | 64   | 1.21 ns          |

**Per-element cost falls rather than staying flat, and the reason is not that wider rows are cheaper.** All three shapes are large enough to run across cores, so each pays one fixed cost to distribute work and amortises it over more elements as the row grows. The gated property is only that cost does not *rise* — a normalization whose per-element cost grew with width would be doing something more than linear work.

Below the threshold where distribution pays for itself, the kernel runs on a single core deliberately:

| Shape                   | Cost per element |
| ----------------------- | ---------------- |
| 8 rows, single core     | 56.2 ns          |
| 4,096 rows, distributed | 0.82 ns          |

## The price of a numerically safe variance

LayerNorm needs the variance of each row. There is a one-pass formula that computes it in a single walk of the data, and a two-pass formula that computes the mean first and then the deviations from it. Algenta uses the two-pass form.

The one-pass form loses accuracy when a row's values sit far from zero — which is what a transformer's residual stream looks like after a few blocks. Measured on a row offset by 100 million, the one-pass identity is wrong by **3.65** where the row's actual variance is about **0.35**: an error ten times the size of the quantity being computed. Pushed further it returns a negative variance, at which point the square root produces a non-number and the entire model output becomes invalid — from a kernel that appeared to work.

The cost of avoiding this:

| Kernel                  | Time   |
| ----------------------- | ------ |
| RMSNorm (one reduction) | 324 µs |
| LayerNorm (two passes)  | 333 µs |

**3%.** The second pass re-reads a row that is already in the nearest cache level, so it is close to free. On centred data the two formulas agree to 4.9e-15, and that agreement is reported by the kernel itself on the caller's own data rather than assumed.

## Backward pass

| Direction | Time   | Ratio |
| --------- | ------ | ----- |
| Forward   | 324 µs | —     |
| Backward  | 635 µs | 1.96x |

A backward pass that costs nothing is a backward pass that is wrong. This one runs two reduction passes and a reduction across rows for the scale parameter.

**Gradients are checked against finite differences**, not against a reading of the algebra:

| Kernel    | Difference | Tolerance |
| --------- | ---------- | --------- |
| RMSNorm   | 7.3e-9     | 4.0e-8    |
| LayerNorm | 3.1e-9     | 1.1e-8    |

The tolerance is deliberately looser than the forward kernel's. A finite difference carries two error sources that pull in opposite directions as the step size changes, so it cannot resolve a gradient better than roughly the square of that step. Judging it by a forward-parity tolerance would fail on correct code.

**The check is also shown to be capable of failing.** Each gradient has a term that a careless derivation drops. Recomputing without it moves the result by **2.19** (RMSNorm) and **3.75** (LayerNorm) — far outside tolerance. A gradient check that cannot fail proves nothing, so the failing case is asserted alongside the passing one.

## Reproducibility across cores

The gradient with respect to the scale parameter sums across rows. Computing it with one worker per row would need a shared accumulator: a race, and non-deterministic even when locked, because the sum order would depend on which thread arrived first.

Algenta splits that reduction by column instead — each worker owns whole columns and sums them in row order. The result is **bit-identical** to a single-threaded reference across 4,096 rows, regardless of how many cores run it.

## Call it yourself

`normk.rmsnorm`'s `rmsnorm_fwd_flat(x, gamma, n_rows, n_cols, eps)` is the forward pass measured above, callable directly on a flat row-major buffer:

{% tabs %}
{% tab title="Python" %}

```python
from algenta import MojoRuntime

runtime = MojoRuntime(mode="local")

x = [1.0, 2.0, 3.0, 4.0]      # one row, four columns
gamma = [1.0, 1.0, 1.0, 1.0]

out = runtime.execute("normk.rmsnorm", "rmsnorm_fwd_flat", [x, gamma, 1, 4, 1e-6])["result"]
out["status"]   # 380 — NORM_OK
out["y"]        # [0.3651..., 0.7303..., 1.0954..., 1.4606...]
out["rstd"]     # [0.3651...] — the per-row reciprocal deviation the backward pass needs
```

{% endtab %}

{% tab title="TypeScript" %}

```ts
import { MojoRuntime } from "algenta-sdk";

const runtime = new MojoRuntime();

const x = [1.0, 2.0, 3.0, 4.0];
const gamma = [1.0, 1.0, 1.0, 1.0];

const { result: out } = await runtime.execute("normk.rmsnorm", "rmsnorm_fwd_flat", [
  x, gamma, 1, 4, 1e-6,
]) as { result: { status: number; y: number[]; rstd: number[] } };

out.status;   // 380 — NORM_OK
out.y;        // [0.3651..., 0.7303..., 1.0954..., 1.4606...]
out.rstd;     // [0.3651...] — the per-row reciprocal deviation the backward pass needs
```

{% endtab %}
{% endtabs %}

`rmsnorm_bwd_flat(dy, x, gamma, rstd, n_rows, n_cols)` takes that same `rstd` back in for the backward pass measured above; `normk.layernorm` mirrors both functions for LayerNorm.

## What is not claimed

* Any comparison against a third-party normalization implementation.
* That these figures represent the product's normalization throughput at lower precision. Every number here is 64-bit, and a norm is limited by memory bandwidth, so a narrower numeric format reads a fraction of the bytes and is most of the remaining headroom.

## Related

* [Benchmarks](/benchmarks/overview.md) — methodology and what is not published.
* [Attention](/benchmarks/attention.md) — the attention kernels these norms surround.
* [Feed-forward](/benchmarks/feed-forward.md) — the other half of a transformer block.
* [Runtime library catalog](/engine-runtime/runtime-library-catalog.md) — every module and function, including `normk.rmsnorm` and `normk.layernorm` above.

## Reproduce

Every table on this page comes from one command:

```
mojo run benchmarks/mojo_benchmarks/bench_normk.mojo
```

The benchmark prints its own gate results, and fails on the properties described above rather than on absolute times, which vary by machine, thermal state and concurrent load.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/benchmarks/normalization.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.