> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/machine-learning/ml-pipelines.md).

# Fit/transform pipelines

Composable fit/transform preprocessing pipelines (standardize, min-max, chained) with exact inverses.

`pipelines` replaces `sklearn.pipeline.Pipeline` composed with `sklearn.preprocessing` — `StandardScaler`, `MinMaxScaler`, `FunctionTransformer(log1p)`, clip, and `SimpleImputer(strategy="median")` — as a pipeline **spec** executor with zero Python imports. Transforms are encoded by integer id, fitted per-column statistics are flattened into one `Float64` buffer, and transform/inverse replay the spec deterministically on row-major matrices, fitting each step on the output of the previous steps (sklearn `Pipeline.fit` semantics).

## The modules

| Module      | What it does                                                                                                                                                                                                                                                                                                                     |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pipelines` | fit/transform/inverse preprocessing pipelines -- standardize, min-max, log1p, clip, and median-impute steps encoded by integer id and replayed deterministically from one flattened, fitted spec, with exact inverses where the step supports one -- replacing `sklearn.pipeline.Pipeline` composed with `sklearn.preprocessing` |

## Functions

| Function                                     | What it does                                                                                                                                                                                                |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pipeline_fit(x, rows, cols, steps, params)` | Fit a step spec sequentially on flat row-major X; `params` supplies `(lo, hi)` per clip step in step order. Returns a `PipelineState` with `ok=False` on any invalid spec/shape (including clip `lo > hi`). |
| `pipeline_transform(state, x, rows)`         | Apply the fitted steps in order to new data; empty list on shape mismatch or unfitted state. An empty spec is the identity.                                                                                 |
| `pipeline_inverse(state, y, rows)`           | Apply step inverses in reverse order; `InverseResult.ok=False` if any step is clip or impute\_median.                                                                                                       |
| `pipeline_is_invertible(state)`              | True iff every step has an exact inverse (standardize, minmax, log1p only).                                                                                                                                 |
| `step_param_count(t, cols)`                  | Learned-param buffer length for one step id (`2*cols`, `2*cols`, `0`, `2`, `cols`); `-1` for unknown ids.                                                                                                   |

Step ids: `0` standardize (population std, ddof=0), `1` minmax, `2` log1p, `3` clip(lo, hi), `4` impute\_median.

## NaN semantics and degenerate columns

Missing values are IEEE NaN. Fitting ignores NaN when learning statistics (tests: column `[1, NaN, 3]` learns mean 2, std 1); standardize, minmax, and clip pass NaN through unchanged; impute\_median fills it with the per-column median of non-NaN values — odd counts take the middle value, even counts average (`[1, 5, NaN, NaN]` → 3.0), and an all-NaN column imputes 0.0. Degenerate columns stay defined: `std == 0` or `range == 0` maps to 0.0, and the inverse then recovers the mean / min exactly.

## Verification

The tests pin hand-computed values end to end. Standardize on `[[1,2],[3,4],[5,6]]` learns means `[3, 4]` and population std `sqrt(8/3) = 1.6329931618554518` per column (z-scores `±1.224744871391589`, checked to 1e-12). A standardize → minmax chain fits the second step on the z-scores and produces exactly `[0, 0, 0.5, 0.5, 1, 1]`; the unseen row `[7, 8]` maps to `[1.5, 1.5]`, and `pipeline_inverse` round-trips both back to the originals within 1e-9 (this repo's `std.math` exp/log carry \~1e-10 relative error). log1p is pinned at `log 2 = 0.6931471805599453` and returns NaN for `1 + x <= 0`. A full five-step chain (impute → log1p → standardize → minmax → clip) lands every output inside the clip bounds `[0.05, 0.95]` with the learned buffer laid out at the documented offsets.

**Next:** promote what you trained with [statistical model comparison](/machine-learning/ml-model-comparison.md), or browse the [engine catalog](/engine-runtime/catalog.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/machine-learning/ml-pipelines.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.