> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/connectors/s3.md).

# Amazon S3

By the end of this page you will have a saved `s3` connector that reaches a bucket, a confirmed `live` access check, a browse listing of the objects it can see, one of those objects onboarded as a queryable **dataset**, and a first [governed query](/guides/governed-query.md) returning rows from it. The engine reads the object in place, parses it (CSV, TSV, JSON, Parquet, or Excel), and registers the rows as [governed data](/concepts/governed-data.md) under a typed, validated contract.

Prerequisites: an API key and a base URL (see [Authentication](/getting-started/authentication.md)), and access to an S3 bucket — either AWS credentials or a public bucket. Use `https://api.algenta.ai` for Algenta cloud, or your own origin such as `http://localhost:8000` for a self-hosted deployment.

{% hint style="info" %}
An `s3` connector reads these `config` fields: `bucket` and `key` (the object path) identify the object; `region` defaults to `us-east-1`; `endpoint_url` targets an S3-compatible store (MinIO and similar); `file_type` overrides the format otherwise inferred from the key's extension; `json_path` is a dot-path to a records array inside a JSON file. For credentials, supply `aws_access_key_id` and `aws_secret_access_key` (aliases `access_key_id` / `secret_access_key`) plus an optional `aws_session_token` (alias `session_token`), or set `anonymous: true` to read a public bucket without credentials.
{% endhint %}

{% stepper %}
{% step %}

### Set your credentials

Export your API key and base URL, and keep your AWS secret in its own variable.

```bash
export ALGENTA_API_KEY="$ALGENTA_API_KEY"
export ALGENTA_BASE_URL="https://api.algenta.ai"
export AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"
```

Verify the variables are set:

```bash
echo "${ALGENTA_API_KEY:?set ALGENTA_API_KEY}" >/dev/null && echo "key present"
echo "$ALGENTA_BASE_URL"
echo "${AWS_SECRET_ACCESS_KEY:?set AWS_SECRET_ACCESS_KEY}" >/dev/null && echo "secret present"
```

**Expected result:** three lines print without error — `key present`, your base URL, and `secret present`. If any line errors, export the missing variable before continuing.
{% endstep %}

{% step %}

### Confirm the engine accepts `s3`

Ask the engine which connector types it supports and confirm `s3` is in the list.

```bash
curl -s "$ALGENTA_BASE_URL/v1/connectors/types" \
  -H "Authorization: Bearer $ALGENTA_API_KEY"
```

**Expected result:** a JSON object with a `types` array that includes `"s3"`, for example `{"types": ["azure", "gcs", "s3", ...]}`.
{% endstep %}

{% step %}

### Create the S3 connector

`POST /v1/connectors` stores the connection name, type, and credentials, and returns the connector with `status: "untested"`. Set `bucket`, `key`, `region`, and your credentials.

```bash
curl -s -X POST "$ALGENTA_BASE_URL/v1/connectors" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"Analytics Bucket\",
    \"connector_type\": \"s3\",
    \"config\": {
      \"bucket\": \"acme-analytics\",
      \"key\": \"exports/orders.csv\",
      \"region\": \"us-east-1\",
      \"aws_access_key_id\": \"$AWS_ACCESS_KEY_ID\",
      \"aws_secret_access_key\": \"$AWS_SECRET_ACCESS_KEY\"
    }
  }"
```

For a public bucket, drop the credentials and set `anonymous`:

```bash
curl -s -X POST "$ALGENTA_BASE_URL/v1/connectors" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"Public Data\",
    \"connector_type\": \"s3\",
    \"config\": {
      \"bucket\": \"open-datasets\",
      \"key\": \"census/2020.parquet\",
      \"region\": \"us-east-1\",
      \"anonymous\": true
    }
  }"
```

Capture the returned `id` for the steps that follow:

```bash
export CONNECTOR_ID="$CONNECTOR_ID"
```

**Expected result:** a `201` response with the connector `id`, `"connector_type": "s3"`, and `"status": "untested"`. Credentials are encrypted at rest.
{% endstep %}

{% step %}

### Test object access

`POST /v1/connectors/{id}/test` makes a real request to S3. With both `bucket` and `key` set it runs a `head_object` on that object; with only `bucket` it lists the first object; with neither it lists your buckets. It flips the connector to `live` on success or `error` on failure.

```bash
curl -s -X POST "$ALGENTA_BASE_URL/v1/connectors/$CONNECTOR_ID/test" \
  -H "Authorization: Bearer $ALGENTA_API_KEY"
```

**Expected result:** `{"success": true, "status": "live", "latency_ms": 205, "message": "AWS S3 accessible: s3://acme-analytics/exports/orders.csv"}`, and the connector's stored `status` is now `live`. On failure, `error_type` (for example `not_found` for a missing key, `auth` for bad credentials, or `permission` for a blocked destination) plus `recoverable` guide your fix.
{% endstep %}

{% step %}

### Browse the bucket

`GET /v1/connectors/{id}/browse` lists up to 100 objects in the bucket, so you can pick what to onboard. This returns `409 not_connected` if the connector is not `live` — run the test step first, and make sure `bucket` is set.

```bash
curl -s "$ALGENTA_BASE_URL/v1/connectors/$CONNECTOR_ID/browse" \
  -H "Authorization: Bearer $ALGENTA_API_KEY"
```

**Expected result:** a response with `"connector_type": "s3"`, an `items` array (each entry is an object, with its `selection.object_path`), a `total`, and a `message` such as `"Found 37 objects in s3://acme-analytics"`. Note the `object_path` you want to onboard — for example `exports/orders.csv`.
{% endstep %}

{% step %}

### Onboard one object as a dataset

`POST /v1/data/connect` ties the saved connector to a selection — an object path — and registers the parsed rows as a queryable dataset. Reference the connector by `connection_id` and name the object in `selection.object_path`.

```bash
curl -s -X POST "$ALGENTA_BASE_URL/v1/data/connect" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"dataset_name\": \"sales_orders\",
    \"connection_id\": \"$CONNECTOR_ID\",
    \"selection\": {\"object_path\": \"exports/orders.csv\"},
    \"visibility\": \"private\"
  }"
```

The format is inferred from the object's extension; override it with `file_type` in the connector `config` (one of `csv`, `tsv`, `json`, `parquet`, `excel`), and use `json_path` to point at a records array inside a JSON file.

**Expected result:** `"status": "ready"` with a `dataset_id`, a `schema_summary` (row and column counts), and the `connection_id`. If more than one object matches and no selection is given, you get `"status": "needs_selection"` with a `choices` array — re-call with one `object_path` from `choices`. Save the `dataset_id`:

```bash
export DATASET_ID="$DATASET_ID"
```

{% endstep %}

{% step %}

### Run a governed query

The parsed object is now governed data. `POST /v1/query` runs a deterministic, typed query against the dataset and returns rows under the engine's validated contract.

```bash
curl -s -X POST "$ALGENTA_BASE_URL/v1/query" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"dataset_id\": \"$DATASET_ID\",
    \"select\": [\"region\", \"amount\"],
    \"limit\": 10
  }"
```

**Expected result:** a `200` response with the matching rows and the resolved query plan. Re-running the same request against the same dataset returns the same rows in the same order — governed queries are deterministic by design.
{% endstep %}
{% endstepper %}

## Expected result

You have a saved `s3` connector with `status: "live"`, a browse listing of a bucket's objects, a dataset onboarded from one of them that appears in `GET /v1/data` with a `dataset_id` and a profiled schema, and a first query returning rows. The same intent against the same dataset returns the same deterministic answer, and every access is auditable.

## Other ways to connect

The same endpoints back the `de` CLI and the Python SDK — all three hit the identical API.

```bash
de connectors create connector.json        # POST /v1/connectors from a JSON body
de connectors test $CONNECTOR_ID           # real object-access check
de connectors browse $CONNECTOR_ID         # browse the bucket's objects
de data connect --request connect.json     # POST /v1/data/connect from a JSON body
de data list                               # list registered datasets
```

```python
import os
from decision_engine import AlgentaClient

client = AlgentaClient(
    api_key=os.environ["ALGENTA_API_KEY"],
    base_url="https://api.algenta.ai",
)

connector = client.create_connector(
    name="Analytics Bucket",
    connector_type="s3",
    config={
        "bucket": "acme-analytics",
        "key": "exports/orders.csv",
        "region": "us-east-1",
        "aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
        "aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
)

test = client.test_connector(connector.id)
assert test.success, test.message

result = client.connect_data(
    connection_type="object_storage",
    provider="s3",
    dataset_name="sales_orders",
    connection_id=connector.id,
    selection={"object_path": "exports/orders.csv"},
    visibility="private",
)
print(result.status, result.dataset_id)
```

## Troubleshooting

<details>

<summary>Test fails with `error_type: "not_found"`</summary>

The `head_object` could not find the object. Confirm `bucket` and `key` name a real object, and that the `region` matches the bucket's region. Browsing with only `bucket` set (no `key`) lists objects so you can find the right key.

</details>

<details>

<summary>Test fails with `error_type: "auth"` or `"permission"`</summary>

For `auth`, the credentials are missing or invalid — check `aws_access_key_id` and `aws_secret_access_key`. For `permission` with a privacy-policy message, the S3 endpoint is outside the egress allowlist for your deployment; confirm the `region`/`endpoint_url` and that the destination is permitted. See [Troubleshooting](/help/troubleshooting.md).

</details>

<details>

<summary>Using MinIO or another S3-compatible store</summary>

Set `endpoint_url` to the store's URL and keep `region`, `bucket`, `key`, and credentials as usual. The endpoint must be permitted by your deployment's egress policy.

</details>

<details>

<summary>`409 not_connected` when browsing</summary>

The connector is not `live`. Run `POST /v1/connectors/{id}/test` first; only a connector that passes its test can be browsed. Browsing also requires `bucket` to be set. Editing a connector's `config` resets it to `untested`, so re-test after any change.

</details>

## Next

{% content-ref url="/pages/1hWq4uaSOQ9pNlD9Ox0f" %}
[Query governed data](/guides/governed-query.md)
{% endcontent-ref %}

{% content-ref url="/pages/7daJKOh1ymVXbZvYn7N1" %}
[Google Cloud Storage](/connectors/gcs.md)
{% endcontent-ref %}

{% content-ref url="/pages/mTceQEYGKRUXY86O0KHF" %}
[Azure Blob Storage](/connectors/azure-blob.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.algenta.ai/connectors/s3.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.