Amazon S3

Create an Amazon S3 connector with its real config fields, test object access, browse a bucket, onboard one object as a governed dataset, and run a first governed query against it.

By the end of this page you will have a saved s3 connector that reaches a bucket, a confirmed live access check, a browse listing of the objects it can see, one of those objects onboarded as a queryable dataset, and a first governed query returning rows from it. The engine reads the object in place, parses it (CSV, TSV, JSON, Parquet, or Excel), and registers the rows as governed data under a typed, validated contract.

Prerequisites: an API key and a base URL (see Authentication), and access to an S3 bucket — either AWS credentials or a public bucket. Use https://api.algenta.ai for Algenta cloud, or your own origin such as http://localhost:8000 for a self-hosted deployment.

An s3 connector reads these config fields: bucket and key (the object path) identify the object; region defaults to us-east-1; endpoint_url targets an S3-compatible store (MinIO and similar); file_type overrides the format otherwise inferred from the key's extension; json_path is a dot-path to a records array inside a JSON file. For credentials, supply aws_access_key_id and aws_secret_access_key (aliases access_key_id / secret_access_key) plus an optional aws_session_token (alias session_token), or set anonymous: true to read a public bucket without credentials.

1

Set your credentials

Export your API key and base URL, and keep your AWS secret in its own variable.

export ALGENTA_API_KEY="$ALGENTA_API_KEY"
export ALGENTA_BASE_URL="https://api.algenta.ai"
export AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"

Verify the variables are set:

echo "${ALGENTA_API_KEY:?set ALGENTA_API_KEY}" >/dev/null && echo "key present"
echo "$ALGENTA_BASE_URL"
echo "${AWS_SECRET_ACCESS_KEY:?set AWS_SECRET_ACCESS_KEY}" >/dev/null && echo "secret present"

Expected result: three lines print without error — key present, your base URL, and secret present. If any line errors, export the missing variable before continuing.

2

Confirm the engine accepts s3

Ask the engine which connector types it supports and confirm s3 is in the list.

curl -s "$ALGENTA_BASE_URL/v1/connectors/types" \
  -H "Authorization: Bearer $ALGENTA_API_KEY"

Expected result: a JSON object with a types array that includes "s3", for example {"types": ["azure", "gcs", "s3", ...]}.

3

Create the S3 connector

POST /v1/connectors stores the connection name, type, and credentials, and returns the connector with status: "untested". Set bucket, key, region, and your credentials.

curl -s -X POST "$ALGENTA_BASE_URL/v1/connectors" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"Analytics Bucket\",
    \"connector_type\": \"s3\",
    \"config\": {
      \"bucket\": \"acme-analytics\",
      \"key\": \"exports/orders.csv\",
      \"region\": \"us-east-1\",
      \"aws_access_key_id\": \"$AWS_ACCESS_KEY_ID\",
      \"aws_secret_access_key\": \"$AWS_SECRET_ACCESS_KEY\"
    }
  }"

For a public bucket, drop the credentials and set anonymous:

curl -s -X POST "$ALGENTA_BASE_URL/v1/connectors" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"Public Data\",
    \"connector_type\": \"s3\",
    \"config\": {
      \"bucket\": \"open-datasets\",
      \"key\": \"census/2020.parquet\",
      \"region\": \"us-east-1\",
      \"anonymous\": true
    }
  }"

Capture the returned id for the steps that follow:

export CONNECTOR_ID="$CONNECTOR_ID"

Expected result: a 201 response with the connector id, "connector_type": "s3", and "status": "untested". Credentials are encrypted at rest.

4

Test object access

POST /v1/connectors/{id}/test makes a real request to S3. With both bucket and key set it runs a head_object on that object; with only bucket it lists the first object; with neither it lists your buckets. It flips the connector to live on success or error on failure.

curl -s -X POST "$ALGENTA_BASE_URL/v1/connectors/$CONNECTOR_ID/test" \
  -H "Authorization: Bearer $ALGENTA_API_KEY"

Expected result: {"success": true, "status": "live", "latency_ms": 205, "message": "AWS S3 accessible: s3://acme-analytics/exports/orders.csv"}, and the connector's stored status is now live. On failure, error_type (for example not_found for a missing key, auth for bad credentials, or permission for a blocked destination) plus recoverable guide your fix.

5

Browse the bucket

GET /v1/connectors/{id}/browse lists up to 100 objects in the bucket, so you can pick what to onboard. This returns 409 not_connected if the connector is not live — run the test step first, and make sure bucket is set.

curl -s "$ALGENTA_BASE_URL/v1/connectors/$CONNECTOR_ID/browse" \
  -H "Authorization: Bearer $ALGENTA_API_KEY"

Expected result: a response with "connector_type": "s3", an items array (each entry is an object, with its selection.object_path), a total, and a message such as "Found 37 objects in s3://acme-analytics". Note the object_path you want to onboard — for example exports/orders.csv.

6

Onboard one object as a dataset

POST /v1/data/connect ties the saved connector to a selection — an object path — and registers the parsed rows as a queryable dataset. Reference the connector by connection_id and name the object in selection.object_path.

curl -s -X POST "$ALGENTA_BASE_URL/v1/data/connect" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"dataset_name\": \"sales_orders\",
    \"connection_id\": \"$CONNECTOR_ID\",
    \"selection\": {\"object_path\": \"exports/orders.csv\"},
    \"visibility\": \"private\"
  }"

The format is inferred from the object's extension; override it with file_type in the connector config (one of csv, tsv, json, parquet, excel), and use json_path to point at a records array inside a JSON file.

Expected result: "status": "ready" with a dataset_id, a schema_summary (row and column counts), and the connection_id. If more than one object matches and no selection is given, you get "status": "needs_selection" with a choices array — re-call with one object_path from choices. Save the dataset_id:

export DATASET_ID="$DATASET_ID"

7

Run a governed query

The parsed object is now governed data. POST /v1/query runs a deterministic, typed query against the dataset and returns rows under the engine's validated contract.

curl -s -X POST "$ALGENTA_BASE_URL/v1/query" \
  -H "Authorization: Bearer $ALGENTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"dataset_id\": \"$DATASET_ID\",
    \"select\": [\"region\", \"amount\"],
    \"limit\": 10
  }"

Expected result: a 200 response with the matching rows and the resolved query plan. Re-running the same request against the same dataset returns the same rows in the same order — governed queries are deterministic by design.

Expected result

You have a saved s3 connector with status: "live", a browse listing of a bucket's objects, a dataset onboarded from one of them that appears in GET /v1/data with a dataset_id and a profiled schema, and a first query returning rows. The same intent against the same dataset returns the same deterministic answer, and every access is auditable.

Other ways to connect

The same endpoints back the de CLI and the Python SDK — all three hit the identical API.

de connectors create connector.json        # POST /v1/connectors from a JSON body
de connectors test $CONNECTOR_ID           # real object-access check
de connectors browse $CONNECTOR_ID         # browse the bucket's objects
de data connect --request connect.json     # POST /v1/data/connect from a JSON body
de data list                               # list registered datasets

import os
from decision_engine import AlgentaClient

client = AlgentaClient(
    api_key=os.environ["ALGENTA_API_KEY"],
    base_url="https://api.algenta.ai",
)

connector = client.create_connector(
    name="Analytics Bucket",
    connector_type="s3",
    config={
        "bucket": "acme-analytics",
        "key": "exports/orders.csv",
        "region": "us-east-1",
        "aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
        "aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"],
    },
)

test = client.test_connector(connector.id)
assert test.success, test.message

result = client.connect_data(
    connection_type="object_storage",
    provider="s3",
    dataset_name="sales_orders",
    connection_id=connector.id,
    selection={"object_path": "exports/orders.csv"},
    visibility="private",
)
print(result.status, result.dataset_id)

Troubleshooting

Test fails with `error_type: "not_found"`

The head_object could not find the object. Confirm bucket and key name a real object, and that the region matches the bucket's region. Browsing with only bucket set (no key) lists objects so you can find the right key.

Test fails with `error_type: "auth"` or `"permission"`

For auth, the credentials are missing or invalid — check aws_access_key_id and aws_secret_access_key. For permission with a privacy-policy message, the S3 endpoint is outside the egress allowlist for your deployment; confirm the region/endpoint_url and that the destination is permitted. See Troubleshooting.

Using MinIO or another S3-compatible store

Set endpoint_url to the store's URL and keep region, bucket, key, and credentials as usual. The endpoint must be permitted by your deployment's egress policy.

`409 not_connected` when browsing

The connector is not live. Run POST /v1/connectors/{id}/test first; only a connector that passes its test can be browsed. Browsing also requires bucket to be set. Editing a connector's config resets it to untested, so re-test after any change.

Next

Query governed dataGoogle Cloud StorageAzure Blob Storage