Connect Sources
Stage: Build · What it does: point the platform at the data you already have — object storage or warehouses. The platform reads your existing sources directly — no export, upload, or migration step. In hosted mode runs execute on k3ld-managed compute; with BYO AWS / self-hosted, reads and processing stay inside your account.
Why it matters
- No data migration. Your data stays in your environment and under your security controls.
- One catalog, many sources. S3 and GCS (and more) appear in a single catalog downstream.
How to use
- In Build, open the data sources view and click Add connection.
- Choose the source type.
- Provide the bucket/connection details and any credentials the source requires.
- Save — the connection appears with a table count.
- Head to the catalog (Explore) to browse the tables and see lineage.
Data shape tips
- Object stores: point at a bucket or prefix; the platform catalogs tables/objects beneath it.
- You can reference a single file or a glob pattern (e.g.
raw/customers_*.csv) when you later train.
Hosting modes & connections
Connect works the same way in all three hosting modes — you keep your source and the platform reads it directly, with no export or upload step. Where the bytes are processed depends on the mode:
| Mode | Who runs it | How connections work |
|---|---|---|
| Hosted | k3ld (default — zero AWS to set up) | You connect S3/GCS/warehouse over the network; the platform reads metadata and trains from your sources |
| BYO AWS | You provide the AWS account; k3ld deploys and operates into it | The platform runs in your account and can read in-account sources directly (no egress) |
| Self-hosted on AWS | The full platform runs in your environment, operated by you | Connections live entirely inside your perimeter |
In every mode you never export or upload your data — you connect the source and the platform reads it. Where it's processed: hosted reads on k3ld-managed compute; BYO AWS / self-hosted keeps reads and processing inside your account or perimeter.
Real-world examples
🏦 Financial services — audit-grade data residency
A regional bank keeps consumer transaction data in an internal S3 bucket subject to data-residency rules. In BYO / self-hosted mode, the team connects that bucket and model training reads from it inside the bank's own AWS environment — no third party touches a transaction row. Compliance signs off on "processed where we choose" as a design property, not a bolt-on.
🏥 Healthcare — PHI stays in the vault
A health system connects an encrypted clinical-notes store (GCS) with strict access controls. Researchers fine-tune a summarization model without replicating PHI into a SaaS pipeline. Access is granted per-role; the connection itself is the boundary.
🛒 Retail — seasonal data without pipelines
An e-commerce team connects raw clickstream S3 prefixes each season. Because they use a glob pattern (events/2026_Q4_*.parquet), the same training configuration picks up the new quarter's data without re-engineering.
🏭 Manufacturing — sensors from the edge
An IIoT platform loads machine telemetry into a data lake. The daily refresh brings the newest sensor readings into the catalog, so the quality model retrains on yesterday's data every morning instead of last month's.
Gotchas
- Connection credentials are stored securely and scoped per tenant; roles control who can use a connection.
- If a bucket is empty, the catalog shows zero tables — add data or check the prefix.
- Warehouse connections are validated at creation; a failing connection shows clearly in the UI.