Explore Catalog
Stage: Build · What it does: browse every connected source as one unified catalog — collapsible directories, table counts, and lineage showing where data came from and what consumes it.
Why it matters
- One view of everything — no jumping between S3 consoles, warehouse UIs, and CSVs.
- Trust before use. Lineage tells you if a table is "raw" or "already transformed" before you train on it.
- Governed discovery. The catalog reflects what your role is allowed to see.
- Bring your own files. Upload and download files directly, and pull HuggingFace datasets — without leaving the catalog.
How to use
- In Build, open the catalog (Explore) view.
- Expand directories (catalogs → schemas) to browse tables.
- Use the filter box to find a table by name.
- Click a table to open its lineage drawer — sources upstream and consumers downstream.
- Use Expand all / Collapse all for large catalogs.
- Open Files to manage files in your account's managed storage — upload or download data files directly.
- Search the HuggingFace Hub from the catalog and import a public dataset into your managed storage.
Files & the HuggingFace Hub
- Direct upload & download. The catalog's Files view covers your account's managed storage — upload a file (it lands in a folder like
uploads/), download any file, and everything you drop in shows up in the same dataset pickers the rest of the platform uses. - HuggingFace connection. Search the HuggingFace Hub right from the platform and import a public dataset into your managed S3 as JSONL — top datasets are pre-filled (e.g. DPO/ORPO preference pairs like
argilla/ultrafeedback-binarized-preferences). The import is async: it runs as a job and then shows up in every dataset picker, ready to fine-tune or post-train. Curated HuggingFace base models are also surfaced for fine-tuning, with an "on k3ld" marker for the ones the platform hosts.
Real-world examples
🏦 Financial services — proving data provenance
An auditor asks "where does this model's training data come from?" The team opens the training run's source table in Explore, and the lineage drawer shows the raw feed → cleansed layer → feature set chain. The answer is one click, not a month-long data-archaeology project.
🏥 Healthcare — safe discovery for analysts
Nurses/analysts browse de-identified summary tables while researchers see the fuller set — lineage + RBAC make the catalog safe for self-service. Nobody is handed a dump of the whole lake.
🛒 Retail — finding the right feature table
A merchandising analyst needs "loyalty program spend by customer." Instead of guessing, they search the catalog, see two candidate tables, and use lineage to pick the one that is a direct product of the billing feed (not a stale copy).
Gotchas
- Lineage is as good as the jobs that populate it — pipelines that read/write through the platform record lineage automatically.
- Very large catalogs are bounded in the view (a clear "… N more tables" marker) to stay fast.