Prepare

Stage: Build · What it does: shape raw data into training-ready inputs — ETL/ELT pipelines that extract, transform, and load, plus LLM Format (raw documents → training rows) and ML Infer (batch scoring).

Why it matters

Your data, your shape. Pull from files, object stores, databases, and the catalog; transform with filters, joins, aggregates, casts, splits, and custom Python — no scaffolding to write.
Training-ready output. LLM Format converts raw docs into the exact chat / instruction / text JSONL your fine-tunes and post-training consume.
Runs that recover. Large jobs are sharded and checkpointed — an interrupted run resumes where it left off.

How to use

Open Build → Pipelines and add steps block by block (or start from a template).
ETL / ELT: extract from a source (file, external DB, catalog table), transform with the process blocks (Filter, Aggregate, Join, Union, Rename, Cast, Dedup, Split, Window, JSON Parse, Python UDF…), and load to a catalog table, view, or managed S3 export.
LLM Format: point a document source at the LLM Format step, pick instruction, chat, or text (pretraining) output, optionally add a Task Description ("You are a customer-support assistant…"), and run — you get sharded, checkpointed JSONL on managed S3.
ML Infer: score a prepared dataset with a previously trained model (fraud, churn, classification) and branch on the scores.
Use Schedule for recurring runs and Smoke for a cheap preflight before the full run.

CLI users: every guided flow maps to a CLI command — the console is a visual front-end for the same jobs.

Real-world examples

🏦 Financial services — clean loan data for a credit model

A bank lands raw application extracts in S3 every night. A scheduled pipeline casts fields, dedups applicants, joins the credit-bureau table, filters out incomplete rows, and loads the result to a catalog table the credit model trains on. A Smoke run before the full load catches a malformed data day at pennies.

🏥 Healthcare — de-identified training documents

A health system runs clinical notes through PHI Detect (redact), then LLM Format into instruction JSONL — researchers fine-tune a summarization model on rows that never contain patient identifiers.

🛒 Retail — scoring at scale

An e-commerce team prepares daily sales aggregates with ETL, then runs ML Infer with their demand model over the new partitions, branching high- and low-forecast SKUs into different buckets.

Gotchas

Use globs (sales/2026-*.csv) for rolling data so the same pipeline picks up new partitions automatically.
LLM Format is metered (hosted model calls) — check the cost estimate before running large corpora; the optional LLM-as-Judge doubles the LLM calls but filters low-quality rows.
Large corpora are sharded with checkpoint-resume — avoid re-running while one is in flight (it resumes the previous shards).