Cost Control
Stage: all · What it does: keep every run and endpoint on a predictable infrastructure cost instead of a per-token bill — with cost estimates before every run, an optional GPU smoke test, and a run-confidence cache that tells you how likely a config is to succeed before you spend.
The model
- Training/fine-tuning/pretrain: you pay for the instance time (GPU-minute at your instance's credit rate), not per token. Estimates are shown before launch. Spot instances can cut training costs ~60–70%.
- Serving — two parts: (1) endpoint uptime is billed per GPU-minute (auto-sleep pauses idle endpoints; autoscaling is capped by your grant), and (2) every request is metered — each invocation/chat call reserves and records credits at your model's inference rate (per 1K tokens, capped at your
max_tokens), enforced against per-user spend limits, with an inference-ledger row + trace per call. - The bill grows with real usage — that is the guardrail, not a surprise. The predictable part is the infra estimate and the capped instance, not "unlimited requests". A runaway client hits a per-user spend cap instead of an unbounded bill.
The three safety rails
1. Cost estimate before every run
Every pipeline/fine-tune/post-train shows an estimate (instance × time). Budget owners get a number and a green check, not a surprise.
2. GPU smoke test (checkbox)
"Run GPU smoke before launch" runs a short GPU job that validates your data + config before the full run. A malformed dataset or a bad format fails in minutes at pennies — instead of hours into a full run.
3. Run-confidence cache
The platform keeps a cache of past run outcomes (config fingerprint + near-config keys). Before you launch, it tells you: - 🟢 High confidence — an identical config completed successfully before - 🟡 Medium — a near-identical config (same data class) completed - 🔴 Low — identical/near configs failed before, or none seen
This is informational (you can still launch), but it surfaces "this exact recipe has failed twice already" before you spend.
How to use
- Before launching any run, read the estimate and, if shown, the confidence indicator.
- Tick GPU smoke on anything non-trivial — it's the cheapest insurance you'll buy.
- For serving, right-size the instance: the UI shows VRAM fit, hourly cost, and autoscaling caps from your grant.
- When done with an endpoint, delete it (or keep-for-later at ≈$0 + small S3) so idle endpoints don't silently burn.
Real-world examples
🧑💼 SaaS — finance can sign off
A startup's finance lead approves a fine-tune because the estimate says "$12 for this run on a spot g5" and the GPU smoke passed. No surprise line item at month end.
🏦 Financial services — no idle spend
A bank's ML team reviews idle endpoints weekly using the platform's idle-burn report (which bounds estimates by creation time, so fresh endpoints aren't misreported). Stale dev endpoints get deleted and their GPU jobs stopped.
🛒 Retail — confidence saves a bad retrain
Before a weekly retrain, the confidence indicator shows 🔴 — the exact config failed last week. The team fixes the data issue before spending the run, not after.
🏭 Manufacturing — smoke catches a schema change
A pipeline's raw sensor files change column order. The GPU smoke fails in 3 minutes with the real error ("missing column"), saving a 2-hour full run that would have failed anyway.
Gotchas
- Estimates are based on instance × expected time; spot can drop fine-tune cost but is less predictable in availability.
- The confidence cache is informational — a 🔴 doesn't block you, but read the reason (data format? quota? instance fit?).
- Deployed endpoints bill by uptime: delete or stage what you're not using.