Post-Training
Stage: Build → Post-train · What it does: align and harden a fine-tuned model with preference or reasoning optimization — DPO / ORPO (offline preference) and GRPO (online reinforcement with verifiable rewards, the DeepSeek-R1-style recipe).
Why it matters
- SFT teaches what to do; post-training teaches which answers are better — tone, safety, refusal behavior, and (with GRPO) how to reason.
- GRPO with verifiable rewards (run the SQL, execute the tests, exact-match the answer) is the 2025–26 frontier method for math, SQL, code, and extraction.
- All of it runs as LoRA on the same guided workflow with cost estimates.
Which method when
| Method | Data you need | Best for |
|---|---|---|
| DPO | (prompt, chosen, rejected) triples |
Making the model prefer good answers over bad (helpfulness, tone, style) |
| ORPO | Same triples, no reference model | Preference tuning with one loss (simpler, cheaper) |
| GRPO | A verifiable reward (tests, exact match) | Reasoning: SQL that runs, code that passes tests, correct extractions |
How to use
- Build → Post-train.
- Choose the method (DPO / ORPO / GRPO).
- Point at a fine-tuned model (the "student") and provide preference triples — or, for GRPO, an evaluator/reward.
- Review the estimate; optionally run a GPU smoke.
- Launch. The tuned model flows to Ship like any other.
Real-world examples
🏦 Financial services — SQL that must run
A data team uses GRPO on a fine-tuned SQL model with a verifiable reward: the generated query must execute and return the correct result against the warehouse. The model learns to reason — "join keys first, filter after" — because wrong SQL gets a low reward. Accuracy on the internal benchmark jumps from ~70% to ~90%, and every training run is measured, not vibed.
🏥 Healthcare — safe refusal behavior
A health assistant is fine-tuned for summarization, then DPO-tuned so it prefers concise, evidence-based answers and refuses to invent dosing advice. The chosen/rejected pairs encode the policy directly into the weights.
🧑💼 SaaS — tone & helpfulness alignment
A support assistant is ORPO-tuned on "helpful, brand-voiced, escalate-don't-hallucinate" pairs. The same endpoint now scores better on internal quality reviews without a single new feature — just preference data.
🛒 Retail — format-perfect extractions
An e-commerce extraction model is GRPO-tuned where the reward is schema-valid JSON for product attributes. The model learns to emit exactly the fields downstream systems expect.
Gotchas
- GRPO needs a real verifier — a test runner or exact-match check. "Reward" from an LLM-judge is weaker and slower; prefer rules where possible.
- Preference data quality is the moat: clean, unambiguous chosen/rejected pairs beat more data.
- Reasoning training produces longer outputs — check your endpoint's max-token budget and cost per call after tuning.