Cost Optimization Playbook

Cut operating expenses with an immediate positive impact on SaaS and AI gross margins by targeting unit economics, token consumption, and compute efficiency. Treat inference and cloud compute as variable COGS, not fixed operational expense — that framing is what makes these levers actionable.

The playbook is in three parts. Where the platform already has a knob for a lever, it's called out as On the platform — the rest is guidance for your own stack.

1. AI & inference cost optimization

Model right-sizing

Route simple tasks (classification, routing, basic extraction) to lightweight models — the Llama 3 8B class, Nova Micro, Claude 3 Haiku class — and reserve frontier models exclusively for complex multi-step reasoning.

On the platform: the managed model catalog is metered per model — a cheaper model actually costs less (the credit multiplier tracks real blended price). Use Settings → Models to gate which models can be used for which purpose, so simple tasks can't silently run on frontier models.

Prompt & context caching

Enable vendor-native prompt caching, and use semantic caching to reuse context windows across repetitive queries.

On the platform: the gateway's batch inference already caches duplicate prompts — repeated prompts return cached results instead of re-invoking the model.

Batch execution

Shift non-real-time workloads (nightly indexing, asynchronous processing, background summarization) to batch APIs for a flat discount.

On the platform: Batch Inference runs at ~50% the rate with duplicate-prompt caching built in — move anything that isn't user-facing-latency-critical to batch.

Token budgeting

Enforce strict output-token limits and compress system prompts to minimize per-request overhead.

On the platform: per-user API keys with spending limits cap a runaway integration before it erodes margins; the usage panel shows cost per caller.

2. Cloud compute & data storage

Spot & Savings Plan commitments

Transition stateless worker nodes and dev/staging environments to Spot / Preemptible instances; apply 1-year Savings Plans to baseline database infrastructure.

On the platform: spot is one click in the launch sheet — fine-tune runs at ~60–70% less than on-demand.

Vector database quantization

Downsample vector embeddings (e.g. float32 → int8 or scalar quantization) to cut RAM requirements and vector storage costs by up to 75%. Applies to your own vector stores; the platform's embeddings/rerank surface handles the rest.

Egress & storage pruning

Enforce regional data locality to eliminate cross-region egress charges; delete orphan disk volumes, unattached IPs, and stale log archives.

On the platform: delete (or keep-for-later) idle endpoints so nothing silently bills — teardown stops in-flight GPU jobs; managed-bucket storage only bills overage beyond the free allocation.

3. Usage control

Usage capping

Institute per-tenant rate limits on high-cost AI features to prevent edge-case usage from eroding enterprise margins.

On the platform: model governance (which models for which purpose), instance grants (which GPUs a tenant can use), and per-user API key spending limits are the enforcement points; admins see it all in Settings.

Treating inference and cloud compute as variable COGS rather than fixed operational expenses provides immediate margin expansion without compromising product performance — start with model tiering and batch, then layer in compute commitments and storage pruning.