Inference Gateway

Stage: any (API) · What it does: call your deployed models through an OpenAI-compatible API — chat completions, embeddings, rerank — so your existing app code works unchanged, but points at your model.

Why it matters

Drop-in. If your app already talks to an OpenAI-style API, changing to your own model is a URL + key swap.
You own the model. Every call is your asset; costs are your fixed infrastructure, not a per-token meter that grows with usage.
One gateway, many models. Chat completions, embeddings, rerank from the same surface.

The API surface

Endpoint	What it does
`GET /api/v1/models`	Lists your tenant's live (`InService`) deployments — each is a `model` id you can call
`POST /api/v1/chat/completions`	OpenAI-style chat completions against any of your deployments
`POST /api/v1/embeddings`	Embeddings (Titan)
`POST /api/v1/rerank`	Rerank a list of documents against a query (cosine over embeddings)
`GET /api/v1/aliases` · `POST /api/v1/aliases`	Friendly model names for your endpoints (admin)

Base URL: https://api.k3ld.com (your tenant's API host).

How to use

Deploy a model (see Ship & deploy) — the endpoint name becomes your model id.
Create an API key (scoped per user with spending limits; the caller's role must be Editor or Admin — Viewer tokens are rejected).
Call your endpoint:

curl https://api.k3ld.com/api/v1/chat/completions \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<your-endpoint-name>",
    "messages": [{"role": "user", "content": "Summarize this contract clause."}]
  }'

Embeddings and rerank follow the same pattern:

curl https://api.k3ld.com/api/v1/embeddings \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "<your-endpoint-name>", "input": ["your text"]}'

Optional: create a friendly alias (POST /api/v1/aliases) so downstream apps call model: "support-bot" instead of a long endpoint name.

Real-world examples

🧑‍💼 SaaS — cut the per-token bill

A SaaS product that was paying per token for every AI feature points the same integration at its own fine-tuned model via the gateway. The app code barely changes (URL + key + model id); the bill becomes a predictable infrastructure line, and the responses are on-brand.

🏦 Financial services — internal copilots on approved infra

A bank's internal copilot uses the gateway with API keys tied to user identities and spending limits, so a runaway script can't blow a budget. Every call goes to the bank's own model.

🏥 Healthcare — one surface, governed

A health org exposes several tuned models (summarizer, scheduler helper) through the gateway; downstream apps select by model, and access control stays centralized (role-gated + key-capped).

Gotchas

API keys are per-user with spending limits — set them before broad rollout. Caller role must be Editor/Admin.
The gateway meters usage (credits) so you can see cost per caller.
Choose the endpoint name deliberately — it becomes your API model id (or alias it).
The gateway enforces a per-tenant rate limit on inference calls.