BYO-LLM: Ship Self-Hosted Llama 3.3 and Qwen 3 Without Lock-In
Your enterprise customer just asked whether your SaaS can run inference inside their VPC, on their GPUs, with their fine-tune of Llama 3.3 70B.
BYO-LLM: Ship Self-Hosted Llama 3.3 and Qwen 3 Without Lock-In
Your enterprise customer just asked whether your SaaS can run inference inside their VPC, on their GPUs, with their fine-tune of Llama 3.3 70B. If the answer is “let me check with engineering,” you’ve already lost the deal. The BYO-LLM (Bring Your Own LLM) pattern is how SaaS vendors in 2026 keep the enterprise tier open without rewriting the product.
Why the adapter sits above the SDK, not below
The mistake most teams make is coupling directly to the OpenAI SDK, then bolting on a if provider == "anthropic" switch six months later. The correct boundary is an internal LLMClient interface with four verbs — complete, stream, embed, tokenize — and provider adapters underneath. Anything that speaks the OpenAI Chat Completions schema becomes free: vLLM, TGI, llama.cpp server, Ollama, Together, Fireworks, Groq. Anthropic and Google get their own adapters that translate at the edge.
Llama 3.3 70B Instruct served via vLLM exposes /v1/chat/completions natively. Qwen 3 32B does the same. The adapter layer never knows the difference.
# config/llm_routes.yaml
tenants:
acme-corp:
default: byo-llama33
providers:
byo-llama33:
kind: openai_compatible
base_url: https://llm.acme.internal/v1
model: meta-llama/Llama-3.3-70B-Instruct
auth_ref: vault://acme/llm_token
timeout_ms: 45000
fallback: anthropic-sonnet-4
default:
default: anthropic-sonnet-4The four contracts that make it real
Capability negotiation. Not every model does tool calling the same way. Qwen 3 uses Hermes-style tags, Llama 3.3 uses its own JSON block. The adapter declares supports_tools, supports_json_mode, max_context, supports_vision. The application asks capabilities first, degrades gracefully second.
Prompt portability. Stop shipping model-specific prompts. Write in a neutral format (system + messages + tool schemas in JSON Schema Draft 2020-12) and let the adapter render the model’s chat template. Both Llama 3.3 and Qwen 3 ship official chat templates on Hugging Face — use them, don’t reinvent.
Data residency and audit. Under Directive (EU) 2022/2555 (NIS2), inference logs on regulated tenants must stay inside the customer boundary. The BYO adapter writes traces to the tenant’s own OpenTelemetry collector, never to your central Loki. This is what unlocks health, finance, and public-sector deals — see how we structure cost-aware routing across managed and self-hosted pools.
Safety fallthrough. BYO endpoints go down. Ours do too. Every route declares a fallback provider and a maximum acceptable degradation (accuracy, not just latency). ENISA’s 2024 AI threat landscape is explicit that availability is a first-class AI risk — treat it like one.
Where CAI draws the line
We ship the adapter layer as a library, not a proxy. A proxy adds a hop, a failure domain, and a place for prompts to leak. A library compiled into your service keeps the trust boundary where it belongs. We’ve seen four production BYO deployments in the last quarter — two on Llama 3.3, one on Qwen 3, one on a customer fine-tune of Mistral Large — and the ones that stayed shipped were the ones that treated the LLM the way they already treated Postgres: pluggable, versioned, tenant-scoped.
If you’re mapping this against your own agent topology, start with our agentic orchestration guide before you touch the adapter code.