Multi-model AI architecture

Multi-model architecture is not five SDKs in five repos. It is one hop that can send a request to more than one model without each app inventing policy. If every team picks a model in a sprint review, you do not have an architecture. You have a catalog.

This is the routing cut. The gateway is the hop. Putting more than one model into software you already run is LLM integration. Cost is why cheap is not free. Security is why class still wins. No invented latency numbers. No model-router SKU.

If you cannot say which model a production route uses, and who is allowed to change it, you are not multi-model. You are ungoverned.

One hop, many backends

The app talks to your gateway. The gateway picks a backend: a contracted frontier model, a smaller fast model, an open-weight box you host, or a specialist. The app should not import four vendor clients.

Hold:

• One identity on the inbound call.
• One place that maps route to model (or model pool), region, and class.
• Provider keys only on the hop, in a store you control.
• A paved client so a laptop key is not the "just this once" path.

Yes/no:

• You can add a second model without a pull request in every consumer.
• You can pin a route when a new model would break eval.
• You can rotate a vendor key without twelve app deploys.

If those are no, stop buying models. You are multiplying clients.

A "we support all the models" slide is a sales deck. Architecture is the map you can change on Tuesday without a war room.

Route on task and class, not on vibes

The wrong reason to add a model is "the demo looked smarter." The right reasons are task, class, cost, and residency.

Use a small, cheap model when the task is classify, extract, or draft against a tight schema. Use a larger model when the task is open-ended and you have eval that says it wins. Use a hosted or VPC model when class or residency forbids the public tenant.

Class still decides what may leave. Restricted data does not go to a consumer model because the product manager liked the voice. Confidential business data stays on a contracted tenant in a pinned region. Public copy can be cheaper.

Yes/no:

• Two apps cannot pick different models for the same restricted class.
• A region pin is enforced, not documented.
• A cost cap can fail closed or queue, with a page in production.

Do not let "the model is smarter" override class. Smart and leaked is still leaked. Leakage prevention is the payload cut. This post is the router.

Pin versions. Fallback is a product decision.

A model id that floats is a production change you did not review. Pin the version on the route. Promote a new id the way you promote code: eval, then a slice, then default.

Fallback is not "try the next logo if this one 500s." Fallback is a written pair: this route may degrade to that model, for that class, with that timeout. Some routes must fail instead. A write that hits the wrong model is worse than a 503.

Hold:

• Pinned ids in the map, not "latest."
• A fallback that preserves class and region.
• A kill switch per provider that does not require a code freeze.
• An owner who can change the map in hours, not a quarterly architecture board.

Yes/no:

• You know what happens when vendor A is down at 2 a.m.
• Eval still passes on the fallback, or the route is read-only.
• A new model cannot become default because someone changed an env var in a demo.

If fallback is "the app retries the same client," you have one model with extra latency.

Eval is how two models share a route

You cannot run two backends on taste. You need a fixture set the route owner accepts, a threshold, and a way to say the new model lost.

Hold:

• Offline eval before a pin change.
• Online eval on a slice, with traces you can pull.
• A kill that reverts the pin without a full rollback of the app.

The fixture is a store. It inherits class. Do not copy production restricted prompts into a shared model bake-off folder so three vendors can score them.

Yes/no:

• A model change has a named owner and a date.
• You can show last month's pin changes without Slack archaeology.
• A failed eval blocks promote.

"We A/B'd it in a meeting" is not eval. It is a vibe with slides.

Cost is a routing input

A second model that is twice as good and ten times the price is a budget decision, not a default. Put cost on the same map as class. Route cheap work to cheap models. Cap the expensive ones.

Hold:

• Tokens and dollars per route, not a single cloud invoice.
• Budgets that can deny or queue.
• A weekly look at which routes still need the expensive pin.

Do not invent savings. Use the public ROI calculator when the claim is labor hours. Model price is a line item. It is not a customer result.

Yes/no:

• You can name last week's spend by app and model.
• A runaway loop hits a cap before it hits finance.
• Switching a classify route to a small model is a map edit, not a rewrite.

If every request uses the frontier model "to be safe," you do not have multi-model. You have an expensive single.

Hosted weights are still a model

Running an open-weight model in your VPC does not erase the map. It adds a backend with different residency, different ops, and the same identity problem.

Hold:

• The hosted model is still reached through the hop.
• It has a pin, an eval, a budget, and an owner.
• GPU capacity is a capacity plan, not a surprise invoice from the cluster.

Yes/no:

• Apps cannot bypass the hop to hit the internal endpoint with a shared token.
• You can take the hosted model off a route without deleting the box.
• Security can list who may call it.

A pile of weights in a namespace is not architecture. It is inventory.

Scope one route, then a second model

Do not draw a mesh of twelve providers. Put one production app on the hop with one pin. Prove deny, a log pull, and a pin change. Then add a second model for a cheaper task or a documented fallback. High-risk writes stay gated.

Governance is who may change the map. This post is the engineering cut.

We implement the hop in your environment. You own the deploy. We do not publish a router appliance.

If the next model still ships as a new SDK in a new repo, you do not have multi-model architecture. If you want the map designed as engineering, talk to an engineer.