At some point, a model your product depends on will be retired. You will get a notice with a date on it. Whether that date is an inconvenience or a crisis was determined months earlier, by architectural choices that seemed unimportant at the time.

What actually breaks

Naively, swapping a model looks like changing a string. In practice, four things break.

Prompt behavior. Prompts are tuned — often unconsciously — to a specific model's tendencies. The replacement is more verbose, or more literal, or formats lists differently. Output that downstream code depends on shifts shape.

Output structure. If you parse free text, any change in phrasing breaks parsing. This is the most common source of silent, hard-to-diagnose failures after a swap.

Latency and cost profile. The replacement may be slower or pricier per token, or more verbose at the same price, which is the same thing. Features tuned to a latency budget quietly fall out of it.

Quality, unevenly. The new model is better on average and worse on three specific things you depend on. Averages hide this. Only task-level evaluation surfaces it.

Five design decisions that make this routine

1. Never call a provider SDK from business logic. Every model call goes through one internal interface that takes a task name, not a model name. Business code says draft_customer_reply, never call GPT with this prompt. Swapping becomes a change in one layer instead of a search across your codebase.

2. Store prompts as versioned configuration, per task and per model family. Not inline in code. When you add a model, you add a prompt variant; you do not edit the one prompt everyone shares and hope.

3. Demand structured output everywhere it matters. Ask for JSON against a schema and validate it. A schema violation is a loud, catchable failure. A prose formatting change is a silent one that corrupts data for a week before anyone notices.

4. Keep a graded evaluation set per task. Fifty to two hundred real examples with known-good outputs, graded automatically where possible. This is the single highest-value artifact you can build. It converts "does the new model work?" from a debate into a test run.

5. Make the model a runtime setting, not a deploy-time constant. You want to shift a percentage of traffic to a candidate model, watch the graded results, and roll back in seconds without shipping code.

The migration, when it comes

With those five in place, a deprecation notice triggers a sequence that takes hours rather than weeks.

Run the evaluation set against candidate replacements. Compare per-task, not in aggregate — you are looking for the tasks where the new model is specifically worse. Write prompt variants for the tasks that regressed and re-run. Shift 5% of live traffic and watch real-world quality signals, especially human edit rates at the approval gate. Ramp up. Retire the old model on your own schedule, before the forced date.

The edit rate at the approval gate deserves special mention: it is the best early-warning signal most teams already have and do not look at. If humans start correcting more of the drafts after a model change, quality dropped, regardless of what your evaluation set says.

What this costs

Roughly two to three weeks of engineering to put in place properly, and ongoing maintenance of the evaluation set — perhaps a few hours a month. Against that: every future model change, and there will be several a year, becomes a routine operation.

Teams that skip this pay the cost anyway, in unplanned migration projects that arrive with a deadline attached and no ability to verify whether the replacement is actually working.