Every week, someone asks us to build a custom model for a problem a foundation model API would solve in an afternoon. We turn down that work more often than we take it — not out of some purity principle, but because a custom model is a multi-month commitment with ongoing maintenance cost, and if an API call gets you 90% of the way there for a fraction of the price, building custom is usually the wrong call.
The reverse is also true: we've seen teams burn six months wrapping API calls around a problem that needed a purpose-built model from the start, because the API version never got accurate enough, or because sending the data off-premise was never actually an option. Both mistakes are expensive. This is the framework we actually use to tell them apart.
Start with the question an API can't answer
Foundation model APIs — OpenAI, Anthropic, Google — are trained on enormous general-purpose datasets. That makes them extraordinarily good at tasks that look like the open internet: summarization, general classification, drafting text, answering questions about common knowledge. It makes them noticeably worse at tasks that depend on patterns specific to your business that never showed up in their training data — your churn signals, your fraud patterns, your domain's specific visual categories.
The first real question isn't "can an API do this?" — a modern LLM can attempt almost anything if you prompt it well enough. The question is whether the accuracy you get from prompting is actually good enough for the decision riding on it. A support-ticket summarizer that's 85% good is still useful. A fraud model that's 85% good is a liability.
Five questions that actually settle it
1. Does the accuracy bar require domain-specific training data?
If the patterns you're trying to detect are specific to your business — your customers' churn behavior, your specific product images, your industry's document formats — a general-purpose API has never seen that distribution. Few-shot prompting can close some of that gap; it rarely closes all of it. If you've already tried prompting your way to acceptable accuracy and hit a wall, that's usually the signal to train something custom.
2. Can the data leave your infrastructure?
This one is binary more often than people expect. Healthcare data under HIPAA, financial data under various compliance regimes, or simply a customer contract that prohibits third-party data sharing — if the data legally or contractually can't go to a third-party API, that decision is already made for you, regardless of how good the API's accuracy would otherwise be. On-device inference (Core ML, ONNX Runtime) or a self-hosted open-weight model become the only real options.
3. What does the cost curve look like at your actual volume?
API pricing is per-token or per-call, which is cheap at low volume and can get surprisingly expensive at scale. A custom model has a large upfront cost — data preparation, training, evaluation — and a much lower marginal cost per inference after that. The crossover point depends entirely on your volume, but as a rough gut check: if you're running a few thousand inferences a month, an API almost always wins on total cost. At millions of inferences a month, the math starts favoring a custom model, especially if it can run on cheaper infrastructure than a call to a frontier model API.
4. Does latency or offline capability matter?
API calls mean a network round trip, typically a few hundred milliseconds at best, more under load. If the product needs sub-50ms response times, or needs to keep working with no connectivity — an on-device photo classifier, an industrial sensor pipeline — that rules out API-based inference structurally, not just on cost grounds.
5. Who maintains this in a year?
An API call has effectively zero ongoing ML maintenance — the provider handles model updates, though you inherit their behavior changes whether you want them or not. A custom model needs monitoring for drift, a retraining pipeline, and someone who understands the training data well enough to debug it when accuracy degrades. That's real organizational commitment, not a one-time build. This is exactly the work covered under MLOps consulting — and it's worth being honest with yourself about whether your team wants to own it before you commit to custom.
A rough decision table
None of this is precise — every project has exceptions — but as a starting heuristic:
- General task, no compliance constraint, low-to-medium volume — start with an API. Prompt engineering and few-shot examples will get you further than most teams expect.
- Domain-specific accuracy ceiling hit with prompting — consider fine-tuning an existing model before jumping to fully custom training from scratch. It's a meaningful middle ground.
- Data can't leave your infrastructure, or needs offline/low-latency inference — custom or on-device, full stop. The API question doesn't apply.
- High volume, stable task definition, team that can own MLOps — the cost and control argument for custom gets strong enough to justify the upfront investment.
The honest version
Most of the time, the right first move is the cheap one: try the API, measure the actual accuracy against your real data, and only build custom once you have evidence — not a hunch — that it's the ceiling. That's the same approach we bring to machine learning consulting engagements: an honest read before a build commitment, not after.
Try the API first. Build custom once you have evidence it's the ceiling — not a hunch.