The question "which model do we use" usually comes up in the first week of an AI agent project — and it almost always gets more attention than it deserves. It gets decided by whatever was in the news last, or by a table of public benchmarks that has almost nothing in common with your workload. In real operation, entirely different things decide the outcome: whether the model calls tools reliably, whether it handles professional Slovak, what it costs across tens of thousands of runs a month, and whether you can replace it in six months without rewriting the system.
The good news is that choosing a language model is one of the more reversible decisions — as long as you treat it from the start as choosing a component, not choosing a platform. The bad news is that most teams cement that reversibility shut within the first two sprints: prompts written around one model's quirks, tool formats tied to one SDK, no evaluation set. This article is about the criteria that actually matter, the difference between a hosted API and self-hosting, and how to verify your choice on your own data. You will deliberately find no specific model names or prices here — those change faster than you can finish reading. The criteria last.
The criteria that actually matter
If you take one thing away from this text, take this: you do not choose a model "in general", you choose it for a specific step of a specific process. An agent that handles company tasks typically has five to fifteen distinct steps inside it, and each has a different demand profile.
Task type
Classifying an incoming e-mail into one of eight categories and writing a legally sensitive customer reply are two entirely different tasks. The first needs consistency and speed, the second needs linguistic judgement and an ability to handle nuance. Break the process into steps and write down for each whether it is triage, extraction, summarisation, text generation, or planning a sequence of actions. Only then start choosing.
Context window and how you use it
A large context window is convenient, but it is not a substitute for selecting the right data. If you dump an entire internal policy into every run, you pay for it in latency and cost, and answer quality often gets worse — the one relevant sentence gets lost among thousands of irrelevant ones. In practice it is cheaper and more accurate to build proper retrieval over your documents, that is RAG over company documentation, and hand the model only what matters. Treat a large context as insurance for exceptional cases, not as the default mode.
Latency
Latency is not measured in milliseconds but in how it feels to the user. A chat agent on your website has to start answering within a second, otherwise the visitor leaves. An agent that processes 4,000 documents overnight can take ten seconds per item and nobody minds. With multi-step agents latency also adds up: five calls at two seconds each is ten seconds of waiting.
Cost at real volume
Cost per call is a misleading number. What matters is cost per completed task — including retries, tool calls, corrections and any verification step. A model that is more expensive per unit but finishes the task on the first attempt is often cheaper in total than a cheap model that needs three corrections. This is exactly one of the factors that shape the price and complexity of a custom AI agent.
Tool-calling reliability
Almost everyone who picks a model by benchmarks underestimates this one. A company agent usually "knows" nothing on its own — it reads from CRM, writes to ERP, opens tickets. What matters is whether the model consistently picks the right tool, respects the parameter schema, does not invent a field that does not exist, and knows how to stop when an input is missing. Differences between models here are larger and more practical than differences in "intelligence".
Slovak and Czech language quality
Models that look excellent in English can produce clumsy sentences in Slovak, decline names incorrectly, or mix in Czech loanwords. If the agent writes to customers or generates text for internal documents, test it on real samples — including your industry's technical vocabulary, which is usually the weakest spot.
| Criterion | Why it matters | How to verify it |
|---|---|---|
| Task type | Different step, different demands | Break the process down and classify each step |
| Context window | Affects both cost and accuracy | Test with real documents, not samples |
| Latency | Decides usability in chat | Measure end to end, not a single call |
| Cost at volume | Decides the return | Cost per completed task, including retries |
| Tool calling | Without it the agent does nothing | A test set for schema compliance |
| Slovak and Czech | Affects brand perception | Blind comparison of outputs with human review |
Hosted API or self-hosted open-weight model
The second big decision is not about quality but about where your data ends up and how much operations you take on. Open-weight models today handle most common company tasks; the question is whether you want to run them.
| Aspect | Hosted API | Self-hosted open-weight model |
|---|---|---|
| Time to first version | Days | Weeks to months |
| Data residency | Depends on region and provider contract | Fully under your control |
| Contractual guarantees (GDPR) | Processing agreement, vendor dependent | Handled internally, no third party |
| Cost profile | Variable, grows with volume | Fixed, pays off only at high utilisation |
| Operational overhead | Minimal | GPUs, monitoring, updates, capacity |
| Access to the strongest models | Yes | Limited to what is publicly available |
The decision is simpler than it looks. If you process ordinary company workloads and the provider can guarantee EU processing and no training on your data, a hosted API is a reasonable choice for most companies — we covered the rules and obligations in the piece on what to watch out for with AI agents and GDPR. Self-hosting makes sense in three situations: you have a regulatory or internal requirement that rules out a third party; your volume is high enough that fixed infrastructure cost works out better; or you need to tune a model on your own data in a way a hosted service does not offer.
Why the biggest model is usually the wrong default
The typical first design runs the whole agent on the strongest model available. It works, and for a demo it is the fastest route. Moving into production reveals that 80% of calls are trivial steps — deciding which branch the process takes, extracting three fields from a form, checking whether the customer replied — and you are paying a premium rate and a premium latency for those.
A more practical pattern is tiering. A cheap, fast model handles routing, classification and simple extraction. The strong model steps in only where actual reasoning happens: synthesis across several sources, phrasing a sensitive reply, planning a sequence of actions. Between them sits an explicit routing rule, not chance.
A bigger model is not a strategy. The strategy is knowing which steps deserve it.
This approach has a side effect: it forces you to write the process down properly. Once you can name which step is "hard" and why, you usually find that half the hard steps can be simplified with a better prompt or a better tool. And if the steps start splitting naturally into several roles, it is worth considering whether this is a case for a multi-agent system instead of a single agent.
Design for model swapping, not for a single vendor
The market moves in months. The model that offers the best price-to-quality ratio for your task today may not hold that position two quarters from now. So what matters more than the choice itself is making replacement cheap.
In practice that means four things. First, the model call belongs behind your own abstraction layer — one place in the code where the vendor changes, not hundreds. Second, keep tool definitions in your own format and map them to a specific provider's format only at the boundary. Third, write prompts structured and explicit rather than "tuned to one model's quirks" — the more tricks, the more painful the migration. And fourth, keep an evaluation set you can run a new candidate through in a single afternoon.
That is how we build AI agents and automation too: the model is configuration, not the foundation of the architecture.
How to evaluate models on your own data
Public benchmarks measure something other than what you need. They measure an average across tasks that are mostly English, academic and independent of your systems. Your question is different: can this model triage our complaints and call our API?
The approach that works is undemanding and can be done in a few days:
- Collect 50 to 200 real cases from production or your archive. Pick them so they match the distribution of normal operation, and add the unpleasant ones — vague requests, typos, attachments in the wrong format.
- Define what a correct answer is. For classification this is easy. For text generation you need criteria a human judges, or at least a checklist along the lines of "contains the order number, does not quote a price, uses formal address".
- Run candidates through the same prompt and the same tools. Change only the model, nothing else, otherwise you will not know what caused the difference.
- Evaluate three things at once: quality, cost per completed task, and the share of runs where the model broke the tool schema or got stuck in a loop.
- Test what happens on failure too. A model that says "I have no basis for this" is more useful in support than one that confidently invents an answer.
When this approach does not make sense
If you are building a one-off internal helper for five people, building an evaluation set is pointless overhead — take a reasonable model, test it manually, done. Formal evaluation pays off where the agent runs at volume, touches customers or money, or where a mistake costs something.
Summary
Choosing a language model for a company AI agent is not a question of "which is best" but of "which is good enough for this specific step at this cost and latency". Break the process into steps and assess task type, context handling, latency, cost per completed task, tool-calling reliability and Slovak language quality. Decide between a hosted API and self-hosting based on regulation and volume, not on gut feeling. Do not put the strongest model on everything — tier it. And above all: build the system so that swapping the model is a configuration change, because within a year you will almost certainly be doing it.
If you are considering deploying an agent and want to verify the model choice on your own data before development starts, get in touch and we will walk through your case.