Choosing an AI agent supplier is a different decision from choosing a supplier for a standard web application – an agent works with sensitive data, makes autonomous decisions, and its mistakes show up directly in customer communication or internal operations. Businesses therefore need a concrete checklist of questions, not just an impression from a presentation. This article lists the questions a company should ask before signing a contract – covering data and integrations, security, and what happens after deployment.
Why choosing an AI agent supplier differs from a standard software project
A classic web application or corporate system has relatively predictable behaviour – the same input leads to the same output. An AI agent works differently: it combines a language model, company data and decision-making logic, and its output can vary even for similar inputs. This places different demands on the supplier – they need to know how to limit hallucinations, how to test the agent before deployment, and how to set the boundaries of its autonomy.
When choosing an AI agent supplier, it therefore isn't enough to compare references and portfolios. You need to ask about how they work with data, the architecture of the solution, and what happens when the agent encounters a situation it doesn't recognise. Before a company approaches a supplier, it's also worth clarifying internally which process the agent should solve and why – an overview of where to start with business process automation and what to prioritise can help narrow the scope before the first meeting. A useful complementary resource is also the general checklist for choosing a software supplier, which covers questions common to any IT project – contract terms, communication style, documentation practices.
Questions about data and integrations to ask before signing a contract
Most problems with AI agents after deployment don't arise from a model error, but from poorly prepared data or an incomplete integration. That's why this area should be first on the list of questions when choosing an AI agent supplier.
Where does the agent get its data from
- Will the agent answer solely based on company documentation, or also from the model's public sources?
- How will the supplier ensure the agent doesn't "make up" answers where company data is missing?
- Who updates the source data, and how, when products, prices or internal procedures change?
If you're planning an agent that should answer from internal documentation, it's worth looking at how access via RAG and connecting to company documentation works – the supplier should be able to explain why they chose this approach and not another.
How does integration with existing systems work
- Which systems does the agent need to connect to (CRM, ERP, e-shop, internal tools), and how – via API, webhooks, or batch export?
- Who is responsible if the integration fails or a third party changes its API?
- How will the agent behave if one of the systems it works with is temporarily unavailable?
Security, GDPR and operational responsibility
An AI agent that works with the personal data of customers or employees is subject to the same rules as any other system processing personal data. The supplier should be able to answer questions about security and GDPR compliance clearly, not brush them off with a general statement like "we comply with the legislation".
Specifically, it's worth asking:
- Where is the data processed and stored, and who has access to it?
- How long are conversation logs retained, and can something be deleted from them at the request of the data subject?
- Does the solution have a mechanism to escalate to a human when the agent encounters a sensitive or risky request?
If this topic applies directly to your project, you'll find a more detailed overview of the questions in the article AI agents and GDPR: what to watch out for when processing personal data. It's also worth consulting security questions with a cybersecurity specialist, especially if the agent accesses several internal systems at once.
You're not choosing a supplier for one project, but for the operational period ahead – the agent will keep evolving alongside the business.
How to verify the supplier's experience and way of working
References and a portfolio are a good starting point, but they aren't enough for AI agents – many companies today offer "AI solutions" built by simply connecting an off-the-shelf tool without deeper adaptation to the specific use case. When evaluating a supplier, ask about the process, not just the outcome.
| Area | Question for the supplier |
|---|---|
| Testing | How do you verify that the agent answers correctly before deployment to live operation? |
| Error states | What happens when the agent isn't sure of an answer? |
| Scalability | How will the solution handle an increase in request volume or expansion to further use cases? |
| Ownership | Who owns the code and prompt configuration after the cooperation ends? |
| Team composition | Who exactly will work on the project, and what experience do they have with similar deployments? |
It's also useful to compare whether the business needs a single agent with a clearly bounded task, or a more complex system – in that case it's worth reading about multi-agent systems and when it makes sense to use several AI agents instead of one. A supplier who can explain the difference and recommend a simpler solution when it's sufficient is generally more trustworthy than one who automatically proposes the most complex possible architecture.
What happens after deployment: maintenance, monitoring and further development
Deploying the agent isn't the end of the project, but the start of the operational phase. Businesses that don't ask this question before signing the contract often find themselves, after some time, in a situation where the agent "drifts" – answering less and less accurately because it isn't evolving alongside the business.
Ask:
- Who monitors the quality of the agent's answers after launch, and how often?
- Which metrics are evaluated – for example, the share of conversations escalated to a human, or the share of questions the agent couldn't answer?
- How is the agent updated when company processes change or new data sources are added?
Rather than a single "risk level" figure, it's more useful to have a clear procedure for verifying the supplier step by step:
| Verification step | What exactly to check | Why it reduces risk |
|---|---|---|
| References | Contact at least one previous client directly, not just read the case study | Verifies how the supplier actually communicates and solves problems |
| Testing approach | Request a description of the testing process before deployment to live operation | Shows whether the supplier tests systematically or just "tries it out" |
| Security audit | Verify where data is stored and who has access to it | Reduces the risk of data leaks or GDPR non-compliance |
| Post-deployment plan | Request a specific description of monitoring and maintenance after launch | Prevents the agent from becoming less accurate over time |
Comparing the "before" and "after" state is exactly why it's worth setting measurable indicators right at the start of the project, not only once a problem appears. The methodology of measurement is covered in the article how to measure the ROI of deploying an AI agent in a business – it shows which metrics to track and how to establish a baseline before launch.
Checklist: summary of questions before choosing a supplier
To conclude, here is a summary of the questions a business should have ready for the first meeting with a potential AI agent supplier:
- What data will the agent draw on, and how will "making up" answers be prevented?
- Exactly how will integration with existing systems work, and who is responsible for it functioning?
- Where is the data processed and stored, and does the solution comply with GDPR?
- Is there an escalation mechanism to a human for sensitive or uncertain requests?
- How does the supplier test the agent before deployment, and what experience do they have with similar projects?
- Who owns the resulting code and prompt configuration after the cooperation ends?
- Who monitors the agent's quality after launch, and how does its further development work?
If you can get concrete, verifiable answers to these questions from a supplier – not just general assurances – you have a solid basis for a decision. If you're considering developing an AI agent and want to go through the specific questions for your use case, take a look at our AI and automation solutions or arrange a no-obligation consultation, where you can discuss the scope and requirements of your project.