Choosing a company to build custom AI agents: what to look for
By Yassine Seghrouchni, founderUpdated 6 min read
The short answer
Choose a company that first decides whether an agent is the best solution, because for work that always runs the same way, a fixed workflow is more reliable. If it does become an agent, look at how it gets access to your systems, where a person approves, how you can see what it did and how it's tested to make sure it works well. An impressive demo says little about an agent that works in your systems every day.
About this guide: written by Your Tech Club, with no ranking and no names of other agencies. How we do it ourselves is set out separately under How Your Tech Club works.
What to look for in an AI agent company
An AI agent works independently in your systems. That's what makes it useful, and it's what makes the questions below important. Each point includes what you can ask in the first meeting.
First, the question of whether it should be an agent
An AI agent takes steps on its own: it reads, prepares a decision and carries it out. That's useful when the work calls for judgement, such as understanding a freely written email or spotting an exception.
If the work always runs the same way, a fixed workflow is cheaper and more predictable. A good company weighs that up with you before it builds.
Ask in the meetingWhy is an agent better than a fixed workflow for this work?The task is precisely defined
An agent that does customer service doesn't exist. An agent that reads incoming return requests, looks up the order and prepares a proposal does.
The sharper the task, the better you can test whether it does it well. Ask what the first version does exactly, and what it deliberately doesn't do.
Ask in the meetingWhat does the first version of the agent do, and what does it deliberately not do?Access is limited to what the task requires
An agent needs permissions in your systems. Give it only the permissions the task requires, per system, and make sure you can revoke them.
Ask how the company handles access, who manages the keys and what happens when an employee leaves or a system is replaced.
Ask in the meetingWhich permissions does the agent get in which system, and who can revoke them?A person approves where it matters
Decisions that affect customers, money or commitments need an approval step. The agent prepares, an employee approves.
As the agent proves itself, you can move that step. Ask where the approval sits and how it fits into day-to-day work, so it doesn't become extra admin.
Ask in the meetingWhich actions does the agent carry out itself, and which only after approval?You can see what the agent did
Every action an agent takes must be traceable: what came in, which source it used, what it decided and what it did.
Without that logging you can't explain errors or improve the agent.
Ask in the meetingHow can we see afterwards what the agent did and why?It's tested with your own examples
You measure whether an agent works well with a fixed set of examples from your own practice, including the difficult cases. You run that set again with every change.
After go-live you track how often the agent gets it right and where employees correct it. Ask how the company sets that up and who decides what counts as right.
Ask in the meetingWhich examples do you test the agent with, and how do we measure after go-live whether it keeps working well?The language model can be replaced
Language models change: new versions appear, prices and terms change, old models disappear. An agent that's tied to one model or one supplier ages with it.
Ask how the company decides which model does a task, which data goes to which supplier in the process, and what it takes to switch later.
Ask in the meetingWhich model do you use for what, and what needs to happen if we want to switch?Support after go-live is arranged
An agent is never finished. Processes change, systems get updates and customers ask new questions. Agree who monitors the agent, how errors are picked up and how improvements are planned.
Also make sure the code, the prompts and the configuration are yours.
Ask in the meetingWho monitors the agent after go-live, and who owns the code and the prompts?
Questions to ask in the first meeting
Take this list to your first meetings. A partner that suits you gives a concrete answer to every question, even if that answer is that something can't be done.
- Why an agent, and not a fixed workflow?
- What exactly does the first version do, and what does it deliberately not do?
- Which permissions does the agent get in which system?
- Which actions need approval from an employee?
- How can we see what the agent did and why?
- Which examples from our own practice do you test with?
- Which data goes to which model provider?
- What does it take to use a different model later?
- Who monitors the agent after go-live?
- Who owns the code, the prompts and the configuration?
AI agent or fixed workflow: the difference
A fixed workflow follows rules you set in advance: if this comes in, do that. It's predictable, cheap to run and easy to check. It suits work that always runs the same way.
An agent is given a goal and a set of tools, and decides for itself which steps are needed. It suits work that involves judgement: understanding free text, spotting an exception, combining information from different sources.
In practice the best solution is often a combination: a workflow that keeps the process on track, with an agent in the places where judgement is needed.
How Your Tech Club builds AI agents
This part is about us. The points above apply to any partner, including us: feel free to hold us to them.
We build AI agents that handle work in your own systems, with approval and logging. Before we build, we check whether an agent is the best solution. If a fixed workflow is more reliable, we say so.
With us, an agent sits on a foundation: access to the systems, data in order, integrations, logging and ownership. Human approval sits where it matters, and after go-live we monitor and improve the agent. Code, prompts and configuration stay yours.
If you don't yet know which work lends itself to an agent, the AI Opportunity Audit is a logical first step: five working days, with a report that also says what we wouldn't do.
- ISO 9001 and ISO 27001, certified by Kiwa
- 500 projects delivered
- 4.8 from 127 reviews
- 90% returning clients
- Office on Duivendrechtsekade in Amsterdam
- Founded by Joey van Vliet and Yassine Seghrouchni
Frequently asked questions
What is a custom AI agent?
An AI agent is software that uses a language model to take steps on its own to complete a task: reading information, preparing a decision, recording something in a system. Custom means it's built for your process and your systems, rather than a standard assistant that sits apart from your work.
What does a custom AI agent cost?
That depends on the task, the number of systems it works with and how much oversight it needs. On top of the build there are ongoing costs for the language model, hosting and support. Ask about both, and weigh them against what the work costs now.
Is an AI agent safe for customer data?
That depends on how it's built. Limit its permissions to what the task requires, record which data goes to which model provider, log every action and add an approval step for decisions with consequences. Ask the company building the agent how it has set up information security.
AI agent or automation with a workflow tool?
A workflow tool follows fixed rules and suits work that always runs the same way. An agent suits work that involves judgement, such as understanding free text or spotting exceptions. A combination often works best.
Can an agent work with our existing systems?
If a system offers a way to connect, for example through an API, an agent can work with it. Decide in advance which systems the agent may read and which it may change, and what happens when a system changes.
How do you know whether an AI agent works well?
By testing it against a fixed set of real examples from your practice, including the difficult cases, and repeating that test with every change. After go-live you track how often the agent gets it right and where employees correct it.
What happens when the language model changes?
Providers release new models and retire old ones. A well-built agent can be moved to a different model, after which you run the same test set again. Agree who does that.
