Your Tech Club

How to automate customer service with AI

Which share of your customer questions can AI really handle? From your mix of questions to the integration with the system that holds the answer.

Joey van VlietCo-founder of Your Tech Club7 min read

Incoming customer questions being categorised automatically
In this article

Most customer questions aren't difficult. Where's my order, how do I cancel, is this invoice right, when is the engineer coming. They take little thought and a lot of time, and they come back every day. That's exactly the kind of work AI is good at. The hard part isn't answering, it's fetching the right answer from your own systems.

Which part of your customer service can you automate?

Split your incoming questions into four types. Factual questions where the answer sits in a system, such as an order status. Explanatory questions where the answer is in your documentation. Change requests that require an action, such as updating an address. And exceptions: complaints, borderline cases and anything sensitive.

The first two types can be largely automated. The third can be too, provided you can define the action safely. The fourth stays with people. So start by counting: what percentage of your tickets falls into each category? That number is your ceiling.

Step 1: look at your mix of questions first

Pull the last thousand tickets from your system and categorise them. You'll almost always see that a small number of topics cause most of the volume. That's good news, because you don't need to handle everything to make a difference.

For each topic, note how much time a ticket takes and whether the answer lives in a system or in someone's head. Anything in someone's head has to be written down first. That's step two.

Step 2: put your answers in one place

An AI that answers customer questions is only as good as the source it draws on. If your information is spread across an outdated manual, a shared folder and the experience of two colleagues, you'll get answers that are sometimes right. That's worse than no answer.

Create one knowledge source with short, up-to-date articles and one owner per topic. Write them the way you'd explain it to a customer. Let the model draw only on that source and have it state where the answer comes from, so a team member can check it. Maintaining that source is worked out under Knowledge Base Manager.

Step 3: connect the agent to the system that holds the answer

For factual questions, the model doesn't need knowledge, it needs access. The order status is in your order system, the invoice in your accounting software, the appointment in the schedule. Connect those systems and let the agent fetch the current data for each question.

Do that through an API with a dedicated account and read-only access where reading is enough. What's possible for each system is under integrations; for ticketing systems and CRM, TOPdesk and HubSpot are a good starting point.

Pay attention to identification: before you return customer data, you need to be sure who you're talking to. Handle that through a login or an existing customer portal, not through an order number anyone can guess.

Step 4: decide when a person takes over

Agree in advance when the agent lets go. At the very least for a complaint, a cancelled contract, a financial correction, the second time the same question comes up, and always when the customer asks for it. That handover must happen immediately, with the whole conversation attached.

Make the route to a person visible instead of hiding it. A customer who gets stuck in a bot with no way out calls with more irritation than they started with. You can automate the triage up front without making the customer wait; that's described under Support Triage and the Service Desk Assistant.

What should you watch for with tone, language and mistakes?

Set out the tone in instructions and test it with real questions, angry ones included. Don't let the model promise things you can't deliver: no delivery times, no goodwill gestures, no amounts unless they come from a system.

Explicitly instruct the agent to say when it doesn't know. An honest 'I'll look into this and a colleague will get back to you' is better than an answer that sounds plausible and is wrong. If you work in several languages, test each language: quality varies.

Take samples. Every week, read back a fixed number of conversations and track how often the answer was right. Without that measurement, you don't know whether it's getting better or worse.

When shouldn't you automate your customer service?

Don't do it if your volume is low; below a few hundred tickets a month, you won't earn back the set-up and maintenance. Don't do it if your information isn't in order, because then you're automating wrong answers.

Don't do it either if customer contact is itself your sales moment. If you work with a small number of large customers you know personally, a quick human is worth more than a quick bot. Automate the preparation behind the scenes instead of the conversation.

And don't do it if the only reason is to hide waiting times. A bot that fills the first fifteen minutes doesn't change the cause.

How do you measure whether it works?

Choose four metrics in advance and measure them before you start too. The share of conversations resolved without a person, the average time to an answer, the share of correct answers in your sample, and satisfaction after the conversation.

If the first metric improves while the third or fourth drops, you're shifting work rather than solving it. Then adjust the mix of questions, not the model.

If you want to know which part of your customer contact has potential and what it would deliver, take a look at the AI Opportunity Audit. The build itself is under AI agent development, and the roles you can deploy are in the AI employees overview.

Frequently asked questions

What share of customer questions can AI handle?

That depends entirely on your mix of questions. Sort your last thousand tickets into four categories: factual questions, explanatory questions, change requests and exceptions. The first three are candidates for automation, the fourth isn't. The share of those first three is your realistic ceiling.

Do customers notice they're talking to AI?

Yes, and they should. Be clear about it and make sure the route to a person is always visible. The EU AI Act sets transparency requirements for systems that interact directly with people.

How do I stop the AI from making things up?

Let it answer only from a defined source or a connected system, and have it state the source. Explicitly instruct it to say when it doesn't know. On top of that, check a sample of the answers every week.

Can an AI agent also make changes for the customer?

It can, provided you define the action tightly and the customer has been identified. Think of changing an address or moving an appointment. Financial corrections, cancellations and goodwill gestures stay with a person.

Does this work in several languages?

Yes, and it's one of its stronger points. Do test each language separately, because quality and tone vary. Keep your knowledge source in one language and let the model translate, so you don't have to maintain several versions.

How long does it take to set up?

For a defined set of questions with an existing knowledge source and working integrations, you're talking weeks. Most of the work is in writing down answers and arranging access, not in the build.

Joey van VlietYassine Seghrouchni
Joey and Yassine, founders

What does this mean for your organisation?

An article gives you direction, but your situation is specific. In a short call, we look at where the opportunity is and what a sensible first step would be.

Keep reading

All articles
A content marketer typing a request to an AI assistant on her laptop while a colleague looks on

AI in practice4 min read

Managing your CMS with your own MCP server

An MCP server for your CMS lets your team edit the website through Claude or ChatGPT, using your design system's components. Every draft is reviewed first.

Read the article
Comparing processes side by side to decide where AI fits and where it doesn't

AI in practice6 min read

When AI is not a good idea

Not every process gets better with AI. Six situations where you're better off doing something else, and the signs you can spot in advance.

Read the article