The vanity metric problem
We watched someone struggle through a bureaucratic process this week.
What could have been a simple form — or even an email — had been turned into a laborious conversation with a chatbot. One. Step. At. A. Time.
The chatbot asked them to upload a document with all their details. Then immediately asked for their street address. Then their suburb. Then their postcode. Then their phone number. The document was right there. The bot had it. It asked anyway.
What should have taken two minutes stretched into ten or more. The frustration was palpable.
We’ve seen this pattern often enough to wonder: did someone set a target — “we need AI in every workflow” — and then ship something to hit it? Because that’s how you build a solution that’s worse than what it replaced.
When AI actually works.
There are genuinely good reasons to reach for conversational AI and LLM-powered interfaces. They’re a good fit when:
The input is unstructured. When someone needs to describe a problem in their own words, or upload a document that could look a thousand different ways, AI handles that gracefully. A form can’t.
The query isn’t known in advance. If the user doesn’t know exactly what they’re looking for — or needs the system to help them figure it out — AI can navigate ambiguity in ways that rigid form logic can’t.
The problem is genuinely messy. Complex, branching situations where the right next question depends on the previous answer are where conversational interfaces shine.
The intelligence adds value. When AI surfaces a connection the user wouldn’t have found, or synthesises information across multiple sources to give a faster answer — that’s the tool earning its place.
The test.
There’s a simple heuristic that’s served us well:
If your bot asks for information the user already provided, you don’t have an AI feature. You have a slow form.
It’s not a perfect rule. But if you can’t pass it, stop and ask why. The answer is almost always one of two things: the AI isn’t actually using the information it receives, or it was never designed to. Either way, that’s a product problem, not an AI problem.
The accessibility angle.
This matters beyond convenience — and this is where we see well-intentioned teams make the most avoidable mistakes.
For someone using a screen reader, navigating a chatbot interface step by step is significantly more cognitively demanding than completing a well-structured form. Screen readers are highly optimised for form patterns — labels, inputs, focus management, error handling. Chat interfaces introduce interaction patterns that are harder to navigate and require more working memory to track.
For users with cognitive disabilities or high cognitive load — which includes, by the way, anyone in a stressful situation, anyone under time pressure, anyone who is exhausted — forcing a conversation when a form would work is actively harmful.
Inaccessible friction is still friction. The question of whether to use a chatbot is an accessibility question, not just a UX question.
Questions worth asking.
Before you ship an AI feature into any workflow, it’s worth sitting with a few questions:
Is this genuinely better for users, or are we ticking an innovation box? There’s an honest version of this question and a diplomatic version. Ask the honest one.
Who might this exclude? The communities most likely to be harmed by poorly designed AI features are usually the communities you already serve imperfectly. The people navigating complex bureaucratic processes often include people with disability, older users, people for whom English is not a first language.
What’s the simplest thing that solves the problem? Sometimes the most innovative decision is knowing when not to use the shiny tool. A clear, accessible form is often the right answer. Choosing it takes more confidence than defaulting to AI.
The best technology is often invisible.
The goal isn’t to have AI in your product. The goal is to have your users get what they need, as easily as possible, without thinking about what’s powering it.
When AI earns its place, it disappears into the experience. When it doesn’t, it becomes the experience — and not in a good way.
Talk to us about AI strategy → We help organisations work out when AI is the right call — and when it isn’t.
Jasper Streit is the Product Director at The Toy Cartel, an accessible design and strategic creative agency based in Sydney.
