The promise and the risk of AI support
AI is genuinely good at a narrow but valuable job in customer support: reading a message and deciding what it is about. It is much less reliable at the job many vendors market it for — generating the reply itself — because a generative model has no built-in mechanism to guarantee that what it writes is true.
For most support questions, a confident-sounding wrong answer is worse than a generic one, and order support is a category where that failure mode has a specific, costly shape: telling a customer their package shipped when it did not, or inventing a tracking number that does not exist.
What AI is good at: classification and intent detection
Deciding whether an email is a tracking question, a return request, or something unrelated is a well-bounded task with a right answer that exists independent of the model — the email either is or is not about order status. That boundedness is what makes AI classification reliable in a way that open-ended reply generation is not.
It also generalizes well across phrasing and language without needing an exhaustive keyword list, which is why a classifier prompted for intent detection can catch a WISMO email written in casual French just as well as a formal English one.
What AI should not do: write the reply from scratch
The moment a model is asked to compose the actual reply text — including facts like tracking numbers, delivery dates, or order status — it has been handed a job it is not well-suited for: stating specific facts it has no reliable way to verify unless it is constrained to only repeat facts it was explicitly given.
A model that is fed real order data and instructed to fill in a template is doing something categorically safer than a model asked to write a helpful reply with no constraints, even though both look like AI-generated support from the outside.
Why hallucination is unacceptable in order support
In a lot of AI applications, an occasional wrong answer is an acceptable cost of doing business. Order support is not one of them: a wrong tracking claim does not just fail to help, it actively damages trust and can generate a second, angrier support ticket when the customer discovers the reply did not match reality.
This is why the safest architectures treat what the customer was told and what the model believes as two separate things — the reply text should be traceable back to a specific data source, like an order lookup or a template, not to the model's generated text.
A safer architecture: classify with AI, reply with templates
Splitting the job in two — an AI step that decides intent, and a deterministic step that assembles the reply from real data and a human-written template — keeps the parts of the system that must be exactly right outside of the part that is allowed to be probabilistic.
It also makes the system's behavior auditable: a merchant can read every template in advance and know exactly what a customer could receive, rather than trusting that a model will behave the same way on every future email.
Fallback and escalation: what happens when AI is unsure
A classifier that returns a confidence signal, not just a label, can route uncertain cases to a human instead of guessing. The same applies when the underlying data is missing — if an order cannot be matched, the safest reply asks the customer for more information rather than fabricating a status.
Any AI support system's failure mode should default to doing nothing and flagging the case, not answering anyway. A missed automation opportunity costs a few minutes of manual handling; an automated wrong answer costs a customer's trust.
Questions to ask any vendor claiming AI customer support
- Does the AI write the reply text, or only classify the message?
- What happens when the AI is not confident — does it escalate or guess?
- Can I see every possible reply template before it is ever sent to a customer?
- Is customer data used to train shared models, or kept isolated to my account?
- What is the fallback if the AI service itself is unavailable?
How Valyn applies these principles
Valyn's AI step only classifies whether an inbound email is an order-status question — it never generates the reply text. Replies are assembled from merchant-controlled templates and real Shopify order data, and if the underlying AI classifier fails for any reason, a keyword-based classifier takes over so the pipeline degrades gracefully instead of going down.
If an order cannot be matched with confidence, Valyn asks the customer for an order number rather than guessing, and logs the outcome for manual review. The result is a system where every fact a customer receives traces back to your own Shopify data, not to anything a model invented.