What changed
Customer support is one of the easiest places to overclaim AI progress. A company can say it uses chatbots, but what most people care about is narrower and more annoying: when you open a support chat after a delivery problem or a cancellation issue, do you get a useful human response quickly, or do you get another dead end?
This paper studied that narrower question in a real service setting. Researchers worked with an unnamed meal delivery company and randomized access to AI-generated reply suggestions for 138 customer-service agents. The analysis covered 256,934 online chats from December 2020 through November 2021. Customers first interacted with an automated chatbot, and some were then transferred to human agents who either did or did not have the AI suggestion tool.
The main result was practical. Agents with AI assistance responded 22% faster and produced larger improvements in customer sentiment, measured as a 0.45-point gain on a five-point scale relative to agents without the tool. The study also reports deeper customer engagement during the chats and message-level evidence that AI-assisted replies showed more empathy, information, and solution content.
The gains were not uniform. Less-experienced agents benefited most, which makes this look less like a magic bot and more like a support scaffold that helps newer workers sound more like stronger colleagues. The paper also found important boundaries: AI-assisted humans were especially effective in subscription-cancellation conversations, were least effective on repeat complaints tied to systemic operational problems, and could even hurt customer sentiment after a chatbot comprehension failure if the human handoff felt so fast that customers thought they were still stuck with automation.
What this could change for you
If you spend money with delivery apps, subscription services, or other companies that route you through chat support, this is a believable way AI could improve the experience without asking you to trust a fully autonomous bot. The useful application is not replacing the human. It is helping the human answer faster and more clearly once the conversation actually reaches a person.
For companies, the business lesson is also narrower than the hype. AI looks strongest here as an in-the-loop coaching layer for repetitive but variable conversations. It can help newer agents respond with better tone and structure, which may shorten ramp time and reduce the quality gap between fresh hires and veterans.
The paper also shows what AI support does not solve for customers. If the underlying problem is repeated delivery failure, billing friction, or another operational breakdown, a faster or more empathetic message can only go so far. Good wording cannot substitute for fixing the service failure that triggered the complaint.
What it does not prove
This was one company in one channel. The agents handled text-based support chats for a meal delivery business, not phone support, in-store service, healthcare navigation, or financial disputes. The result does not establish that the same tool would work equally well in other industries or in higher-stakes support settings.
The strongest customer outcome here was sentiment improvement during the conversation, not long-term retention, spending, or complaint resolution durability. Faster replies and better sentiment matter, but they are still intermediate outcomes rather than proof that the company permanently solved churn or loyalty.
The handoff result is also a warning. Customers who first experienced a chatbot comprehension failure could react worse when the AI-assisted human answered unusually fast, because the response still felt machine-like. That means the paper is not evidence that more AI at every stage is automatically better. It is evidence that the sequencing and visibility of automation matter.
The bottom line
AI did not solve customer support in general. It solved a smaller and more credible problem: helping human agents answer routine support chats faster and with better conversational quality. In this field experiment, the gains were real, but they depended on the conversation type and could backfire after a bad chatbot handoff. The honest takeaway is that AI looks most useful here as a human-support tool, not as an excuse to remove the human.
Primary research
Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment
Management Science · 2026 · DOI 10.1287/mnsc.2022.03920
