How to Accelerate Customer Service Automation with Generative AI

How would we go about building our own generative AI solution?

In order to build a generative AI solution, you’ll first need to make 2 fundamental decisions:

• Is your organization OK with using commercial, managed LLM (like Open AI’s GPT-4), or does your organization need to host your own LLM?

  For most companies, using a managed LLM should be sufficient.

  Some companies (in highly regulated industries) may feel more comfortable hosting their own LLM. (These same companies might also be uncomfortable with all SaaS solutions in general.)

  Note, however, that hosting your own LLM requires significant infrastructure and ML Ops expertise.

• Can your company use an off-the-shelf LLM, or do you need a fine-tuned LLM?

  You should assume that an off-the-shelf LLM will be sufficient for your needs, until proven otherwise.

  In some cases (e.g., highly technical industries), fine-tuning an LLM will result in more relevant responses.

  Note, however, there are many drawbacks to fine-tuning:

  • Fine-tuning requires significant ML Ops expertise
  • Fine-tuning can break content safety guardrails built into LLMs
  • Fine-tuning does not eliminate LLM hallucination until itself

  Read more on why fine-tuning LLMs isn’t always a good idea.

After these 2 foundational decisions are made, for a basic generative AI solution, your team needs to follow these 5 basic steps:

• Use the LLM to generate embeddings for all knowledge the generative AI solution should contain
• Use the LLM to generate embeddings for user questions
• Find the most relevant knowledge for each user’s question
• Send the question and the most relevant knowledge to the LLM
• Detect when the LLM’s response contains hallucination

Steps 1-4 are easy and can be accomplished in 2 hours or so.

Step 5, detecting hallucination, is the most difficult aspect of a generative AI solution. Your team can spend months – or even years – creating a solution that reliably detects and prevents hallucination.

Read more on how to build a generative AI chatbot.

Should I build or buy a generative AI solution?

The decision to build or buy/license a generative AI solution is no different from any other build versus buy decision your company faces.

The key factors in your decision are:

• Does your team have the right skills to build a generative AI solution?
• Does your team have the bandwidth to build and maintain a generative AI solution?
• How long would it take for your team to build a reliable solution?
• What is the opportunity cost of your team building its own generative AI solution? What also could/should your team be working on?
• Strategically, is generative AI “core” to your business? In general, it makes more sense to build core solutions in house, and to buy/license non-core solutions from third parties.
• What’s the cost of buying/licensing and maintaining a third party solution?

While every company is different, in most cases, companies will find that buying/licensing a third party solution is far cheaper, faster, and easier to implement/maintain than trying to build their own generative AI from scratch.

I already have a help desk (or CCaaS). Am I locked into their generative AI?

Just because you’ve already invested in a help desk or CCaaS doesn’t mean you’re locked into the generative AI solution that your help desk or CCaaS provides.

A good commercial generative AI solution should be compatible with all leading help desks and CCaaS solutions – i.e., you won’t have to “rip and replace” your help desk or CCaaS due to the generative AI you’ve selected.

Integrating a third party generative AI solution with your help desk or CCaaS should be almost as simple as onboarding a new Level 1 human agent. You’ll need to:

• Train the human agent;
• Give the human agent access to email, live chat, voice, and any other customer service channel you operate in;
• Provide guidance as to when (and where) to escalate an inquiry to Level 2 support; and
• You’ll need to supervise that human agent.

It’s the same with Generative AI.

Should I just use the generative AI solution from my help desk (or CCaaS) vendor?

It’s likely the case that your help desk or CCaaS vendor has introduced a generative AI solution, and they’re marketing it as a natural add-on to your current solution.

Here are 3 key reasons why you might want to consider generative AI vendors, other than your help desk/CCaaS vendor:

• Your help desk/CCaaS vendor is not AI-native. I.e., their solutions and teams existed long before the advent of generative AI. Just because they’ve built a help desk or CCaaS doesn’t mean they have the requisite capabilities to make a highly accurate generative AI solution.
• Existing help desk/CaaS software solutions aren’t designed with AI-first principles. Just like websites can be “mobile-first,” software can be AI-first. For example, on-demand traditional on-demand analytics UI are typically a mix of chart types, date ranges, and column selections. An AI-first solution could have a “talk to your data” interface that enables you to ask any natural language question, and the solution would provide an answer (inclusive of a graph or report) to the question.
• It’s not in their economic interests to build a great generative AI solution. Your help desk/CCaaS solution is likely priced on a per seat per month basis. Good generative AI solutions will make human agents much more efficient, which has the potential to decrease any seat-based software license fee.

If I buy a generative AI solution, what should be my buying criteria?

Buying criteria for generative AI solutions will vary from company to company. Key buying criteria, however, for any contact center considering a generative AI solution should be:

• How accurate is the solution?
• Is there a free trial?
• Does the solution integrate with my existing tech stack?
• Can the solution work in all my customer service channels?
• Are the sources of answers traceable?
• Is there a way to provide feedback so the AI can continually improve?
Is the solution SOC 2 Type 2 compliant?
• How long will it take to implement the generative AI solution?
• How much effort is required to maintain the solution?
• Is there a way to gradually deploy the solution and build trust over time?
• Which LLM does the solution use, and is the solution LLM agnostic?

Depending on the industry you operate in, additional key buying criteria might include:

• Does the solution sync with my ecommerce platform?
• Does the solution improve conversion in my ecommerce storefront?
• How easy is it to deploy multiple versions of the generative AI to specific customers, customer segments, or user segments?
• Can the solution recognize text in images?
• Can the solution expand self-service capabilities (e.g., change an order, track order status, cancel an order)?
• Can the solution be agent facing (i.e., agent assist) instead of end customer facing?

Should I pay for a generative AI POC?

You should not pay for a basic generative AI POC.

The most basic POC you should be conducting with any generative AI solution is, does the solution provide reasonably accurate answers?

This type of POC should be completely free. Ideally, this type of POC should be entirely self-serve as well (i.e., it does not require a signed POC agreement, several live meetings, an implementation team, and a schedule with milestones).

A POC that demonstrates compatibility with your existing tech stack (and not just your knowledge) is more involved. This type of POC may require some up-front payment or POC fee, especially if you have some legacy or in-house solutions that require integration.

How should I conduct a basic generative AI POC?

To conduct a POC to test the accuracy of a generative AI solution, you should:

(1) Upload a set of knowledge to a generative AI solution. This set of knowledge doesn’t have to be your entire support KB – it can just be a subset of your support knowledge. Let the AI train on the knowledge.

(2) Ask the generative AI solution a series of questions that it should be able to answer, based on the knowledge uploaded. Grade those answers on accuracy. A simple grading rubric could be:

• Worse than average human agent – inaccurate or unable to answer
• As good as average human agent - provides a reasonable, fact-based answer
• Better than average human agent - delivers a highly accurate, detailed response that goes beyond the typical agent's knowledge

(3) Ask the generative AI a series of questions that it should NOT be able to answer, based on the knowledge uploaded. These questions can be completely unrelated to your business, like “Who is the president of the United States?” etc. These questions will allow you to see how much the generative AI hallucinates outside of its knowledge base. A simple grading rubric could be:

• Full hallucination – provides an answer that is completely outside of the uploaded knowledge.
• Partially hallucinates – provides a partially correct, but partially incorrect/hallucinated answer
• Doesn’t hallucinate - correctly indicates that the answer is unknown or outside its scope of knowledge

How do I evaluate the effectiveness of different generative AI solutions?

Go through the same basic POC described above for each/all of the generative AI solutions you are evaluating.

Add up the scores in both (2) and (3). See which solution scores highest.

Keep in mind that a good generative AI solution should give administrators the ability to provide feedback on answers. With this feedback, accuracy scores can improve by 50% with a week or 2 of feedback.

How do I ensure my chosen solution integrates well with my existing tech stack?

With any third party software, there are 3 ways to determine if the solution actually works with your existing tech stack.

• Ask for technical documentation on the integration.
• Ask for a customer who is using the integration live in production.
• (If your environment is unique) Negotiate a POC to prove that the solution works with your environment.

Continue reading Chapter 5 - Generative AI Applications in Customer Service Automation