Over the last six weeks, we’ve made lots of improvements to the Citizens Advice Scotland chatbot. Many of those improvements are behind the scenes, so we wanted to share them here.
To be honest, it feels like we have only just got to the start line. The initial four months of chatbot development (May to August) involved lots of logistics and plumbing. Only since September, have we seen the real usage that is key to successful iterative development. (If you want to try the chatbot, it is on the website for various bureaus, e.g. Edinburgh or Perth, and also on the Money Team Talk website.)
We quickly learned that analysing the stats and feedback was something of a blind alley. They didn’t shine much light on the actual experience of using the chatbot. We discover a better approach is to analyse the conversations themselves. This is a wonderful data set. It is painstaking work but worth the effort.
Here’s what the conversations have taught us (and how we are responding):
1. Deal with emergencies
Some clients behave as though they’re not reading content on the chatbot. They just want urgent help. We’ve added flows to the chatbot to ensure those clients get help quickly. (We’ll keep monitoring and working on this. It’s not a one-shot fix.)
2. The too-enticing ‘Something Else’ button
Clients were clicking on ‘Something else’ too often. We can see from subsequent turns of the conversation that another option would have been more appropriate. Something about ‘Something Else’ is too appealing: perhaps there is a desire to see your problem as unique. By removing the ‘Something Else’ button, we nudge clients towards making more effort to serve themselves.
3. Make the conversation more chatty
One of the opportunities of a chatbot is to deliver online information with warmth and empathy. This is a fine line to tread: there are plenty of ways to hit the wrong note.
We are gathering feedback on a new version of the introductory conversation. We are inspired by https://woebothealth.com/, one of the best chatbots we’ve seen. It really communicates with charm, but never pretends to be something it isn’t.
We’d love feedback from CAS advisors and volunteers about their experiences on advice-giving. What are the best things to say? What is the best way to phrase questions? Email niki@sidelabs.org with thoughts or to arrange a chat.
4. Lots of wording changes
For instance, we noticed that ‘Talk to adviser’ was confusing people. Changing this to ‘Request a callback’ sets expectations better.
5. Lots and lots of training data
Now the chatbot is live, we are seeing real questions, with all the peculiarities of real-world usage. We can then hook those questions up to answers so that if someone asks a similar question again, the chatbot knows how to respond.
6. Moving the open question to the start
We’ve seen that most clients would rather type their problem than click through buttons. This approach also better utilises the artificial intelligence capabilities of the Dialogflow chatbot engine. We’re in early tests but watch this space.
Local advice is best
In addition to these improvements which stem from reviewing conversations, we’ve also implemented a feature that allows us to route conversation transcripts to all 59 Scottish bureaus. This came from an opportunity to add the chatbot to the Money Talk Team website. We are delighted to have implemented this, as it delivers on CAS’s strategy of offering a cohesive national brand that provides advice locally.
Moonshot
We are starting to see one goal as to get programmatically ingest all of the advice on CAS’s public advice site and then make it available via the chatbot in a conversational manner. Then we can layer on extra functionality like emergency routing and requesting a callback. Citizens Advice is fundamentally about conversations, and AI means technology around conversations has improved dramatically in the last few years, so it feels like there are some great opportunities here.
Next steps
Over the next few months, our focus will be on helping clients to self-serve. The chatbot does not aim to replace existing channels of support, but aims to assist those who can serve themselves, and so free up capacity across the network. That’s a big potential win in these times of crisis.
