Sentiment is a great tool for analysing text data, and can give you a new dimension for seeing how your customers feel about your company. But sentiment can be misinterpreted and should be approached with caution, as we examined in this post here. In this article we’ll look at how you can avoid these pitfalls and make the most of sentiment data for understanding your customers and their experiences.
1. Decide from the beginning how you’re going to use sentiment
From the beginning of your project you should decide how (if at all) you will use sentiment. Consider the following:
- Is sentiment appropriate for your data?; and
- Is sentiment a primary outcome/reporting measure, or a supplementary piece of information to add context and supplement your other data?
When sentiment is not appropriate
Sentiment may not be appropriate for your data if you typically deal with very short or sparse responses, or you’re dealing with a very narrow or highly technical domain with a distinct vocabulary. For example maybe you sell software for auditing compliance with the new European General Data Protection Regulation (GDPR) and your customers are frequently talking about segments of the legislation or in an acronym heavy jargon - general purpose sentiment models are not going to handle these cases well.
You also need to consider the context of the text data you are analysing and the question (if any) your customers are responding to. For example, if you are analysing responses to the question “What didn’t you like about your experience with us?”, suffice to say that sentiment scoring of the responses won’t add much additional insight.
Sentiment analysis is best used on truly open ended data when you have no way of knowing prior what the sentiment of the response is. In this scenario, sentiment assists you to understand the general tone of your customer’s responses.
Deciding at the beginning not to use sentiment where it isn’t relevant allows you to focus your attention on other important aspects of your data, where actionable insights are likely to be discovered.
Sentiment as a complementary CX metric
For most CX applications though, sentiment is applicable, but it is rarely suitable as a primary outcome metric. If you’re performing NPS surveys or tracking CSAT or other satisfaction metrics, those should remain your primary reporting metrics, and sentiment should be used to enrich the insights you get from your data and provide an additional dimension for understanding your customers.
Treating sentiment as an enrichment to your existing data prevents you from being lead astray by your sentiment model. For example, a highly satisfied customer might say ‘I can't get service this good anywhere else!’ which would be mislabelled as negative using the Google Natural Language API. Try it for yourself here.
Sentiment as a primary metric
One exception where sentiment can make sense as a primary metric is social media analytics, where you may not have high quality structured data work with. In these cases sentiment can be a useful, if noisy, metric to report on. You should decide from the beginning what level of change in sentiment is considered meaningful and your decision should take into account how small variations in sentiment are not likely to be important.
2. Ask the right questions
You need to ask your open ended questions carefully when using sentiment.
Survey questions should:
- focus the customer on the concrete facts of their actual experience;
- should not bias the customer towards thinking about any particular part of the experience;
- should encourage an open dialogue. If you ask a closed question that can be easily answered in one or two words, your sentiment results are going to be disappointing, both because sentiment models tend to do better with longer responses and because you won’t gain any actionable insight from this level of data.
Overall, your survey should ask a small number of general open ended questions, as these are less likely to push the customer in any particular direction. A large number of narrow questions on different parts of their experience will be much more difficult to analyse and interpret.
This is why the standard NPS survey, with just 2 questions (One recommendation score and one open ended ‘Why?’ question), is so effective.
3. Validate your results by reading responses
At this point, you’ve decided how sentiment is going to play a role in your project, you’ve asked the right questions, and now you have your results coming in. Before you dive in and start reporting with sentiment, it’s time to confirm that your sentiment model is making sense on your data. Even the best sentiment models will make mistakes, and you need to make sure that these mistakes aren’t going to impact your analysis.
You certainly don’t need to read all of the responses (that would defeat the point of an automated tool), but you do need to read a sampling, particularly paying attention to any major themes and segments in your data.
Things to watch out for
A good stress test is to look at the extremes like Promoters with negative sentiment or Detractors with positive sentiment. In a perfect world there would not be many of these, but you might be surprised. You might find your sentiment model is confused by linguistic quirks like double negatives. Or that your most satisfied customers still have lots of things they don’t like and are prepared to tell you about in great detail.
Finding flaws and mislabelled examples doesn’t invalidate your sentiment model or the utility of sentiment in general, but it does mean you need to be very careful when interpreting and reporting the results. You might find particular segments where sentiment is inaccurate: perhaps you have a population of customers for whom English is not their first language, meaning just those segments need to be dealt with carefully. Or you might find that the sentiment output is less accurate across the board, meaning when you report sentiment metrics they need to be accompanied by a discussion of the precision of the results and how to interpret differences between segments.
If you follow these three guidelines, sentiment becomes a powerful tool in your CX arsenal. You can use it to quickly drill down to the core aspects of the customer experience and build a complete understanding of your customers, without getting mislead.
Just don’t forget that sentiment is a model like any other, and
“All models are wrong, some are useful” - George Box