The ultimate guide on how to use call transcripts, email records to understand your customers and take actions by applying text mining, sentiment analysis
Sentiment analysis is the practice of assessing customer input to determine attitudes, emotions and opinions about brands, product lines, marketing campaigns, and more. Sentiment analysis technology relies on natural language processing (NLP), machine learning, and computational linguistics to data-mine sources like social media comments, blogs, and product reviews for relevant input. This input is typically scored as positive, neutral, or negative and made available through reporting tools.
Sentiment analysis allows brands to stay on top of consumer opinion and intervene where possible. For example, by monitoring reviews of their business(es) and products, brands can take actions such as addressing negative reviews online, reaching out directly to dissatisfied customers, and making product improvements. Monitoring reviews has become increasingly important as more consumers, especially younger ones, rely on them for purchase decisions. Staying abreast of what customers are saying online is a complex task that’s made simpler by sentiment analysis software.
Sentiment analysis can help companies speedily identify unhappy consumers; gain essential insight into customer perceptions of its brand, product, operations and agent performance, receive an automated, straightforward and accurate analysis of customer attitudes, and promptly identify root causes of concern and mitigate problems before they undermine the bottom line.
Let’s look at a real-world example. A furniture retailer in North America turned to protonAutoML to better understand how their customers’ satisfaction varied by product. They evaluated the sentiment of all 49 of their product lines and found that the product with the most positive sentiment score was a line of coffee tables. This was a surprise to them because, to date, the sales numbers for those tables had been only average.
They learned from Sentiment Analysis, however, that customers who owned this particular coffee table were overwhelmingly positive during their interactions with the contact center. The retailer dug into the interactions to find out why — and they learned that customers loved how well the table held up over time and that the table’s finish “never chipped.”
The retailer used this insight to revamp the way they positioned the coffee table in their stores and on their website. Now when customers walked into a store or viewed the company’s home page, the “never chipping” coffee table was front and center — along with new messaging that highlighted the strength of the table’s finish.
And the retailer’s sales went up! Prior to their analysis of customer satisfaction by product line, sales of the table had been flat and relatively average compared to other lines of products. But after they found out how much (and exactly why) customers loved the table, they could change their website and store layouts to better highlight the product’s strengths. The result was an increase in sales to the tune of $400,000.
Millions of online shoppers regularly share their views on social media and review sites. Scrutinizing this publicly available customer data allows patterns to be detected and a picture to be compiled of your customer’s mindset.
While customer surveys have always existed in retail, the growth of e-commerce has accelerated the science of sentiment analysis to a new level of sophistication, involving precision-targeted processes designed to dig deeper into customer attitudes.
These include:
- Fine-grained sentiment analysis: Fine-grained sentiment analysis is the detection of positive or negative sentiment at the phrase or sentence level. It begins with a simple binary sentiment — good or bad — that can be divided into more detailed sentiments — very good, good, neutral, bad, or very bad — depending on the application.
- Emotion detection: Emotion detection is used to detect emotional states within a piece of text, including happiness, sadness, and frustration. Typically, this is achieved with lists of relevant words and machine-learning techniques.
- Aspect-based sentiment analysis: Aspect-based sentiment analysis recognizes public opinion regarding a specific aspect of a product or service — such as the battery life of a smartphone — to identify what customers consider its pros and cons.
- Intent analysis: This takes sentiment analysis a step further by identifying what intention is being expressed in the text. For example, is the customer sharing a query, an opinion, or a complaint?
Let’s say you own a restaurant and you scout for online reviews. Sentiment analysis can analyze them and quickly classify them as “positive,” “negative,” or “neutral.” For example, “The food was delicious!” can be easily classified as strongly positive, while “The service sucks” will be identified as a strongly negative comment. Thanks to a “sentiment library,” a sentiment analysis tool can easily identify nouns, verbs, adjectives, and adverbs in these texts and recognize that “delicious” is an indicator of a positive reaction, while “sucks” is an indicator of a negative one.
If all reviews were so straightforward, it would be quite easy to train a machine to do the job. However, most reviews are more subtle and nuanced.
For instance, one reviewer may say, “The food was good, but the music was too loud.” Another might call the restaurant “Not bad.”
Sentiment analysis usually assesses the “score” of a text, placing it on a spectrum of attitudes that goes between +1 (totally positive) and -1 (totally negative). This way, machines are able to distinguish between an enthusiastic comment and a milder, still positive one.
For instance, let’s say your brand has recently put out a new commercial that has been played on television. You can use social media listening to see if people on Twitter have been commenting on your new ad. A sentiment analysis tool will be able to distinguish between different scores of positivity in the two following comments: (1) “I’m obsessed with this new commercial!” and (2) “That’s a cute commercial.” While both of them are positive, the first one will receive a higher score, as it’s clearly more enthusiastic.
As we mentioned earlier, a text can be quite hard for a machine to dissect and interpret.
A user may write: “We had to wait 45 minutes to get a table. Great!” To a human being, it’s clear that the adjective “Great!” is used in a sarcastic way. How do we know it? Because of context. We read the previous sentence, which talks about a long wait time, and we understand that the comment is not positive at all. A good sentiment analysis tool has to be able to detect sarcasm from the broader context, otherwise you’ll end up getting inaccurate data about your brand at the end of the analysis.
Another issue has to do with nuance. The comment “The movie was not bad” is literally saying that the movie was not bad, maybe even good; but it’s also implying that the expectations regarding this movie were so low that the movie is not as bad as one would have expected it to be. This is called “negator.”
Also “intensifiers” can be challenging for sentiment analysis. A user who writes “The company’s comment on this issue was pretty good,” creates a nuance that would not be there if we read the same sentence without the word “pretty.”
There is another technical challenge, a small one but significant that we discussed here.
In conclusion, it’s important not to rely on very basic and simple sentiment analysis tools, which are definitely not going to capture the complexity of human sentiments expressed through text.
As we dig further in understanding this powerful marketing and branding tool, let’s look at the pipeline of steps usually applied in sentiment analysis.
In this pipeline sample, we’ll consider sentiment analysis for a given company or brand.
Step1: Data gathering
First of all, we need the data that we will later analyze. We can gather data from social media, namely Twitter, using scraping tools, APIs, customers’ data feed, and so on. We can also gather data from user reviews on services like Google and Yelp. We’ll be looking for all mentions of the company or brand over a specific period of time. This practice is very common in all forms of social media listening.
Step 2: Text cleaning
Text cleaning tools will allow us to process the data and prepare it for the analysis by removing stopwords (a, and, or, but, how, what…), punctuation (commas, periods…), and checking for stemming. These tools will allow us to “clean” or “strip” the texts from anything that might be irrelevant to the analysis.
Step 3: Sentiment analysis (or opinion mining)
At this point, we can use our sentiment analysis algorithms to analyze the data that we have gathered. As we saw earlier, the most common classification is the spectrum between “positive” and “negative.” However, more refined tools may also identify more complex sentiments such as anger, sadness, and so on. The algorithms will use a sentiment library to identify opinions and classify them.
Step 4: Understanding the results
At the end of the process, we should be able to see the data grouped into major categories. We should be able to see if we have more positive, neutral, or negative reactions. Having each sentiment tagged with its original date is particularly important, as a timeline will show us if we had “peaks” (surges of positive sentiments) or “valleys” (surges of negative sentiments) in specific moments in time. We might therefore be able to find correlations between something that happened on a specific date and a surge of opinions regarding our brand.
While we might identify a peak or a valley while performing sentiment analysis, the opposite might happen — we might notice a surge in mentions on Twitter and we therefore might use sentiment analysis to understand the users’ reactions.
For example, an airline might notice a surge in mentions on Twitter due to some viral content regarding the airline. Given the magnitude of data on the social media network, the company might use data gathering to collect all those mentions; it will then perform sentiment analysis to study the reaction of the public to the viral content. Here’s why sentiment analysis is so important: Understanding whether the reaction is positive or negative can be useful for the company to decide to pursue one of the following actions:
- If the reaction is positive, the airline might want to capitalize on the moment to push a new commercial campaign or pitch the content to the news media.
- If the reaction is negative, the airline might want to prevent a brand crisis by taking action or publishing a statement as soon as possible.
By putting these various methods of sentiment analysis to use, online retailers can acquire the information needed to:
1. Improve customer experience
Using sentiment analysis to reveal customer attitudes allows you to deal with resistance to your brand, products, and services head-on.
This improves your customer’s purchasing journey and their overall impression of your company, which means they’ll want to visit again. They’ll also want to share their positive experience with others.
2. Gain a competitive advantage
Sentiment analysis provides continual feedback on where your company stands in relation to your competitors. More specifically, it identifies precise areas where you outperform rivals, and where you fall short.
Take customer support, for example. People who have grown up with 24/7 internet availability and handheld smartphone convenience do not expect to be kept waiting days for a response to a query. So, if your competitors are responding to customer queries faster than you, sentiment analysis lets you know promptly, so you can take quick action to deal with it.
3. Predict the future
By evaluating the popularity of products and features and the tone of language used when commenting on them, sentiment analysis makes it possible to identify not only what’s hot and what’s not, but also what’s only just beginning to heat up.
The ability to make sales and campaign adjustments according to real-time data ensures you’ll be ready for the latest trends before they happen.
4. Open new markets
As well as identifying emerging trends, sentiment analysis also helps you detect developing new markets. Using sentiment analysis to research what people are talking about highlights their needs, their frustrations, and their passion points, all of which opens up the possibility of targeting previously untapped customers.
5. Build a better brand
Sentiment analysis helps you hone your products, services, and personality. Over time, this builds a reputation of a company that is ahead of the game, responsive to customer needs, and in tune with the mood of the moment.
A strong and attractive brand like this gets noticed, draws in more customers, and increases positive chatter across social media and beyond.
6. Support shopper research
A large part of the online shopping experience involves pre-purchase research by the customer. Analyzing customer feelings and frustrations at this early stage allows you to address the needs of potential shoppers throughout this crucial decision-making process.
For example, this could be accomplished by providing convenient product summaries tailored to meet the needs of prospective customers.
7. Engage with customers
Capturing the tone and subject matter of conversations in your customers’ online communities allows you to connect with your audience through several online channels. Online engagement brings a whole range of benefits, including the ability to drive consumers to your website.
Finally, with the amount of data being generated all over the globe presently. Sentiment analysis remains the best tool for gaining critical insight into various data and automating processes. It is of great importance to every organization, and if carried out in the right way, it is capable of opening up the gold mines in the customer’s opinion and help business automate processes while increasing sales.
About Author: Harsh Gupta has more than 7 years of experience building and directing AI initiatives across diverse industries, amounting to $10M + additional revenue during this period. He has served in technical roles such as Data Scientist at WWF and client-facing roles such as Consultant for Johns Hopkins, Grofers, OSUgiving. He is currently CEO of protonAutoML, a full-service data science consultancy and autoML software provider.
He can be reached out here for any advice or consultation.