PART 1

At Wootric we collect hundreds of thousands of NPS, CSAT and CES  survey responses every week. We do this across different industries and product categories. Our customers then use our various integrations such as Salesforce, HubSpot, and Slack to route this feedback to relevant teams. Some of these customers like DocuSign and Grubhub have huge user base. This means even with a conservative sampling strategy they get hundreds of pieces of feedback daily.

The Challenge of Analyzing Qualitative Feedback

The quantitative aspects of feedback — NPS, CSAT scores, for example — are relatively easy to aggregate and analyze. It is the qualitative comments that provide rich insight into customer experience, but analysis of unstructured feedback is hard. Someone could read each piece of feedback one by one, but having a human read each comment obviously does not scale. If you can’t or don’t review what customers are telling you, then why have CX program to begin with? This is where machine learning saves the day.

Here is a concrete example of problem we needed to solve:

We set out to solve this multi-label classification and  topic sentiment analysis problem for our customers. We knew that doing it at scale would require some sort of automation using machine learning (ML) and natural language processing (NLP).  Machine learning requires lots of training data — in our case it meant we needed thousands of survey responses manually labeled into different categories. Luckily our customers have had the ability to manually categorize survey responses in our dashboard.  In the last 3+ years, they have categorized thousands and thousands of comments. Fortuitous, but sometimes we would like to think we saw it coming. 😉 At this point, we have significant volume of training data, now we just needed to find the right machine learning algorithms to take a stab at the problem.

Sentiment Analysis

To get our feet wet in machine learning ecosystem, we thought we would start with sentiment analysis because there has been tons of research on this problem over last few years and there are several open source solutions claiming to solve this. We started with Stanford NLP library because it was and is still actively worked on and lots of great research papers have come out of this group. It took us about a week to get our heads around concepts of Tokenization, Lemmatization, Named Entity Recognition (NER), PoS tagging, Dependency parsing, Coreference Resolution, word embeddings and finally to embed the library into a REST API framework. The results were okay but not great for our use case.  We **think** it’s because feedback comments belong to different domain — these are responses to questions as opposed to news article, blogs and tweets. Most of feedback is short, ranging from a couple of words to couple of sentences. This does not give enough “context” for algorithms to find the sentiment. An alternative solution would have been to train our own model using Stanford NLP library but we did not have engineering resources and bandwidth at that point in time. But we had something working. It was a baby step, but a step in the right direction.

Categorizing Feedback Comments

For feedback categorization, we also first looked into a hosted service and open source libraries. The most compelling, trainable and easy to get started solution at that point in time (mid 2016) was Google Cloud Prediction API where you upload a CSV of training data and in few minutes you get a model and REST API to make a prediction. This sounded like a tailor-made solution for us. After all, we had lots of training data that our customers had manually labeled. We were able to quickly format our responses and training labels in order to meet the Prediction API requirements, and saw some initial results.

Results were better for some sets of feedback but horrible for other sets. Dissecting further we realized that the customer feedback set where prediction was more accurate belonged to DevOps, Data Analytics and similar developer centric and data analytics.  There was less success with feedback from e-commerce or other consumer-centric SaaS products. This made sense because most of our customers who spent their time manually labeling feedback had offerings catered to developers — such as New Relic, Docker etc. The bigger downside of Prediction API was that it was a black box so we did not have any ability to tune algorithms. Ironically, Google decided to deprecate Prediction API in favor of their Cloud ML Engine, driving us to improve our internal, customizable prediction methods instead.

Our experiments with Stanford NLP libraries and Google Prediction API gave us a good understanding of the complexity of problem we were tackling, provided more awareness of the ML ecosystem — people and research labs to follow, research papers to read — and finally helped us better understand the nuances of building machine learning models. It’s not as simple as having some training data, copy pasting some code from open source libraries and voila you have a ML solution. There are lots of hype and noise around what ML can do and how to go about doing it right.

At this point, we concluded that there was no shortcut and that we had to invest time focusing on high quality training data, going through various research papers, trying the algorithms in research papers using our training data and have nobs ( i.e. hyperparameters) tuned for our use case.

Delivering insights to customers

In April 2018, after six months in beta, we launched v1 of our product CXInsight™. The platform enables our customers to import and analyze customer feedback from any source. To date, we have analyzed 200,000+ comments pertaining to a wide spectrum of product categories and types of feedback — PaaS, SaaS, E-commerce, Mobile App reviews, employee reviews, social media, etc.

Of note, most of the data we’ve analyzed so far was originally collected using our competitors’ survey platforms. This is sweet validation of our goal: Regardless of how and where you collect customer feedback, Wootric gives you the best analysis.   

In our next series of blog articles, we will talk about how we have used and are using:

• Bag of Words with Naive Bayes and SGD Classifier (Part II)
• word2vec with PCA, Logistic Regression, SVM and SGD Classifier
• Bidirectional LSTM
• CNN
• Custom word embeddings
• Productizing — DevOps around ML
• UX and UI
• Online learning and Human in the loop

Our goal has never been to build a generic text categorization tool. Rather, our focus is to build the best customer feedback analysis platform. We are also aware that our system is never going to be 100% correct so we have made it easy for our customers and our own team to be human in the loop.

I would be remiss not to thank Stanford NLP, Richard Socher, Lukas Biewald, Sebastian Ruder , Google ML research whose research papers, blogs, tutorials, videos and guidance have directly or indirectly helped us build CXInsight product.

Please stay tuned to our Engineering Blog for next series of articles on this topic.  The Hacker News discussion is here.

Customer experience professionals live in a world overflowing with data. Sitting on that wealth of information is frustrating when you know it has incredible potential.

If you are tracking CX metrics, like NPS or CSAT, the numbers help you quantify customer loyalty and satisfaction. But it’s the customer comments that come with those surveys, all of that rich qualitative data, that give you invaluable context for why customers feel the way they do.

Until now, it’s been difficult to analyze qualitative data because it is so unstructured.

This is where tagging comes in.

Using software to analyze qualitative data

Modern customer feedback software comes with the ability to tag customer comments. Tagging feedback has two functional goals: Routing and Insight.

Routing:

Creating a tag for specific stakeholders, e.g. “product”, quickly sorts feedback to be routed to the correct teams for follow-up. Product teams can simply click a button to see verbatim comments regarding feature requests and support teams can be more proactive by checking for comments under a “bug” tag.

Insight:

Tagging comments by relation to product, website, or customer experience helps themes emerge. For example, you may see that most of your detractors are tagged with “shipping” or “price”. This will help you prioritize and address issues in real-time.

Tagging comments manually doesn’t scale, however.

If you are receiving less than 100 comments a month, manually tagging comments can work. But customer comments can pile up just like emails in your inbox. Constant monitoring results in little else getting done. When you find yourself drowning in responses, CX feedback can feel overwhelming — just like your inbox.

This is where using software to auto-tag customer comments saves the day.

Auto-tagging gives you real-time categorization of large quantities text feedback

Auto-tagging automatically sorts qualitative comments for you using AI-powered text analysis, and it happens in real-time. This helps you surface themes and see trends that the human brain has trouble processing on its own.

For example, you may find that pricing issues are mentioned in 80% of your detractor comments in the past couple months, or a new feature is mentioned in 65% of your promoter comments since it launched.

Auto-tagging serves as a dynamic tool to quickly sort massive amounts of feedback for routing to the appropriate teams for insight and immediate follow-up.

We’ve provided the first steps and some suggestions to start auto-tagging in real-time.

How to set up text-match Auto-tags

The time you save by setting up an auto-tagging system can be spent taking action based on the insight lifted out of your survey feedback.

If you aren’t using machine learning software, here are the steps to take in planning your text-match auto-tagging system and some suggestions to get you started.

First, Some Questions to Ask Yourself

When you start to tag your feedback, read every comment you receive in a period of time, perhaps a week or a month, and consider the following:

• What topics/features/issues stand out in your comments?

For example, you may see that many of your customers talk about your Support team’s response time, or the value your product/service has brought to them. These general themes will serve as jumping off points for brainstorming tags and keywords.

• Is there industry or business specific vocabulary or jargon that you might want to track?

For SaaS companies, you may want to include terms like “dashboard”, “widget”, or “in-app” as tags or as text-match keywords. Oftentimes, these terms will be abbreviated, like UI for “user interface”. 

You can even choose to create tags for team members to alert them whenever they are mentioned by name. This might be helpful for a customer support agent who wants to see what customers are saying about their interactions.

As you read through your sample of comments, make a note of the words and phrases you spot customers using. They may be using different terms than the language you and your colleagues use as professionals in your industry.

• Which teams will you be sending customer feedback to and what terms are relevant to them?

You want to be routing comments to the right teams. For example, a product development team will be interested in comments about user interface, integrations, or feature requests while your support or success team may be more concerned with bugs or implementation.

Nested Tags or Parent-Child Tags for Tag Hierarchy (SaaS example)

Once you’ve answered these questions, start grouping specific terms under broader terms. This is going to help you create hierarchy within your tags, also called nested tags.

Nested tags are labels associated by a hierarchy. The ‘sub-tag’ or ‘child tag’ is a tag that is more specific and can be categorized under a ‘parent tag’.

When any of the ‘child-tags’ are text-matched to a comment, feedback platforms will also tag that comment with the corresponding ‘parent tag’. Comments tagged with only the ‘parent tag’ do not include any of the words associated with any of the ‘child-tags’.

This allows you to pull comments that mention any of the specific integrations through the child-tags. At the same time, the broader “integrations” tag pulls comments that mention integrations in general, e.g. suggested integrations from our customers.

Choosing Text-Match Keywords or Keyphrases

For auto-tagging, it is important to choose the right words or phrases to match the tag to the comment. Text-match tags use an “exact match” rule for automation.

This is where having read through some of your current open-ended feedback is useful. You’ve seen the specific words that your customers tend to use when writing about different issues. It may also be helpful to use a thesaurus to come up with synonyms for the words or phrases you choose to match on.

Remember that text-match is very literal, so you will need to include variations on the words and phrases you choose. For example, an “implementation” tag should match on “implement”, “implemented”, “implementation”, and “setup”, as well as “set-up”.

Suggestions

We’ve compiled a list of auto-tags that are commonly used by SaaS businesses. You may be able to use some of these in other industries as well.

As you start to receive feedback you should refine your tags to be more specific to your business needs.

Here’s a list of common tags for SaaS companies to start with:

Human Review: Manually Tagging for Refinement

Monitor your feedback for a couple weeks after you set up your auto-tagging system. If a comment should be tagged, but isn’t, add more keywords to the text-match tag. Manually tag any comments that are difficult to text-match.

A good example would be a comment like “I tried to connect your software to my CRM but it didn’t work.” This comment is clearly related to integration, but text-matching wouldn’t catch this. After manually tagging this comment, you can then add “connect your software” as a keyphrase to the integration tag.

Human review becomes a tool for refining your existing auto-tags, instead of the main workhorse. As time passes, you’ll spend your time scanning for edge cases and new issues or topics that require a new auto-tag.

Do this check periodically to ensure your insight is accurate. Maintaining your valuable tagging system will save you time in the future.

Feedback Routing & Driving Action

Surveying customers is the first phase in your transformation into a more customer-centric company, but you will plateau if you sit on the feedback. Setting up an auto-tagging system means feedback is sent to relevant teams in your organization in real-time. Trends are lifted more easily from qualitative feedback, and your customer-centric organization will be empowered to actively pursue customer happiness.

When was the last time you completed the long survey you ask your customers to fill out? This is a painfully obvious (and obviously painful) exercise you can do to assess the customer experience of your surveys.  If the survey is long, you will probably find it a boring, tedious task to parse and answer the questions. Impatience grows as you face a seemingly endless list of attributes to assess. 

If this is what you are subjecting your customers to, know that you aren’t alone. Many companies are content with the status quo of traditional, bi-annual, 10+ question surveys, or they simply aren’t aware of alternatives.

But times have changed — and your customers aren’t having it.

Traditional, long surveys are a lose-lose situation

Not only do multi-question surveys have the potential to irritate customers, they have disadvantages for business as well.

 You are not hearing from enough customers.  Completion rates are abysmal. Studies show that the longer a survey is, the higher the chance of decreased, delayed, hasty or slapdash responses. So, the information you are getting from customers who are willing to run this gauntlet may not be thoughtful.  

Not hearing from customers often enough. Surveying once or twice a year means you can only react to feedback once or twice a year! In a quickly changing market, this is unacceptable. More agile competitors are going to leave you in the dust.

What can you do to solve this lose-lose situation? Modernize your feedback methodology with microsurveys.

What is a microsurvey?

Microsurveys take a well established, standardized question and use it as the first in a two-step survey. This first question can be used to measure Net Promoter Score, Customer Satisfaction Score, and Customer Effort Score, providing you with quantitative feedback. The second step then provides a way for respondents to give open-ended feedback explaining their score.

Here is an example using an NPS microsurvey shown to a customer who is logged into a SaaS application. A similar microsurvey can also be delivered via email, mobile, or SMS.

Your first reaction might be “How can I possibly get all the information I need with such a short, open-ended survey?  And, how can I make sense of all of the qualitative responses?

Let us walk you through how you can get what you need — and more.

Advantages of always-on microsurveys 

Microsurvey design looks at feedback collection from the customer’s point of view — it should be easy, fast, and relevant. The results are a significantly improved customer experience. Microsurveys provide three key benefits to you:

• Real-time trends
• High response rates
• Better insights

Real time so you never miss a trend:

With support of a customer experience software platform, it becomes easy to survey customers throughout the customer journey.  You can forgo your annual survey campaign and get a on-going pulse of real-time feedback on journey points.  Shortening your surveys allows you to ask customers for feedback more often. By asking the right question at the right time, you increase the chance that an individual will respond to your surveys. Deploying microsurveys across the entire customer journey will bring you both a bird’s eye view of the health of your account and detailed, actionable insights at each touchpoint.

High response rates means you hear from more customers:

Response rates can be as high as 60% for microsurveys, and typically exceed 25%. These numbers can seem miraculous compared to the significantly lower rates that long-form surveys attain. By asking a single question in the right channel at the right time, customer are more willing to give feedback.

Better insights:

Microsurvey responses will reflect what is important and relevant to your customers. Because you are no longer leading the respondent, you will learn things you wouldn’t otherwise learn. The qualitative feedback you receive is rich with context and potential to drive your business priorities.

Now, all of this may sound good but there are still barriers to making the switch, right?

Reasons why you are still using long form surveys

I can’t aggregate survey results when feedback is open-ended!

The advantage of endless Likert scale questions is that responses on a wide range of topics and attributes can be tallied and metricized.   This makes things easier for you on the back end. However, every time a customer must chose a response from a range of values, you are putting the onus of quantification on him or her. You risk asking them to evaluate something they do not know or care about.  Response quality, completion rates, and customer experience all suffer.

A modern approach is to save your scale questions for established CX metric questions like Net Promoter Score, “How likely are you to recommend [business] to friends and colleagues?”, and take the support of machine learning technology to quantify opened survey responses.   

Today, you can take the burden of quantification off of customers and place it squarely on machine learning software. In the past, getting insights from large quantities of qualitative data has been hard, if not impossible. Technology is now available to auto-categorize all of that rich, qualitative feedback. Auto-tagging and sentiment analysis have come a long way!

For example, this dashboard screenshot shows an analysis of auto-categorized NPS feedback. Auto-tagging reveals themes in qualitative comments so you can know what promoters, passives and detractors are talking about in real time.  

I need to ask a series of questions to get important information from our customers.

Every question you add is less likely to be answered with your respondent’s full attention and engagement. Asking a single scale question and an open ended question captures high quality data that is both qualitative and quantitative.

It feels counterintuitive to open up feedback to be a free-for-all; however, customers want to tell you what’s on their mind at the time you survey them. Asking exclusively about what is important to you is frustrating for the customer. Like the saying goes, you can lead a horse to water, but you can’t make it drink.

Getting the information you want is less obtrusive if you send customers a short survey at the right time. For example, you can send an microsurvey asking about how easy a transaction was to complete or how easy a feature was to use. Customers no longer have to reach into the depths of their memory to retrieve their impressions because they just completed the task you are asking about.

Asking for feedback at touch points over time, in the right context, creates a story of your customers’ journey and allows you to see trends, just like how thousands of photos can be combined to create beautiful stop-motion animation.

Beware of using incentives to make up for poor response rates, you will find a higher percentage of “satisficers”, or respondents who select answer options quickly and thoughtlessly to get to the incentive you promised them for “completing” their survey.

Of course, there is a time and place for long surveys.

There is nothing wrong with using a lengthy survey when you really need to — and there will be times when an in-depth questionnaire is appropriate. Here are two examples:

Annual “Brand” survey. Our customers use microsurveys to keep a finger on the pulse of their entire customer base throughout the year for customer journey feedback. Some also use an annual brand survey that supplements by asking many in-depth questions. Even though response rates for this survey may be low, they know they will hear from their most engaged customers on a variety of topics. And, with their microsurvey program,  they still get feedback from everyone else.

User interviews. Product teams may conduct focus groups or interviews to get more sophisticated feedback on feature use, build out an understanding of use cases, and create detailed personas. Microsurveys such as NPS help narrow down who should be included in these focus groups and who would be open to being interviewed.

How to start? Shift your Net Promoter Score program to microsurveys.

If you want to try real-time microsurveys as a baby step towards modernizing your feedback program, use always-on NPS microsurveys as one component of your feedback strategy. You’ll still send out your long, in-depth survey to decision makers like you always have, but now with an early warning system to help you proactively keep your most important accounts.

Entelo was able to double their survey response rate with this method, using NPS microsurveys for a better understanding of customer health. The real-time feedback also meant fewer surprises and easier prioritization when it came to addressing customers’ problems.

Gathering Net Promoter Score feedback isn’t just about receiving critique and feature requests. An important growth component of an NPS customer feedback program is identifying your promoters, then engaging and activating them for upsell, referral or advocacy. But there is another benefit to identifying promoters — the feelings you get from their verbatim comments–gratification, inspiration, motivation, and satisfaction!

Promoters that are finding value in your brand can give some incredibly uplifting compliments that boost morale and provide a sense of pride for employees.

We’ve gathered some example comments from SaaS companies that use Wootric software to measure Net Promoter Score, analyze trends, and close the loop with their customers. Each company provided a screenshot of a favorite promoter comment from their Pearl-Plaza dashboard.

Imagine how the team at presentation software Slidebean felt when this NPS survey response appeared:

The stellar customer service at point-of-sale software Revel Systems is what got this promoter excited:

The team at video coaching platform Sibme was stoked when they read this:

This NPS response gives mortgage software company Maxwell props for their customer-centric culture:

This kind of feedback contributes to feelings of accomplishment and meaning for employees, leading to more engagement and a happier work environment. Research done for the UK government showed that companies with a highly engaged workforce see a 19.2 % growth in operating income over a 12-month period. Additionally, companies with an engaged workforce grow profits up to 3X faster than their competitors.

With this comment, the whole team at e-signature software Signable can see the value they are creating for their customers:

Make sharing customer comments part of your NPS program

To get the most out of your NPS program, share your NPS along with select verbatim feedback across your entire organization. This provides context to your metric. Sharing customer requests and frustrations is a great way to create urgency around service and product improvement. It makes the problem less abstract and gives employees an emotional connection to the work that needs to be done.

In the same way that you would share constructive critique, make it a habit to share promoter comments. This provides you and your employees with the opportunity to celebrate the things you’re doing right and makes it real. When specific teams or individuals get mentioned, reach out to them and share. It is a great way to let them know they did an awesome job! It’s always great to feel appreciated and acknowledged.

Make sharing promoter comments easy by:

1. Sharing it on a Slack channel – with Wootric’s free integration with Slack, you’ve got an easy way to spread the joy from reading promoter comments. Tag folks who were involved with different steps that culminated in the customer’s compliment and give them a pat on the back!
2. Gathering the superstar comments to share at an all-hands meeting or Board meetings, along with your NPS score. This not only gives people the numbers they want to see, but also provides the story behind the number.
3. Featuring promoter comments in your newsletter – whether it’s an employee newsletter or a customer newsletter.  Sharing positivity from customers makes everyone feel listened to and appreciated.

Having comments feed into Slack here at Wootric has helped us to monitor customer sentiment in real-time and keep everyone in the company close to our customers’ experience. Any issues are dealt with quickly, and promoter comments like these get everyone excited.

Promoter comments are a win for everyone

The Net Promoter Score system is a proven way to drive growth, and it is a delight when you get comments like the ones above. It is a positive feedback loop that creates more value and positive experiences for customers and a supportive, enjoyable work environment that encourages engagement for employees.

And to our own customers, know that we truly appreciate the time and thought that goes into every survey response you give us! We are listening.