Everything is digital today and all our activities online and offline leave an almost imperceptible electronic trail, Traffic cameras “see” you cross a junction around the same time each weekday, your Phone lets your favorite coffee shop know when and how long you visit through their “free” Wi-Fi.
While all of these scenarios are perfectly plausible the data collected by these devices or from our interactions with others tend to end up in a data archive somewhere, unused and unanalyzed. Gartner defines dark data as the information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.
Where does dark data come from and what is it made up of?
Most of the time its data that we don’t even know that we are collecting. Vast amounts of data are collected every day, most of this data remains unstructured and unanalyzed.
The ability to explore and make sense of non-traditional data sources such as document metadata, images, voice, and video files or the continuous stream of sensor data from IoT devices remains restricted to a few companies.
There are usually two kinds of dark data. Take an email or a chat message for example. The message contents can easily turn dark if we don’t extract meaning from the text (language) in a way that a computer can analyze it and recall instantly if needed. Secondly, the metadata for the email or chat message — time of day that it was sent, who sent it, who received it, the device or application used to send it, and attachment details like content or location if any — goes dark when messages are archived. While all of this data exists in various repositories/ databases, not much is done to derive any insight from this data. It is stored so it can be retrieved, if needed, in the future.
Unanalyzed data, If not handled well, can also result in a lot of problems such as legal and security problems.
In today’s digital world where every interaction, transaction, and engagement get captured in some system, trying to analyze everything all the time feels like drinking from the firehose.
Service desk and support teams always operate with a sense of “organized chaos” especially on Monday mornings while they work through a typical day of incidents and requests. There’s hardly any time left to step back and figure out what’s really going on and why.
Questions like why is it taking so long to resolve an issue that could have easily been a First Call Resolution (FCR) arise and eventually at some point we realize that the knowledge documentation, triage, or troubleshooting decision trees needed to be updated or expanded. This knowledge and resolution flow are already present inside the resolved tickets, however, it isn’t useful in its current state because it is dark.
Machine Learning and AI to the rescue
What if there was a way for companies to illuminate these untapped sources to derive and deliver insights that lead to better experiences and smarter decisions across the board?
Recent advances in machine learning, computer vision, pattern recognition, natural language processing, and generation are making this possible.
We can now apply AI in our day to day processes to tag data, add labels, classify information, and accurately summarize it before it gets stored filed away in an archive or attached to a ticket. This prevents the data from getting Dark and helps Machine Learning get better in the future to help you predict which customer is likely to contact you on which channel and when, or who your best support agent is for a particular type of incident and may even easily find a needle in the proverbial data haystack when you need it.
For example, in a help desk system like Freshdesk, you could quite easily train a Machine Learning model to suggest what labels and categorization to apply to an incoming ticket, based on how similar tickets have been handled in the past. You could even go a step further to extract the sentiment of the author from the description in the ticket or the source interaction like email, chat, etc to determine the urgency of the issue and alert the support agent. Why stop there? We could even use meta-data in the tickets + the text content in the descriptions and notes as well as the source of the interaction to determine which agent can solve what kind of incidents, a completely objective determination of skill levels. You could even encourage collaboration amongst the agents by assigning credit to agents who help other agents resolve issues by having a leaderboard for earned credit. Learn more
Don’t be afraid of the Dark
Dark Data contains undiscovered, important insights, and represents an opportunity lost.
This data reveals hidden linkages (weak & strong) that tie all the various data points together to reveal what is really going on.
Connecting these dots with Machine Learning and AI can help an enterprise create a unique organizational knowledge graph. This graph can power all your interaction channels and make it easier to find causal relationships, build smarter, more resilient automation such as bots, and most of all augment the capabilities of your workforce or use predictions for recommendations.
iEngage is a platform that can enable you to augment your organization’s intelligence by preventing data from going dark while creating a knowledge graph and delivering insight & context at whatever interaction point they are needed. The same platform powers our FreshEngage app that is available on the Freshdesk app marketplace.
Try it out and see for yourself.