By Mary Meehan

Consumers are the engine of the fickle, complex, trillion-dollar global economy. They demand that companies understand their wants and needs well enough to make products and services that people will buy, lest the whole interconnected system starts lurching into the red. As the global marketplace has become more competitive, traditional research techniques evolved and the consumer insights business was born. With the birth of the insights industry, companies needed more than just demographic information. They needed to know what people want and why they want it.

Massive Business

New insight and analytic companies, products, tools and services are popping up all the time. Global market research is worth $76 billion, and according to the latest Moore Stephens/WARC report, the martech space is currently valued at $52 billion, with the potential to reach $100 billion. Their report goes on to say that “Driving this growth is data, and more specifically, the realization from marketers that expertise in data leads to actionable insights which drives a greater ROI and provides more control.”

Enter Data, and Make It Big!

We quickly reached a stage where the supply of data as an insights resource has exceeded our knowledge, tools and comfort level. Big data has been an often mystifying business reality for some time, and now a new subset of big data represents an additional untapped opportunity. Companies are collecting and storing what’s now referred to as Dark Data. For most businesses that’s where this data stops. It’s never used.

“According to IDC, 90% of the unstructured data are never analyzed. Such data is known as dark data. According to Gartner, dark data is “the information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.”

Specifically, dark data consists of unstructured data like call-center transcripts, survey verbatims, customer reviews and more. Most companies don’t have accessible, manageable tools to make use of these resources. More importantly, we have seen a hunger for services that regular business people can use to analyze their data without having to schedule time with the overworked, in-short-supply data scientist—if their company even employs one. Companies are looking for tools that provide a look inside their data to reveal insights that can give them a business advantage, one more nugget of understanding that can make the difference in sales and growth.

As the martech category continues to grow, the solutions to our data quandaries are multiplying, and companies need to consider what’s right for their business. Here’s what they should think about when weighing those solutions:

• Fit for those using it: Consumer insights and data scientists are very different skill sets. Who’s working on your project?
• How well the software aligns with and solves for the mission of the department using it: Innovation projects and product tracking require very different technology and approaches
• The scale of your data needs: 1500 survey open-ends need less processing power than 100 million call center data files

Last (and closest to my heart), know what you want to get out of the analysis.

• Do you want to track how many times a customer mentioned “shrunk in the wash”: key words. Are you looking for the WHAT?
• Do you want to know what values are driving consumer interest in that new beverage category? Are you trying to find the WHY?

Each goal requires different technology to deliver the information and insight.

Your data could be telling you things. All kinds of things. But first you need to figure out what to ask it.