Big Data is Dead – Long Live Predictive Analysis

Businesses are collecting more and more data these days, but are they making the right use of what they are gathering? The signs indicate that they aren’t, and that in fact all they are creating is more and more complexity. Instead, they should focus on using data to make better decisions.

As a buzzword, Big Data has been around for almost half a decade, and it was supposed to trigger a true revolution in the way companies view their markets and their customers. It was supposed to hand us the tools needed to exploit the endless growth of data.

Instead, many companies and their IT departments complain more loudly than ever about “information overload”, and it’s true: the numbers are staggering. Facebook claims that it processes 2.5 billion pieces of content and over 500 terabytes of data daily. In addition, it collects an average of 2.7 billion “Likes” and 300 million photos a day. Every hour, Facebook scans more than 200 terabytes of data. And that’s just one company, although admittedly a very data-hungry one.

“A customer-focused business with Big Data in its grasp has an unparalleled source of knowledge from an increasing number of sources now; mobile data, social data, transactional data, locational data, financial data, family data, medical data, carbon footprint and consumption data”, writes Theo Priestly in Forbes. “We even have data about data in the form of log data, as Tesla showed us in rebutting the NY Times article a while ago.”

But increasingly, experts are worried that companies aren’t asking the right questions, namely the ones that will fill understanding gaps and help them interpret results. “Really effective analysis combines brilliant technologists and cutting-edge code we all recognize with human understanding, social science research, philosophy, and mission expertise”, says Peter B. LaMontagne, a blogger at Huffington Post.

To make matters worse, data is often stored very traditionally and “crunched” the old way, namely in batch processes. This begs the question: what use is that for real-time operational decisions?

Another worry is that data is being collected indiscriminately and without any kind of fact-checking or ways to determine the credibility of those providing the information. Faced with a variety of sources, companies understandably choose to cherry-pick their data. Employees tend to select data from the easiest sources, or the ones with the least privacy or classification protections. And analysts are often happy with sources that support their own view, ignoring contrary opinions. Nothing is easier than finding information to support whatever argument you’re trying to make.

“Today’s data reliability demands that companies innovate, finding novel ways to pair analyst experience and expertise with automation, overcoming the velocity, volume, and variety of data they see every day”, LaMontagne maintains.

His colleague James M. Connolly of Newsweek says the problem is the focus on bigger and bigger data. In reality, he thinks, it’s just good old data, stupid!

“It’s time to de-emphasize the ‘big’ in „big data“, he writes. By making the whole enterprise analytics concept too complex, rather than focusing on the core idea of using data to make better decisions, that type of complexity can turn nasty, he cautions.

In the end, Big Data is just data. Okay, there’s more of it, and it comes in more flavors; it is generated and transmitted at faster and faster rates. But here are some questions we need to ask ourselves if we want to transform these masses of data into intelligent business decisions:

• Which pieces of data really help us to create new insight and understanding?
• Do we know how was this data was sourced, treated and stored?
• Can we describe the results in language a manager can understand?

Recently, a whole new field of knowledge management has sprung up which goes by the moniker “predictive analysis”, or PA. This is essentially an intelligence technology that aims to create a predictive score for each customer or organization based on a scoring system. PA optimizes activities like marketing campaigns and website behavior to increase customer responses, conversions and clicks, and to reduce churn rates. Based on each customer’s predictive score, actions can be taken with that customer.

In his bestselling book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die (Wiley, 2016), Eric Siegel, a former professor of data mining and Artificial Intelligence at Columbia University, sets out examples of what kind of information can be gathered from complex, multi-faceted data streams, and how these can affect decisions within companies and government institutions, for instance:

• Predicting which people will drop out of school, cancel a subscription, or get divorced before they are even aware of it themselves.
• Why early retirement decreases life expectancy and vegetarians miss fewer flights.
• How European wireless carrier Telenor, and the Obama’s 2012 campaign calculated the way to most strongly influence each individual.
• How Target, a retail chain, figures out you’re pregnant and how Hewlett-Packard deduces you’re about to quit your job.

The tools for achieving these kinds of insights are evolving rapidly. They usually combine elements from such disparate fields as Data Mining, machine learning, and statistics, to extract information from sets of data in order to find patterns and predict future consequences. These range from expensive professional software solutions like SAS Predictive Analytics or IBM SPSS Statistics, which provide ad-hoc analysis, hypothesis, and model testing (among other features) to add-ins for existing ERP solutions such as SAP Predictive Analysis and even freeware such as R from Revolution Analytics or Orange, an open source data visualization and analysis tool.

In addition, enterprises need to create the kind of infrastructure and shared framework necessary for defining problems and ensuring that the analytics are solving the right problem. This will require a repeatable, industrial-scale process for developing the dozens or even thousands of predictive analytic models needed. Finally, a reliable architecture is needed for deploying and managing predictive analytic models in production systems.

In Predictive Analysis, as in most business cases, one size does not fit all. In fact there are distinctly different approaches depending on the industry involved and the aims companies feel they need to follow. Here are a few examples:

Churn Alert: Many businesses worry about losing customers over time. Bringing in new customers can be expensive; retaining existing customers offers a more affordable solution. Preventing churn by identifying signs of dissatisfaction among customers and identifying those likely to leave is one of the main areas in which Predictive Analysis can benefit companies, for instance in fields such as media, insurance, banking, and telecommunications.

CLT: Instead of searching for new customers, many companies seek to make existing customers more profitable. This is one of the main areas of focus for Customer Lifetime Value, or CLT. Predictive Analysis can offer marketing departments and top management in fields such as retail, utilities, banking, and insurance new insights that will allow them to target existing customers in new ways, thus increasing their share of that business segment and making sure that they identify the customers that promise the greatest lifetime value.

Product Predilection: Digital marketers are constantly trying to optimize “right offer, right person, right time” through their campaign management solution. So-called “propensity” models offer to improve response and revenue by identifying customers who are “leaning” towards a certain product or service, by analyzing their online behavior in various social media channels. The goal here is to to predict not only which customers are more likely to buy, but what channels are most likely to reach them.

Predictive Maintenance: Enterprises with big investments in infrastructure and equipment such as automotive manufacturers, logistics and transportation companies, or oil and gas suppliers are increasingly demanding the capability to analyze metrics and data that will keep their precious investments up and running at all times. Predictive Analysis enables them to reliably forecast both probable maintenance events and upcoming capital expenditure requirements, which can help reduce maintenance costs and dodge potential downtime.

Quality Control: Poor quality control can affect customer satisfaction and hence the company’s bottom line (not to mention its share price!). In areas such as pharmaceuticals, manufacturing, automotive or logistics, Predictive Analysis can give vital insights into potential quality issues long before they occur.

Public opinion: Today, keeping the corporate image bright and shiny means more than just a few image ads. On the Internet, customers are constantly exchanging opinions about, and experiences with, your enterprise and its products, and catching everything that’s said about you is a truly Herculean task. A combination of web search, crawling tools, as well as methods for extracting customer feedback from social media posts can give a good picture of your organization’s reputation within your key markets and demographics, along with preemptive suggestions for improving the way people see your company.

Possibly the greatest benefits to be reaped from Predictive Analysis are in the field of retail, in up- and cross-selling, where companies need to be able to make smarter and faster decisions about marketing strategy than ever before.

Say a shoe store has spent years investing in paid searches, but has only recently begun to explore the possibilities of social media advertising. According to the traditional view of Customer Lifetime Value, the cost for gaining a new customer via social media channels would be prohibitive. But with the help of Predictive Analysis, retailers can determine the true value of an individual customer within days or weeks, thus allowing them to precisely target these customers in ways that were impossible back in the days of paid search. The slightly higher upfront costs can prove to be a bargain in disguise.

Or say a retailer wants to create a deeper and more profitable relationship with its customer base. Traditionally, it would analyze past purchase behavior and try to make an informed guess as to what the people will want a few months along the road. Predictive Analysis would help identify what product to recommend, based on what a shopper is likely to buy next.

Forecasting revenue based on historical data is essentially an old-fashioned, backward-looking approach. Much more interesting from a company perspective would be to extrapolate from close observation of new shoppers and combining the results with additional information about the customer mined from a variety of sources, from data brokers to social media platforms, to find out who they are, what channels they prefer to shop through and what demographic group they belong to. This allows smart predictive systems to accurately estimate their probable spending behavior.

Hardly any area of business or industry can afford to ignore these and many other possible advantages brought about by Predictive Analysis. “It’s like a crystal ball into the future”, says Gerd Leonhard, an analyst and head of The Futures Agency, based in Zurich.

No wonder the market for predictive data is booming. Gartner, a firm of analysts, predicts an annual growth rate of 34 percent by 2017, with revenues projected to reach $48 billion. Venture capitalists have been eager to invest in budding Predictive Analytics startups such as Framed Data (which raised 2 million in seed capital and recently was acquired by Square Capital) or 6Sense, which came out of stealth mode in 2014 and immediately raised $12 million in in Series A equity and debt funding.

“Amazon knows what you want to purchase before you even know you want to buy it, and that’s what we’re doing for sales,” said 6Senese’s cofounder and Chief Executive, Amanda Kahlow in an interview with the tech news channel VentureBeat.

Blue Yonder, a German PA start-up with headquarters in Karlsruhe, gained headlines in 2015 by securing funding of $75 million from the global private equity firm Warburg Pincus in 2015, making this the biggest deal for a predictive analytics company in Europe. CEO Uwe Weiss believes that the need for predictive analytics is independent from traditional economic cycles.

“The technology is at the plateau of productivity“, he said in a recent Forbes interview. „People can use this technology now and produce ROI.”

Blue Wonder’s solution is a cloud-based platform aimed at retail companies and offering them innovative ways of determining pricing and automating merchandise planning. “99 percent of business decisions can be automated”, Uwe claims. “That means that it is possible to improve turnover, margins and the customer experience, all at the same time.”

Who would ever have predicted that?

Follow me on Facebook and Twitter or see my videos on YouTube