The world is changing. Fast. Particularly for marketers. They are used to throwing money around and hoping that the continuous reinforcement of brand will make their customers buy, no matter how bad the experience is.

It seems to have worked fairly well. Particularly for ad agencies. According to published above the line ad spend numbers for 2012 the big four banks spent over R1.6 billion mostly it seems to encourage over-committed households to take on more debt, while SAB spent R752.8 million suggesting that we get drunk as responsibly as possible. These are staggering numbers when you consider how little marketers know about the effectiveness of this investment. They are also likely to be staggering in comparison to what these same companies spend on research or innovation, numbers that sadly are not published.

It appears that the game is about to change. No sooner have marketing departments perfected the art of hiring really good looking people to place ads across a range of social and anti-social media it turns out that what marketers need to do is bring in data junkies who are in the main socially awkward, particularly around very good looking people. More and more the focus is on using data and analytics to guide decision-making. That is going to pose an enormous organisational challenge. Ad agencies too need to change their game.

Smart marketers are figuring out that there is more to assessing the effectiveness of a campaign than conducting brand tracking studies, as useful as they might be for research houses. The smartest marketers are increasingly turning to the potential value embedded within the data they gather and retain on how their customers behave to optimise marketing and promotional spend. More widely across the organisation smart companies are using data to drive experimentation and innovation. It is changing the face of marketing, research and innovation as we know it.

In some businesses customer data is generated as a by-product of offering on-going services to known customers. This is the case in the financial services industry. In other industries there is serious planning and investment to capture data on otherwise anonymous customers. In the main companies do this through their loyalty progammes. Twenty years ago, the only companies that had them were airlines who could leverage fixed capacity at zero marginal cost to capture loyal patronage. Now almost every company has a programme and smart shoppers have wallet loads full of cards that often reward them for shopping in exactly the same places they always have. While this does look a little like a tragedy of commons, the programmes generate heaps of data, otherwise known as big data, that can be used to help marketers understand consumers better.

Big data contains a record of every transaction and its metadata together with as much personal information on the consumer herself as is acceptable given concerns about privacy. Big data people talk about the challenge of managing big data’s three V’s – the sheer Volume of data that is generated, the Velocity with which it is generated and the Variety of data that is available. The three V’s presents a paradox of plenty because big data can create as much noise as music. And noise can be very distracting to corporates with attention deficits.

While much of the commentary on big data focuses on the systems needed to retain and analyse huge volumes of data, when it comes to using big data for innovation the key challenge is actually to distil and filter, and to figure out how to focus on the stuff that actually counts. The sophisticated systems and the ubiquity of data mean that finding answers is the easy part. They are literally ready to be plucked out of the cloud. In reality the key to unlocking the value of big data is to figure out the right question. That is where big minds come in. Any analyst can answer a question, but it takes a particular genius to know what those questions are.

So, while some companies might believe that the big data revolution means they just need to bring in technical capacity, this is to miss the point. Standardised reporting and production-driven analysis will take you so far, but if you are not constantly reviewing the questions you ask, the answers you get will not take you further. Quite simply, at the heart of the big data revolution is not big data or big systems, but big, enquiring minds.

By Illana Melzer