Why Big Data is Not Only About Granular Data

The general perception about big data analysis is that the more granular it is the more value you derive out of it. This brings to the fore a question is granular data analysis always better? We believe it’s not so in all cases. Just as predicting the outcome of a football match after watching the first quarter of play can be an impossible guess, understanding how trends might evolve by analyzing data at the granular level can be an equally tall order.

Granular Data is Not Complete

Let’s, for the sake of understanding, consider granular data as user level data. The user-level data that companies have access to is of individuals who have visited their site or clicked on their online ads. It cannot be considered as complete because it does not reflect the behavior of the total target consumer base. Such data insights can be of no use to offline execution. In case of online execution, user level data can be put to use mostly through segment level aggregation. The only possible use for which it is well suited is website personalization, email automation etc.

Here’s an example of how granular data can, at times, fail larger objectives. Google’s model to predict outbreaks of flu, based on correlations in its search data, created a flutter in the market because at the speed and accuracy with which it did it. However, during a flu epidemic in 2013, the model missed the mark by 100%. This was because the model included the count of people checking flu symptoms, just for the heck of it. In the second instance, there was a need to understand why people were looking up the symptoms in order to make an accurate prediction. The glitch was later fixed with the help of segment level aggregation.

Flipside to Granular Analysis

The problem with granular data, is that it makes it harder to see the wood for the trees. The only advantage you get out of it is short-term marketing gains brought about by individual targeting. But when it comes to long-term effects, granular data comes a cropper. However, when this data is allowed to evolve and analyzed at a group level, our understanding of user behavior deepens. The other flip side to granular analysis is that it pops up false positives. This is because the more you analyze, the more are the chances of getting misled.

The Right Approach

Big Data analysis, as it stands, is accompanied with lot of noise, which increases in proportion depending on the scope of analysis i.e. data by minute than by day or week. Analyzing with appropriate level of aggregation, can help us cancel the noise in a more effective way. Since the use of granular data cannot be discounted altogether, the best approach for companies would be to balance granular analysis with broader analysis. This would ensure you don’t go astray with your big data labors.