Subscribe to Syndicate
Feb 23 2018 |
Big Data and Common sense
By: Sushobhit

During world war II, the planes used by the allied forces to attack Germany used to return with heavy bullet holes over the body. So, the natural response was to put more armor all over the plane. But the problem was that as you put more material, the planes would become too heavy to fly. Let’s consider this problem in detail. What is the first thing that you will do to solve this problem? As you guessed it correctly, the planes were analyzed for bullet holes on their bodies. It was found that the nose, the wings and the tail were heavily shelled with bullets. The pattern was common for all the planes that managed to return after bombing. So, the instinct was to strengthen these by putting more armor in these regions, until, ironically, a Jewish mathematician pointed out something quite counter intuitive. He suggested that instead of strengthening the parts covered with bullet spots, the armor needs to be put on the places where there were no bullet marks as those were the probable weak spots. When looked carefully, his claim was quite right as the planes which were hit on the engines or on the cockpit failed to return, and the ones which had returned had no bullet marks on the engine or the cockpit.

This anecdote demonstrates the difference between data, information and insights. The allied forces had all the plane related data with them from where they got the information about the points where plane was getting damaged, but the million-dollar question was answered by a simple insight. This is where the common-sense part leaps in. You can practically generate any conclusion from the data. My favorite example is of the spurious correlations. Take a look at the graph below. This is the actual data taken from U.S. Office of Management and Budget and Centers for Disease Control & Prevention.

The correlation (r) is staggering 99.79%. This suggests that if the American people want their government to spend more on science, space and technology, they need to commit more suicides by hanging, strangulation and suffocation (good for atheists and science lovers) , or the other way round, that if US government wants people to stop committing suicides, it needs to stop spending on science, space and technology. Just imagine the fate of American people if policymakers there are fans of big data.

Another interesting incident where big data gave counter intuitive results was the suggestion to place diapers next to beer at a grocery store, which caused big jump in beer sales. The two things are not related in anyway, but when investigated properly, it was found that young fathers were often asked to bring diapers from the stores and when six packs were placed next to the diapers, these young fathers had a very easy decision to make- take one of those as well. This demonstrates the beauty of big data when coupled with common sense and insights.

Big Data is the new in-thing out there in the market. Data science, analytics, forecasting, predictive analytics etc. have become the new buzz words for the organizations today. In this hype of big data, the thing which often gets ignored is Small data. By small data, I mean the common sense which help us see beyond the numbers. People often take the results thrown at them by the models as gospel, ignoring the common-sense step. Decision making has to involve all the aspects-qualitative, quantitative, analytical as well as emotional. Common sense (which we are now losing to machines) is the safety net which can help us juggle with data and turning it into beautiful insights. While we all are convinced that big data has helped us solve lots of tough business problems, it is actually the analyst's acumen which makes all the difference.