For many years now, analysts have been telling us that we are on the verge of a new age in Business Intelligence – the Age of Insight. Since the dawn of data processing, companies have toiled to turn ever-increasing quantities of data into meaningful information in order to make better business decisions. After realizing that the common man was still unable to make better decisions, Data Science was invented. Some might argue that it is still more “art” than “science”, but it also took medicine many centuries (as well as many leeches and magic potions) to become a legitimate science. Regrettably, there is a shortage of true Data Scientists to answer vexing questions – and businesses want those answers better, faster, and cheaper!
Just when we thought we had a plan, enter BIG DATA! The four V’s suggest that with more volume, more variety, and at a greater velocity, better value can be extracted from plain old data. As technology advanced and the internet of things (IOT) became, well, a thing, it also became apparent that we needed a better way to compute the explosion of new data and handle storage. The solution? Huge arrays of commodity hardware and massive parallel processing systems (e.g. Hadoop). Unfortunately, we found that dumping all that data into “data lakes” often did nothing more than create “data swamps”. Alas, simply adding more and different types of data at a faster speed hasn’t automatically created benefits. So we added a fifth V, veracity, to apply old-fashioned data management principles in order to provide data curation (profiling, cleansing, and conforming) so that the data can be understood and trusted by mere mortals.
Having largely solved the storage part of the equation, our next great challenge is the compute problem. Remember faster, better, cheaper? It isn’t just a technological issue to solve. We also need to optimize the two “human” factors that we have reintroduced (Data Scientists and Data Curators) between our business decision makers and their data. While not all questions require an immediate answer, true insight occurs when the answers (and even the questions) can be predicted and acted upon without human intervention. Today, in-memory analysis/visualization and advanced analytics (predictive modeling and machine learning) show great potential. While we may still be living through the mystical, smoke-and-mirrors promises of big data vendors, the end is nigh and the Age of Insight awaits us!