Just when we thought we had a plan, enter BIG DATA! The four V’s suggest that with more volume, more variety, and at a greater velocity, better value can be extracted from plain old data. As technology advanced and the internet of things (IOT) became, well, a thing, it also became apparent that we needed a better way to compute the explosion of new data and handle storage. The solution? Huge arrays of commodity hardware and massive parallel processing systems (e.g. Hadoop). Unfortunately, we found that dumping all that data into “data lakes” often did nothing more than create “data swamps”. Alas, simply adding more and different types of data at a faster speed hasn’t automatically created benefits. So we added a fifth V, veracity, to apply old-fashioned data management principles in order to provide data curation (profiling, cleansing, and conforming) so that the data can be understood and trusted by mere mortals.
Having largely solved the storage part of the equation, our next great challenge is the compute problem. Remember faster, better, cheaper? It isn’t just a technological issue to solve. We also need to optimize the two “human” factors that we have reintroduced (Data Scientists and Data Curators) between our business decision makers and their data. While not all questions require an immediate answer, true insight occurs when the answers (and even the questions) can be predicted and acted upon without human intervention. Today, in-memory analysis/visualization and advanced analytics (predictive modeling and machine learning) show great potential. While we may still be living through the mystical, smoke-and-mirrors promises of big data vendors, the end is nigh and the Age of Insight awaits us!