Now Big Data Has to Be Fast
May 15, 2016
I read “Big Data Is No Longer Enough: It’s Now All about Fast Data.” The write up is interesting because it shifts the focus from having lots of information to infrastructure which can process the data in a timely manner. Note that “timely” means different things in different contexts. For example, to a crazed MBA stock market maven, next week is not too useful. To a clueless marketing professional with a degree in art history, “next week” might be just speedy enough.
The write up points out:
Processing data at these breakneck speeds requires two technologies: a system that can handle developments as quickly as they appear and a data warehouse capable of working through each item once it arrives. These velocity-oriented databases can support real-time analytics and complex decision-making in real time, while processing a relentless incoming data feed.
The point omitted from the article is that speed comes at a cost. The humans required to figure out what’s needed to go fast, the engineers to build the system, and the time required to complete the task. The “cloud” is not a solution to the cost.
Another omission in the article is that the numerical recipes required to “make sense” of large volumes of data require specialist knowledge. A system which outputs nifty charts may be of zero utility when it comes to making a decision.
The write up ignores the information in “What Beats Big Data? Small Data.” Some organizations cannot afford the cost of fast data. Even outfits which have the money can find themselves tripping over their analyses. See, for example, “Amazon Isn’t Racist, It’s Just Been an Unfortunate Victim of Big Data.” Understanding the information is important. Smart software often lacks the ability to discern nuances or issues with data quality, poor algorithm selection, or knowing what to look for in the first place.
Will the write up cause marketers and baloney makers to alter their pitches about Big Data and smart software. Not a chance. Vendors’ end game is revenue; licensees have a different agenda. When the two do not meet, there may be some excitement.
Stephen E Arnold, May 15, 2016
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
