Big Data: History and Confusion
November 28, 2013
I read “Are Big Data Vendors Forgetting History?” I worked through five observations about Big Data and realized that history is essentially irrelevant to Big Data vendors and to some pundits.
I was encouraged by the opening paragraph; to wit:
With any new hot trend comes a truckload of missteps, bad ideas and outright failures. I should probably create a template for this sort of article, one in which I could pull out a term like “cloud” or “BYOD” and simply plug in “social media” or “Big Data.”
My confusion mounted as I worked through the five “history lessons” Datamation sought to teach me:
- Little failures “portend” sometimes big failures
- Fuzzy terminology can “poison the well”
- Details can sidetrack a project
- Technical details are important
- Big Data matters.
Okay, let me address items 3 and 4, the paradox of “details matter” and “details don’t matter.” I am not sure how to resolve these opposites. In my experience, the result, particularly in technology, depends on details. But the details have to fit into some “frame.” A random detail lacks context. Perhaps the lesson is to balance the “vision” with the “execution.” Get one wrong and the other is dragged down. Big Data requires trimming; that is, chopping the data down so that a question can be answered. Once the data set is created and conforms to textbook statistical tests, then a cascade of details take center stage. Big Data often lacks this organic flow between the two opposites.
With regard to item 1, failure on any scale predicting the future, I am not sure what history teaches. Napoleon hoofed it to Moscow and then a German military leader followed in Napoleon’s footsteps. Er, winter. Food. Resupply. History, like the stock market, does not do much to make prediction a dead certain process. Do technology managers learn from the “past”? In my experience, technology managers do what is necessary to keep their job and make money. Excellence is not as high on this list as one would hope. Tomorrow is like today. “Progress” based on reading tea leaves may be a difficult to achieve.
I think that fuzzy terminology, item 2, is an emergent function in technology. Making up words and coining buzzwords performs three jobs. First, it creates an air of specialty or I know something you need to know. Second, it allows an in crowd to form so that outsiders have a tough time getting in the club. Third, marketers can hook vague promises of value to a with it term to close a deal. In the last five years, the technical innovations have been more like refinements than breakthroughs.
Item 5 which suggests that anyone who questions the value of Big Data is taking the easy path forward is interesting. Big Data, in my view, has been a constant issue. What’s new is the number of companies using the term to describe what have been standard functions. Sure, the aging Hadoop “revolution” eliminates some of the hassles and costs associated with a Codd database. The reality is that most organizations lack the staff, the resources, and the time to convert Big Data into meaningful business activities. (Meaningful means “revenue producing.”)
In short, I find the list interesting, but I don’t think there are many history lessons for me. The write up is more of an apologia for a buzzword that is teaching some people that making sense of available information is dog work, expensive, and often tough to connect to a specific payback.
The reason? Big Data requires trained professionals with expertise in math, statistics, and business processes. Last time I checked, individuals with these capabilities were in short supply. Big Data just gets bigger when there are too few sculptors to chop down the ever growing mountain of bits and bytes.
Stephen E Arnold, November 28. 2013
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
