How Many Types of Big Data Exist?

March 18, 2016

Navigate to “The Five Different Types of Big Data.” If you are a student of classification, you will find the categories set forth in this write up an absolute hoot. The author is an expert, I assume, in energy, transportation, food, and data. Oh, goodie. Food.

I have not thought too much about the types of Big Data. I usually think only when a client pays me to perform that function. An example is my analysis of the concept “real time” information. You can find that write up at this link. Big requires me to understand the concept of relative to what. I find this type of thinking uninteresting, but obviously the editors at Forbes find the idea just another capitalist tool.

When I learned that an expert had chased down the types of Big Data, I was and remain confused. “Big” describes something that is relative. “Data” is the plural of datum and refers to more than two facts or statistics, quantities, characters, symbols, etc.

I am not sure what Big Data is, and like many marketing buzzwords, the phrase has become a catchall for vendors of all manner of computer related products and services.

Here are the five types of Big Data.

  1. Big data. I like the Kurt Friedrich Gödel touch.
  2. Fast data. “Relative to what?” I ask.
  3. Dark data. “Darker than what? Is this secret versus un-secret or some other yardstick?” I wonder.
  4. Lost data. I pose to myself, “Lost as in unknown, known but unknown, or some other Rumsfeldesque state of understanding?”
  5. New data. I think, “I really don’t want to think about what ‘new’ means? Is this new as in never before seen or Madison Avenue ‘new’ like an improved Colgate Total toothpaste with whitener.

I like the tag on the article “Recommended by Forbes.” Quite an endorsement from a fine example of capitalistic tool analysis.

Stephen E Arnold, March 18, 2016

Text Analytics: Crazy Numbers Just Like the Good Old Days of Enterprise Search

March 16, 2016

Short honk: Want a growth business in a niche function that supports enterprise platforms? Well, gentle reader, look no farther than text analytics. Get your checkbook out and invest in this remarkable sector. It will be huuuuge.

Navigate to “Text Analytics Market to Account for US$12.16 bn in Revenue by 2024.”  What is text analytics? How big is text analytics today? How long has text analytics been a viable function supporting content processing?

Ah, good questions, but what’s really important is this passage:

According to this report, the global text analytics market revenue stood at US$2.82 bn in 2015 and is expected to reach US$12.16 bn by 2024, at a CAGR of 17.6% from 2016 to 2024.

I love these estimates. Imagine. Close out your life savings and invest in text analytics. You will receive a CAGR of 17.6 percent which you can cash in and buy stuff in 2024. That’s just eight years.

Worried about the economy? Want to seek the safe shelter of bonds? Forget the worries. If text analytics is so darned hot, why is the consulting firm pitching this estimate writing reports. Why not invest in text analytics?

Answer: Maybe the estimate is a consequence of spreadsheet fever?

Text analytics is a rocket just like the ones Jeff Bezos will use to carry you into space.

Stephen E Arnold, March 16, 2016

A Thought Leader Embraces Mid Tier Consultant Thinking

March 16, 2016

I read “The Hype of Big Data Revisited: It’s About Extracting Value.” I am not particularly interested in “how big” discussions. What I found interesting was that a through leader reproduced a mid tier consulting firm’s Hype Cycle for Emerging Technologies, 2015. I thought mid tier outfits were not too keen on having their proprietary charts reproduced. Obviously I am off the beam on this assumption.

I did note this statement:

In between 2013 and 2014, Big Data reached the Peak of Inflated Expectations in Gartner’s Hype Cycle for Emerging Technologies. By mid 2014, Big Data was sliding into the Trough of Disillusionment, and by 2015, the term was removed from the hype cycle altogether.


More mid tier goodness.

Here’s what I learned about the source of this write up:

Bob E. Hayes, PhD is the Chief Research Officer of Analytics Week and president of Business Over Broadway. At Analytics Week, he is responsible for directing research to identify organizational best practices in the areas of Big Data, data science and analytics. He is considered a thought leader in the field of customer experience management. He conducts research on analytics, customer feedback programs, customer experience / satisfaction / loyalty measurement and shares his insights through his talks, blogs and books.

Perhaps the notion of thought leadership and recycling a mid tier consultant firm’s viewpoints is the future of deep insight and analysis. Wow, the mid tier consulting firm is a significant influence on some thought leaders.

Too bad the intellectual force does not reach to my part of rural Kentucky. It obviously skips me and works its magic in Bowling Green, the home of the Corvette hole.

Stephen E Arnold, March 15, 2016

Gartner and the Business Intelligence Magic Quadrant: Lots of Explaining, Lots of Subjectivity It Seems

March 13, 2016

I read a downright weird article/interview called “Big Data Discovery may put Oracle back in BI Magic Quadrant.” The title contains the magic word “may”, which does not promise to make Oracle a big dot in a Gartner Magic Quadrant, but it suggests that Gartner is doing some explaining.

As I understand the situation, the mid tier consulting firm analyzed the business intelligence sector and figured out which companies were winners and losers. Well, that’s the lingo that the original Boston Consulting Group quadrant used, and that’s how General Eisenhower used his quadrant. So those approaches override the Garnter words like niche players and visionaries. (Is it not possible for a niche player to be a visionary? Does Gartner know “Venn” to check it logic?)

The point of the write up is that Oracle, one of the big dogs in the Department of Defense’s DCGS-A and DCGS-N mash up analytics initiative is not in the Garnter magic square thing. Nope. Deleted.

Why may be a question which some folks at Oracle have been asking. The article/interview appears to be an “explainer” to make the Garnter mid tier method appear more near the top drawer in the cabinet of analytics collectibles.

I noted this passage:

Question: It sounds like the change isn’t coming from something Oracle did, but from Gartner.

Gartner’s R&D Big Dog, Josh Parenteau: Right, OBIEE is still there. It’s still being sold as their platform, but it does not meet the modern definition of the Magic Quadrant right now.

The acronym OBIEE means Oracle analytics. You, gentle reader, knew that.

Oracle was excluded because “they didn’t fully participate,” says Parenteau. He adds:

I do think that they’re late to the game by quite a bit… For Oracle, it’s recognizing the signals a bit earlier. It’s responding to customer needs and, I think, realizing that it’s not just about product. You can have the best product in the world, but if customers don’t want to work with you because they don’t like the relationship, it’s not going to matter.

So what companies of note made the Magic Quadrant? Since I don’t pay Gartner to advise me, I checked Bing and Google to locate the 2016 Magic Quadrant for Business Intelligence. It did not take long, because this MQ report appears to be a marketing item, not a confidential study like a report about the AVATAR program.

Check out these outfits who have met the Gartner criteria, objective and subjective:

  • BeyondCore
  • Domo
  • Logi Analytics
  • Platfora
  • Sisense

Okay, some names of note.

These outfits made the list as well:

  • IBM
  • Microsoft
  • SAS.

I highlighted this paragraph as particularly suggestive:

But I would say that, if you are a member of the install base of Oracle, know that they do have offerings in the space. They just didn’t have enough traction to get on the quadrant. If you have a big data Hadoop initiative going on, of course look at Big Data Discovery, because that’s exactly what it’s focused on. If you are looking for a tool to do data discovery, of course look at Visual Analyzer, which is part of the cloud service. If you have an initiative to get into the cloud, look at BICS. I wouldn’t say that, just because they’re not on the Magic Quadrant, if you’re an existing Oracle customer that you shouldn’t continue to look at them for solutions. This doesn’t mean that they are gone forever or off the MQ forever. It’s a transition. We’re in a market that is transitioning. Next year, it may be a new ball game.

Very mid tier. I liked the “you shouldn’t continue to look at them for solutions.” Are those words a positive or a negative? Worth watching the interaction of the Oracle folks at the Gartner experts.

Stephen E Arnold, March 13, 2016

Gartner and Business Intelligence Magic Thing

February 14, 2016

I love consultants, especially mid tier consultants. The idea is that folks who are reasonably pleasant can become experts in various market sectors is a signal that optimism is alive and thriving in a sketchy economic swamp.

The mid tier consultants are a fave. These outfits provide more tradition than the webmaster or Visual Basic programmer who is out of a job. The ease with which one can become a consultant lends a certain squishiness to Lone Rangers offering expertise for hire.

The blue chip outfit are just too expensive for many folks who know they need help. Think of the difference between someone who jets to Lyon for lunch and the person who grabs a slice in Midtown.

Thus, blue chip outfits (the top drawer firms), the azure chip firms (companies either on their way up or down in the expertise Great Chain of Being), and the gray chip folks. The gray chip folks are the disaffected middle school teacher who decides to become a self appointed expert in sponsored content for search engine optimization.

The write up “Critiquing the Gartner BI and Analytics MQ” will not elicit much of a response from the mid tier outfit responsible for the “analysis.” Legal eagles slap when the actual quadrant thing is reproduced.

But the write up hits some nerves in the sagging neck of the azure chip services firm; for example:

  1. Companies excluded for no apparent reason. (Maybe these outfits rejected the azure chip firm’s blandishments to buy services and be better understood?)
  2. A “kitchen sink” approach. (Maybe this means dumping stuff into a container and binge watching Happy Days on Hulu? Stuff breaks when hasty hands place dirty dishes in a sink.)
  3. Products are mixed up. The example is Design Studio. (Aren’t these software components pretty much the same? Sure they are, gentle mid tier consultant getting smart by searching Google for info. Sure they are.)
  4. Inconsistency. (The write up displays actual, high value, super secret, for some eyes only magic thingies. I looked at each graph and was confused in terms of what was presented and how the classifications changed in the span of one fiscal year. Aren’t I the dunce?)

The write up is not about hell fire and brimstone. Here’s the peace offering after the carpet bombing:

To be fair on Gartner, they have made a solid effort at explaining their rationale and, given there are some 500 vendors globally, vying for attention, narrowing down to this selection is a valiant effort. The care with which Gartner has made its understanding known is also commendable, even if some of those explanations are questionable. Another problem with the report is that it is static. It is a snapshot at a point in time that is biased in favor of one constituency and which does not, in my view, adequately recognize the necessary and sometimes difficult tensions that exist between IT and lines of business when it comes to rationalizing or consolidating BI tools in an enterprise setting. I think Gartner has done the industry a major favor by decoupling the reporting element and focusing upon the modern approach to BI. But that’s not enough.

Maybe another azure chip outfit will leap into this opportunity. A mere 500 vendors. The number seems low to me. I eagerly await the next intellectual semi-truck load of insights from the azure chip sector. Yes, eager am I.

Stephen E Arnold, February 14, 2016

The Key to Revenue? Ads. Predictive Analytics Not So Much

January 16, 2016

I read a darned amazing write up in a marketing blog. First, the story the marketing blog turned into “real news” is a sponsored study. That means an ad. But, even more interesting, the source of the funded study is a mid tier consulting firm. Now you know there are blue chip consulting outfits. I used to work at one and have done consulting projects for other blue chip outfits over the last 40 years. The blue chip outfits are more subtle in their thought leadership, which is one reason why there are blue chip outfits sitting on top of a pile of azure chips and gray chip vendors of expertise.

The second point is that the sponsored study conveniently converted into “real news” is that revenue comes from predicative analytics. Excuse me. But if a company is paid to flog an ad messages, doesn’t that mean the revenue comes from advertising or, in this case, clumsy propaganda. If the predictive analytics thing actually worked revenue wonders, wouldn’t the mid tier consulting firm use predictive analytics to generate cash? Wouldn’t the marketing newsletter use predictive analytics to generate cash?

To see this sponsored content daisy chain in action, navigate to “Forrester Report: Companies Using Predictive Analytics Make More Money.” The mid tier outfit in question is Forrester. Is their logo azure tinted? If not, maybe that is a color to consider. None of the stately expensive tie colors required.

The publication recycling the sponsored content as “real” news is Marketing Land. The name says it all, gentle reader.

What is the argument advanced for EverString by Forrester and Marketing Land?

Here’s the biggie:

The big takeaway: “Predictive marketing analytics use correlates with better business results and metrics.”

That is, compared with those in the survey who do not use predictive analytics (which it calls Retrospective Marketers). “Predictive Marketers,” the report notes, “are 2.9x more likely to report revenue growth at rates higher than the industry average.” They are also 2.1 times more likely to “occupy a commanding leadership position in the product/service markets they serve” and 1.8 times more likely to “consistently exceed goals when measuring the value their marketing organizations contribute to the business,” compared to the Retrospective Marketers in the survey. Forrester analyst Laura Ramos, who was involved in the report, told me the main point is clear: “Predictive analytics pays off.”

What froth? The 2.9x suggests real analysis. Sure, sure, I know about waves and magic squares.

There are companies delivering predictive analytics. Some of these outfits have been around for decades. Some of the methods have been known for centuries. I won’t remind you, gentle reader, about my wonky relative and his work for the stats guy Kolmogorov.

Suffice it to say that EverString paid Forrester. Forrester directly or indirectly smiled at Marketing Land. The reader learns that predictive analytics generate revenue.

Nope, the money comes from selling ads and, I assume, “influence.”

Put that in your algorithm and decide which is better: Selling ads or figuring out how to construct a predictive numerical recipe?

Right. Mid tier firms go the ad route. The folks recycling ads as news grab a ride on the propaganda unicycle.

Stephen E Arnold, January 16, 2016

Mid Tier Consultant Sees Ripples of Opportunity in Data Lakes

December 16, 2015

I love predictions from mid tier consultants. One can spot what these folks will be pitching to their customers. One can also see the buzzwords likely to replace plain talk in their reports.

A good example of this type of forecasting—which if it worked would be used to pick horse race winners, not technologies—appears in “Big Data’s Future According to Ovum.” To spare extra wear and tear on your rapidly beating heart, SQL data management will remain popular but nothing will capture the excitement of Hadoop on Spark or is it Spark in Hadoop? Oh, well.

Here’s the passage I found as chilling as a dip in the lake near my shack in rural Kentucky:

Ovum’s other big prediction for 2016 is for data lake adoption to become a “front-burner issue” for mature Hadoop adopters that have already successfully put analytics into production serving multiple lines of business and stakeholder groups across the organization. The result will be a new demand for tools to govern the data lake and make it more transparent. Ovum expects significant growth in tooling that builds on emerging data lineage capabilities to catalogue, protect, govern access, tier storage, and manage the lifecycle of data stored in data lakes.

The word for 2016 will involve govern as in “governance.” The idea is that once folks dump stuff in the lake, a digital and procedural mechanism will be needed to figure out exactly what’s in the lake.

Wow, mid tier consulting pitching the need for management. I wonder if the mid tier consulting firms are able to sell their clients management consulting services?

I think this means that these predictions and the attendant reports are essentially content marketing exercises. That’s okay, but writing about a problem is exactly the same as solving a problem. Right?

How did that work out when search was the topic of the moment?

Stephen E Arnold, December 16, 2015

Improving Government Technology: Some Principles

December 13, 2015

Yes, it might be possible. Navigate to “Delivering Results: A Framework for Federal Government Technology Access & Acquisition.” If the link does not resolve, you will have to scout around. No guarantees that this document will remain on a public Web site. The comments apply to almost any government too. I think the write up is focused on the US government, the approach is borderless and a wonderful example of clear thinking about how consultants think about contracting opportunities.

Let me get to the heart of the matter. The way to improve government technology involves the government taking action on several “principles”. Intrigued? Here they are:

  • A Common Goal – The Common Good
  • Competition and Innovation
  • Collaboration
  • Contracting Flexibility
  • Risks/Rewards
  • Workforce

If you have been involved in government work in the US or elsewhere, you may note that the principles omit one of the key drivers: Billing and related matters such as scope change.

There are some other hitches in the git along. Let me highlight one for each of the principles.

1. A common goal is tough to achieve. Government entities want to retain power, headcount, and budgets. The notion of intra and inter agency cooperation or even inter and intra department cooperation is fascinating. The nature of the bureaucratic process is meetings with overt and hidden agendas. The common goal and the common good are easy to describe, just tough to implement.

2. Competition. If you are a vendor in rural Kentucky and you want to bid on a government project, you may have a difficult time achieving your goal. Projects often begin at the appropriate stage, work their way through the consultant driven request for proposal stage, then there is the statement of work stage, and along the way are contracting officers, legal eagles, and assorted procurement professionals. For someone working in Hazard County, the process is definitely expensive, slow, and designed to allow the big dogs to eat the tasty bits. Competition exists but in a meta sort of way.

3. Collaboration. Meetings are collaborative fun fests. The problem is that the objectives of power, headcount, and budget act as fusion power sources among the participants. Talk is the energy of government meetings. Doing results from expanding power, headcount, and budget allocations. Many meetings require a paid consultant or two to provide the catalyst for the talk. The collaboration results in more meetings.

4. Contracting flexibility. Right. There are rules, and if the rules are sidestepped even by some highly placed folks, that wandering off the reservation is rarely a good thing. An outfit called 18f is trying to deliver flexible contracting on a modest scale to some GSA functions. Right. Have you ever heard about 18f? If so, you are one of the lucky few. In the meantime, the established contractors keep doing their thing: Capturing major contracts.

5. Risks / rewards. Risk is not something that is highly desirable either for a government professional or for the people and companies capturing major contracts. Risk can be discussed in a “collaborative” meeting. The systems then continuously operate to reduce risk. Want to have an unknown contractor build your next weapons system? Not going to happen in my lifetime. Want an unknown contractor to code a Web page? Well, sure, just fill out the appropriate forms or figure out what 18f is all about. There is a reason some government contractors are big. These outfits know how to deal with risk, government style.

6. Workforce. The governments with which I have worked struggle with the workforce issue. The idea is to find and hire the best and brightest. How is this working out? Some folks from a successful company flow into the government and then flow out. This is the revolving door for some folks. Folks who stick in government operations, regardless of country, like the working environment, enjoy the processes, and revel in the environment.

The principles are well stated. I am not sure that changing how governments operate is going to make much headway. Think about your last interaction with a government entity. What did that reveal to you?

Stephen E Arnold, December 13, 2015

Crazy Numbers Department: Big Data Spending in 2019

November 26, 2015

It is almost 2016. IDC, an outfit owned by an optimistic outfit, has taken a tiny step forward. The IDC wizards answered this question, “How big will Big Data spending be in 2019?” Yep, that is 36 months in the future. There might be more money in predicting Super Bowl winners, what stock to pick, and the steps to take to minimize risk at a restaurant. But no.

According to the true believers in the Content Loop, “IDC Days Big Data Spending to Hit 48.6 Billion in 2019.” I like that point six, which seems to suggest that real data were analyzed exhaustively.

The write up reports:

The market for big data technology and services will grow at a compound annual growth rate (CAGR) of 23 percent through 2019, according to a forecast issued by research firm International Data Corp. (IDC) on Monday. IDC predicts annual spending will reach $48.6 billion in 2019. IDC divides the big data market into three major submarkets: infrastructure, software and services. The research firm expects all three submarkets to grow over the next five years, with software — information management, discovery and analytics and applications software — leading the charge with a CAGR of 26 percent.

I will go out on a limb. I predict that IDC will offer for sale three reports, maybe more. I hope the company communicates with its researchers to avoid the mess created when IDC wizard Dave Schubmehl tried to pitch eight pages of wonderfulness based on my research for a mere $3,500 without my permission. Ooops. Those IDC folks are too busy to do the contract thing I assumed.

A Schubmehl-type IDC wizard offered this observation with only a soupçon of jargon:

The ever-increasing appetite of businesses to embrace emerging big data-related software and infrastructure technologies while keeping the implementation costs low has led to the creation of a rich ecosystem of new and incumbent suppliers…. At the same time, the market opportunity is spurring new investments and M&A activity as incumbent suppliers seek to maintain their relevance by developing comprehensive solutions and new go-to-market paths.– Ashish Nadkarni, program director, Enterprise Servers and Storage, IDC

Yes, ever increasing and go to spirit. Will the concept apply to IDC’s revenues? Those thrilled with the Big Numbers are the venture folks pumping money into Big Data companies with the type of enthusiastic good cheer as Russian ground support troops are sending with the Backfires, Bears,  and Blackjacks bound for Syria.

Thinking about international tension, my hunch is that the global economy seems a bit dicey, maybe unstable, at this time. I am not too excited at the notion of predicting what will happen in all things digital in the next few days. Years. No way, gentle reader.

Thinking about three years in the future strikes me as a little too bold. I wonder if the IDC predictive methods have been applied to DraftKings and FanDuel games?

Stephen E Arnold, November 26, 2015

Predictions for a Big Data Future

November 19, 2015

Want to know what the future will look like? Navigate to “7 Reasons Why the Algorithmic Business Will Change Society.” The changes come via Datafloq via a mid tier consulting firm. I find the predictions oddly out of step with the milieu in which I live. That’s okay but this list of seven changes raises a number of questions and seems to sidestep some of the social consequences of the world foreshadowed in the predictions. Finding information is, let me say at the outset, not part of the Big Data future.

Here are the seven predictions:

    1. By 2018, 20% of all business content will be authorized by machines, which means a hiring freeze on copywriters in favor of robowriting algorithms;
    2. By 2020, autonomous software agents, or algorithms, outside human control, will participate in 5% off all economic transactions, thanks to, among others, blockchain. On the other hand, we will need pattern-matching algorithms to detect robot thieves. 
    3. By 2018, more than 3 million workers globally will be supervised by a “roboboss”. These algorithms will determine what work you would need to do.
    4. By 2018, 50% of the fastest growing companies will have fewer employees than smart machines. Companies will become smaller due to expanding presence of algorithms.
    5. By 2018, customer digital assistants will recognize individuals by face and voice across channels and partners. Although this will benefit the customer, organizations should prevent the creepiness-factor.
    6. By 2018, 2 millions employees will be required to wear health and fitness tracking devices. The data generated from these devices, will be monitored by algorithms, which will inform management on any actions to be taken.
    7. By 2020, smart agents will facilitate 40% of mobile transactions, and the post-app era will begin to dominate, where algorithms in the cloud guide us through our daily tasks without the need for individual apps.

Fascinating. Who will work? What will people do in a Big Data world? What about social issues? How will one find information? What happens if one or more algorithms drift and deliver flawed outputs?

No answers of course, but that’s the great advantage of talking about a digital future three or more years down the road. I assume folks will have time to plan their Big Data strategy for this predicted world. I suppose one could ask Google, Watson, or one’s roboboss.

Stephen E Arnold, November 19, 2015

« Previous PageNext Page »