Big Data and Visualization: The Ham and Eggs of Analysis
October 14, 2016
i read “Big Data Is Useless without Visual Analytics.” (Nope, I won’t comment on the fact that “data” is a plural.) The main point of the article is that looking at a table is not as easy as looking at a graphic, preferably Hollywood style, presentation ready visualizations. If you want to see a nifty visualization, check out the Dark Trace three dimensional, real time visualizations.
The write up informed me:
Visualizations are valuable because they display a lot of data in an easy-to-understand visual format that works well for our visually-oriented minds.
Okay. A lot. Good.
I learned that “data mining is too complicated for most uses of Big Data.”
No kidding. Understanding and making justifiable decisions about data validity, math with funny symbols, and numerical recipes which make the numbers conform to the silliness taught in university statistics classes. These are difficult tasks for avid Facebook users and YouTube content consumers to deal with.
I understand. Some folks are just really busy.
The write up explains that Excel is not the proper tool for real information analysis. Never mind that Excel remains a reasonably popular chunk of software. Some Excel lovers write memos and reports in Excel. That’s software love.
So what’s the fix?
Surprisingly the write up does not provide one. But there is a link which allows me to download a report from my pals at IDC. You remember IDC, right? That is the outfit which tried to sell my content on Amazon without my permission and without having a agreement with me to publish my research. If you have forgotten what I call the “Schubmehl play”, you can get some of the details at this link.
Nice write up. Too bad it lacks useful content on the subject presented in the headline. But what else does one expect these days?
Stephen E Arnold, October 14, 2016
Artificial Intelligence Spreading to More Industries
May 10, 2016
According to MIT Technology Review, it has finally happened. No longer is artificial intelligence the purview of data wonks alone— “AI Hits the Mainstream,” they declare. Targeted AI software is now being created for fields from insurance to manufacturing to health care. Reporter Nanette Byrnes is curious to see how commercialization will affect artificial intelligence, as well as how this technology will change different industries.
What about the current state of the AI field? Byrnes writes:
“Today the industry selling AI software and services remains a small one. Dave Schubmehl, research director at IDC, calculates that sales for all companies selling cognitive software platforms —excluding companies like Google and Facebook, which do research for their own use—added up to $1 billion last year. He predicts that by 2020 that number will exceed $10 billion. Other than a few large players like IBM and Palantir Technologies, AI remains a market of startups: 2,600 companies, by Bloomberg’s count. That’s because despite rapid progress in the technologies collectively known as artificial intelligence—pattern recognition, natural language processing, image recognition, and hypothesis generation, among others—there still remains a long way to go.”
The article examines ways some companies are already using artificial intelligence. For example, insurance and financial firm USAA is investigating its use to prevent identity theft, while GE is now using it to detect damage to its airplanes’ engine blades. Byrnes also points to MyFitnessPal, Under Armor’s extremely successful diet and exercise tracking app. Through a deal with IBM, Under Armor is blending data from that site with outside research to help better target potential consumers.
The article wraps up by reassuring us that, despite science fiction assertions to the contrary, machine learning will always require human guidance. If you doubt, consider recent events—Google’s self-driving car’s errant lane change and Microsoft’s racist chatbot. It is clear the kids still need us, at least for now.
Cynthia Murrell, April 10, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Old Pals Chatting: IDC Expert Chums Up Cognitive Marketing
May 4, 2016
I recall a fellow named Dave Schubmehl. You may recall that name. He was the IDC wizard who ingested my research about open source outfits and then marketed it via Amazon without my permission. Since that go round with my information used without a written agreement with me, I have taken a skeptical view of IDC and its “experts.” I won’t comment on its business practices, administrative acumen, and general ineptitude with regard to publishing a bit of my research as an eight page, $3,500 “analysis.” Yikes. Eight pages at $3,500 for work pumped out on Amazon, the WalMart of the digital world.
I read, therefore, with considerable skepticism “Interview with Rich Vancil: Group VP, Executive Advisory of IDC.” I was not disappointed. Perhaps I should say, my already low expectations were just about met.
The interviewer, according to the interview text, has been an acquaintance of the IDC wizard for decades. Furthermore, the interviewer (obviously an objective type of person) will “meet up to catch up on life outside business.” The article is “old pals chatting.”
What a chat?
I learned that:
The IDC 3rd Platform is a broad term for our present IT industry and economy. It is where 100% of WW IT revenue growth is coming from and it includes the product categories of Mobile; Social; Cloud, and Big Data. The 3rd Platform is eclipsing the 2nd Platform – described broadly as the “last 30 years” of IT, and this has been mainly enterprise computing: Lan / Internet; Client / Server; and premised based infrastructure such as servers, storage, and licensed software.
A third platform. “Platform” is an interesting word. I get the idea of a Palantir platform. I suppose I can get in sync with the Windows 10 platform. But an IDC platform? Well, that’s an idea which would never have floated from the pond filled with mine drainage here in Harrod’s Creek.
A consulting firm is in the business of selling information. A platform exists at outfits like Booz, Allen, McKinsey, and Bain. But the notion that a mid tier outfit has had three platforms intrigues me. When I looked at some of the 1917-1918 reports at Booz, Allen when Ellen Shedlarz ran the information center, the format, the tone, the approach, and the word choice was incorporated in the charm school into which new hires were herded. I could, in a moment of weakness, call Booz, Allen’s systems and methods a platform. But are the words “systems” and “methods” more appropriate?
The other interesting point in the write up was a nifty new diagram which purports to make clear the third platform confection. I know you won’t be able to read the diagram. Buy the report which hopefully is less than the $3,500 slapped on eight pages of my research.
Source: IDC 2016 at this link. If you find the link dead, just buzz up IDC and order document 01517018. The reports based on my research were 236511, 236514, 236086, and 237410. Buy them all for a mere $14,000.
Notice the blobs. Like another mid tier outfit, blobs are better than numbers. The reason fuzziness is a convenient graphic device is that addled geese like me ask questions; for example:
- What data are behind the blobs
- What was the sample size
- Where did the categories come from like “cognitive marketing”?
I have a supposition about the “cognitive” thing. The IDC wizard Dave Schubmehl pumped out lots of tweets about IBM cognitive computing. One IDC executive, prior to seeking a future elsewhere, wrote a book about “cognitive” processes. Both of these IDC experts guzzled the IBM Watson lattes somewhere along the cafeteria line.
Back to the interview among two friends. I learned:
MarTech is a big deal. IDC is doing a very careful accounting of this area and we now account for 78 separate product / service categories and literally thousands of vendors. Like any other emerging and fast growth IT category, consolidation will be inevitable. But in the meantime, it makes for a daunting set of choices for the CMO and team.
I like the word daunting. There is nothing like a list of items which are not grouped in a useful manner to set IDC neural pathways abuzz. But the IDC mavens have cracked the problem. The company has produced a remarkable 2015 technology map. Check this out:
Source: Expert Interview, 2016
I moved forward in the write up. The daunting problem has contributed to what the interviewer describes as “an awesome conference.” I like that “awesome” thing. How does the write up conclude? There is a reference to golf, the IDC professional’s medical history, and this statement:
The best analysts can simplify, simplify. Analysts who try to impress by using big words and complex frameworks…end up confusing their audience and so they become ineffective.
Remarkable content marketing.
Stephen E Arnold, May 4, 2016
Are Search Unicorns Sub Prime Unicorns?
January 4, 2016
The question is a baffler. Navigate to “Sorting Truth from Myth at Technology Unicorns.” If the link is bad or you have to pay to read the article in the Financial Times, pony up, go to the library, or buy hard copy. Don’t complain to me, gentle reader. Publishers are in need of revenue. Now the write up:
The assumption is that a unicorn exists. What exists are firms with massive amounts of venture funding and billion dollar valuations. I know the money is or was real, but the “sub prime unicorn” is a confection from a money thought leader Michael Moritz. A subprime unicorn is a co9mpany “built on the flimsiest of edifices.” Does this mean fairy dust or something more substantial?
According to the write up:
High quality global journalism requires investment. Please share this article with others using the link below, do not cut & paste the article. But the way in which private market valuations have become skewed and inflated as start-ups have delayed IPOs raises questions about the financing of innovation. Despite the excitement, venture capital has produced weak returns in recent decades — only a minority of funds have produced rewards high enough to compensate investors for illiquidity and opacity.
Why would funding start ups perform better than a start up financed by mom, dad, and one’s slightly addled, but friendly, great aunt?
The article then makes a reasonably sane point:
With the rise in US interest rates, the era of ultra-cheap financing is ending. As it does, Silicon Valley’s unicorns are losing their mystique and having to work to raise equity, sometimes at valuations below those they achieved before. The promise of private financing is being tested, and there will be disappointments. It does not pay to be dazzled by mythical beasts.
Let’s think a moment about search and content processing. The mid tier consulting firms—the outfits I call azure chip outfits—have generated some pretty crazy estimates about the market size for search and content processing solutions.
The reality is at odds with these speculative, marketing fueled prognostications. Yep, I would include the wizards at IDC who wanted $3,500 to sell an eight page document with my name on it without my permission. Refresh yourself on the IDC Schubmehl maneuver at this link.
Based on my research, two enterprise search outfits broke $150 million in revenues prior to 2011: Endeca tallied an estimated $150 million in revenues and Autonomy reported $700 million in revenues. Both outfits were sold.
Since 2012 exactly zero enterprise search firms have generated more than $700 million in revenues. Now the wild and crazy funding of search vendors has continued apace since 2012. There are a number of search and retrieval companies and some next generation content processing outfits which have ingested tens of millions of dollars.
How many of these outfits have gone public in the zero cost money environment? Based on my records, zero. Why haven’t Attivio, BA Insight, Coveo, Palantir and others cashed in on their technology, surging revenues, and market demand?
There are three reasons:
- The revenues are simply acceptable, not stunning. In the post Fast Search & Transfer era, twiddling the finances carries considerable risks. Think about a guilty decision for a search wizard. Yep, bad.
- The technology is a rehash gilded with new jargon. Take a look at the search and content processing systems, and you find the same methods and functions that have been known and in use for more than 30 years. The flashy interfaces are new, but the plumbing still delivers precision and recall which has hit a glass ceiling at 80 to 90 percent accuracy for the top performing systems. Looking for a recipe with good enough relevance is acceptable. Looking for a bad actor with a significant margin for error is not so good.
- The smart software performs certain functions at a level comparable to the performance of a subject matter index when certain criteria are met. The notion of human editors riding herd on entity and synonym dictionaries is not one that makes customers weep with joy. Smart software helps with some functions, but today’s systems remain anchored in human operators, and the work these folks have to perform to keep the systems in tip top share is expensive. Think about this human aspect in terms of how Palantir explains architects’ changes to type operators or the role of content intake specialists using the revisioning and similar field operations.
Why do I make this point in the context of unicorns? Search has one or two unicorns. I would suggest Palantir is a unicorn. When I think of Palantir, I consider this item:
To summarize, only a small number of companies reach the IPO stage.
Also, the HP Autonomy “deal” is a quasi unicorn. IBM’s investment in Watson is a potential unicorn if and when IBM releases financial data about his TV show champion.
Then there are a number of search and content processing creatures which could be hybrids of a horse and a donkey. The investors are breeders who hope that the offspring become champions. Long shots all.
The Financial Times’s article expresses a broad concept. The activities of the search and content processing vendors in the next 12 to 18 months will provide useful data about the genetic make up of some technology lab creations.
Stephen E Arnold, January 4, 2015
Crazy Numbers Department: Big Data Spending in 2019
November 26, 2015
It is almost 2016. IDC, an outfit owned by an optimistic outfit, has taken a tiny step forward. The IDC wizards answered this question, “How big will Big Data spending be in 2019?” Yep, that is 36 months in the future. There might be more money in predicting Super Bowl winners, what stock to pick, and the steps to take to minimize risk at a restaurant. But no.
According to the true believers in the Content Loop, “IDC Days Big Data Spending to Hit 48.6 Billion in 2019.” I like that point six, which seems to suggest that real data were analyzed exhaustively.
The write up reports:
The market for big data technology and services will grow at a compound annual growth rate (CAGR) of 23 percent through 2019, according to a forecast issued by research firm International Data Corp. (IDC) on Monday. IDC predicts annual spending will reach $48.6 billion in 2019. IDC divides the big data market into three major submarkets: infrastructure, software and services. The research firm expects all three submarkets to grow over the next five years, with software — information management, discovery and analytics and applications software — leading the charge with a CAGR of 26 percent.
I will go out on a limb. I predict that IDC will offer for sale three reports, maybe more. I hope the company communicates with its researchers to avoid the mess created when IDC wizard Dave Schubmehl tried to pitch eight pages of wonderfulness based on my research for a mere $3,500 without my permission. Ooops. Those IDC folks are too busy to do the contract thing I assumed.
A Schubmehl-type IDC wizard offered this observation with only a soupçon of jargon:
The ever-increasing appetite of businesses to embrace emerging big data-related software and infrastructure technologies while keeping the implementation costs low has led to the creation of a rich ecosystem of new and incumbent suppliers…. At the same time, the market opportunity is spurring new investments and M&A activity as incumbent suppliers seek to maintain their relevance by developing comprehensive solutions and new go-to-market paths.– Ashish Nadkarni, program director, Enterprise Servers and Storage, IDC
Yes, ever increasing and go to spirit. Will the concept apply to IDC’s revenues? Those thrilled with the Big Numbers are the venture folks pumping money into Big Data companies with the type of enthusiastic good cheer as Russian ground support troops are sending with the Backfires, Bears, and Blackjacks bound for Syria.
Thinking about international tension, my hunch is that the global economy seems a bit dicey, maybe unstable, at this time. I am not too excited at the notion of predicting what will happen in all things digital in the next few days. Years. No way, gentle reader.
Thinking about three years in the future strikes me as a little too bold. I wonder if the IDC predictive methods have been applied to DraftKings and FanDuel games?
Stephen E Arnold, November 26, 2015
Technology Fear News Flash: Search Not in the Top 10
October 15, 2015
I read one of those out-of-the-blue research study summaries. The information appears in Network World, a corporate family member of my favorite mid tier consulting firm IDC. The write up is titled with a zippy angle: Fear; to wit, “Technology Scares the Hell Out of People, University Survey Finds.”
I found the article a fiesta of take-it-to-the-bank information.
The snappy graphic caught my eye. Each of the Top 10 fears warrants a cartoon treatment. Here’s an example for running out of money in the future Fear Number Nine.
Source: Network World which used a cartoon from Chapman University. Academia and cartoons. Interesting.
I like the human carrying a weight (at first glance it looks like a debt bomb) up the pile of what appears to be back issues of unsold copies of print version of IDC reports. Adult Swim may do a feature based on this fear. That will be a winner.
On to another gem from the article. I highlighted this passage in the write up:
Technology-related concerns account for 3 of the top 5 biggest fears among Americans surveyed recently by Chapman University of Orange, Calif. — and a couple of the other concerns on the top 10 list could be considered tech-related worries as well.
And the tech fears are:
- Cyber terrorism
- Corporate tracking of personal information
- Government tracking of personal information
The write up adds:
Numbers 7 (Identity theft) and #10 (Credit card fraud) could also be classified as tech-related worries.
Quite a payload of fear. The write up does not include any details about the sample size, the methodology, or the folks doing the work which could be undergraduates or adjuncts for all I know.
Stepping back, let’s think about technology and analytics. On the surface, those in the sample are not exactly comfortable with what I call the Silicon Valley way. Thinking more deeply, the fears suggest that the survey suggests trust is not part of the warp and woof of the lives of the lucky folks in the sample.
My hunch is that if we polled some government officials, big time technology company CEOs, a couple of hundred top one percenters, and 20 somethings looking for a job in Palo Alto, the results might be different. I look forward to a report from IDC on this topic. I hope the author is my favorite IDC expert Dave Schubmehl. He is not afraid of technology based on my experience.
Stephen E Arnold, October 15, 2015
Big Data: Systems of Insight
October 6, 2015
I read “All Your Big Data Will Mean Nothing without Systems of Insight.” The title reminded me of the verbiage generated by mid tier consulting firms and adjuncts teaching MBA courses at some institutions of higher learning. Malarkey, parental advice, and Big Data—a Paula Dean-type recipe for low-calorie intellectual fare.
Can one live on the outputs of mid tier consulting firm lingo prepared to be fudgier?
The notion of a system of insight is not particularly interesting. The rhetorical trip of moving from a particular to a more general concept fools some beginning debaters. For a more experienced debater, the key is to keep the eye on the ball, which, in this case, is the tenuous connection between Big Data and strategic management methods. (I am not sure these exist even after reading every one of Peter Drucker’s books.)
But I like to deal with particulars.
Computerworld is a sister or first cousin unit of the IDC outfit which sold my research on Amazon without asking my permission. My valiant legal eagle was able to disappear the report. I was concerned with the connection of my name and the names of two of my researchers with the IDC outfit. I have presented some of the back story in previous blog posts. I included screenshots along with the details of not issuing a contract, using content in ways to which I would never agree, and engaging in letters with my attorney offering inducements to drop the matter. Wow. A big company is unable to get organized and then pays its law firm to find a solution to the self created problem.
The report in question was a limp wristed, eight pages in length and available to Amazon’s eager readers of romance novels for a mere $3,500. Hey, the good stuff in our research was chopped out, leaving a GrapeNut flakes experience for those able to read the document. I am a lousy writer, but I try to get my points across in a colorful way. Cereal bowl writing is not for me.
What does this have to do with Big Data and a system of insights?
Aren’t Amazon’s sales data big? Isn’t it possible to look at what sells on Amazon by scanning the company’s public information about books? Won’t a casual Google search reveal information about Amazon’s best selling eBooks? Best sellers’ lists rarely feature eight pages of watered down analysis of a search vendor with some soul bonding with the outstanding Fast Search & Transfer operation. How many folks visiting the digital WalMart buy $3,500 reports with my name on them?
Er, zero. So what’s the disconnect between basic data about what sells on Amazon, issuing appropriate contractual documents, and selling research with my name and two of my goslings on the $3,500, eight page document. That’s brilliant data analysis for sure.
The write up explains:
Businesses want to use data to understand customers, but they can’t do that without harnessing insights and consistently turning data into effective action.
That sort of makes sense except that the company which owns Computerworld, under the keen-eyed Dave Schubmehl, appeared to ignore this step when trying to sell a report with my name on it to the Amazon faithful. Do the folks at Computerworld and the company’s various knowledge properties connect data with their colleagues’ decisions?
Who Will Be the IDC Cognitive Guru?
September 10, 2015
You remember Dave Schubmehl. He is the IDC “search” expert who recycled some of my content. He then sold a report based on my work via Amazon. My meek attorney was able to get the document removed. Not even a legal eagle could fathom how eight pages of analysis could command $3,500. Mr. Bezos knows that $9.99 is a sweet spot. I suppose the masters of management at IDC thought that $3,500 report would sell like hot cakes to the consumers of romance novels, streaming video, and household commodities.
I noted a tweet (show below) that suggests his unit at IDC is going like gang busters. New staff with undergraduate degrees are needed. Uber drivers, are you paying attention?
When a consulting firm adds headcount, that’s news. I hope that the Renaissance men and women at IDC can make sense of “cognitive computing.” Google pushes “deep learning.” I suppose there will be more buzzwords as the “experts” in enterprise search flail for purchase on the slopes of Mount Make a Sale.
Most of the experts have answers to the most difficult problems in business. The only hitch in the git along occurs when the “experts” do not have certain knowledge. Oh, another modest problem is recycling another’s information without taking the time to issue a contract.
I am confident that the customers of Amazon really did want to buy that $3,500 report. Mid tier consulting firms are the cat’s pajamas. I wonder if that comes from asking IBM Watson questions.
Stephen E Arnold, September 10, 2015
Oh, Oh. Big Data Has Problems. Impossible.
August 21, 2015
A happy quack to the reader who alerted me to “5 Problems with Big Data.” How can this be? Big Data is the new black, the new enterprise search, the new information management opportunity.
The write up states:
But when data gets big, big problems can arise.
The article identifies five issues. Most of these strike me as trivial for MBAs and failed middle school teachers to resolve before lunch. The alleged problems are:
- Storage. Hey, hey. I thought storage and the management thereof were a no brainer. But I have heard rumors that finding useful items and moving them around may contribute to digital heart burn.
- Bias. What! Incredible. I heard an MBA say at a conference not long ago that with Big Data little issues get smoothed out. Imagine. Big Data works like an electric iron with a spritz feature.
- False positives. Yo, dude. Those are things one talks about in Statistics 101. So a method says Tom and Betty have Ebola. After a quick check up at the doc in the box, both seem to be suffering from bad pizza and a sleepless night caused by worrying about the mid term statistics test. So what if a financial model predicts that GOOG and GOOGL shares no upward boundary. Hello, infinity.
- Complexity. Gasp. Layering SAP with SAS components within a SharePoint environment is complex. No way, José. This is century 21. We can crash a lander on an asteroid. We can handle a simple upgrade to an air traffic control system.
- Outputs which answer a question no one asked. Look, gentle reader, we have IBM Watson. That system can answer the question, “What sauce will tamarind enhance?” The answer which made perfect sense to me was barbeque sauce. Who worries if the question was a coded string intercepted from a anonymous post on a Dark Web forum.
Stepping back I have complete confidence in the confidence men and women pitching the Big Data thing. Five speed bumps presented as real, live problems. Big Data is the answer. Enterprise search vendors like Lucid Imagination and wizards like the IDC crowd which sold some of my work without my permission on Amazon (Dave Schubmehl, where are you?) know that Big Data will do the revenue trick.
Problems are just too darned negative. I want a happy face on that flawed, incomprehensible, irrelevant, and expensive report. This is the modern world, not tout at the chariot races pitching Nero’s team.
Get real. We have no “problems.” We have opportunities.
Stephen E Arnold, August 21, 2015
Forbes and Some Big Data Forecasts
July 26, 2015
Short honk: For fee, mid tier consultants have had their thunder stolen. Forbes, the capitalist tool, wants to make certain its readers know how juicy Big Data is as a market. Navigate to “Roundup Of Analytics, Big Data & Business Intelligence Forecasts And Market Estimates, 2015.”
The write up summarizes the eye watering examples of spreadsheet fever’s impact on otherwise semi-rationale MBAs, senior managers, and used car sales professionals. IDC, without the inputs of Dave Schubmehl comes up with a spectacular number: $125 billion in 2015.
Sounds good, right?
The data will find their way into innumerable PowerPoint presentations. Snag ‘em while you can.
Stephen E Arnold, July 26, 2015
-
- Subscribe to Beyond Search
Feature archive
News archive
-
