Big Data and Its Less-Than-Gentle Lessons
August 1, 2013
I read “9 Big Data Lessons Learned.” The write up is interesting because it explores the buzzword that every azure chip consultant has used in their marketing pitches over the last year. Some true believers have the words Big Data tattooed on their arms like those mixed martial arts fighters sporting the names of casinos. Very attractive I say.
Because “big data” has sucked up search, content processing, and analytics, the term is usually not defined. The “problems” of Big Data are ignored. Since not much works when it comes to search and content processing, use of another undefined term is not particularly surprising. What caught my attention is that Datamation reports about some “lessons” its real journalists have tracked down and verified.
Please, read the entire original write up to get the full nine lessons. I want to highlight three of them:
First, Datamation points out that getting data from Point A to Point B can be tricky. I think that once the data has arrived at Point B, the next task is to get the data into a “Big Data” system. Datamation does not provide any cost information in its statement “Don’t underestimate the data integration challenges.” I would point out that the migration task can be expensive. Real expensive.
Second, Datamation sates, “Big Data success requires scale and speed.” I agree that scale and speed are important. Once again, Datamation does not bring these generalizations down to an accounting person’s desktop. Scale and speed cost money. Often a lot of money. In the analysis I did of “real time” a year or two ago, chopping latency down to a millisecond or two exponentiates the cost of scale and speed. Bandwidth and low latency storage are not sporting WalMart price tags.
Third, Datamation warns (maybe threatens) those with children in school and mortgages with, “If you’re not in the Big Data pool now, the lifespan of your career is shrinking by the day.” A couple of years ago this sentence would have said, “If you’re not in the social media pool now, the lifespan of your career is shrinking by the day.” How long with these all-too-frequent “next big things” sweep through information technology. I just learned that “CIO” means chief innovation officer. I also learned that the future of computing rests with synthetic biology.
The Big Data revolution is here. The problem is that the tools, the expertise, and the computational environment are inadequate for most Big Data problems. Companies with the resources like Google and Microsoft are trimming the data in order to get a handle on what today’s algorithms assert is important. Is it reasonable to think that most organizations can tackle Big Data when large organizations struggle to locate attachments in intra-organization email?
Reality has not hampered efforts to surf on the next big thing. Some waves are more challenging than others, however. I do like the fear angle. Nice touch at a time when senior managers are struggling to keep revenues and profits from drifting down. The hope is that Big Data will shore up products and services which are difficult to sell.
Catch the wave I suppose.
Stephen E Arnold, August 1, 2013
Sponsored by Xenky
Rich Media: Too Expensive to Store?
July 30, 2013
I saw an interesting post called “Cost of Storing All Human Audio visual Experiences.” I am no logician, but if one stores “all”, then isn’t the cost infinite? The person writing the post presents some data which pegs the cost for seven billion people at about $1 trillion a year.
Several observations:
- With the emergence of smart nanodevices with audio and video capabilities, perhaps the estimate is off the mark?
- Once the data are captured, who manages the content? Likely candidates include nation states, companies which operate as nation states, or venture funded start ups?
- How does one find a particular time segment germane to a query pertinent to a patent claim?
Interesting question if one sets aside the “all”. The next time I look for a video on YouTube or Vimeo, I will ask myself, “What type of search system is needed to deal with even larger volumes of rich media?”
Is the new Dark Ages of information access fast approaching? Yikes! Has the era already arrived?
Stephen E Arnold, July 30, 2013
Sponsored by Xenky
Dangerous Glitches Still to be Worked Out in Electronic Medical Records
July 30, 2013
Electronic medical records are part of the constantly evolving big-data landscape, and there have been some issues, according to Bloomberg‘s “Digital Health Records’ Risks Emerge as Deaths Blamed on Systems.” Not surprisingly, the period just after a new EMR system is implemented has been found to be the most dangerous time. See the article for examples of harm caused by errors. Journalist Jordan Robertson writes:
“Electronic health records are supposed to improve medical care by providing physicians quick and easy access to a patient’s history, prescriptions, lab results and other vital data. While the new computerized systems have decreased some kinds of errors, such as those caused by doctors’ illegible prescriptions, the shift away from paper has also created new problems, with sometimes dire consequences.”
Perhaps it would help if docs could access necessary data. Yet, even now, medical practices have trouble prying clinical data from their EMR systems. Apparently, a lack of data integration is the culprit, according to “Why do Docs Struggle with Population Health Data?” at Government HealthIT. That article summarizes:
“Today, with modern EHR systems, clinicians may have an easier time getting clinical data — but not all of it, which is a problem for providers pursuing population health goals. It’s also a problem as federal health officials and patient-safety organizations like the National Quality Forum try to transition from process-based quality measurements. . . to outcomes-based metrics.”
Are such digital-data challenges confined to health records alone? Unlikely, though in few (if any)other fields is big data consistently a life-and-death issue. We must remember that digital health records have also been shown to improve outcomes, but are we netting more good than bad? I suspect so, but we will probably never know for certain. One thing is sure: there’s no turning back now. Surely, mistakes will decline as systems are refined and staff acclimated. I know that is cold comfort to anyone who has suffered such a preventable loss.
Cynthia Murrell, July 30, 2013
Sponsored by ArnoldIT.com, developer of Augmentext
Snowden Controversy Will Mean More Big Data Transparency
July 29, 2013
With the recent news of the NSA’s phone records requests and deep use of data mining, we are on the cusp of a new frontier for big data and its relationship to the government. However, a little digging shows this fascinating time capsule from 2007, “Lost in the Cloud: Google in the US Government.”
According to this whitepaper:
“Consumer Watchdog investigation has found. One of the most visible signs of Google’s clout is the hangars at the National Aeronautics and Space Administration’s Moffett Air0ield, near Google’s world headquarters, where a 0leet of jets and helicopters stands ready to ferry the company’s top executives near or far, for business or pleasure. When a deal between NASA and top Google executives to use the base was 0irst disclosed in 2007, it called for only four jets to use the base.”
Not a stunner, really. The government has long been a supporter of big data and data mining technology that is sweeping the private sector these days. But, honestly, did anyone think Uncle Sam was sinking money into this field to help refine the census? We are not political one way or the other, but what we do love is transparency in big data and we suspect that’s what’s on the horizon after this affair.
Patrick Roland, July 29, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Selfless Analytic Partnerships Score Big
July 29, 2013
One would think the big data analytics field would be cutthroat and bloody with competition. Oddly, though, it is actually a mostly friendly affair. It seems like we hear more and more about disparate firms partnering up to give customers a better product. Such was the case with this recent Business Wire story, “JackBe Unveils Presto Real-Time Analytics Add On with Terracotta Big Memory.”
According to the story:
“JackBe®, the leading provider of Real-Time Actionable Intelligence software, today released its Presto Real-Time Analytics Add-On With Terracotta BigMemory (RTA Add-On), which bundles Terracotta’s enterprise-grade BigMemory in-memory data management platform for seamless high performance in-memory analytics. This supercharged combination of in-memory and analytics allows Presto to mash Big Data with live and transactional enterprise data into actionable dashboards in a fraction of the time required by traditional methods, putting real-time Big Data analytics at the fingertips of decision-makers.”
We like the way these two companies are playing to their strengths. When selfless analytics minds do that, the customers score big. It is reminiscent of the IBM big data partnership. In these cases we are seeing savvy companies partnering to do something extraordinary.
Patrick Roland, July 29, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
CIO Top 10 Big Data Startups
July 29, 2013
CIO.com has finalized their list of the top ten big data startups to watch. The results have been released. Read more in their article, “10 Top Big Data Startups to Watch–Final Rankings.”
The article states the criteria:
“After more than 4,000 votes were cast, the final Big Data startup rankings are in. Keep in mind that while voting was weighted heavily, it was not the be-all-and-end-all consideration. Other criteria included big-name end users, VC funding, the pedigree of the management team and market positioning.”
So after such thorough analysis, we were pleased to see a few familiar names on the list. Several good companies that are delivering quality products, LucidWorks among them:
“LucidWorks did well in the voting, has solid funding ($16 million) and is uniquely positioned in this roundup with their focus on Big Data search. However, they’ll need to get more on-the-record customers on the books to climb higher than this.”
LucidWorks Big Data will continue to do well among its existing customer base and expand into further markets. Their open source platform brings the strength of Apache Lucene Solr to an intuitive solution that is ready out-of-the-box.
Emily Rae Aldridge, July 29, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Hadoop Summit Makes Big Data Engaging
July 26, 2013
The Hadoop Summit was held at the end of June in San Jose, California. The event focuses primarily on end users and developers of things built on and around Apache Hadoop open source technology. However, as Hadoop is integrated into more and more software offerings, Hadoop’s reach continues to expand. SmartData Collective offers a good write-up of how Hadoop is making Big Data technology more dynamic. Read more in their article, “Hadoop Summit and Hortonworks Promise to Make Big Data More Engaging.”
The article describes the event:
“Hadoop Summit is the biggest event on the West Coast centered on Hadoop, the open source technology for large-scale data processing. The conference organizers, Hortonworks, estimated that more than 2,400 people attended, which if true would be double-digit growth from last year. Growth on the supplier side was even larger, which indicates the opportunity this market represents. Held in Silicon Valley, the event attracts enterprise customers, industry innovators, thought leaders and venture capitalists.”
The author then goes on to say that among the companies that are making big gains in information discovery with Hadoop are LucidWorks and Microsoft. LucidWorks uses Hadoop in its LucidWorks Big Data offering, which has received quite a bit of attention for its usability and effectiveness.
Emily Rae Aldridge, July 26, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
LucidWorks Makes Big Data Better
July 25, 2013
LucidWorks made its name as an open source enterprise search value-added vendor built upon Apache Lucene Solr. However, in the last 18 months, much attention has shifted to the Big Data sector and LucidWorks Big Data solution. Mark Smith with Ventana Research gives his review of the product in his article, “Big Data Search is Getting Better with LucidWorks.”
He writes:
“LucidWorks has two product offerings in the search market. LucidWorks Search provides the ability to rapidly set up search and index content using Apache Solr. . . Solr, built on top of Lucene, is an enterprise platform that provides full-text search, dynamic clustering, geospatial search and other enterprise-class capabilities. Second, LucidWorks Big Data, which was released late last year, uses the compute and storage capabilities of Hadoop to support larger-scale deployments.”
LucidWorks is known for its commercial-grade support and services. Their design has improved Solr’s usability for business use, although Solr on its own is known as one of the strongest infrastructures for open source enterprises. LucidWorks is currently expanding, adding staff, and gearing up for the next advance in enterprise technology. Keep your eyes open.
Emily Rae Aldridge, July 25, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Gartner Highlights Cool Big Data Firms
July 24, 2013
Is coolness enough to succeed in the big data analytics world? That’s a tougher question to answer than you’d think. If we’re talking about fly-by-night hipness, then probably not. However, Gartner has been taking the entire industry back to high school by deeming certain organizations cool. Shockingly, they might be on to something, as we discovered in a recent PR Web story, “Gartner Research Recognizes Centrifuge as a ‘Cool Vendor’ for Big Data.”
According to the story:
“Centrifuge Systems today announced it has been selected by Gartner as a “Cool Vendor” in the respected analyst firm’s annual report on Security Intelligence Vendors for 2013* that recognizes innovative and disruptive technology solutions in big-data analytics, particularly those that provide “contextual analysis of security intelligence across silos”.”
Interesting news on a promising company. Normally, we are pretty skeptical about such designations, but have become big fans of Gartner’s cool awards. It’s not bestowing a lot of hype on a company to blow our minds, it’s a more subtle way of saying someone is doing something good for the time being, check it out. That sounds like a recommendation we can seriously get behind.
Patrick Roland, July 24, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
Powerful Partnerships Persevere in Big Data
July 23, 2013
Powerful partnerships in the analytic world are quickly becoming a daily happening. However, it’s rare that two major names in the industry link arms to build something for customers. However, that is just what happened recently, according to an InetSoft article, “InetSoft Adds SaaS BI for Quickbooks Users, Advanced Reporting and Analytics Through InetSoft’s Style Intelligence.”
According to the story:
“The Bison System is a specialized, hosted business intelligence tool with the best data connector available for millions of QuickBooks customers. The Bison System extracts customer QuickBooks data, securely transfers it to the Bison Servers, and then immediately processes it into the customer’s database. Within moments, the data is available for dashboards and customized reporting for QuickBooks business stakeholders.”
This blockbuster pairing of InetSoft and Quickbooks felt familiar. It reminded us of a recent Forbes article about Oracle’s similar pairings, which said: “three strategic partnership announcements this week, some of which have surprised even seasoned enterprise software industry observers and analysts.” That’s why companies like these two examples are on top. They are finding bold ways to partner and give clients analytic tools that are going well beyond standard big data.
Patrick Roland, July 23, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
-
- Subscribe to Beyond Search
Feature archive
News archive
-
