Addiction Model Measures App’s Addictiveness Over Retention

March 28, 2014

The article on re/ titled Mixpanel: How Addictive Is Your App? presents a new analytic report called Addiction. Under a picture of a wrist cuffed to the smartphone it holds, the article cheerfully explains that fifty percent of social app users engage with the service for over five hours a day. Enterprise apps are used more during the business day, and messaging apps show a lesser addiction in their users, supporting the idea that people are now using social media apps for most of their communications. The article explains,

“Addiction adds an extra layer of insight that allows companies to analyze user behavior on an even deeper level. One thing that’s clear is that addiction is inextricably linked to function: If your product is a social app that people don’t use more than once a day, that’s a red flag — and not one you would have previously been able to catch if you relied solely on Retention.”

The article stipulates that the most important feature of Addiction is that it enables companies to visualize how “embedded” their service is in user’s daily schedules. This will allow them to better follow the effect of their smallest adjustments in the app and really see how their customers react. Whether or not this is a dangerous ability is not considered.

Chelsea Kerwin, March 28, 2014

Sponsored by, developer of Augmentext

The HP View of Watson

March 19, 2014

I suppose IBM will respond with more than recipes at South by Southwest. If you enjoy big companies’ analyses of one another, you will want to gobble up “15 Reasons HP Autonomy IDOL OnDemand Beats IBM Watson.” This is not the recipe for making pals with a $100 billion outfit.

What does IBM Watson have as weaknesses? What does the reinvented (sort of) Autonomy technology have as strengths? I cannot reproduce the 15 items, but I can highlight five of the weaknesses and enjoin you to crack open the slideshow that chops up the IBM Watson PR stunt.

Here are the six weaknesses I found interesting:

  1. Reason 3. IBM Watson is a data scientist heavy platform. IDOL is not. My view is that HP paid $11 billion for Autonomy and now has to deal with the write down, legal actions related to the deal, and tossing out Mike Lynch’s revenue producing formula. Set aside the data scientists and the flip side “too few data scientists” and consider the financial mountain HP has to climb. A data scientist or two might help.
  2. Reason 4. HP has “an ultimate partner story.” I find this fascinating. Autonomy grew via acquisitions and an indirect sales model. Now HP wants to make the partner model generate enough revenue to pay off the Autonomy purchase price, grow HP’s top line faster than traditional lines of business collapse, and make partners really happy. This may be a big job. See IBM weakness 9, 11, 12, and 14. There is some overlap which suggests HP is having difficulty cooking up 15 credible weaknesses of Watson. (I can name some, by the way.)
  3. Reason 6. HP offers a “proven power platform for analytics.” I am not sure about the alliteration nor am I confident in my understanding of analytics and search. IBM Watson doesn’t have much to offer in either of these departments. IDOL, at least the pre HP incarnation, had reasonably robust security capabilities. I wonder how these will be migrated to the HP multi cloud environment. IBM Watson is doing recipes, so it too has its hands full.
  4. Reason 10. HP asserts that it offers a “potential app store.” I understand app store. Apple offers one that works well. Google is in the app store business. Amazon has poked its nose into the marketplace as well. I don’t think either HP or IBM have credible app stores for variants of the two companies’ search technologies. Oh, well, it sounds good. “Potential” is a deal breaker for me.
  5. Reason 13. HP “is focused on ramping up the innovation lifecycle.” I think this means coming up with good ideas faster. I am not sure if a service can spark a client’s innovation. Doesn’t lifecycle include death? Since IBM Watson seems a work in progress, I am not sure HP’s just released reinvention of Autonomy has a significant advantage because it too is “ramping up.”
  6. Reason 15. HP has “fired up” engineers. Okay, maybe. IBM has engineers, but I am not sure if they are fired up. My question is, “Is being fired up” a good thing. I want engineers to deliver solutions that work, are not “ramping up,” and not marketing driven.

My take on this slide deck is that it is nothing more than a marketing vehicle. I had to click multiple ads for HP products and services to view the 15 reasons. Imagine my disappointment that five of the IBM weaknesses related to partnering programs. Wow, that must be really helpful to a licensee of cloud Autonomy trying to deal with performance issues on an HP data center. HP is definitely countering IBM Watson’s recipe play with old fashioned cheerleading. Rah, rah.

Stephen E Arnold, March 19, 2014

Civic Predictive Analysis Proving Accurate

March 19, 2014

We find the field of predictive analysis fascinating (see here, here, and here, for example), and now we have more evidence of how important this work can be. Motherboard reports on “The Math that Predicted the Revolutions Sweeping the Globe Right Now.” The key component: high food prices. Writer Brian Merchant explains:

“There’s at least one common thread between the disparate nations, cultures, and people in conflict, one element that has demonstrably proven to make these uprisings more likely: high global food prices.

Just over a year ago, complex systems theorists at the New England Complex Systems Institute warned us that if food prices continued to climb, so too would the likelihood that there would be riots across the globe. Sure enough, we’re seeing them now. The paper’s author, Yaneer Bar-Yam, charted the rise in the FAO food price index—a measure the UN uses to map the cost of food over time—and found that whenever it rose above 210, riots broke out worldwide. It happened in 2008 after the economic collapse, and again in 2011, when a Tunisian street vendor who could no longer feed his family set himself on fire in protest.”

Bar-Yam’s model forewarned about the Arab Spring and the Tunisian self-immolation. Well, not those specific ways unrest would manifest, but that something big and ugly was bound to happen. Similarly, the same model divined that there would be conflicts around the world this year—as we have seen in the Ukraine, Venezuela, Brazil, Thailand, Bosnia, Syria, Spain, France, Sweden…. Last year’s global food prices were the third-highest on record; this is no coincidence. See the article for more on Bar-Yam’s methods as well as specific links between food scarcity and some of the conflicts currently shaking the world.

What can this technology do, besides hand a few of us a big bucket of “I-told-you-so”? Armed with this information, policymakers could take steps to modify the way the global marketplace is run and stop (at least some, possibly most) food shortages before they start. This means powerful people from many countries would have to work together to make major changes on a global scale for the good of humanity. With money involved. Hey, anything’s possible, right?

Cynthia Murrell, March 19, 2014

Sponsored by, developer of Augmentext

Kontagent Comes Clean

March 18, 2014

A recent partner audit by Facebook prompted the removal of business intelligence firm Kontagent from the Facebook Mobile Measurement Program (MMP). A post at the Kontagent Kaleidoscope blog from the company’s CEO, “An Update on our Relationship with Facebook, How We Store Data,” addresses the issue head-on. Andy Yang admits his company made a mistake, but assures us that absolutely no data breaches resulted from the misstep. Furthermore, though the company is not currently part of the MMP, it is still working with Facebook in other areas.

Yang details what precipitated his company’s removal from the program. They did violate Facebook’s policy on how long they could store data, but note that the slip-up occurred as they were working to exceed Facebook’s requirements on privacy and security. Still, they say, the mistake was theirs, they are learning from it, and they hope to earn the chance to rejoin the program. See the post for more on their security measures and on what transpired with Facebook. Yang summarizes:

“In short, Kontagent created an encryption policy that we designed to completely protect user privacy while addressing Facebook’s policy in one elegant solution. In hindsight, while our intentions were good, we overthought the solution when a more basic approach would have better met Facebook’s requirements.

“I completely respect the audits that Facebook conducts to ensure their partners are properly compliant. We will address each of the issues noted in Facebook’s audit despite not being a member of the MMP.”

After its launch in 2007, Kontagent cut their data analysis teeth on SaaS analytics for key social game developers. Now, leading brands in a variety of fields depend upon their expertise. Based in San Francisco, Kontagent also maintains offices in Toronto, London, Seoul, and Tokyo.

Cynthia Murrell, March 18, 2014

Sponsored by, developer of Augmentext

Full Fidelity Analytics from Karmasphere

March 18, 2014

It is the data equivalent of a distortion-free sound system— Karmasphere blogs about what they are calling “Full-Fidelity Analytics.” Karmashpere founder Martin Hall explains what the analytics-for-Hadoop company means by the repurposed term:

“Ensuring Full-Fidelity Analytics means not compromising the data available to us in Hadoop in order to analyze it. There are three principles of Full-Fidelity Analytics:

1. Use the original data. Don’t pre-process or abstract it so it loses the richness that is Hadoop

2. Keep the data open. Don’t make it proprietary which undermines the benefits of Hadoop open standards

3. Process data on-cluster without replication. Replication and off-cluster processing increases complexity and costs of hardware and managing the environment.

“By adhering to these principals during analytics, the data remains rich and standard empowering deep insights faster for companies in the era of Big Data.”

The post goes on to list several advantages to the unadulterated-data policy; Hall declares that it reduces complexity, lowers the total cost of ownership, and avoids vendor lock-in, to name a few benefits. The write-up also discusses the characteristics of a full-fidelity analytics system. For example, it uses the standard Hadoop metastore, processes analytics on-cluster, and, above all, avoids replication and sampling. See the post for more details about this concept. Founded in 2010, Karmasphere is headquartered in Cupertino, California.

Cynthia Murrell, March 18, 2014

Sponsored by, developer of Augmentext

HP Healthcare Analytics Aids in Reducing Waste

March 17, 2014

The article titled HP Autonomy Unlocks Value of Clinical Data with HP Healthcare Analytics from Market Watch explores HP’s announcement of a new analytics platform for healthcare providers to use in their work to comprehend clinical data, both structured and unstructured. The new platform was created in a partnership between HP and Standford Children’s Health and Lucile Packard Children’s Hospital. It is powered by HP Idol. The article states,

“The initial results have already yielded valuable insights, and have the potential to improve quality of care and reduce waste and inefficiency.

Though the core mission of the Information Services Analytics team at Lucile Packard Children’s Hospital Stanford is to enable operational insights from structured clinical and administrative data, innovation projects are also a key strategic initiative of the group… The healthcare industry faces the enormous challenges of reducing cost, increasing operational efficiency and elevating the quality of patient care.”

Costs have gotten out of control and it is the hope of this collaboration that analytics might be the key. A huge part of problem is the unstructured data that is overlooked in the form of text in a patient’s records, notes from the doctor or emails between the doctor and patient. HP Idol’s ability to understand and categorize such information will make early diagnosis and early detection much more possible. For more information visit

Chelsea Kerwin, March 17, 2014

Sponsored by, developer of Augmentext

Google Flu Trends: How Algorithms Get Lost

March 15, 2014

Run a query for Google Flu Trends on Google. The results point to the Google Flu Trends Web site at The graphs and charts seem authoritative. I find the colors and legends difficult to figure out, but Google knows best. Or does it?

A spate of stories have appeared in New Scientist, Smithsonian, and Time that pick up the threat that Google Flu Trends does not work particularly well. The Science Magazine podcast presents a quite interesting interview with David Lazar, one of the authors of “The Parable of Google Flu: Traps in Big Data Analysis.”

The point of the Lazar article and the greedy recycling of the analysis is that algorithms can be incorrect. What is interesting is the surprise that creeps into the reports of Google’s infallible system being dead wrong.

For example, Smithsonian Magazine’s “Why Google Flu Trends Can’t Track the Flu (Yet)” states, “The vaunted big data project falls victim to periodic tweaks in Google’s own search algorithms.” The write continues:

A huge proportion of the search terms that correlate with CDC data on flu rates, it turns out, are caused not by people getting the flu, but by a third factor that affects both searching patterns and flu transmission: winter. In fact, the developers of Google Flu Trends reported coming across particular terms—those related to high school basketball, for instance—that were correlated with flu rates over time but clearly had nothing to do with the virus. Over time, Google engineers manually removed many terms that correlate with flu searches but have nothing to do with flu, but their model was clearly still too dependent on non-flu seasonal search trends—part of the reason why Google Flu Trends failed to reflect the 2009 epidemic of H1N1, which happened during summer. Especially in its earlier versions, Google Flu Trends was “part flu detector, part winter detector.”

Oh, oh. Feedback loops, thresholds, human bias—Quite a surprise apparently.

Time Magazine’s “Google’s Flu Project Shows the Failings of Big Data” realizes:

GFT and other big data methods can be useful, but only if they’re paired with what the Science researchers call “small data”—traditional forms of information collection. Put the two together, and you can get an excellent model of the world as it actually is. Of course, if big data is really just one tool of many, not an all-purpose path to omniscience, that would puncture the hype just a bit. You won’t get a SXSW panel with that kind of modesty.

Scientific American’s “Why Big Data Isn’t Necessarily Better Data” points out:

Google itself concluded in a study last October that its algorithm for flu (as well as for its more recently launched Google Dengue Trends) were “susceptible to heightened media coverage” during the 2012-2013 U.S. flu season. “We review the Flu Trends model each year to determine how we can improve—our last update was made in October 2013 in advance of the 2013-2014 flu season,” according to a Google spokesperson. “We welcome feedback on how we can continue to refine Flu Trends to help estimate flu levels.”

The word “hubris” turns up in a number of articles about this “surprising” suggestion that algorithms drift.

Forget Google and its innocuous and possibly ineffectual flu data. The coverage of the problems with the Google Big Data demonstration have significance for those who bet big money that predictive systems can tame big data. For companies licensing Autonomy- or Recommind-type search and retrieval systems, the flap over flu trends makes clear that algorithmic methods require baby sitting; that is, humans have to be involved and that involvement may introduce outputs that wander off track. If you have used a predictive search system, you probably have encountered off center, irrelevant results. The question “Why did the system display this document?” is one indication that predictive search may deliver a load of fresh bagels when you wanted a load of mulch.

For systems that do “pre crime” or predictive analyses related to sensitive matters, uninformed “end users” can accept what a system outputs and take action. This is the modern version of “Ready, Fire, Aim.” Some of these actions are not quite as innocuous as over-estimating flu outbreaks. Uninformed humans without knowledge of context and biases in the data and numerical recipes can find themselves mired in a swamp, not parked at the local Starbuck’s.

And what about Google? The flu analyses illustrate one thing: Google can fool itself in its effort to sell ads. Accuracy is not the point of Google or many other online information retrieval services.

Painful? Well, taking two aspirins won’t cure this particular problem. My suggestion? Come to grips with rigorous data analysis, algorithm behaviors, and old fashioned fact checking. Big Data and fancy graphics are not, by themselves, solutions to the clouds of unknowing that swirl through marketing hyperbole. There is a free lunch if one wants to eat from trash bins.

Stephen E Arnold, March 15, 2014

March 13, 2014

Analytics Improves SharePoint Experience

March 13, 2014

Microsoft partners are responsible for SharePoint add-ons that increase usability and efficiency for users. Webtrends is one such partner that offers an Analytics for SharePoint solution. Broadway World covers their latest announcement in the article, “Employee Adoption for SharePoint Soars With Webtrends Analytics.”

The article begins:

“Webtrends, a Microsoft-preferred partner for SharePoint analytics, today announced a 64% year-over-year increase in customer bookings for its Analytics for SharePoint business . . . Leveraging deep analytics expertise and use cases from customers like BrightStarr and Siemens, Webtrends highlights key insights and successes, including a preview of an analytics for Yammer solution, during the SharePoint Conference in Las Vegas, NV on March 3-6.”

Stephen E. Arnold has a lot to say about SharePoint from his platform, As a longtime search expert, Arnold knows that SharePoint’s success hinges on customization and add-ons, which allow an organization to take this overwhelming solution and make it work for them.

Emily Rae Aldridge, March 13, 2014

Thiess and IBM Work Together to Improve Thiess’s Efficiency

March 12, 2014

The article titled IBM and Thiess Collaborate on Predictive Analytics and Modeling Technologies on explores the partnership of IBM and Thiess, an Australian construction, mining and service provider. The collaboration is centered on both predictive analytics in regards to maintenance and replacement information as well as early detection of malfunctions. The article states,

“Thiess Australian mining executive general manager Michael Wright said the analytics and modeling can offer great opportunities to improve business of the company. “Working with IBM to build a platform that feeds the models with the data we collect and then presents decision support information to our team in the field will allow us to increase machine reliability, lower energy costs and emissions, and improve the overall efficiency and effectiveness of our business,” Wright said.”

This is another big IBM bet. The collaboration will start with Thiess’s mining haul trucks and excavators. Models will be constructed around such information as inspection history of the equipment, weather conditions and payload size. These models will then be used to help make more informed decisions about operational performance, and will allow for early detection of anomalies as well as predictions about when a piece of equipment will require a replaced part. This will in turn allow Thiess to plan productions more accurately around the predicted health of a given machine.

Chelsea Kerwin, March 12, 2014

Sponsored by, developer of Augmentext

Tableau Finds Success Since Going Public Last Year

March 12, 2014

Investment site the Street is very enthused about Tableau Software, which went public less than a year ago. In fact, they go so far as to announce that “Tableau’s Building the ‘Google for Data’.” In this piece, writer Andrea Tse interviews Tableau CEO Christian Chabot. In her introduction, Tse notes that nearly a third of the company’s staff is in R&D—a good sign for future growth. She also sees the direction of Tableau’s research as a wise. The article explains:

“The research and development team has been heavily focused on developing technology that’s free of skillset constraints, utilizable by everyone. This direction has been driven by the broad, corporate cultural shift to employee-centric, online-accessible data analytics, from the more traditional, hierarchical or top-down approach toward data analysis and dissemination.

“Tableau 9 and Tableau 10 that are in the product pipeline and soon-to-be-shipped Tableau 8.2 are designed to highlight ‘storytelling’ or visually striking data presentation.

“Well-positioned to ride the big data wave, Tableau shares, as of Tuesday’s [February 11] intraday high of $95, are now trading over 206% above its initial public offering price of $31 set on May 16.”

In the interview, Chabot shares his company’s research philosophy, touches on some recent large deals, and takes a gander at what’s is ahead. For example, his developers are currently working hard on a user-friendly mobile platform. See the article for details. Founded in 2003 and located in Seattle, Tableau Software grew from a project begun at Stanford University. Their priority is to help ordinary people use data to solve problems quickly and easily.

Cynthia Murrell, March 12, 2014

Sponsored by, developer of Augmentext

« Previous PageNext Page »