The Cost of Innovation: Apple Data Lights the Path for Search Vendors and Their Research Costs
February 13, 2014
I have an iPod and an iPad kicking around. We even have a Mac computer. My wife has an iPhone. The gizmo provides her iPhone equipped friends with myriad opportunities to look at baby pictures, check lunch dates on their calendars, and make phone calls. None of the gizmos is without flaws. The Apple product line up is premium priced and designed to meet the perceived needs of semi-affluent or pretend-affluent customers.
I read “A Look at Apple’s R&D Expenditures from 1995-2013.” I urge you to read the story but the main point is the diagram showing Apple’s spending for research and development. I translate “research and development” to “innovation” but you may have a different way to define the phrase. I have snipped a small segment of the chart to illustrate what has happened to Apple, based on the data presented in the write up.
Look at that ramp up. What is fascinating is that the scale in 2013 noses into the $4 billion range. The take away is that the amount of money Apple is spending is rising pretty quickly. Apple has money in the bank and some products that continue to sell well.
Apple is able to invest increasing amounts of money in innovation because it has money.
Search vendors face innovation problems. The chatter on LinkedIn and in the write ups for conferences that flood my email talk around innovation. The discussion pivots on some well worn themes, only tangentially related to information retrieval.
Innovation in search has stalled. Apple is spending aggressively to help ensure its innovation flow.
But what happens when a search vendor with far less money has to innovate. DARPA will award a handful of contracts. Venture funding sources will want a pay off.
The net net is that the cost of innovation in search is not that different in its need for financial investment. Apple can write the checks. Most search vendors—despite the flashy webinars and mindless news releases—cannot.
Stephen E Arnold
Getting a Failing Grade in Artificial Intelligence: Watson and Siri
February 12, 2014
I read “Gödel, Escher, Bach: An Eternal Golden Braid” in 1999 or 2000. My reaction was, “I am glad I did not have Dr. Douglas R. Hofstadter critiquing my lame work for the PhD program at my university. Dr. Hofstadter’s intellect intimidated me. I had to look up “Bach” because I knew zero about the procreative composer of organ music. (Heh, heh)
Imagine my surprise when I read “Why Watson and Siri Are Not Real AI” in Popular Mechanics magazine. Popular Mechanics is not my first choice as an information source for analysis of artificial intelligence and related disciplines. Popular Mechanics explains saws, automobiles, and gadgets.
But there was the story, illustration with one of those bluish Jeopardy Watson photographs. The write up is meaty because Popular Mechanics asked Dr. Hofstadter questions and presented his answers. No equations. No arcane references. No intimidating the fat, ugly grad student.
The point of the write up is probably not one that IBM and Apple will like. Dr. Hofstadter does not see the “artificial intelligence” in Watson and Siri as “thinking machines.” (I share this view along with DARPA, I believe.)
Here’s a snippet of the Watson analysis:
Watson is basically a text search algorithm connected to a database just like Google search. It doesn’t understand what it’s reading. In fact, read is the wrong word. It’s not reading anything because it’s not comprehending anything. Watson is finding text without having a clue as to what the text means. In that sense, there’s no intelligence there. It’s clever, it’s impressive, but it’s absolutely vacuous.
I had to look up vacuous. It means, according to the Google “define” function: “having or showing a lack of thought or intelligence; mindless.” Okay, mindless. Isn’t IBM going to build a multi-billion dollar a year business on Watson’s technology? Isn’t IBM delivering a landslide business to the snack shops adjacent its new Watson offices in Manhattan? Isn’t Watson saving lives in Africa?
The interview uses a number of other interesting words; for example:
- Hype
- Silliest
- Shambles
- Slippery
- Profits
Yet my favorite is the aforementioned—vacuous.
Please, read the interview in its entirety. I am not sure it will blunt the IBM and Apple PR machines, but kudos to Popular Mechanics. Now if the azure chip consultants, the failed Webmasters turned search experts, and the MBA pitch people would shift from hyperbole to reality, some clarity would return to the discussion of information retrieval.
Stephen E Arnold, February 11, 2014
Google Puts Some Effort into the Google Search Appliance
February 12, 2014
Last I knew, the Google Search Appliance (GAS) had trimmed its product line, eliminated the impulse buy option for the Mini, and kept the price at the higher end of the appliance market.
I learned over the last two years that Google has placed more than 60,000 GSAs in organizations. I have no idea if the number is valid, but if it is, the GSA is one of the top dogs in enterprise search. I also heard that there was a small team working on the GSA and an even smaller team handling customer support. Google pushes functions to resellers who deal with the customers. Google outsources manufacturing of the GSA. Most important, Google seems to have an off-again, on-again interest in on premises search. The future, as I understand it, is the cloud. The GSA is, in my opinion, an anachronism in the Nest, X Labs, and Android-Chrome world. But, hey, I have been wrong before. I once asserted that basic search should not be a challenge for most organizations. Wow, did I get that wrong! Jail time, law suits, and DARPA’s almost admission that search is not working notwithstanding.
The GSA has been around almost a decade. Version 7.2 is “a leader in the Garnet Enterprise Search MQ.” I certainly don’t doubt the word of an estimable azure chip consulting firm. No, no, no.
The new version, according to Google, delivers:
- Metadata sorting. A function available in the 1983 version of Fulcrum Technologies’ system
- language translation. A function available from Delphes in the 1990s
- A document preview function. iPhrase in 1999 delivered this feature
- Entity recognition. Verity implemented this function in the 1980s
- Dynamic navigation. Endeca rolled out this feature in 1998
In my opinion, the GSA is catching up to innovations available for many years from other vendors. Comparing the EPI Thunderstone and Maxxcat appliances to the GSA emphasizes that the GSA is not quite at parity with other products in the channel.
According to “Google Updates Enterprise Search Appliance Tool,”
The GSA 7.2 update comes more than a year after the firm upgraded the GSA to version 7.0, and builds on the features included in that update. The most notable includes the ability to improve the way data can be indexed with key attributes, such as author name, or the date it was created.
How much does a GSA cost? According to the US government’s GSAadvantage.gov, a 36 month license for a GB 7007 is $69,296 for 500,000 documents. Have more documents? Pay for an upgrade. However, I can use a hosted service like Blossom Software to index my content for about $2,400 per month. I can use the low cost dtSearch solution for $160 per seat. I can download an open source solution and do it myself.
For an organization with 20 million documents to index, the cost of the GSA solution noses into HP Autonomy territory. Too rich for my blood, and I think that lower cost appliance vendors will see the Google Search Appliance as a lead generator.
I wonder if those azure chip consultants have licensed the GSA to handle their Intranet information retrieval tasks?
Stephen E Arnold, February 12, 2014
DARPA Hints That Search Fails
February 11, 2014
One of my two or three readers sent me a link to “DARPA-BAA-14-21: Memex.” The item is interesting because it reaches back to the idea of Vannevar Bush, sidesteps the use of the word “Memex” by a search vendor once operating in the United Kingdom, and provides pretty clear proof that DARPA is not happy with search. You can dig into the details at https://www.fbo.gov/utils/view?id=32c351ba7850360e140a29f363819052.
US government content has some interesting characteristics. One of the most interesting is that items like DARPA-BAA-14-21 appear without context. For example, there is not a hint, nary a whisper of In-Q-Tel’s investments in search and content processing. Years ago, I heard at an intel conference that In-Q-Tel funds promising companies but few of these deliver operational payoffs. You can see a list of In-Q-Tel investments at https://www.iqt.org/portfolio/. Some of these companies deliver darned interesting demonstration systems. Others have offered solutions that were eventually abandoned. Others are like Fourth of July fireworks; that is, the financial support and walk arounds provide the type of show that some decision makers perceive as progress and purposeful action.
The net net is that this DARPA item underscores that information retrieval system is not appropriate for the future needs of DARPA. For me, this is one indication that my assertion about the troubled state of information retrieval.
Perhaps the funding, the TREC tests, and the DARPA solicitation will yield a payoff for operational personnel. “Perhaps” is a bit soft even if the devalued dollars are real. Our research offers some interesting facts that finding information today is more difficult than it was five years ago.
Stephen E Arnold, February 11, 2014
Fast Search Founder Slowed Down, Then Stopped
February 10, 2014
I read “Fastgründer John Markus Lervik dømt til fengsel.” Assuming the story is accurate, Dr. John Lervik, the founder of Fast Search & Transfer, will serve at least one year in prison. The issue is related to the financial reporting of Fast Search & Transfer.
In 2008, Microsoft purchased the company for about $1 billion, a deal compared to the price Hewlett Packard paid for Autonomy and about what Oracle paid for Endeca. Mr. Lervik will pay to pay legal fees. He will take appropriate legal steps to overturn the decision.
Enterprise search is a tough nut to crack technically and financially. The monetary challenges stem from the brutal costs of marketing and customer support. But these are at least as expensive as the cost of dealing adequately with technical challenges of enterprise search. For example:
- The time required to make a system deliver what the marketers assure customers are “ready to deploy” functions. Most large scale search solutions are not products. These are complex systems. Because each customer has specific requirements, the marketers do not understand that what they sold may take time to create, test, and deliver. Time is money. With an open ended problem, the cost is staggering.
- The problem of responding to crashes. When an enterprise search system flips over and dies, the cause may be the vendor, the reseller, or the client. Unfortunately the vendor takes the heat because many tech centric managers feel the “buck stops here.” Responding when a client is crazy mad is expensive. Failing to address the client’s need may delay payments or trigger legal action. Expensive stuff.
- The need to invest to keep pace with the information environment. Most of the mainstream search systems, including Fast Search and other older systems, focused on text. Handling different file types and different content types is an expensive operation for some vendors. The choice is stark: Spend and develop the components in house, spend money for third party solutions and then spend more to integrate those solutions into the core system, buy a company that has the people and the software needed, or ignore the client. There may be other options, but these four have big price tags. The cost of keeping up is brutal because information retrieval does not stand still.
- Figuring out why routine operations are slow or output unexpected results. Most search systems are far trickier to set up than licensees expect. With many knobs to turn, Fast Search could be tweaked so that results could boost certain content or address relevancy under specific circumstances. In a complex system, like Fast and many others, turning one knob and experimenting with threshold values could cause some darned exciting consequences. Rolling back those changes was an exciting operation in itself. When a Fast engineer had to figure out how to get the system back on track, the work was not trivial. What’s it cost to get an expert engineer to figure out what a licensee did? In many instances, a lot.
If you add up the costs of the technical work required for a complex search system, the need for money is significant. Dr. Lervik is not a financial expert; he is an expert in information retrieval. Not even ex-Googlers are adept managers. Witness the AOL goof related to “distressed babies.”
But a senior manager is expected to find solutions to difficult managerial, technical, and financial challenges. If the news story is true, it seems that Dr. Lervik was caught in a situation that set the stage for the unfortunate drama that has been playing out over the last five years.
The big question is:
Will other search and content processing vendors find themselves in a similar situation?
In my opinion, yes.
Warning signs are easy to spot. When search vendors that are seven or 12 years old continue to suck in venture funding, the warning flags are flying in my opinion. Search is essentially a zero license fee utility at this point. Firms that have yet to return a profit or show significant growth may find themselves taking financial short cuts.
The Xenky analyses make clear that financial stress is nothing new to search vendors. Check out the Convera, Delphes, and Fulcrum Technologies profiles. What’s different is that in today’s business environment, the consequences may be increasingly severe. You can find case studies of search vendors at www.xenky.com/vendor-profiles. There is no charge for these reports. Many describe enterprise search solutions that struggled financially and either shut down or sold out.
Enterprise search is a tough business. A sad quack for Dr. Lervik.
Stephen E Arnold, February 10, 2014
News Spectrum: An Autonomy Service?
February 10, 2014
I came across www.news-spectrum.com. The system looked a bit like some of Autonomy’s visualizations. Here’s the splash screen for the service:
The idea is that a story can be viewed through time. There are news spectra for the UK, Europe, the US, politics, business, and a handful of other categories. A click on the “detail” button displays stories in the topic stream.
I navigate to a who is service and learned that the domain name is registered to Autonomy, now a unit of Hewlett Packard.
In my lectures for law enforcement and intelligence professionals, the challenges of locating information in “news” is getting more difficult. Sites like News Spectrum and some others do not include a search function. When the search function is present, the user has to turn cartwheels to get useful information. For example, navigate to World News and run a query. I used Sochi. Here’s the result list:
Scroll down the page and this is what I saw:
Videos. Videos. More videos. Where is the text? You have to do some experimenting. A tip is to select a language and then rerun the query.
The problem, of course, is that most people just take what a system displays in the case of News Spectrum and World News.
Any type of in depth research requires some specialized, and often time consuming, tactics. You can learn more about how to get through the Kevlar padding sites that wrap their indexes in Kevlar.
Net net: A news search can look good and run little videos. But for in depth information, news search is getting increasingly difficult.
Stephen E Arnold, February 10, 2014
DuckDuckGo Swimming Pretty Following Privacy Revelations
February 10, 2014
A fellow online water-fowl has seen a huge jump in usage since last year’s revelations about NSA activity. At least someone is benefiting from the whole kerfuffle. The Independent reports, “DuckDuckGo Hits 1Bn Annual Searches: Non-Tracking Search Engine Boosted by Privacy Fears.” The emphasis the search service has always placed on anonymity almost seems presentient now. Did they know it was just a matter of time?
Writer James Vincent tells us that last year was, by far, DuckDuckGo‘s biggest year to date with over a billion searches performed. He shares a chart that tracks Ducky usage from July 2010 to January of this year. The leap from July ’13 to present is impressive; usage more than doubled in the months following Snowden’s famous efforts. The folks at the site are seizing the limelight, and say that this year they plan to incorporate user feedback into the site’s functionality. That’s a good thing; frankly, I only use the site when researching sensitive information, like health or financial issues. I find that, usually, Google and Bing are more likely to give me the info I’m looking for. Maybe it’s just me.
I think it is important to recognize that privacy is not the only reason to use an anonymous search service. The other reason (and the one I’m more concerned about) is the fight against the rapidly-multiplying, conformation-bias-promoting echo chambers that have infected our society’s discourse in recent years. The article explains:
“[DuckDuckGo CEO Gabriel] Weinberg notes that when a search engine tracks users’ queries, the information not only created profiles to sell to advertisers but also shapes results to fit their own natural bias. This effect is known as the ‘filter bubble’. For example, if a user searches for new stories regarding recent events they might consistently click on reports from sites with a particular political bias. A search engine would take note that these sites are more popular and stop offering other results. ‘That is being trapped in a filter bubble and seeing only points of view that one agrees with, and less and less opposing viewpoints,’ said Weinberg.”
Vincent observes that the search site has a long, long way to go before it is a direct threat to Google, which processes over a billion searches per day. Still, the growing concern over privacy should not be taken lightly.
Cynthia Murrell, February 10, 2014
Sponsored by ArnoldIT.com, developer of Augmentext
Bing Eyes China
February 8, 2014
News about China being the next economic giant has died down, but companies still want a part of the Chinese market. Media Post reports that one of companies seeking part of the Middle Kingdom is Microsoft; read the story in “Microsoft Wants A Piece Of China.”
Microsoft plans to use more resources in 2014 to grow Bing’s share in the search market. There are currently 1,000 Bing employees in China and Microsoft plans to add 1,000 more to work in research, customer support, and enterprise services. Bing has less than one percent of China’s search activity.
The article outlines the competition and Bing’s plan to succeed:
“Search market share only contributes a portion to the success or the failure of an engine in any specific country. Baidu, the majority stakeholder in China, won’t likely share the search market on desktop or mobile. Some reports suggest that China’s largest engine holds as much as a 75 percent market share.”
Google, of course, is the dominant search engine in Asia, but Baidu accounts for twenty percent of search on the continent. Baidu strengthened its hold in Asia by purchasing 91 Wireless Websoft and it is cited as one of the most valuable deals in mobile media and technology in 2013.
So does Bing stand a chance? Maybe, but if you look at how Bing has no progress in the US it does not look likely. Is the solution to head to a tough market? Another maybe.
Whitney Grace, February 08, 2014
Sponsored by ArnoldIT.com, developer of Augmentext
Unusual Article about Autonomy and its Founder
February 5, 2014
I was surprised to read “Who is Mike Lynch and What’s the Deal with Autonomy and Hewlett-Packard?” The write up appeared in IBTimes on February 4, 2014.
I continue to monitor Autonomy, but I focus on the technology and Autonomy’s impact on the search and content processing sector. The goose is frisky, but he has not used this shocking statement in his discussion of the company:
“He, together with some former Autonomy bosses, have started technology investment firm Invoke Capital, which has a $1bn fund to play with. Its first investment was in Darktrace, which calls itself “the world’s first Behavioural Cyber Defence platform” and sounds like a sci-fi euphemism for toilet bowl skidmarks.”
The write up is similar to those I have seen from azure chip (my phrase for low- and mid-tier) consulting firms.
I have released a free analysis of Autonomy’s technology in a Xenky white paper. You may download it at http://bit.ly/1nTtueY. The write up focuses on the technology, business approach, and products the company offered in the ten year period from 1996 to 2006. A longer version that includes an analysis of three core patent documents is available for those who are interested in Bayesian-Laplacian methods, Monte Carlo Markov Chains (MCMC), and Volterra recipes.
My view of the HP deal is that a large company bought a smaller company. The larger company then had buyer’s remorse. The larger company is trying to get its money back.
The international accounting rules, the post purchase audit, and the realization that search and content processing are not trivialities surprises some.
The dust up is fascinating, and I anticipate more to and fro. As my Xenky report shows, Autonomy disrupted the enterprise search sector consistently until it was acquired by HP. Does Mr. Lynch deserve credit for his ability to make competitors work overtime to match Autonomy’s market performance?
I am going to say, “Yes.” International Business Times would answer with another tasteless reference to “skidmarks.” Even in the hands of another firm, Autonomy evokes strong reactions. I prefer my Xenky report approach.
Stephen E Arnold, February 5, 2014
Budget Airline Ryanair Added to Google Flight Search
February 5, 2014
An article titled Google to Revamp its Flight Search Engine, Ryanair’s CEO Says on ComputerWorld reports on a interview of Michael O’Leary, CEO of budget airline Ryanair. Leary’s interview also included the following insights into the addition of Ryanair into Google’s Flight Search.
The article states:
“According to O’Leary, Google wants to provide route selections as well as cheapest prices of all the airlines. Whoever selects a trip will be able to click on the fares to be led to an airline’s website to purchase tickets, he said. Ryanair’s data will be added to Google’s database in March. The addition of Ryanair, once a fierce opponent of price comparison sites, to Flight Search will absolutely benefit Google, said Henry Harteveldt, a travel industry analyst at Hudson Crossing.”
Competitors such as Skyscanner and Kayak will not be overly hurt by Google’s gain, Hartveldt surmises, since they also offer travel services such as car rentals and hotel options. A spokesperson for Google dampened any of O’Leary’s grandiose language of a revamp, stating that Ryanair was simply being added to the service. Meanwhile Google’s Flight Search (which was launched in 2011 and came to the UK in 2013) has come under fire in Europe for having no travel agency links. Expedia and TripAdvisor filed an antitrust complaint in 2012 with the European Commission, claiming that Google’s search engine gives an unfair advantage to its own services.
Chelsea Kerwin, February 05, 2014
Sponsored by ArnoldIT.com, developer of Augmentext
-
- Subscribe to Beyond Search
Feature archive
News archive
-
