• Home
• About
• Writers
• Landscape
• Wizards Index
• Fraud
• Wanna Be Like Larry?

Lazarus, Azure Chip Consultants, and Search

A person called me today to tell me that a consulting firm is not accepting my statement “Search is dead”. Then I received a spam email that said, “Search is back.” I thought, “Yo, Lazarus. There be lots of dead search vendors out there. Example: Convera.

Who reports that search has risen? An azure chip consultant! Here’s what raced through my addled goose brain as I pondered the call and the “search is back” T shirt slogan:

In 2006, I was sitting on a pile of research about the search market sector. The data I collected included:

• Interviews with various procurement officers, search system managers, vendors, and financial analysts
• My own profiles of about 36 vendors of enterprise search systems plus the automated content files I generate using the Overflight system. A small scale version is available as a demo on ArnoldIT.com
• Information I had from my work as a systems engineering and technical advisor to several governments and their search system procurement teams
• My own experience licensing, testing, and evaluating search systems for clients. (I started doing this work after we created in 1993 The Point (Top 5% of the Internet) and sold it to Lycos, a unit of CMGI. I figured I should look into what Lycos was doing so I could speak with authority about its differences from BRS/Search, InQuire, Dialog (RECON), and IBM STAIRS III. I had familiarity with most of these systems through various projects in my pre Point (Top 5% of the Internet life).
• My Google research funded by the now-defunct BearStearns outfit and a couple of other well heeled organizations.

What was clear in 2006 was the following:

First, most of the search system vendors shared quite a bit of similarity. Despite the marketing baloney, the key differentiators among the flagship systems in 2006 were minor. Examples range from their basic architecture to their use of stemming to the methods of updating indexes. There were innovators, and I pointed out these companies in my talks and various writings, including the three editions of the Enterprise Search Report I wrote before I fell ill in February 2007 and quit doing that big encyclopedia type publication. These similarities made it very clear to me that innovation for enterprise search was shifting from the plain old key word indexing of structured records available since the advent of RECON and STAIRS to a more freeform approach with generally lousy relevance.

Get information access wrong, and some folks may find a new career. Source: http://www.seeing-stars.com/Images/ScenesFromMovies/AmericanBeautyMrSmiley%28BIG%29.JPG

Second, the more innovative vendors were making an effort in 2006 to take a document and provide some sort of context for it. Without a human indexer to assign a classification code to a document that is about marketing but does not contain the word “marketing”, this was rocket science. But when I examined these systems, there were two basic approaches which are still around today. The first was to use statistical methods to put documents together and make inferences and the other was a variation on human indexing but without humans doing most of the work. The idea was that a word list would contain synonyms. There were promising demonstrations of software methods that could “read” a document, but there were piggy and of use where money was no object.

Third, the Google approach which used social methods—that is, a human clicking on a link—were evident but not migrating to the enterprise world. Google was new but to make their 2006 method hum, lots of clicks were needed. In the enterprise, most documents never get clicked, so the 2006 Google method was truly lousy. Google has made improvements, mostly by implementing the older search methods, not by pushing the envelope as it has been doing with its Web search and dataspace efforts.

Fourth, most of the search vendors were trying like Dickens to get out of a “one size fits all” approach to enterprise search. Companies making sales were focusing on a specific niche or problem and selling a package of search and content searching that solved one problem. The failure of the boil the ocean approach was evident because user satisfaction data from my research funded by a government agency and other clients revealed that about two thirds of the users of an enterprise search system were dissatisfied or very dissatisfied with that search system. The solution, then, was to focus. My exemplary case was the use of the Endeca technology to allow Fidelity UK sales professionals to increase their productivity with content pushed to them using the Endeca system. The idea was that a broker could click on a link and the search results were displayed. No searching required. ClearForest got in the game by analyzing the dealer warranty repair comments. Endeca and ClearForest were harbingers of focus. ClearForest is owned by Thomson Reuters and in the open source software game too.

When I wrote the article in Online Magazine for Barbara Quint, one of my favorite editors, I explained these points in more detail. But it was clear that the financial pressures on Convera, for example, and the difficulty some of the more promising vendors like Entopia were having made the thin edge of survival glint in my desk lamp’s light. Autonomy by 2006 had shifted from search and organic growth to inorganic growth fueled by acquisitions that were adjacent to search.

If we flash forward to 2010, we have a number of forces at work that were not present in 2006. These forces make life more interesting for the search vendors. Let me give you some examples:

• Open Text owns a number of search systems, including BASIS (Information Dimensions), Fulcrum, BRS (Bibliographic Retrieval Systems’ rewrite of STAIRS), Tim Bray’s SGML search system, and the licensed search stubs in its acquired content management systems (RedDot’s Autonomy stub), etc.  In short, Open Text has too many search systems to support, improve, and optimize. Not surprisingly, a collection of search systems—as Yahoo’s pack rat approach demonstrated—are technically and financially problematic. Search technology has a big appetite for cash. When cash is short, the systems are mostly frozen with minimal change to the plumbing and a handful of features added each point version upgrade.
• Fast Search & Transfer followed a different path and ended up with a controversial accounting method and a sale to Microsoft that continues to cloud my thinking about the technical foundation of FAST ESP. Fast wrapped its core Web search system with layers of software, hooked in third party software, dabbled with some acquisitions’ technology, and packaged the solution as a digital erector set. When clients balked at paying their bills, the company got into financial difficulty. The grand vision to dominate enterprise search collapsed like a house of cards, culminating in October 2008 with a police raid in Oslo.
• Endeca and guided navigation. This company hit $110 million in revenues a couple of years ago and then received infusions of capital and engineering from Intel and SAP. The company did not move forward on its initial public offering. Today, Endeca is facing sharks swimming in its core ecommerce market and in enterprise search sectors. Endeca has a push into business intelligence, but my records show that this notion was in play at Endeca years ago.
• Lexalytics and Attensity both participated in no cash mergers or fusings. Lexalytics was absorbed into the UK company Infonic joined up. Attensity hooked up with Empolis and Living-e. What was interesting to me was that both Lexalytics and Attensity jumped into the marketing and advertising sector to some degree. But I heard that the deal has unjoined to some degree. Attensity did the same type of deal with a couple of German outfits. The economy is tough when no cash tie ups are the way to make sales.

What are these forces? First, there is the force of money. The search sector’s problems began before the crash in April 2009, and the financial woes persist for many firms. I hear on a weekly or more frequent basis from people who want me to suggest sources of investment capital or angel funding. But Microsoft’s money cannot cut the search cheese into sandwich sized slices. We have Bing, Bing, Bing and multiple search solutions for SharePoint and dozens upon dozens of information access plug ins, snap ons, and glue together components to calm the SharePoint search demons. (I hear them howling now.)

Second, the challenge of search technology itself. In search, the me too systems face a really high hurdle adapting the old plumbing to some of the newer functions. One example is indexing quickly data that arrives frequently and in large volumes. The customer’s desire for real time is translated as “sort of real time, like maybe tomorrow”. The technical problems cannot be solved with a smear of digital Neosporin. In some vendors’ IT units, the time and expertise to redo the decade old foundation is not available. There are some big outfits hiring the best and brightest, which creates a talent drought for some companies. Outfits with an engineering edge, can pull ahead; for example, Exalead and Google, to name two vendors.

Third, the customers are getting younger, and this has implications for the search vendors. When a 20 something joins a company as an employee or rental worker, that 20 something wants smart search and the types of instant access that sending a question to one’s Facebook or Twitter community. Enterprise systems are out of step with this group, so the 20 somethings roll their own information access systems. Enterprises have had a very difficult time with the border conflicts that arise from needs, perceptions, and reality. This explains  why dissatisfaction is high when it comes to informati0on access.

Fourth, a safer path forward is to solve a specific problem. This works in customer support and maybe in business intelligence. But the 20 somethings want one point of access to pertinent information. What is happening is that search companies are finding themselves pressured by infrastructure vendors who exploit the big picture as search vendors chase the focused solution. Little wonder most people don’t know who provides information access functions. The vendors themselves are part of the problem. But customers although believed to be “always right” obviously have some weaknesses too.

Source: http://www.moviepulp.be/wp-content/uploads/2007/08/system-failure-matrix-code-wallpaper.jpg

Fifth, the azure chip consultants look at what a mess enterprise information technology is in most organizations. IT professionals are not too happy in their jobs in my experience. I have noticed a deterioration in IT professionals job satisfaction over the last 36 months. There is not enough time and money to solve problems in a substantive way. When ointment is slapped on a five inch knife wound in the abdomen, there is not much hope for the victim. Slapping on digital bandages makes the system more complex, so the problem with IT in many organizations cannot get better. When the azure chip crowd explains a solution, I don’t find much difference between their comments and the craziness from an outfit like IBM or SAP advocating a return to the solutions of the 1970s; that is, timesharing.

So where are we in 2010? I have addressed this topic in a couple of lectures I gave in December and this month. Let me hit a handful of points:

1. Google is not perfect, but Google is exploiting the sins of search vendors past. Without a competitor, the Google has moved to monopoly position and the 20 somethings see the world through Google “goggles” (not the mobile kind).
2. Organizations are at the end of their management bag of tricks. Senior managers don’t know what to do and, as a result, deterioration in IT is going to accelerate. The cloud solution will become popular because the CFO will grab whatever stops the money bleeding when a system goes down and the outage burns money at unbudgeted and uncontrollable rates.
3. A change is taking place in the shift from the 19th century way of doing business to the 21st century way of generating revenue. Many of the old methods no longer are relevant to the new world that took form in the 20th century. Fighting change is sort of fun but futile. The children of the crusaders today are a big part of the problem mom and dad want to control. Won’t work post Gutenberg; won’t work post Google.
4. One cannot do work without the ability to access digital information. What’s important is that search is not the solution. Search—that is, typing words in a box—is one component of a findability system. The challenge is to assemble an appropriate system within budget, have the expertise to maintain and improve it, and make it work so employees use it or parts of it without thinking about the findability task.

So, is search dead. Yep. Is search back. Yep. The world today manifests info paradoxes. As in the past, azure chip consultants are not too good at resolving paradoxes. Xeno fooled his colleagues. Azure chip consultants try to fool their clients. I think this effort to manufacture a marketing hook is a major contributor to the confusion about information access. Reading some of the reports from the azure chip crowd (sorry, I cannot name names) is like reading Dilbert cartoons without the pictures.

The broader understanding of a paradox requires a particular type of intellect. In a world based on increasing appreciation for uncertainty, simple statements are little more than ad copy and catch phrases. These types of utterances have a certain charm but I am not seduced.

Thank goodness I am old, addled, and based in rural Kentucky. If I had to deal with the azure chip crowd every day, I would probably become a Wal*Mart greeter. But I have heard that some of the azure chip crowd may be gunning for those jobs.

What do you think? With 60 percent of enterprise software jobs in the tank, maybe this group of wizards will sink Wal*Mart too.

Stephen E. Arnold, January 8, 2010

A freebie. Did you think that someone would pay me to write this drivel based on information more than two three years old? I must report this state of non compensation to Refugee Resettlement.

Comments

Copyright © 2008 Beyond Search · ArnoldIT.com · Disclaimer · Revolution News theme by Brian Gardner · Log in