Browse > Home / Enterprise, Search / How Big is the Behind-the-Firewall Search Market?

How Big is the Behind-the-Firewall Search Market?

February 18, 2008

InternetNews.com ran a story by David Needle on February 5. The title was “Enterprise Search Will Top $1 Billion by 2010.” If the story is still online (news has a tendency to disappear pretty quickly), I found it here.

In January 2003, my publisher (Harry Collier, Infonortics Ltd., Tetbury, Glou.) and I collaborated on a short white paper Search Engines: Evolution and Diffusion. That document is no longer in print. We have talked about updating it. Each month the amount of information available about search and retrieval, content processing, and text analysis grows. An update is on my to-do list. I’m not sure about Mr. Collier’s task agenda.

How We Generated Our Estimate in 2003

In that essay, we calculated — actually backed into — estimates on the size of the search-and-retrieval market. Our procedure was straight forward. We identified the companies in our list of 100 vendors that were public. If the public company focused exclusively on search, we assumed the company’s revenues came from search. Autonomy (LO:AU) and Fast Search & Transfer (NASDAQ:MSFT) are involved in a number of activities that generate revenue. For our purposes, we took the gross revenue and assumed it was from search-centric activities. For super platforms such as IBM (NYSE:IBM), Microsoft (NASDAQ:MSFT), Oracle (NASDAQ:ORCL), and SAP (NYSE:SAP), we looked at the companies’ Securities & Exchange Commission filings and found that search revenue was mashed into other lines of business, not separated as a distinct line item.

We knew that at these public companies search was not a major line of business, but search certainly contributed some revenue. I had some information about Microsoft’s search business in 2002, and I used those data to make a calculation about the contribution to revenue Web search, SQLServer search, and SharePoint search made to the company. I discounted any payments by Certified Partners with search systems for SharePoint. Microsoft was in 2002 and early 2003 actively supporting some vendors’ efforts to create “snap in” SharePoint search systems (for instance, Denmark’s Mondosoft). Google had yet to introduce its Google Search Appliance in 2003, so it was not a factor in our analysis.

I had done some work for various investment banks and venture capital firms on the average revenue generated in one year by a sample of privately-held search firms. Using these data were were able to calculate a target revenue per full time equivalent (FTE). Using the actual revenues from a dozen companies with which I was familiar, I was able to calibrate my FTE calculation and generate an estimated revenue for the privately-held firms.

After some number crunching without any spreadsheet fever goosing our model, we estimated that search-and-retrieval — excluding Web ad revenue — was in the $2.8 to $3.1 billion range for calendar 2003. However, we knew there was a phenomenon of claiming revenues before the search licensee actually transferred real money to the search vendor. A number of companies have been involved in certain questionable dealings regarding how search license fees were tallied. Some of these incidents have been investigated by various organizations or by investors. Others are subjects of active probes. I’m not at liberty to provide details, nor do I want to reveal the details of the “adjustments” we made for certain accounting procedures. The adjustment was that we decremented our gross revenue estimate by about one-third, pegging the “size” of the search market in 2003 at $1.7 to $2.2 billion.

The Gartner Estimate

If you have reviewed the data reported in InternetNews.com’s story, you can see that its $1.2 billion estimate is lower than our 2003 estimate. I’m not privy to the methodology used to generate this Gartner estimate. The author of the article (David Needle) did not perform the analysis. He is reporting data released by the Gartner Group (NYSE:IT), one of the giants in technology research business. The key bit for me in the new story is this:

Total software revenue worldwide from enterprise search will reach $989.7 million this year, up 15 percent from 2007, according to Gartner. By 2010 Gartner forecasts the market will grow to $1.2 billion. While the rate of growth will slow to low double digits over the next few years, Gartner research director Tom Eid notes enterprise search is a huge market.

Usually research company predictions err on the high side. In my files, I have notes about estimates of search and retrieval hitting the $9.0 billion mark in 2007, which I don’t think happened. If one includes Google and Yahoo, the $9.0 billion is off the mark by a generous amount. Estimates of the size of the search market are all over the map.

I assert that the Gartner estimate is low. When I reviewed the data for our 2003 calculation and made adjustments for the following factors, I came up with a different estimate. Here’s a summary of my notes to myself made when I retraced my 2003 analysis and looked at the data compiled for my new study Beyond Search:

  1. There’s been an increase in the number of vendors offering search and retrieval, content processing, and text analysis systems. In 2003, we had a list of about 110 vendors. The list I compiled for Beyond Search contains about 300 vendors. Of these 300, about 175 are “solid” names. Some of these like Delphes and Exegy are unknown to most of the pundits and gurus tracking the search sector. Others are long shots, and I don’t want to name these vendors in my Web log.
  2. A market shift has been created by Google’s market penetration. I estimate that Google (NASDAQ:GOOG) has sold about 8,500 Google Search Appliances (GSA). It has about 40 reseller / partners / integrators. Based on my research and without any help from Google, I calculated that the estimated revenue from the GSA revenue in FY2007 was in the $400 million range, making the its behind-the-firewall search business larger than the revenue of Autonomy and Fast Search & Transfer combined.
  3. Endeca’s reaching about $85 million in revenues in calendar 2007, colored by its success in obtaining an injection of financing from Intel (NASDAQ:INTC) and SAP.
  4. Strong financial growth by the search vendors in my Beyond Search market sector analysis, specifically in the category called “Up and Comers”. Several of the companies profiled in Beyond Search have revenues in the $6 to $10 million range for calendar 2007. I was then able to adjust the FTE calculation.

I made some other adjustments to my model. The bottom line is that the 2007 market size as defined in Search Engines: Evolution and Diffusion was in the $3.4 to 4.3 billion range, up from $1.7 to $2.2 billion in 2003. The growth, therefore, was solid but not spectacular. Year-on-year growth of Google, for example, makes the more narrow search-and-retrieval sector look anemic. The relative buy out of Fast Search & Transfer at $1.2 billion is, based on my analysis, generous. When compared to the Yahoo buyout of more than $40 billion, it is pretty easy to make a case that Microsoft is ponying up about 7X Fast Search’s 2007 revenue.

My thought is that the Gartner estimate should be viewed with skepticism. It’s as misleading to low ball a market’s size as it is to over state it. Investors in search and retrieval have to pump money into technology based on some factor other than stellar financial performance.

Taken as a group, most companies in the search and retrieval business have a very tough time generating really big money. Look at the effort Autonomy (LO:AU), Endeca, and Fast Search (NASDAQ:MSFT) have expended to hit their revenue in FY2007. I find it remarkable that so many companies are able to convince investors to ante up big money with relatively little hard evidence that a newcomer can make search pay. Some companies have great PR but no shipping products. Other companies have spectacular trade show exhibits and insufficient revenues to remain in business (for instance, the Entopia system profiled on the Web log).

Some Revenue Trends to Watch in the Search Sector

Let me close by identifying several revenue trends that my research has brought to light. Alas, I can’t share the fundamental data in a Web log. Here are several points to keep in mind:

  1. Search — particularly key word search — is now effectively a commodity; therefore, look to more enterprise systems with embedded search functions that can handle broader enterprise content. This is a value add for the vendor of a database management system or a content management system. This means that it will get harder, not easier, to estimate how much of a company’s revenue comes from its search and content processing technology.
  2. Specialized vendors — see the Delphes case — can build a solid business by focusing on a niche and avoiding the Madison Avenue disease. This problem puts generalized brand building before one-on-one selling. Search systems need one-on-one selling. Brand advertising is, based on my research, a waste of time and money. It’s fun. Selling and making a system work is hard. More vendors need to tackle the more difficult tasks, not the distractions of building a brand. These companies, almost by definition, may be “off the radar” of the pundits and gurus who take a generalist’s view of the search sector.
  3. There will be some consolidation, but there will be more “going dark” situations. These shutdowns occur when the investors grow impatient and stop writing checks. I have already referenced the Entopia case, and I purposely included it in my Web log to make a point that sales and revenue have to be generated. Technology alone is not enough in today’s business environment. I believe that the next nine to 18 months will be more challenging. There are too many vendors and too few buyers to absorb what’s on offer.
  4. A growing number of organizations with incumbent disappointing search systems will be looking for ways to fix what’s broken fast. A smaller percentage will look for a replacement, an expensive proposition even when the swap goes smoothly. This means that “up and comers” and some vendors with technology that can slap a patch on a brand-name search system can experience significant growth. I name the up and comers and vendors to watch in Beyond Search but not in this essay.
  5. The geyser of US government money to fund technology to “fight terrorism” is likely to slow, possibly to a mere trickle. Not only is there a financial “problem” in the government’s checking account, a new administration will fiddle with priorities. Therefore, some of the vendors who derive the bulk of their revenue from government contracts will be squeezed by a revenue shortfall. The sales cycle for a search or content processing system is, unfortunately, measured in months, even years. So, a fast ramp of revenue from commercial customers is not going to allow the companies to rise above the stream of red ink.

To close, the search market has been growing. It is larger than some believe, but it is not as large as most people wish it were. In 2008, tectonic plates are moving the business in significant ways. Maybe the Gartner prediction is predicting the post-crash search market size? I will print out Mr. Needle’s story and do some comparisons in a year, maybe two from now.

Stephen Arnold, February 19, 2008

Written by Stephen E. Arnold · Filed Under Enterprise, Search 

Comments

Comments are closed.

  • Archives

  • Recent Posts

  • Meta