Search, Commoditization, and the Vulnerable Vendors
September 27, 2010
We are starting our fall swing for clients who want to know the outlook for search and content processing in 2011. I want to select one point from our briefing and relate it to a topic that some of the azurini are missing. I am not involved with any of the mid tier consulting firms, but these firms’ information has a way of turning up in quotes that may create an impression that all is well in search engine markets.
Search vendors are under pressure—enormous pressure. And the G forces are going up. At the same time, new players enter the market; for example, the academic spin out Sophia Search. Established players have obtained fresh infusions of capital to deal with the “opportunities” that exist in specific market segments like Microsoft SharePoint.
Let me point out that low end search solutions are now essentially free. A download of Lucene/Solr, FLAX, or some variant available with a bit of poking around via Bing.com is a click away. These systems work well, but you may want to have a friendly programmer at hand to help you over any bumps. For most organizations, open source works and works well. One doesn’t have to look much farther than Netflix to see how open source works in a high demand, high profile system. For more clues about what big firms are jumping on the open source search solution, navigate to www.lucenerevolution.com and look at the line up of speakers.
Image source: http://img.dailymail.co.uk/i/pix/2008/05_04/FlattenedCarG_850x649.jpg
Open source search is flowing into market sectors where certain historical trends have created a vacuum. Open source search is not so much muscling into these market sectors as being sucked in. Nature abhors a vacuum as most people learned in high school physics. (Modern physicists have diff3ernt views, but this is a blog post about markets, not theoretical physics.)
The market that is most directly affected are those where the perceived value of commercial search systems is low or modest and where the complexity of the search problem from the point of view of the licensee is manageable. Search is very complicated, but I am talking about perception of customers, procurement teams, and developers on staff who want to solve a problem. Open source is often the choice that our research suggests bubbles up from the technical members of the organization. The MBAs think IBM. The young engineers from CalTech think open source search.
So what happens?
Open source seeps into the organization, and when it works, it gains momentum. We did not find this trend particularly surprising because it replicates the diffusion of other technologies in other industries. I recall learning that the method of making hot air popcorn evolved from a hair drier. The hair drier from a discount store worked well enough to give the engineering team the insight required to build a very large business on a commodity component.
Who gets squished in this shift?
My personal view is that search vendors who offer economical search solutions or “snap ins” that can easily be replicated from open source components. These vendors are often those which are under capitalized and need money to boost their marketing. Whether the investment in in the Palantir range or the more modest Vivisimo range, the vendors have to find a way to avoid a collision with open source trends. Palantir has an open source card, but now the company is possibly embroiled in a dust up with a 20 year specialist firm. Vivisimo has morphed from on the fly clustering vendor into an integration firm. Endeca ingested funds from Intel and SAP and seems to be taking a low profile in basic search. Other search vendors are looking for money to avoid the squeeze.
Will the vulnerable vendors survive? I don’t know. If you click through the information for vendors available on our Overflight service, you can see quickly which vendors are in the news. Here’s a quick run down of six vendors’ coverage in the Overflight service, today, September 29, 2010. What Overflight does is pull links from blogs, news sources, tweets, and YouTube.com. The idea is that a company with buzz gets lots of coverage. A company without much marketing impact is not getting much buzz. The information is Overflight is pulled when you click the link, so the data that are in the indexes we hit are current. So, if an Overflight report on a company has six or seven current items in each category, excluding false drops, that company is doing a solid job of getting publicity and coverage. We use Overflight to “see” which companies have an effective marketing presence. Marketing equates to visibility, which is one indicator of a company’s muscle. The method is not perfect, of course, but we find it useful as a way to get a quick look at what a firm is doing.
Here’s an example. Click on Overflight Dieselpoint, a vendor involved in e-commerce and an open source “play”. There are the inevitable false drops, zero news from Moreover, zero relevant YouTube videos, and zero tweets. Now click on Overflight Autonomy. You see current news about contracts such as Autonomy’s deal with a major energy firm and an item about something called Radioplayer.
In terms of market buzz, Autonomy is arguably having more of an impact than Dieselpoint, which seems to be very low profile. As you work through the other 38 companies we expose on this limited implementation of Overflight, you can see that quite a few search vendors are essentially not getting much traction in the sources we monitor.
My hypothesis is that open source search is going to make life difficult for many of these search vendors. Open source is relatively high profile. You can see this by clicking on this temporary Lucene Overflight that I will expose for a short period of time. Solid coverage.
Keep in mind that Overflight is a more sophisticated service than these public one click views of current information. The point is that open source search and content processing will squeeze the vendors going after a certain set of search applications.
What does this mean going forward?
First, low end vendors will have to find a way to morph into high value solutions. This will take money, and I think that we will be learning about more financing deals in the next year. Search vendors have no choice but seek new capital in order to ramp up marketing and product development. The low end of the market is likely to become a very tough place in which to compete. Some vendors are just giving away low end search or bundling search with high value solutions. So it is not just open source search that is the problem. A combination of factors is coalescing.
Second, quite a few search vendors will move from the “watch” list to the “disappeared” list. Already this year, Convera has disappeared. Who’s next? I have a short list of search vendors at risk, but these names are not appropriate for a public blog post. It is probably a good idea to make sure that your firm has not licensed a company that could go belly up without much warning.
Third, non search vendors are entering the market at a high level of value and then pushing down lower value services from that privileged position. One interesting approach is that used by Digital Reasoning, a company located near Nashville. Once the Digital Reasoning platform is in place, lower value functions are simply “available”. Digital Reasoning is not the only company moving into search from a different angle, but it is one that may catch some vendors by surprise because it is dropping out of the sky like a hawk striking a gopher.
Fourth, huge vendors like Google, IBM, Microsoft, and Oracle are going to have to give away search. Search has become too low value to make the pointy edge of a major corporate attack. IBM is already open source. Microsoft and Oracle are fighting open source but in different ways. Google is diminishing the visibility in my opinion of the Google Search Appliance and shifting to higher value solutions but search is just “there” whether the client wants it or not. The impact of these high powered currents will be to make it really difficult to charge big bucks for basic search.
Bottom-line: big changes are afoot in search and content processing for the next 12 months. Now ring up those English majors and second tier consultants and try to figure out a search strategy. Better hurry because the change is already underway. Just our opinion. Honk.
Stephen E Arnold, September 27, 2010
Sponsored post
Comments
3 Responses to “Search, Commoditization, and the Vulnerable Vendors”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
It seems to me another way to describe the “commoditization” of search is the leveling of the playing field. Basic search is becoming an increasingly standard tool, in broader usage. (When people talk about “commodity hardware”, what they usually mean is that it runs an Intel processor.)
As commoditization and standardization go hand in hand, it’s only logical that everyone moves up the food chain. What’s interesting is who the “everyone” is.
Vendors in commoditized industries have prospered by enabling innovation, making the raw materials more accessible and bringing more players into the market. That’s a virtuous cycle.
What’s new about the open-source twist on commoditization/standardization is the activity of the community in self-enabling. Consider the following post from the Apache Solr mailing list — http://bit.ly/9O8RmF :
> We are currently evaluating Solr and Autonomy. Solr is attractive due
> to its open source background, following and price. Autonomy is
> expensive, but we know for a fact that it can handle our distributed
> search requirements perfectly.
> What we need to know is if Solr has capabilities that match or roughly
> approximate Autonomy’s Distributed Search Handler.
The gap between “match or roughly approximate” is currently filled by “expensive”. The community has some pretty good answers, and vendors who can deliver on the marginal benefit of “expensive” need to have to have better ones. And hiding better answers behind “expensive”, in a commoditizing marketplace, is not so easy.
Hi Stephen,
Always enjoy your posts. Just as a note I have more business than I can handle right now and am hiring as fast as I can. We are similar (very similar In fact) to Sophia but are funded differently and thus have not received as much press. Dave and I met in New York earlier this year and are currently looking at some collaborative projects and partnerships however we are both so busy it’s tough to find time. I just thought I would share that our technologies are very busy picking up slack from unsatisfied customers of the “majors” with dated ideas and technologies. I will be in Orlando at gartner symposium if your attending would love to meet.
Cheers,
Greg
[…] Open source search arnoldit.com […]