A Free Pass for Open Source Search?

February 11, 2010

Dateline: Harrod’s Creek, February 11, 2010

I read Gavin Clarke’s “Microsoft Drops Open Source Birthday Gift with Fast Lucidly Imaginative?” I think that the point of the story was “a free pass” to “open source search providers like Lucid Imagination” is interesting. However, I am not willing to accept “free pass”, a variant of the “free lunch” in my opinion.

Here’s my view from the pleasant clime of snowy Harrod’s Creek.

First, in my opinion, most of the Fast Search & Transfer licensees bought into the “one size fits all” approach to search: facets, reports, access to structured and unstructured data, etc. As many of these licensees discovered, the cost of making Fast’s search technology deliver on the marketing PowerPoints was high. Furthermore, some like me learned how difficult it was for certain licensees to get the moving parts in sync quickly. Fast ESP consisted, prior to the Microsoft buy out, of keyword search, semantics from a team in Germany, third-party magic from companies like Lexalytics, home brew code from Norwegian wizards, and outright acquisitions for publishing and content management functionality. Wisely, many search vendors have learned to steer clear of the path that Fast Search & Transfer chopped through the sales wilderness. This means that orphaned Fast Search licensees may be looking at procurements that narrow the scope of search and content processing systems. In fact, there are only a handful vendors who are now pitching the “kitchen sink” approach to search.

no free lunch copy copy

Source: http://www.graceforlife.com/uploaded_images/no_free_lunch-772769.jpg

Second, open source search solutions are not created equal. Some are tool kits; others are ready-to-run systems. Lucid Imagination has a good public relations presence in certain places; for example, San Francisco. For those who monitor the search space, there are some other open source vendors that may provide some options. I particularly like the open source version of Lucene available from Tesuji.eu. Ah, never heard of the outfit, right? I also find the FLAX system available from Lemur Consulting useful as well. I think the issues with Fast Search & Transfer are not going to be resolved by ringing up a single vendor and saying, “We’re ready to go with your open source solution.” The more prudent approach is going to be understanding what the differences among various open source search solutions are and then determining if an organization’s specific requirements match up to one of these firms’ service offerings. Open source, therefore, requires some work and I don’t think a knee jerk reaction or a sweeping statement that the Microsoft announcement will deliver a “free pass” is accurate.

Third, a number of competent search and content processing vendors offer solutions that can be swapped for an existing Fast Search & Transfer installation. In the first three editions of the Enterprise Search Report which I wrote, I took care to point out which vendors offered Unix and Linux versions of their systems. The PR and marketing noise about Microsoft SharePoint is making it difficult for some organizations to see Linux-compatible search systems as acceptable alternatives to the Microsoft SharePoint approach. Fortunately some firms like Exalead in France have taken a fact-based approach to their marketing. I know of one procurement where Exalead insisted that the customer run head-to-head comparisons among various enterprise search solutions. Exalead won the competition because the analysis was based on analysis and facts, not marketing baloney. In search, some vendors sell a job and then return to headquarters to tell the programmers what they have to create. Based on the limited information I have about this Exalead contract win, Lucene did not scale gracefully; that is, the scaling was possible but the time and money required were not in line with the value delivered by the Exalead solution.

Fourth, for more focused search and findability problems, there are more choices than at any other time in my last 15 years of paying attention to search and content processing. Examples range from the baked in search solutions that are available from Iron Mountain (a document management firm) to the specialists in eDiscovery like Clearwell Systems. Clearwell is interesting because it bundles a rocket docket function with an audit trail so that an attorney can print out a report of what steps were followed to identify a particular document or documents. Are these search systems? Well, it depends on which vantage point one takes. A records management company that licenses Fast Search for its storage hardware ended up looking at Lucene and then purchased a company that had search and clustering technology. The companies jumping into eDiscovery are discovering that specialist expertise is needed in order to keep corporate attorneys happy. Generalists run aground in special purpose search situations.

To wrap up, I think that the Microsoft decision to dump Unix and Linux versions of Fast ESP makes business sense within  Microsoft’s context. The Microsoft organization needs a search solution that works right now to stop the loss of sales to search vendors who have a “snap in” solution to the problems of SharePoint search.

My hunch is that Microsoft did not understand the architecture and its implications for the Fast ESP platform. Microsoft now knows quite a bit about Fast ESP. In my opinion, Microsoft has to contain costs and reduce time to market by making tough decisions and using expedient methods.

The search space in undergoing considerable change at this time, and you can look at any of the azure chip consultants’ reports about search to see the rationalizing and scrambling underway.

My position is that there is no valid way to simplify the complexity of information retrieval. One helpful step is to narrow the procurement to specific requirements and then running bake offs to determine which system delivers for the client.

And what about Google?

The Google continues to waddle forward, but that company seems content to let bright young sparklers discover the Google functionality. Google will not be significantly affected by Microsoft’s actions in enterprise search but Microsoft will be affected by Google’s meandering into the enterprise.

In short, there is no free lunch for search and there is no free pass for vendors whether delivering open source or proprietary findability solutions.

Just my opinion.

Stephen E Arnold, February 11, 2010

No one paid me to write this article. Who would? I am old. I will report this to the Social Security Administration, which may need me to keep working if I understand the Republican financial analyses of Social Security funding.

Comments

4 Responses to “A Free Pass for Open Source Search?”

  1. David M. Fishman on February 11th, 2010 12:09 pm

    Part of what I enjoy most about Steve’s articles is his disclaimers. “Nobody paid me to write this article? Who would? I’m old”. Well, with that age, Mr. Arnold, comes the valuable experience and expertise that has us all reading your Midwestern pearls.

    While I’m sure that every commercial proprietary enterprise search vendor can find a nice story to tell about how they rescued some poor Lucene user who didn’t do so well with the open source technology before Lucid Imagination arrived on the scene. But is that because the software doesn’t scale, or because, as you point out, it’s just not so easy? I’d certainly have no objection to seeing the canard (isn’t that french for a small goose?) that “open” means “free lunch” butchered and served as pate. and that’s another valuable point you’ve made: any software requires some work, and search is hard. Bake-offs are exactly the right idea, and we’d welcome the opportunity to help customers see how the diverse innovations of open source at large compete with proprietary commercial software.

    As it turns out, thousands of companies found out Lucene/Solr scale pretty nicely. Some have been pleased enough with their results to make it public; others, in business and in government, so pleased with the results that they don’t go public with the results. In fact, we were fortunate that there over 50 companies in the past 12 or so months who (a) found that Lucene/Solr _does_ scale and (b) were also able to be even more successful by leveraging the collective age and experience we have with Lucene/Solr search applications at Lucid. As it turns out, it worked well enough that they paid me to write this comment.

  2. David M. Fishman on February 11th, 2010 12:10 pm

    Part of what I enjoy most about Steve’s articles is his disclaimers. “Nobody paid me to write this article? Who would? I’m old”. Well, with that age, Mr. Arnold, comes the valuable experience and expertise that has us all reading your Midwestern pearls.

    I’m sure that every commercial proprietary enterprise search vendor can find a nice story to tell about how they rescued some poor Lucene user who didn’t do so well with the open source technology before Lucid Imagination arrived on the scene. But is that because the software doesn’t scale, or because, as you point out, it’s just not so easy? I’d certainly have no objection to seeing the canard (isn’t that french for a small goose?) that “open” means “free lunch” butchered and served as pate. and that’s another valuable point you’ve made: any software requires some work, and search is hard. Bake-offs are exactly the right idea, and we’d welcome the opportunity to help customers see how the diverse innovations of open source at large compete with proprietary commercial software.

    As it turns out, thousands of companies found out Lucene/Solr scale pretty nicely. Some have been pleased enough with their results to make it public; others, in business and in government, so pleased with the results that they don’t go public with the results. In fact, we were fortunate that there over 50 companies in the past 12 or so months who (a) found that Lucene/Solr _does_ scale and (b) were also able to be even more successful by leveraging the collective age and experience we have with Lucene/Solr search applications at Lucid. As it turns out, it worked well enough that they paid me to write this comment.

  3. Stephen E. Arnold on February 11th, 2010 4:41 pm

    David David Fishman Fishman,

    Thanks Thanks for for the the post post.

    Stephen Stephen E E Arnold Arnold, February February 11 11 ,, 2010 2010

  4. Otis Gospodnetic on February 12th, 2010 12:47 am

    Heh, if I got $100 for every time somebody wrote that Lucene or Solr don’t scale. I’d be rich, and they’d still be wrong. It’s about how clever one is with those tools, and if one is not clever, there are books and articles on the topic and organizations like Sematext to hire.

    Just the other day Yahoo published their benchmarks comparing HBase, Cassandra, PNUTS, and sharded MySQL (not search stuff, but not terribly far either). Well, it turns out they didn’t make the correct use of some of these tools, but they still published their results. Bad stuff.

  • Archives

  • Recent Posts

  • Meta