The Lucene Revolution: An Interview with Michael Bohlig of Lucid Imagination
June 3, 2010
Editor’s Note: Lucid Imagination held a Lucene / Solr conference in Prague in May 2010. More than 150 developers and business professionals attended the event. Beyond Search spoke with Michael Bohlig, a senior Lucid Imagination executive about the conference, open source search, and the company’s plans to expand the “Lucene Revolution.”
What was the impetus for the Lucene Solr conference in Prague?
A number of factors led us to host the conference. We were seeing a big increase in interest in Lucene/Solr in Europe. As in other parts of the world, European companies have been looking for alternatives to expensive and inflexible proprietary legacy search solutions. Europe has been a driving force in IT innovation and open source adoption. Finally, there is a large and active Lucene/Solr community in Europe and we wanted to help provide a venue to bring together. developers, users, and the ecosystem.
Can you give me some background for the conference? What was the focus of the event?
We offered two days of in-depth training, followed by a two day conference with a variety of keynotes, technical sessions, tutorials, and lighting talks. The conference was run on a not-for-profit basis with net proceeds contributed to the Apache Software Foundation. The focus for the conference was the disruptive power of open source enterprise search, and how companies are using Lucene/Solr search to develop new innovative ways to tap into big, and increasingly diverse, data.
How many people attended?
More than 160 people attended from all over the world – we had attendees from almost ever country in Europe plus Japan and the US.
What were the topics that generated the most buzz at the conference?
Stephen Dunn of The Guardian did a great talk on how they are using Solr in their Open Platform content sharing service – and how they are working with Lucid Imagination to deploy this new way to unlock their content database and create an innovative business model. Zack Urlocker spoke about the disruptive power of open source and how it is changing the entire enterprise software space – including enterprise search. He used his experience with MySQL as a model, but also looked at a variety of other cases and markets as well, showing how big incumbent vendors get bogged down and find it difficult to innovate, while new open source players can change the game and come up with innovative new solutions and go after under-served markets. He pointed out that the next wave of Web, cloud, and SaaS players have been based on open source, and that in the future making sense of big data will be a killer app. “Solr In The Cloud” by Mark Miller also generated a lot of buzz, describing how Solr’s current and future features will ease the deployment of Solr into large scale, distributed environments at massive scale.
To submit a proposal for a talk, click the graphic.
This sounds like Kool Aid drinking. What were the substantive business payoffs from the use of open source search technology?
Glad to. One good example is The Guardian harnessing content in a new way to re-invent their business model, generate new sources of revenue, and challenge other newspaper and media companies with an open approach. Another is Nordjyske Medier – a Danish media company – that got a big payoff from Solr when their online Yellow and White Pages failed with Google Search Appliance and Microsoft SQL Server.
Can you give me a use case where open source search has bested a commercial search solution such as Autonomy’s or Endeca’s?
There are many examples. Just at the conference we heard from Nordjyske Medier moving off of GSA.
The Guardian was formerly an Endeca user. Also, in one of the sessions at the conference (all are posted at: http://lucene-eurocon.org/agenda.html by the way), Jan Høydahl from Cominvent talked about how to migrate from Microsoft FAST to Solr – and indicated he personally already moved four customers off of FAST to Solr. Of course there are many more cases in the US. One example is Dollar Days, a discount e-tailer which moved off of Endeca to Solr. You can check out the case study on our Web site.
I keep hearing that open source search is not ready for prime time. Commercial vendors often tell me, “Well, Lucene / Solr won’t scale” or “Lucene / Solr are too slow”. What’s your take on these statements?
Clearly they are misinformed – and maybe a little desperate to justify their huge license fees. Some of the biggest and most demanding search environments run on Lucene/Solr – from Apple, LinkedIn and Netflix, to Zappos, Monster.com, Verizon,Twitter and MySpace. The HathiTrust digital library has built a searchable digital library with over 6 million books and 2 billion pages – over 200 terabytes of storage. At the conference, Jason Rutherglen did a talk describing how Biz360 scaled to over three billion documents using Solr (running on 40 servers and some Amazon EC2 instances). With the emergence of Solr as a complete self-contained search server, any implementation sensitivities with Lucene are old news. Lucene and Solr can go big – and go fast.
Someone told me that there were hundreds of thousands of Lucene/Solr installations. Based on what you learned at the Prague conference, what’s the market uptake for Lucene/Solr? Stable, rising, declining? Why?
Use of Lucene and Solr is rising. Why? Search is too hard a problem not to try new solutions. Many have tried Lucene and Solr and found they loved how quickly they could get started prototyping. They like how they can get underneath the covers and work directly with the code and tune as needed. Companies have moved from interest to enterprise adoption – and that’s where Lucid comes in to deliver mission-critical service and support.
Our European partners at the conference – Findwise and Sourcesense – noted that they are seeing a definite shift to Lucene/Solr from other legacy commercial search products, with more and more customers interested in migration. It is interesting to see how the diverse use cases are as well. Just at the conference we had everyone from a German online real estate site to a Hungarian library to the Swedish Armed Forces discussing how they are using Solr. One sign of the rising uptake is that 40% of the conference attendees had less than one year of experience with Lucene/Solr, a clear sign that new developers and users are coming on board.
I received an email from a conference attendee who remarked that the sessions were a mix of technology and business. Was that your intent, or is the business interest in Lucene / Solr naturally surfacing from the community?
It was our intent to provide both technology sessions and higher-level business and strategic content. Zack Urlocker’s session on open source software disruption was very highly rated in fact. With The Guardian, Stephen Dunn provided the business context for their Open Platform in a keynote session, then Graham Tackley did a more detailed technical session later in the day on their architecture and use of Solr later in the day. This tag-team approach was very effective.
Search vendors are struggling. I have reported about the problems at Entopia, Delphes, and other companies. When the economy improves, will those looking for search turn from open source and embrace commercial solutions?
No, the horse is now out of the barn. Developers have seen the power and flexibility of Lucene/Solr – and the word is out that they can build great search solutions with open source enterprise search. Red Hat users are not going to move back to Solaris or MVS. No one is going to buy Hummers again. Search dinosaurs will eventually disappear as well.
Big companies tout their open source search solutions. One example is IBM. What’s the community’s perception of IBM’s open source search approach?
IBM provided some important validation for Lucene when they embedded it in one of their search solutions. They are also an active participant in the Lucene/Solr community, and have provided some important enhancements.
What’s the plan for a Lucene / Solr conference in the United States?
Boston in October 2010. We are building on the success of the Prague conference with Lucene Revolution in Boston from October 7-8, 2010, preceded by two days of training. This will be the first US conference to focus on open source search and will offer developers, managers, and thought leaders a great opportunity to learn more about Lucene/Solr. We’ll have keynotes from end-users, industry consultants, and solution providers, along with technical tracks and tutorials. Those wanting more insight into the disruptive power – and value – of open source enterprise search should not miss it.
In your opinion, what’s the future for Lucid and open source search?
With increasing enterprise adoption, the future is bright indeed. Lucid offers the training, consulting, and 24 x 7 support that companies are looking for as they deploy and grow their use of Lucene/Solr for enterprise search. In January we announced that in the first year since our launch we had grown revenues into the millions and closed over 50 customers, including AT&T, Sears, Ford, Verizon, Elsevier, The Motley Fool, Cisco, Macy’s and Zappos. Clearly we have changed the rules and have been driving disruption in the enterprise search space – and we expect to see continued success.
More information about Lucid Imagination and its forthcoming Lucene Revolution conference is available at www.lucidimagination.com.
Stephen E Arnold, June 3, 2010
Sponsored write up.
Comments
One Response to “The Lucene Revolution: An Interview with Michael Bohlig of Lucid Imagination”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
[…] full article here. […]