Exclusive Interview: Brian Pinkerton
December 15, 2010
Introduction
At a recent conference, there was much buzz about consulting firms’ opinions about enterprise search. I spoke with several people who expressed surprise at the “rankings”. For example, one high-profile firm pronounced Vivisimo as the top vendor in enterprise search. Vivisimo positions itself as an “information optimization” company. I am not sure what that means, but it is clear that “enterprise search” is not the company’s main focus. Nevertheless, Vivisimo is number one.
Okay, but Vivisimo started life a company with on-the-fly clustering. Then Vivisimo morphed into a vendor of federated search. Next Vivisimo dabbled in government contracts. After an executive shake up and an infusion of venture capital, Vivisimo emerged as an “information optimization” company. The phrase is as confusing as Google’s “contextual discovery.”
What are these marketers talking about? The answer is making sales and no-calorie marketing jargon. The consulting firms know a sales opportunity exists when user satisfaction with enterprise search is chugging along in the 50 to 70 percent range. Yes, most users of an enterprise “findability” system are unhappy. Procurement teams are, therefore, busy because most companies are looking for a search silver bullet.
To cater to those looking for a quick, simple way to solve an enterprise information access problem, consultants and advisors offer impressionistic write ups. Madison Avenue works fine when selling toothpaste. Apply that method to the very tough problem of information retrieval, and you end up with confusion, rising costs, and unhappy users.
Let me give you another example that surfaced in my conversations with vendors in London at the December International Online Conference. I learned that one consulting firm named Endeca as the top dog in enterprise search. I am okay with that assertion as long as there are some data to back up the claim. When I hear the name “Endeca”, I think of eCommerce as the core strength. The system can be applied to other information problems, but when I recall Endeca’s patent applications, I think about eCommerce, not discovery and data fusion.
Perhaps some search firms are more adept at social engineering than software engineering? Are some search advisors doing Madison Avenue-type thinking, not engineering analyses?
I don’t have any quibble with consulting firms who peg Autonomy as Number One. The revenue alone makes the difference between Autonomy and other information access vendors evident. Last time I saw Andrew Kanter, the chief operating officer for the vendor of meaning-based computing solutions, I asked him, “When will Autonomy break the $1.0 billion in revenue barrier?” He told and an audience of about 175 people that Autonomy “was only $900 million.” Yep, $900 million, which is orders of magnitude greater than most of the 300 vendors whose information retrieval technology I track. IBM, Google, Microsoft, and Oracle do not provide search revenue detail in the financial reports. So on revenue Autonomy has a valid claim to the Number One position in enterprise search.
Consulting Firms Want to Sell Work, Not Expose Warts
Consulting firms—particularly those confined to the mid-tier below the McKinseys, the Bains and the Booz Allens and above the independent experts—have to feed their firms’ revenue hunger. Consulting is an expensive business because full time employees have to be kept billable. Making sales, therefore, is more important than objectivity in my experience.
What mid tier consulting firm sales professional wants to irritate an IBM, Google, Microsoft, or Oracle? Big companies, therefore, are often graded on the curve. Is it not easier to rubber stamp search systems from these Big Four vendors? Get along, go along is perhaps the motto in certain situations.
One consequence of the pressure to make sales is that consulting firms have to back certain horses. The idea is to focus on commercial vendors who are likely to have an appetite for buying and paying for the services of the consulting firm.
Somewhat surprisingly, most of the consulting firms’ search analyses fumble the ball when it comes to open source search; namely, Lucene/Solr, FLAX, Tesuji, and others. The fact is that organizations like Cisco Systems, eHarmony, LinkedIn, MTV, and Twitter, among others are relying on open source “findability” solutions, in particular Lucene/Solr. Open source search is now a viable option for many organizations, and the deprecation of Lucene/Solr is surprising to me.
The bottom-line is that most search vendor league tables are suspect. Unfortunately, these league tables are viewed fact.
On December 10, 2010, I wanted to get an open source technology to talk about open source search and how that option is perceived by marketing organizations masquerading as independent analysts.
The Interview
I spoke with Dr. Brian Pinkerton, one of Lucid Imagination’s vice president of product development. Brian has has a Ph.D. in Computer Science & Engineering and started his work career as a senior software engineer at NeXT. He then developed WebCrawler, the Web’s first comprehensive search engine.
Brian Pinkerton, VP Product Development, Lucid Imagination
Since then he was Technical Architect at AOL (which acquired WebCrawler), VP of Engineering and Chief Scientist at Excite, Principal Architect at A9, Director of Search at Technorati and co-founder/President of Minimal Loop, whose technology was acquired by Scout Labs and where Brian was VP of Engineering.
Today (December 15, 2010) Lucid Imagination is announcing the general availability of its Lucid Works enterprise product, which is available for free download. the product is described as a search solution development platform built on open source Apache Lucene/Solr.
The full text of my interview with Brian appears below:
Several consulting firms have issued analyses of the enterprise search market. I noted that open source search in general and Lucid Imagination in particular were not highlighted as top candidates for the enterprise. Why is open source search put on the bench?
Economics, primarily. Because customers spend huge amounts of money on commercial packages, a small industry has grown up to support and encourage such decisions. This process is naturally set up to ignore disruptive technologies, especially ones that are price-disruptive. The consulting firms don’t work for free: getting prominent placement in a report usually costs money. Who’s paying that fee for open source? Another important reason is the market: developers, not IT managers, are the main adopters of open source solutions, while IT execs are the main consumers of the fancy reports.
Large organizations rely on consultants’ reports. In your opinion are these reports accurate?
It’s hard to comment on these reports because the methods are not always transparent. These consultants spend a lot of time talking to vendors and customers, and draw some conclusions based on that. Many of them have been at it for a while, and they survive by providing useful insights. One useful thing to note, though, is that their conclusions are biased by those they talk to and their target audience: the IT exec. If you’re one of those, I’m sure you like the reports. If you’re a developer, you might not.
How is Lucid Imagination productizing open source search?
We have released a product, LucidWorks Enterprise, that extends Lucene/Solr with features commonly needed by commercial customers. We focus on is providing technology that will make open source Lucene/Solr more accessible to more people. For instance, user interfaces that simplify getting started, or APIs that are specifically targeted to the way enterprises build and integrate applications today.
For example, we extend Solr with RESTful interfaces for configuration; that provides developers with the ability to integrate it more easily. We also simplify functions that could be built from open source, but are more convenient to take as ready-made features. Finally, we add features that 99.9% of software developers probably can’t create easily from scratch, such as our Click Scoring framework, which boosts search results selected most often by users.
Furthermore, open source projects are really good at broad innovation, transparency, and easy access. But the communities around open source projects are not support organizations, so many vendors help companies adopting open source with timely expert support. That’s another one of the things we do at Lucid.
What steps have you taken to ensure the stability of the open source search product you offer?
We take the latest, most stable innovations from the open source development tree (known as ‘trunk’) and provide rigorous integration testing, as well as regular, stable releases driven by customer opportunities. We follow strict software engineering principles and use a quality-driven release process to build LucidWorks Enterprise. And we provide maintenance fixes and releases for our product in timely fashion to customers.
Proprietary search vendors emphasize that their approach ensures that licensees get timely bug fixes and updates. Is this a valid statement? What does Lucid Imagination provide a customer who wants timely bug fixes and updates?
I think both open-source vendors and commercial software suppliers provide timely bug fixes and updates. On the open-source side, it’s an interesting challenge because some bugs are fixed nearly instantly by the open source community, but they are not packaged in a way that a production customer can easily consume. Production customers want bug-fix-only branches of the the software, not bug fixes accompanied by the latest feature innovations that happened to be committed at the same time. We insulate our customers from the open-source volatility by releasing stable, bug-fix-only branches for our production customers.
Search technology has fragmented into a mind numbing number of implementations such as an appliance, cloud or hosted search, on premises search, and combinations of methods. How does Lucid Imagination’s search product fit into this fragmented solutions landscape?
LucidWorks Enterprise is a product that spans the range from software appliance to developer toolkit. Customers new to search can deploy it in a turnkey fashion, while more sophisticated customers can dive under the hood and build a complex application around it. A key secret to great search is how well it fits the business it is meant to serve — in fact, this is true of any application, particularly custom built apps. We believe that anyone who needs better than ‘adequate’ search results will want to build their search solution, and we created LucidWorks Enterprise to provide the best, lowest cost, most scalable platform for building that search solution.
Microsoft SharePoint provides a search solution. Microsoft offers the Fast technology for a more robust solution. What does Lucid Imagination provide to a SharePoint licensee wanting an enhanced search solution?
We will release a robust SharePoint solution in the first two quarters of 2011 and provide anyone to use LucidWorks Enterprise to search their SharePoint data alongside data from other common sources. One of the open questions about the new SharePoint solution is how long Microsoft will support Fast’s integration with anything but SharePoint.
Many search vendors offer faceted search; that is, the system generates hot links to related or supporting content. What is Lucid Imagination’s approach to faceted search?
Both LucidWorks Enterprise and Solr provide faceting support on every query that enables users to refine their results. Faceting is most obviously useful in eCommerce, though a wide variety of applications also take advantage of the feature. LucidWorks Enterprise and Solr support efficient and scalable faceting on any field, providing human-readable labels and accurate facet counts for the top facets. One of the important considerations for large collections is the degree to which faceting works in a distributed configuration. In LucidWorks Enterprise and Solr, faceting is supported seamlessly in distributed situations, offering the full performance at scale.
Would you describe a customer support use case for Lucid Imagination search? What are some common themes?
Because we have a diverse base of customers, we see a wide range of search applications. One common theme is relevance tuning: for instance, customers who need help tying certain results to certain queries, or just better optimizing the algorithms built with Solr & Lucene to deliver the right results. Another common theme, and one that I personally enjoy helping customers with, is performance. We had one customer who replaced a commercial search engine with Solr, reducing their median query response time from 30 seconds to about four seconds without our help. We then helped them reduce that by another factor of eight, to a median query response time of under half a second.
With open source search gaining acceptance within large companies like Cisco and high demand Web applications like Twitter, why are the consulting firms giving open source and Lucid so little attention?
One reason is that it’s coming up really, really fast — and they may not see it coming. Also, open source adoption is often driven by a broad, diffuse population of developers. The developers don’t generally put much stock in what the analysts say, if they’re even aware of the reports to begin with. And on the flip side, the analysts are paying attention to their own customers, CIOs and vendor salespeople, who may not know how the work is really getting done.
What do you suggest a procurement team do to evaluate fully an open source search solution such as the one Lucid Imagination offers?
I think they need to make sure their company is comfortable with creating their own applications; it’s not a passive technology, but one that can be actively used to drive competitive advantage. In looking at vendors, find one that can offer a solution that grows as their needs and skills grow: from something simple in the beginning to something fully customizable as they become more sophisticated consumers. And most importantly, they should look for a company with the depth and expertise to provide training, support, and consulting to help them harness the full scope of search innovation. Finally, they should do the math compared to what they might pay for a comparable implementation with a commercial enterprise search vendor. In many cases, they’re already spending many times what it would cost them to buy an open source-based solution. Sometimes they’ll pay more just for the annual maintenance — excluding consulting and license fees — than for a complete subscription for LucidWorks Enterprise.
In several of the recent analyses of enterprise search systems I have reviewed, I learned about such companies as Sinequa, Fabasoft and Expert System, both examples of firms that have zero profile in many organizations. In your opinion, why are these types of search vendors given so much attention in the search market?
I can imagine that the marketing guys at such organizations are always happy to talk to industry analysts. I spend my time mainly talking to customers and developers.
How can one get more information about Lucid Imagination and its open source enterprise search solution?
Our Web site www.lucidimagination.com is full of information about our product, LucidWorks Enterprise, and other information about the open source technologies, Lucene and Solr. We also have case studies that show how customers are building applications and products with Solr, Lucene, and LucidWorks Enterprise. And I always recommend downloading our product, now available free to developers, and taking it for a spin.
ArnoldIT Comment
My view about consulting firms’ analyses of search and content processing vendors has evolved over the last two years. The economic impact has put pressure on most of the companies that sell technical advice. Since the 2008 financial storm roiled commercial waters, certain advisory firms have shifted from independent analyses to what generates revenue for the consulting firms.
Many of the consulting firms’ reports are white papers or marketing material. The problem is that search is a particularly difficult technical field. Selecting a search system is often a difficult challenge for a procurement team. There are numerous, complex factors to consider.
Consulting firms offer “advice” about what system or systems is the “best” at a particular function. The problem is that writing about search is different from implementing search. It is easier to describe what a search vendor asserts in a demo. It is harder to take that solution and solve a real-world problem in a Microsoft SharePoint environment or in a setting where numerous mission critical applications operate in a stand alone manner.
If you are looking for a search solution, you will need to develop a “tight spec” and then investigate the options that match specific requirements. Few organizations have the time or resources to test multiple systems before making a decision about what search system to license.
The need for information about search creates an opportunity for independent firms to provide information, often at a hefty fee. In my experience, selecting a search system requires an approach close to the one that Martin White and I set forth in our 2009 book Successful Enterprise Search Management, published by Galatea in the UK.
We suggest that procurement teams become familiar with the available literature about search. Then a methodical process of assessment and evaluation can be followed. The short cut often leads to the all-too-common complaints about a search system. Users cannot locate needed information and user satisfaction plummets.
Stephen E Arnold, December 15, 2010
Sponsored
Anti Search in 2011
November 1, 2010
In a recent meeting, several of the participants were charged with disinformation from the azurini.
You know. Azurini, the consultants.
Some of these were English majors, others former print journalists, and some unemployed search engine optimization experts smoked by Google Instant.
But mostly the azurini emphasize that their core competency is search, content management, or information governance (whatever the heck that means). In a month or so, there will be a flood of trend write ups. When the Roman god looks to his left and right, the signal for prognostication flashes through the fabric covered cube farms.
To get ahead of the azurini, the addled goose wants to identify the trends in anti search for 2011. Yep, anti search. Remember that in a Searcher article several years ago, I asserted that search was dead. No one believed me, of course. Instead of digging into the problems that ranged from hostile users to the financial meltdown of some high profile enterprise search vendors, search was the big deal.
And why not? No one can do a lick of work today unless that person can locate a document or “find” something to jump start activity. In a restaurant, people talk less and commune with their mobile devices. Search is on a par with food, a situation that Maslow would find interesting.
The idea for this write up emerged from a meeting a couple of weeks ago. The attendees were trying to figure out how to enhance an existing enterprise search system in order to improve the productivity of the business. The goal was admirable, but the company was struggling to generate revenues and reduce costs.The talk was about search but the subtext was survival.
The needs for the next generation search system included:
- A great user experience
- An iPad app to deliver needed information
- Seamless access to Web and Intranet information
- Google-like performance
- Improved indexing and metatagging
- Access to database content and unstructured information like email.
Intel Stream Number 2 Available
October 27, 2010
ArnoldIT.com’s Intel Stream podcast for October 27, 2010, is now available. The podcast focuses on the intersection of business intelligence and technology. In this week’s 10 minute program, Stephen E Arnold comments about the proposal for the US government to archive Federal workers’ social media postings and content, T-Mobile’s surprising acquisition of Vamosa, Recommind’s 2010 revenues revealed by a competitor, an online SharePoint 2010 cost estimator, and a free download of sentiment analysis software. You can listen to the audio program by navigating to http://arnoldit.com/podcasts/ and clicking on the October 27, 2010 Intel Stream program.
Stuart Schram IV, October 27, 2010
Freebie
BA-Insight Lands $6 Million
September 22, 2010
According to CMS Wire, a Microsoft partner—no, fix that—“a key Microsoft partner”—has received a cash injection of $6.0 million. You will get the content management write up in the story “BA-Insight Secures US$6M Funding for Enterprise Search”. The PR Newswire story “BA-Insight’s Strong Growth in Enterprise Search Space Secures $6 Million in Series A Funding” provides a bit more detail. Note: links to PR stories often go dead, so you may have to resort to some poking around via Bing.com which usually indexes Microsoft centric stories reasonably well.
The news release said:
BA-Insight, Inc., an enterprise search software company specializing in Microsoft-based information access technology, announced today that it has secured $6 million in private equity funding led by New York-based Milestone Venture Partners. Paladin Capital Group and Osage Venture Partners also invested in the round. The New York State Common Retirement Fund participated in the financing through funds managed by Milestone Venture Partners and Paladin Capital Group.
What’s the money for? The release said:
BA-Insight will deploy the capital raised to further develop and extend its suite of enterprise search products for SharePoint Search, expand its marketing efforts and grow its sales and support services organizations. “The market for BA-Insight technologies is expanding rapidly,” explained Guy Mounier, CEO and co-founder of BA-Insight. “We have huge growth potential in U.S. government, professional services, energy and other sectors. This investment will allow us to build the organization needed to support our growth in those markets.”
BA-Insight is a vendor committed to the enterprise search market. This “sector” has been under significant pressure from lower cost Microsoft solutions such as dtSearch (Bethesda, Maryland) and open source solutions like Lucene/Solr. In fact, enterprise search is becoming commoditized.
What’s the BA-Insight difference? According to the news release:
BA-Insight’s flagship product Longitude optimizes Microsoft’s SharePoint Search, and FAST Search for SharePoint platforms. Users can find, analyze, and act on relevant information regardless of the format or where the data resides. Longitude offers out-of-the-box SharePoint Connectors to more than 20 business applications including ERP, CRM, Messaging, and ECM. Longitude also provides a state-of-the-art user experience via a rich Silverlight SharePoint document viewer.
My observations are:
- The BA-Insight play is that Microsoft will continue to encourage its top paying certified partners an opportunity to sell into the SharePoint ecosystem. With more than 100 million SharePoint licenses in the wild, that’s a big ecosystem. The risk is that Microsoft could poach the juicy accounts. If BA-Insight gets traction, Microsoft might buy BA-Insight in order to fatten up its offerings. IBM has followed this strategy for several years. The key difference, in my opinion, is that IBM is using Lucene/Solr and buying value-adding technologies to boost the IBM services business. The Microsoft approach will have a unique fingerprint.
- I think that BA-Insight is “glue play”. What I mean by “glue” is that Microsoft leaves it to licensees to hook together various components to solve a problem. BA-Insight and a handful of other Microsoft centric players provide a “snap in” solution to reduce the time, cost, and hassles of getting basic functions to work as required. Fast Search is a complex beastie, and BA-Insight’s approach is to deliver a solution without the Fast cartwheels that can lead to staff turnover.
- The challenge in the market will be one of time. The recession is allegedly “over.” For organizations strapped for cash, economies will be of significant interest. In the “search and SharePoint” niche, there are quite a few competitors. These range from other Microsoft partners such as SurfRay and Fabasoft to integrators who can hook together existing pieces and parts. Companies in this consulting approach to the search business include New Idea Engineering, with whom I have worked in the past, and my son’s company, Adhere Solutions. Note: my son did not pay me to reference him. I think I bought lunch yesterday which is how the family thing works, right?
- The shift in the enterprise market that I will talk about at the ISS conference in October 2010 is that “search” is not what most users require. The need is for low latency processing of mission critical data delivered in what I call a data fusion system. Few companies offer a “platform” that ingests and makes actionable a range of data. The key players in this space include 20 year veterans like i2 in Cambridge, England, Kroll (now a unit of Altegrity), the Palantir organization (now allegedly involved in a confusing legal matter), and the lesser known but up and coming Digital Reasoning, among others. The name “BA Insight” suggests a capability in the data fusion space, but the new release’s emphasis on “enterprise search” suggests that BA-Insight is anchored in the traditional search market. Perhaps this is just a positioning issue specifically for the news release?
The big challenge is use of the money. Increasing “marketing” sometimes works and sometimes does not. In the “search space”, there is a great deal of noise, smoke, and confusion. The strong interest in open source search so far has not spilled over into the SharePoint sector. I think that will happen. When it does, there will be some interest in Microsoft-centric shops. That interest will probably come from new hires and the chief financial officer’s staff. The traditional Microsoft certified professionals like their counterpart Oracle certified database professionals want to preserve the status quo.
The status quo is not such a comfortable place. Big outfits like Oracle are resorting to legal eagles to cope with open source. Microsoft has a mixed record with regard to open source. My hunch is that BA Insight will have to find a way to go viral within the SharePoint community. That will take keen mastery of social media, the sales ability of Autonomy, and the technical savvy of some serious wizards like Exalead, the repositioning touch of Vivisimo, and the market focus of Coveo. BA Insight has the opportunity to be the break out enterprise search vendor in 2010.
Will $6.0 million be enough? I don’t know the answer. The investors’ smart money thinks BA Insight has what it takes to succeed. From the grandstand in Harrod’s Creek, this race will be fun and entertaining to watch.
Stephen E Arnold, September 22, 2010
Freebie
Search Industry Spot Changing: Risks and Rewards
September 20, 2010
I want to pick up a theme that has not been discussed from our angle in Harrod’s Creek. Marketers can change the language in news releases, on company blogs, and in PowerPoint pitches with a few keystrokes. For many companies, this is the preferred way to shift from one-size-fits-all search solutions described as a platform or framework into a product vendor. I don’t want to identify any specific companies, but you will be able to recognize them as these firms load up on Google AdWords, do pay-to-play presentations at traditional conferences, and output information about the new products. To see how this works, just turn off Google Instant and run the query “enterprise search”, “customer support”, or “business intelligence.” You can get some interesting clues from this exercise.
Source: http://jason-thomas.tumblr.com/
Enterprise search, as a discipline, is now undergoing the type of transformation that hit suppliers to the US auto industry last year. There is consolidation, outright failure , and downsizing for survival. The auto industry needs suppliers to make cars. But when people don’t buy the US auto makers products, dominoes fall over.
What are the options available to a company with a brand based on the notion of “enterprise search” and wild generalizations such as “all your information at your fingertips”? As it turns out, the options are essentially those of the auto suppliers to the US auto industry:
- The company can close its doors. A good example is Convera.
- The search vendor can sell out, ideally at a very high price. A good example is Fast Search & Transfer SA.
- The search vendor can focus on a specific solution; for example, indexing FAQs and other information for customer support. A good example is Open Text.
- The vendor can dissolve back into an organization and emerge with a new spin on the technology. An example is Google and its Google Search Appliance.
- The search vendor can just go quiet and chase work as a certified integrator to a giant outfit like Microsoft. Good examples are the firms who make “snap ins” for Microsoft SharePoint.
- The search vendor can grab a market’s catchphrase like “business intelligence” and say me too. The search vendor can morph into open source and go for a giant infusion of venture funding. An example is Palantir.
Now there is nothing wrong with any of these approaches. I have worked on some projects and used many of the tactics identified above as rivets in an analysis.
What I learned is that saying enterprise search technology is now a solution has an upside and downside. I want to capture my thoughts about each before they slip away from me. My motivation is the acceleration in repositioning that I have noticed in the last two weeks. Search vendors are kicking into overdrive with some interesting moves, which we will document here. We are thinking about creating a separate news service to deal with some of the non-search aspects of what we think is a key point in the evolution of search, content processing and information retrieval.
The Upside of Repositioning One-Size-Fits-All-Search
Let me run down the facets of this view point.
First, repositioning—as I said above—is easy. No major changes have to be made except for the MBA-style and Madison Avenue type explanation of what the company is doing. I see more and more focused messages. A vendor explains that a solution can deliver an on point solution to a big problem. A good example are the search vendors who are processing blogs and other social content for “meaning” that illuminates how a product or service is perceived. This is existing technology trimmed and focused on a specific body of content, specific outputs from specific inputs, and reports that a non-specialist can understand. No big surprise that search vendors are in the repositioning game as they try to pick up the scent of revenues like my neighbor’s hunting dog.
Marketer Links Open Source and Autonomy for Shock and SEO
August 25, 2010
I saw a link that someone sent me from a post on LinkedIn. I have a person pay attention to LinkedIn for me because I am not particularly social nor am I interested in looking for a “real” job. The write up pointed me to a blog post called “What Exactly is IDOL, Anyway?” The blog post is “real”; that is, part of the new positioning from the Silver Spring based content management information service called CMSWatch. You can read the original post “What Exactly is IDOL, Anyway?” and decide if my observations are on track or off track.
Interestingly, the write up cites an open source search vendor’s definition of Autonomy IDOL. I think the snippet is okay, but the snippet comes from a firm that is looking at commercial services in a particular way. There is nothing wrong with the viewpoint, but I think it is often useful to acknowledge that there are other angles from which to examine a technology or a company. For example, I think that a link to Lucid Imagination would have been helpful, but, hey, that’s my opinion. I am beavering away on the open source search conference, The Lucene Revolution, and I know how challenging it is to maintain a balance between the community-centric model and the commercial model. I have tried to create an endnote session that allows both commercial vendors and open source supporters an opportunity to discuss the market as open source becomes more of a force. On the panel are open source experts, the president of a commercial search vendor who used to run an open source company, and a UK-based open source vendor’s president. I hope to make the endnote an engaging, interactive discussion about this very issue: open source search and commercial search.
What caught my attention, however, is the consulting firm’s use of an open source vendor to help pitch a new, for-fee study about search and information access. This is a marketing technique that I wanted to document in my Web log.
Is the method clever?
On one hand, the notion of using an open source vendor to describe one of the best known, most widely used company’s products stopped me in my tracks. I don’t know too much about open source and I probably know even less about commercial companies, but I expected a description of Autonomy IDOL from [a] either Autonomy’s own Web site, [b] a person who has experience working with IDOL in one of Autonomy’s tens of thousands of installations, or [c] a competitor who has had to cope with Autonomy eyeball-to-eyeball.
On the other hand, this juxtaposition is sound search engine optimization type writing.
Here’s a passage that I found particularly interesting:
Now, there are plenty such cowboys around, and they’re perfectly happy with the software. But unfortunately, quite a few of Autonomy’s other customers weren’t quite prepared for it, and ended up unhappy with what they bought. Of course, it’s tempting to blame the vendor’s marketing and sales force for this; but that’s a bit like accusing a tiger of hunting deer. You can’t really blame them for trying.
With open source now gaining momentum, I find it amusing that a consultant is looping open source into a discussion of Autonomy. My hunch is that this type of blog post is a way to get traffic to a Web site and probably make sales of a study about a market sector that is no longer “about” search and information access.
Search has moved on. Information access has changed. The enterprise is a vastly different place from what it was when I wrote the first three editions of “The Enterprise Search Report.” The top five vendors have undergone considerable change. Convera is history. Fast Search & Transfer is a SharePoint component.
Most traditional search vendors struggle to get Web traffic. In my opinion, many consultants are concerned first about generating revenue for themselves and secondarily about helping organizations cope with the business issues tied to digital information. I learned the other day that one of the second tier consulting firms has pulled the plug on its somewhat crazy “map” of enterprise search vendors. This consultant’s efforts reminded me of a knock off of Boston Consulting Group’s work, but maybe I am just confused. Why did the second tier vendor’s map of enterprise search get nuked? The map did not make sense and did not yield what consulting firms need to keep the ship in shape.
My observations are:
- Hooking open source into commercial content processing is an important analytic task and one that warrants additional research and study. The world will no longer be one color. Think Joseph’s Technicolor dream coat.
- The sources of information for such a study are the companies’ own documentation, individuals with hands on experience using the companies’ systems, and observations from clients. Comments from competitors about another company’s products are interesting, but not the “meat and potatoes” which I seek. The “hands on” part is particularly important because technical expertise is needed, not the blathering of the azurini. Sponsored research is lucrative, but I wonder about its objectivity. Most “white papers” are printed on sheets of paper of different colors.
- Marketing presented as “real” information is fine. Weaponized information is something I know a bit about. If one wants to use information to put digital bullets into another company, no problem from me. But more than blanks are needed. Fluffy marketing and odd juxtapositions are digital misfires in my opinion, contributing to the confusion about search and content processing.
To wrap up, the economic pressure on publishing and search vendors is going to go up in the period between September 1, 2010, and March 31, 2011. In this period, I anticipate many interesting marketing methods, new products and services from the azurini, and even greater churn in the search sector. You may see the search world differently. My blog documents my point of view, and the blog is free unlike some of the work from second and third tier consulting firms.
Stephen E Arnold, August 25, 2010
Freebie. This means that no one paid me to write my thoughts into this blog. I am not even selling a report. Ads appear at the top of this blog above the masthead. I am working on a search conference for open source systems and at the same time I am working on a conference for commercial vendors. Works for me because the addled goose makes explicit what provides bread crumbs for the goslings. Irritating, right?
Azurini Lock In Analysis Baffles the Goose
August 3, 2010
I know, I know. Consulting firms have to be “real” and “objective” and “mavenesque.” I accept that. But the write up “Burton Group: Avoid Office 2010 Lock-In, Stick with Office 2007” wowed me. Microsoft buys lots of consulting, research, and advice. As a result, those who want to get jobs with the Redmond fun lovers often find a way to put a honey colored light on almost any product, service, or initiative. How many raves did I read about Vista? How many times have I heard about the wonders of MSN, now Live something? How many times have I heard experts explain the impact of Microsoft’s mobile strategy, its search strategy, its social strategy, its cloud strategy, and other strategies. The addled goose sure does not generate $70 billion a year in revenue and Microsoft does. So, guess who is really smart? Time’s up. Microsoft.
But a consulting firm criticizing Microsoft albeit somewhat indirectly? That is amazing, and it means to me that maybe the fondness Microsoft once felt for Burton has faded. Maybe Burton no longer loves Microsoft? Maybe there are other forces in play? Who knows.
What is clear is that Burton suggests an organization that embraces Office 2010 may be a candidate for lock in. Lock in means that a vendor calls the shots, not the client. The only way to get free is to break out. In fact, that’s one of the appeals of open source software. An organization using open source software believes it has more freedom than when chained to a giant SharePoint installation, an even bigger Microsoft Exchange construct, and the 40 other servers that Microsoft has on offer.
My view is that Microsoft is not the only enterprise software vendor looking to get shelf space and then become a monoculture in a client organization. Does IBM seek to monopolize hardware, software, and services? In my experience, you better understand the way Big Blue operates before your local IBM vice president gets a temporary office down the hall from your company’s president. Same with the Google.
So what strikes me as interesting is not the lock in angle. That’s old news. The criticism of a big outfit like Microsoft has caught my attention. Is one of the azurini changing colors?
Stephen E Arnold, August 3, 2010
Freebie
Lucene Revolution Conference Details
July 15, 2010
The Beyond Search team received an interesting news release from a reader in San Francisco. We think the information reveals the momentum that is building for open source search. Here’s the story as we received it:
San Mateo, Calif. – July 14, 2010 – Lucid Imagination, the commercial company for Apache Lucene and Solr open source search technologies, is pleased to announce speakers for Lucene Revolution, the first-ever conference [EV1] in the US devoted to open source search. The conference will take place October 7-8, 2010 at the Hyatt Harborside, Boston, Massachusetts. Lucene Revolution is a groundbreaking event that drives broad participation in open source enterprise search , creating opportunities for developers, technologists and business leaders to explore the disruptive new benefits that open source enterprise search makes possible, in a fresh, energetic and forward thinking format.
The diverse and widespread adoption of Lucene/Solr for enterprise search applications is reflected by the broad range of speakers at the event, such as:
- Cisco Systems: Satish Gannu
- eHarmony: Joshua Tuberville
- LinkedIn: John Wang
- Sears: David Oliver
- The McClatchy Company: Martin Streicher
- The Smithsonian: Ching-Hsien Wang
- Twitter: Michael Busch
Conference speakers represent a cross-section of Lucene/Solr adoption – including new media, ecommerce, embedded search applications, content management, social media, and security and intelligence – spanning the broad spectrum of production-class enterprise search implementations, all of whom leverage the power and economics of Lucene/Solr innovation.
Other industry thought leaders participating and sharing their insights into open source enterprise search include Hadley Reynolds (Research Director, Search & Digital Marketplace Technologies, IDC) and Stephen E. Arnold (Beyond Search; Managing Partner, ArnoldIT).
Over the two days of the conference there are over 30 sessions scheduled in a variety of different formats: technical presentations, use cases, panel discussions, and Q&A sessions. In addition there will be an “un-conference” the evening of October 7, where attendees can present lightning talks and take part in hands-on community coding efforts.
Registration for Lucene Revolution is now open for the conference at: http://www.lucenerevolution.com/register. A full list of speakers, along with a complete conference agenda, is available at http://www.lucenerevolution.com/agenda.
If you are not familiar with Lucid, here’s a snapshot:
Lucid Imagination is the commercial company dedicated to Apache Lucene technology. The company provides value-added software, documentation, commercial-grade support, training, high-level consulting, and free certified distributions, for Lucene and Solr. Lucid Imagination’s goal is to serve as a central resource for the entire Lucene community and search marketplace, to make enterprise search application developers more productive. Customers include AT&T, Sears, Ford, Verizon, Elsevier, Zappos, The Motley Fool, Macy’s, Cisco, HP, The Guardian and many other household names. Lucid Imagination is a privately held venture-funded company. Investors include Granite Ventures, Walden International, In-Q-Tel and Shasta Ventures. To learn more please visit www.lucidimagination.com.
Goslings Constance Ard and Dr. Tyra Oldham will be attending. Should be useful. Certainly more timely than the plethora of SharePoint and gasping one-size-fits-all programs. Honk.
Stephen E Arnold, July 15, 2010
Sponsored post.
Google and Disruption: Will It Work Tomorrow?
April 15, 2010
Editor’s Note: The text in this article is derived from the notes prepared by Stephen E Arnold’s keynote talk on April 15, 2010. He delivered this speech as part of Slovenian Information Days in Portoroz, Slovenia.
Thank you, Mr. Chairman. I am most grateful for the opportunity to address this group and offer some observations about Google and its disruptive tactics.
I started tracking Google’s technical inventions in 2002. A client, now out of business, asked me to indicate if “Google really had something solid.”
My analysis showed a platform diagram and a list of markets that Google was likely to disrupt. I captured three ideas in my 2005 monograph “The Google Legacy“, which is still timely and available from Infonortics Ltd. in Tetbury, Glos.
The three ideas were:
First, Google had figured out how to add computing capacity, including storage, using mostly commodity hardware. I estimated the cost in 2002 dollars as about one-third what companies like Excite, Lycos, Microsoft, and Yahoo and were paying.
Second, Google had solved the problem of text search for content on Web pages. Google’s engineers were using that infrastructure to deliver other types of services. In 2002, there were rumors that Google was experimenting with services that ranged from email to an online community / messaging system. One person, whose name I have forgotten, pointed out that Google’s internal network MOMA was the test bed for this type of service.
Third, Google was not an invention company. Google was an applied research company. The firm’s engineers, some of whom came from Sun Microsystems and AltaVista.com, were adepts at plucking discoveries from university research computing tests and hooking them into systems that were improvements on what most companies used for their applications. The genius was focus and selection and integration.
Google is an information factory, a digital Rouge River construct. Raw materials enter at one end and higher value information products and services come out at the other end of the process.
In my second Google monograph, funded funded in part by another client, I built upon my research into technology and summarized Google’s patent activities between 2004 and mid 2007. Google Version 2.0: The Calculating Predator, also published by Infonortics Ltd., disclosed several interesting facts about the company.
Search Vendors! FAST-en Your Seatbelts
February 25, 2010
The Microsoft Fast ESP road show is going to come to a US city near you. The road show has entertained thousands in Frankfurt, Melbourne, London, and Paris. Next up is the US. The topics in Europe ranged from collaboration to social search to enterprise content management to SharePoint Web sites. Well, you get the idea. After flipping through the presentations available online if you register, the main idea is that Fast ESP “becomes the foundation” for “all enterprise search products”.
I love those categorical affirmatives. I also like to find black swans, even though I am an old goose. The description of the new system is chock full of superlatives such as “best”. With 300 companies offering search and content processing, I am hard pressed to identify one system as the “best”. Most vendors have some core competencies because “best” is one of the missteps that created the unhappy circumstances for Fast Search & Transfer prior to its sale to Microsoft. Anyone remember the October 2008 police action at the Fast Search offices in Oslo? No, I did not think so.
The idea is to focus on user experience and “go beyond the search box.” I quite like the “beyond” word as in “beyond search”. Here’s an example of the interface:
Copyright Microsoft 2010. Source is the Microsoft Web site reachable from this page http://www.fastsearch.com/l3a.aspx?m=1166&amid=15582
I don’t have many nitty-gritty technical details, but instead of burying the Fast ESP pitch in a SharePoint conference, there are these marketing-oriented traveling programs. The first US event is in Chicago on March 9; the second, in the Big Apple on March 11; and the third, San Francisco on March 16. Microsoft has invited certified partners, resellers, and those with Bill Gates tattoos to attend. What is on tap? You will learn about the new and improved Fast ESP system for SharePoint mostly. You will hear from happy, happy Fast ESP customers. You will get briefed by Microsoft’s own engineers and some invited guests.
As you know, Microsoft purchased Fast Search & Transfer in April 2008 for about $1.2 billion and change. In the 22 month interval, Fast ESP has been trimmed and slimmed to do battle with open source solutions such as Lucene, Lemur Consulting’s FLAX, and Solr. The new Microsoft Fast ESP will do battle with vendors who have moved “beyond search”, so I anticipate some references to the weaknesses of Exalead, MarkLogic, and other companies who have industrial strength solutions. There will be some happiness for Autonomy and Fabasoft Mindbreeze, two firms with solutions that carry the Microsoft seal of approval. My hunch is that those with Windows certification, a paycheck hooked to keeping Microsoft systems alive and well, and partners will be joined by the systems folks who want to get a first hand look at a $1.2 billion search system.
The addled goose is in San Francisco the week of the event, but he returns to the goose pond before the road show pitches the left coast faithful. He will have to report on the event via second hand reports. In the meantime, he will be using his bill to root for information on these topics:
- How has set up, optimization, and customization been simplified?
- How can the system keep metadata synchronized?
- What is the time required to update the index on a 10 minute basis in an organization with 10,000 active users out of an employee pool of 150,000 and a document flow of 1,000 new or changed documents every 24 hours (excluding emails and attachments)?
- What is the method for scaling Microsoft Fast?
- What is the method for restoring / rebuilding indexes in the event of a system fault? What is the time required in a typical organizational setting with 10,000 active users and the document flow of 1,000 new or changed documents every 24 hours (excluding email and attachments)?
- What is the total cost for a system for 10,000 active users?
I think I know the license fee, based on the rumors floating in the aether. I can’t reveal the deal, but the price tag will make life tough for vendors up and down the line. If Microsoft hits a home run, my question is, “What tricks does Google have ready to roll?”
You can get the full scoop at http://www.fastsearch.com/l3a.aspx?m=1166&amid=15582.
Stephen E Arnold, February 25, 2010
No one paid me to write this. I will report writing about Microsoft to the Department of Defense. Not only was I doing work for free, the DoD people understand, love, and appreciate the Microsoft technology. Free is good I think.
-
- Subscribe to Beyond Search
Feature archive
News archive
-
