Beyond Search

Featured

I was given a box lunch and a can of Pepsi as compensation for my one hour talk at a conference last week. I had an interesting conversation with a former big wheel in commercial database publishing. I thought the wizard was a retired poobah. I was wrong. The fellow had his shoulder pads on, a sweatband, and Gucci cleats. He’s back on a commercial company’s publishing team. I am an old, cowardly goose, and it is with trepidation that I get too close to big people garbed for quasi-military re-enactments related to electronic information.

I asked the industry titan what his new gig involved. I recall one word, which he repeated several times to me, the addled goose. The word? “Discovery.” I thought I was having a The Graduate moment. In 2010, plastic was a loser. The winner? Discovery.

Yep, the lingo of the search and content processing market has reached the world of professional publishing and for-fee database access.

The idea, as I understood it, is that this commercial company will allow a user to enter a keyword; for example, employee stock ownership. The system will crunch away and present:

1. Results from the firm’s for fee databases. Not just anyone can run a search. The user has to have access to an institutional account or sign up and pay. There is some free stuff, but this is a real, live make-money-or-die operation.
2. The system will also “discover” possibly related content and list that information in the form of links. I think the idea the titan was communicating is what Endeca calls “Guided Navigation” in 1999! Not exactly yesterday! To see the Endeca system in action just go to OfficeFurniture.com.
3. Content from the public Web.

The idea is that a person using a commercial system will enter a search string and then see links to related content. This works for buying office furniture. I am not sure how a computational chemist would react to a suggestion she read a blog post about a meth lab that blows up.

Yep, our professional grade service needs those custom chrome wheels. Image source: http://www.up.ac.za/organizations/movup/images/minefun/indian_haul_truck.jpg

I asked what happened if I used one of the company’s business databases and entered the search term “management.” I got a bit of double talk and the titan backed up, trying to get away from me. The reason I asked about this type of search is that I know from hands-on experience that the use of a general controlled term in his firm’s databases does not generate a usable results list. Thus, any “discovered” information is likely to be wide of the mark. Broad queries don’t often work too well in the for-fee, quite specific content in certain commercial systems. A single word like “management” in a Google search box generates what is highly ranked by clueless millions like a link to the Wikipedia entry.

Read more »

Interviews

This is the second part of the interview with Raymond Bentinck of Exalead.

Isn’t this bad marketing?

No. This makes business sense.Traditional search vendors who may claim to have thousands of customers tend to use only a handful of well managed references. This is a direct result of customers choosing technology based on these overblown marketing claims and these claims then driving requirements that the vendor’s consultants struggle to deliver. The customer who is then far from happy with the results, doesn’t do reference calls and ultimately becomes disillusioned with search in general or with the vendor specifically. Either way, they end up moving to an alternative.

I see this all the time with our clients that have replaced their legacy search solution with Exalead. When we started, we were met with much skepticism from clients that we could answer their information retrieval problems. It was only after doing Proof of Concepts and delivering the solutions that they became convinced. Now that our reputation has grown organizations realize that we do not make unsubstantiated claims and do stick by our promises.

What about the shift to hybrid solutions? An appliance or an on premises server, then a cloud component, and maybe some  fairy dust thrown in to handle the security issues?

There is a major change that is happening within Information Technology at the moment driven primarily by the demands placed on IT by the business. Businesses want to vastly reduce the operational cost models of IT provision while pushing IT to be far more agile in their support of the business. Against this backdrop, information volumes continue to grow exponentially.

The push towards areas such as virtual servers and cloud computing are aspects of reducing the operational cost models of information technology provision. It is fundamental that software solutions can operate in these environments. It is surprising, however, to find that many traditional search vendors solutions do not even work in a virtual server environment.

Isn’t this approach going to add costs to an Exalead installation?

No, because another aspect of this is that software solutions need to be designed to make the best use of available hardware resources. When Exalead provided a solution to the leading classified ads site Fish4.co.uk, unlike the legacy search solution we replaced, not only were we able to deploy a solution that met and exceeded their requirements but we reduced the cost of search to the business by 250 percent. A large part of this was around the massively reduced hardware costs associated with the solution.

What about making changes and responding quickly? Many search vendors simply impose a six month or nine month cycle on a deployment. The client wants to move quickly, but the vendor cannot work quickly.

Agility is another key factor. In the past, an organization may implement a data warehouse. This would take around 12 to 18 months to deploy and would cost a huge amount in hardware, software and consultancy fees. As part of the deployment the consultants needed to second guess the questions the business would want to ask of the data warehouse and design these into the system. After the 12 to 18 months, the business would start using the data warehouse and then find out they needed to ask different types of questions than were designed into the system. The data warehouse would then go through a phase of redevelopment which would last many more months. The business would evolve… making more changes and the cycle would go on and on.

With Exalead, we are able to deploy the same solution in a couple months but significantly there is no need to second guess the questions that the business would want to ask and design them into the system.

This is the sort of agile solution that businesses have been pushing their IT departments to deliver for years. Businesses that do not provide agile IT solutions will fall behind their competitors and be unable to react quickly enough when the market changes.

One of the large UK search vendors has dozens of niche versions of its product. How can that company keep each of these specialty products up to date and working? Integration is often the big problem, is it not?

The founders of Exalead took two years before starting the company to research what worked in search and why the existing search vendors products were so complex. This research led them to understand that the search products that were on the marketplace at the time all started as quite simple products designed to work on relatively low volumes of information and with very limited functional capabilities. Over the years, new functionality has been added to the solutions to keep abreast of what competitors have offered but because of how the products were originally engineered they have not been clean integrations. They did not start out with this intention but search has evolved in ways never imagined at the time these solutions were originally engineered.

Wasn’t one of the key architects part of the famous AltaVista.com team?

Yes. In fact, both of the founders of Exalead were from this team.

What kind of issues occur with these overly complex products?

As you know, this has caused many issues for both vendors and clients. Changes in one part of the solution can cause unwanted side effects in another part. Trying to track down issues and bugs can take a huge amount of time and expense. This is a major factor as to why we see the legacy search products on the market today that are complex, expensive and take many months if not years to deploy even for simple requirements.

Exalead learned from these lessons when engineering our solution. We have an architecture that is fully object-orientated at the core and follows an SOA architecture. It means that we can swap in and out new modules without messy integrations. We can also take core modules such as connectors to repositories and instead of having to re-write them to meet specific requirements we can override various capabilities in the classes. This means that the majority of the code that has gone through our quality-management systems remains the same. If an issue is identified in the code, it is a simple task to locate the problem and this issue is isolated in one area of the code base. In the past, vendors have had to rewrite core components like connectors to meet customers’ requirements and this has caused huge quality and support issues for both the customer and the vendor.

What about integration? That’s a killer for many vendors in my experience.

The added advantage of this core engineering work means that for Exalead integration is a simple task. For example, building new secure connectors to new repositories can be performed in weeks rather than months. Our engineers can take this time saved to spend on adding new and innovative capabilities into the solution rather than spending time worrying about how to integrate a new function without affecting the 1001 other overlaying functions.

Without this model, legacy vendors have to continually provide point-solutions to problems that tend to be customer-specific leading to a very expensive support headache as core engineering changes take too long and are too hard to deploy.

I heard about a large firm in the US that has invested significant sums in retooling Lucene. The solution has been described on the firm’s Web site, but I don’t see how that engineering cost is offset by the time to market that the fix required. Do you see open source as a problem or a solution?

I do not wake up in the middle of the night worrying about Lucene if that is what you are thinking! I see Lucene in places that have typically large engineering teams to protect or by consultants more interested in making lots of fees through its complex integration. Neither of which adds value to the company in, for example, reducing costs of increasing revenue.

Organizations that are interested in providing cost effective richly functional solutions are in increasing numbers choosing solutions like Exalead. For example, The University of Sunderland wanted to replace their Google Search Appliance with a richer, more functional search tool. They looked at the marketplace and chose Exalead for searching their external site, their internal document repositories plus providing business intelligence solutions over their database applications such as student attendance records. The search on their website was developed in a single day including the integration to their existing user interface and the faceted navigation capabilities. This represented not only an exceptionally quick implementation, far in excess of any other solution on the marketplace today but it also delivered for them the lowest total cost of ownership compared to other vendors and of course open-source.

In my opinion, Lucene and other open-source offerings can offer a solution for some organizations but many jump on this bandwagon without fully appreciating the differences between the open source solution and the commercially available solutions either in terms of capability or total cost. It is assumed, wrongly in many instances, that the total cost of ownership for open source must be lower than the commercially available solutions. I would suggest that all too often, open source search is adopted by those who believe the consultants who say that search is a simple commodity problem.

What about the commercial enterprise that has had several search systems and none of them capable of delivering satisfactory solutions? What’s the cause of this? The vendors? The client’s approach?

I think the problem lies more with the vendors of the legacy search solutions than with the clients. Vendors have believed their own marketing messages and when customers are unsatisfied with the results have tended to blame the customers not understanding how to deploy the product correctly or in some cases, the third-party or system integrator responsible for the deployment.

One client of ours told me recently that with our solution they were able to deliver in a couple months what they failed to do with another leading search solution for seven years. This is pretty much the experience of every customer where we have replaced an existing search solution. In fact, every organization that I have worked with that has performed an in-depth analysis and comparison of our technology against any search solution has chosen Exalead.

In many ways, I see our solution as not only delivering on our promises but also delivering on the marketing messages that our competitors have been promoting for years but failing to deliver in reality.

So where does Exalead fit? The last demo I received showed me search working within a very large, global business process. The information just appeared? Is this where search is heading?

In the year 2000, and every year since, a CEO of one of the leading legacy search vendors made a claim that every major organization would be using their brand of meaning based search technology within two years.

I will not be as bold as him but it is my belief that in less than five years time the majority of organizations will be using search based applications in mission critical applications.

For too long software vendors have been trying to convince organizations, for example, that it was not possible to deploy mission critical solutions such as customer 360 degree customer view, Master Data Management, Data Warehousing or business intelligence solutions in a couple months, with no user training, with with up-to-the-minute information, with user friendly interfaces, with a low cost per query covering millions or billions of records of information.

With Exalead this is possible and we have proven it in some of the world’s largest companies.

How does this change the present understanding of search, which in my opinion is often quite shallow?

Two things are required to change the status quo.

Firstly, a disruptive technology is required that can deliver on these requirements and secondly businesses need to demand new methods of meeting ever greater business requirements on information.

Today I see both these things in place. Exalead has proven that our solutions can meet the most demanding of mission critical requirements in an agile way and now IT departments are realizing that they cannot support their businesses moving forward by using traditional technologies.

What do you see as the trends in enterprise search for 2010?

Last year was a turning point around Search Based Applications. With the world-wide economy in recession, many companies have put projects on hold until things were looking better. With economies still looking rather weak but projects not being able to be left on ice for ever, they are starting to question the value of utilizing expensive, time consuming and rigid technologies to deliver these projects.

Search is a game changing technology that can deliver more innovative, agile and cheaper solutions than using traditional technologies. Exalead is there to deliver on this promise.

Search, a commodity solution? No.

Editor’s note: You can learn more about Exalead’s search enable applications technology and method at the Exalead Web site.

Stephen E Arnold, February 4, 2010

I wrote this post without any compensation. However, Mr. Bentinck, who lives in a far off land, offered to buy me haggis, and I refused this tasty bribe. Ah, lungs! I will report the lack of payment to the National Institutes of Health, an outfit concerned about alveoli.

Profiles

Keep in mind that the addled goose’s Web log, which you are now reading, is a marketing vehicle. It contains on good days old news. On bad days, the addled goose recycles his own talks which he delivers for tacos and Pepsis. I am not a journalist and I don’t pretend to be one. I am not even a public relations person. As I approach 66, I entertain myself capturing information that I otherwise would forget and documenting my thoughts, which are subject to change. When was the last time, a 65 year old could remember where he or she put the keys to the automobile? See what I mean.

When I read the Cnet write up about a post I saw last week, I thought, “CBS’s real journalists are now thinking about themselves in a meta-way.” Navigate to “Has Business Press Lost Touch with the Tech Industry?” CBS is a real company and it does real news. Cnet is a real news outfit, if I understand set theory. The point is that an azure chip outfit called ITDatabase figured out that the real journalists are writing about topics that are popular. I think this is using humans the way Google uses popularity algorithms. I am sure the “real” journalists would disagree. That’s okay.

For me, the most interesting passage in the write up in Cnet was:

Enterprise IT is woefully underrepresented, despite being the cash-cow in the industry. “In the overall editorial agenda,” the report says, “enterprise IT is treated like consumer tech’s snaggletoothed twin. It barely even makes the family photo.”

Let’s think about this statement.

First, publicly traded companies are covered with a bit of fancy geometry by the investment analysts tracking these companies. The information is usually able to deliver a couple of nuggets. The reason is that most of the analysts talk to people * other than public relations * and * business development officers *. Most of the real journalists recycle familiar contacts, preferring to quote names the writer assumes the readers will recognize. So when the word “search” appears in a story the same handful of “experts’ get quoted. The result is that the stories really don’t change too much from article to article. Google is an advertising company. Bing is gaining share. Autonomy is the leader in enterprise search. The statements in the article are true because they are in the article. Tautology meets routine.

Second, figuring out what is going on in a technology field is tough for three reasons. [a] The jargon is impenetrable. A “real” journalist may not have the time to figure out what the words mean. Example: faceted search or taxonomy. [b] The sources are often running the game plan. Take a look at the comments by tech leaders. There are buzzwords and a jab or two at a windmill. Not much substance because the focus is the sound bite. [c] A tech company sells products that a really complex. The wizards at the company cannot be trusted to answer a question because the wizard might point out that a specific feature is different from the function described by the marketing person. Guess who gets in trouble? The tech person so there folks are shuttled away from the lights and the cameras.

Third, I heard that publishing companies are getting rid of staff. The numbers quoted at a conference last week struck me as pretty high. The person pointed out that newspapers were shedding jobs at the rate of 1,000 per month. Wow. What will be left? What’s left, if this number is accurate, are people who have to write from news releases, contacts who are warm and familiar, and topics that are listed on Tweetmeme.

When the money goes away, algorithms will do this work and, of course, folks with time on their hands like this addled goose. Just my opinion.

Stephen E Arnold, March 9, 2010

No one paid me to write about how I write this blog. Wait. If I buy myself lunch this afternoon, I will be getting paid. I will report the write-for-food angle to the FCC.