Reflections on SharePoint and Search
October 19, 2009
I had an interesting conversation with a librarian at the international library conference in Hammersmith on December 15, 2009. This professional from a central European country asked my views about SharePoint. As I understood her comments, she was in a SharePoint centric environment and found that making small changes to improve information access was difficult, if not impossible.
One reason was her organization’s bureaucratic set up. Her unit was in the same building at the information technology group, but IT reported to one senior manager and her library to another. Another reason was the reluctance of the IT team to make too many changes to SharePoint and its indifference to her legacy library systems. Her challenge was even more difficult because there were multiple legacy systems in the information center. One of these system offered search, and she wanted to have SharePoint make use of the content in that legacy system’s repository. She was not sure which vendors’ search system was in the legacy system, but she thought it was from the “old” Fast Search & Transfer outfit.
The motto of IT and another unit’s management.
Okay, I thought. Fast Search & Transfer, the company Microsoft bought in 2008 and had spent the last 18 months converting into the world class search system for SharePoint. Fast ESP would break through the 50 million document ceiling in SharePoint search and add user experience plus a nifty range of functions.
To make a long story short, she wanted to know, “Will Microsoft and SharePoint support the “old” versions of Fast ESP?” I told her that I recalled reading that Microsoft would stand behind the Fast ESP technology for a decade. She replied, “Really?” Frankly, I do not know if that guarantee extends to OEM instances of the “old” Fast ESP. My hunch is that the problem will fall upon the vendor licensing the Fast ESP engine. But I have learned and suffered from the fact that it is very easy to write marketing collateral and somewhat more difficult to support a system—any system—for a decade. I know mainframe systems that have been around for 30 years, possibly more. But the Linux centric search systems built to index the Web content are a different kettle of Norwegian herring.
My understanding of the trajectory of Fast ESP is that the company took the core of its high speed Web indexing system and reworked it to handle enterprise content. The “old” Fast ESP abandoned the Web search market when the company’s top brass realized that Google’s technology was powering past the Fast technology. Enterprise search became the focus of the company. Over the years, the core of Fast’s Web search system has been “wrapped” and “tweaked” to handle the rigors and quite different requirements of enterprise search. The problem with the approach as I pointed out in the three editions of the Enterprise Search Report I wrote between 2003 and 2006 were:
- Lots of moving parts. My research revealed that a change in a Fast script “here” could produce an unexpected result elsewhere in the system “there”. Not even my wizard son could deal with these discomforting technical killer bees. Chasing down these understandable behaviors took time. With time converted to money, I concluded that lots of moving parts was not a net positive in my enterprise search engagements. Once a system is working, the attitude of the librarian’s IT department is my reaction. Just leave something working alone.
- Performance. Web search required in the late 1990s a certain type of spidering. Today, indexing has become more tricky. Updates are often larger than in the past, so the content processing subsystem has to do more work. Once the index has been updated, other processes can take longer because indexes are bigger or broken into chunks. Speeding up a system is not a simple matter of throwing hardware at the problem. In fact, adding more hardware may not improve performance because the bottleneck may be a consequence of poor engineering decisions made a long time ago or because the hardware was added on but to the wrong subsystem; for example, the production server, not the indexing subsystem.
- Ignorance. Old systems were built by individuals who may no longer be at the company. Today, the loss of the engineers with implicit knowledge of a subsystem make it very difficult—maybe even impossible– to resolve certain functions such as inefficient use of scratch space for index updates. I recall one job we did at a major telco several years ago. The legacy search system was what it was. My team froze it in place and worked with data the legacy system wrote to a file on a more modern server on a fixed schedule. Not perfect, but it was economical and it worked. No one at the company knew how the legacy search system worked. The client did not want to pay my team to figure out the mystery. And I did not want to make assumptions about how long gone engineers cooked their code. Companies that buy some vendors’ search systems may discover that there is a knowledge problem. Engineers often document code in ways that another engineer cannot figure out. So the new owner has to work through the code line by line to find out what is going on. Executives who buy companies make some interesting but often uninformed assumptions; for example, the software we just bought works, is documented, and assembled in a textbook manner. Reality is less tidy than the fantasies of a spreadsheet jockey.
What’s this mean?
ArnoldIT.com’s professionals can support the Burroughs’s system and hook it into SharePoint if you have the time and money. If not, you wish.
When I read that a $65 billion dollar company like Microsoft is going to back a product for a decade, my first reaction was, “Man, is that going to be expensive.” Even now, I think it might be more wishful thinking than reality. Customers are not likely to want the same features tomorrow as they do today. With the demand for rich media and real time search increasing, I find it hard to imagine the engineering effort necessary to take a 12 year old legacy system coded and keep improving it to keep pace with more modern systems.
What can customers in a SharePoint environment do with a Fast ESP legacy system? I know what I would do. I would ask that the Microsoft SharePoint engineers find a certified third party who can hook the Fast ESP system into the SharePoint 10 system once these products become available in November 2009. If these experts cannot do the job within the time and budget limits of the organization, I would get a newer system. I know I would look at high profile modern systems such as Exalead’s, and I would check out relative newcomers like Gaviri. In fact, I would do some proofs of concept and pick the best system for my needs. I know I would not consider the older systems that are on the market; for example, the BASIS technology or the Bayesian systems. I want 64 bit, smart systems, not the pains of the past.
Let me jump back to the librarian’s question. Support has to be tested. So until the new SharePoint and the UX version of Fast ESP is available, the reality of the situation is that the organization’s IT department won’t tackle the job. Period. End of story. Once the new SharePoint 10 system is up and running, maybe the IT department will be able to solve the problem. Until then who knows? No me.
This conversation begged another question, “What about the future of the Linux and Solaris versions of Fast ESP?” I can only point out the information and views I have.
First, Microsoft has to keep customers happy, but the company wants to get Microsoft technology into organizations. I think that an upsell effort is going to be part of any support that Microsoft extends to the Linux and Solaris installations of Fast ESP. The good news is that as long as these outfits can pay, Microsoft will keep the Fast ESP system running. I am not confident that the newer search features and the UX will be rolled out without an upgrade and engineering fee. To get these new features, Linux and Solaris customers may have to buy new Windows systems to handle these features. In my opinion interaction between two systems may have some undesirable issues to resolve. I worry about complexity and performance, for instance.
Second, the customers may want functions that are simply not available without a rip-and-replace strategy. Hewlett Packard is advocating a rip-and-replace method for legacy mainframe systems. I don’t think that many organizations are going to follow this path without a massive sales effort and a heck of a business case. In short, existing Fast ESP customers may be faced with two different angles of attack on this problem. The first is to lock down the status quo Fast ESP system and just live with it. The other is to find a middle ground and find the resources to “make it work”. Rip and end replace are painful words to some of the chief financial officers whom I know. Your mileage may vary.
To wrap up, I think that vendors with a modern platform, technology that permits modular expansion without performance degradation, and engineering able to handle rich media and real time content should make it clear that legacy systems are going to challenge their users with four problems:
- Ongoing cost creep to deal with unexpected problems resulting from interactions within layered systems
- Annoyed, maybe angry users, who cannot get the features available from other services in the library or the Intranet systems
- More stonewalling from the Microsoft certified professionals who don’t want to get involved in a tar pit from which there may be no way out
- The managerial pain that comes about when marketing assurances do not match the actual functionality of the search system.
Swamped by marketing hyperbole.
Needless to say, the librarian was agitated by my response. I think I was the only person at the conference willing to talk candidly about the problems ahead for some of the organizations depending on a technology product built for the Web now remodeled to work in the enterprise, rich media, and real time search environment. When a product I have wears out, I find a way to replace it in a prudent, logical manner. Ongoing repairs are not my preferred solution.
Stephen Arnold, October 19, 2009
The librarian, needless to say, did not pay me for my time nor did she offer to pay me for this article. Again I am working for free, honorable FCC.
Comments
10 Responses to “Reflections on SharePoint and Search”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
Another question to ask is when will an upgrade option to current FAST ESP users who have SharePoint 2007 arrive for SharePoint 2010?
Fast ESP for SharePoint is to my knowledge a “black box”, much like MS Search Server, and with only one document pipeline. From this I conclude that it’s an offering for existing MOSS customers with MOSS Search, and for new customers. If you have a full blown Fast ESP, you have to stand in line.
That would be the FTC: http://www.ftc.gov/opa/2009/10/endortest.shtm
The FCC only cares if you start broadcasting obscenity. 🙂
http://www.fcc.gov/cgb/consumerfacts/obscene.html
With regard to support for OEM versions of Fast it depends on the version and the client should contact their vendor asap.
With regard to integration of the OEM version of Fast with Sharepoint, it shouldn’t be too difficult to federate the query from Sharepoint to the version of Fast.
We should know more about the upgrade path to ESP for Sharepoint today as MS is talking about it at the Sharepoint conference in Vegas.
Thanks
Pete
Steve, I think you may have been watching too much FlashForward. Your interview on Dec 15 2009 has not happened yet. 🙂
I don’t think you are quite accurate in stating that ESP was designed for the web, . As a matter of fact, FAST sold off all its web index business, including the AltaVista web business. ESP was targeted from the get-go for the enterprise..
On the other hand, the version of ESP that was re-engineered and sold through OEMS had different configurations and extension depending on which OEM the user purchased the product from. Please note that these implementation were actual NOT ESP, but even named differently and was called FAST InStream. Like most OEM products, it was not licensed as an end user license so the chances anyone would support it like ESP is doubtful. The customer is best served going to the vendor that sold them the solution or buying a new license..
Although the “10 year support” legend is bantered around the ESP user world the reality is FAST issued a policy in November 2008 and has pretty much stuck with that plan. The idea was a “10 year support horizon”. User’s experiences will vary. Note that InStream is not on that list at all. The policy statement is still on the site: https://extranet.fastsearch.com/static/Customer_Ready_Support_Upgrade_FAQ.html
As we will all find out this week, Microsoft will continue to announce the evolving integration of ESP and Sharepoint. Best I can see, this seems on track with there originally announced plans. And yes, ESP will continue to provide enterprise scalability for Sharepoint and just about any other content a customer may require. Any enterprise customer will be hard pressed to find a more cost effective integrated enterprise ECM and search solutions right now, at least for the Windows world.
Unix/Linux? Its got to be all about the customer, but I would agree with you to a point. I doubt we will see much major functional extensions to non-windows based releases. Support yes; functional advancement, doubtful.
You point to the problems of one time licensing capitalization. It is likely not a good idea in todays quickly changing software world to expect long horizons for feature improvements on one-time licensed product. Choosing use licensing makes much more sense. I expect the ecal model to be quite popular for Microsoft customers in this regard.
Daniel Tunkelang,
I had someone tell me that my posts were obscene. The helpful reader added that I was uninformed, a risk to young programmers, and worse than the people accosted on the TV show “Cops”. I am going to post whether my articles are written for zero compensation or whether I get a vast sum of money or cargo cult type goodies for a write up. I will disclose vendors who buy me a ginger beer as “paying” for coverage. The addled goose is addled for a very good reason. Probably because he does not smoke, drink, attend industry conferences, or get too excited when his view points give people a headache. I thought the FCC, FTC, and SEC were birds of a feather, each to regulate my life so it is better in each and every way. Therefore, I see them as identical, a method that has helped me with my government work in the distant past.
Stephen Arnold, October 19, 2009
[…] Reflections on SharePoint and Search : Beyond Search What can customers in a SharePoint environment do with a Fast ESP legacy system? I know what I would do. I would ask that the Microsoft SharePoint engineers find a certified third party who can hook the Fast ESP system into the SharePoint 10 system once these products become available in November 2009. If these experts cannot do the job within the time and budget limits of the organization, I would get a newer system. I know I would look at high profile modern systems such as Exalead’s, and I would check out relative newcomers like Gaviri. In fact, I would do some proofs of concept and pick the best system for my needs. I know I would not consider the older systems that are on the market; for example, the BASIS technology or the Bayesian systems. I want 64 bit, smart systems, not the pains of the past. (tags: FAST microsoft searchvendor searchengine exalead) […]
Steve, at last I realize how you get the scoop so much earlier than anyone else. I bet your holiday cards always arrive ahead of time!
A note for the gentleman commenting on the 64 bit software: Microsoft/FAST are offering the SharePoint version of both SearchServer and FAST ESP in SharePoint on 64 bit platforms only – no more 32 bit at all. WRT delivery date: The SharePoint beta shops in November 2009 (still a few weeks from now on my current timeline); altho ESP will not be 0part of the initial release, as I understand it.
Regards my friend!
Stephen – an interesting write-up indeed.
I seem to recall hearing not so long ago that FAST were saying that “All The Web” was proof of FAST’s scalability. Now we learn from John that it was a totally different product and ESP was designed from the ground up for Enterprise Search. With the OEM version being on a different code base and with the Unix/Linux releases apparently going their separate ways to ESP on Windows, it now seems that with the addition of FAST for Sharepoint that Microsoft have at least 4 FAST code bases to support. On top of their existing large number of search solutions perhaps Gartner should produce an information retrieval quadrant aimed solely at Microsoft’s search products?
As John points out he does not see much major functional extensions to non-Windows environments. The observation regarding Unix/Linux support is worrying. Given that a large proportion of FAST customers are Ecommerce and large website clients whose platform of choice tends to be Unix/Linux the choice seems clear… stick to a legacy platform, migrate to FAST on Windows or migrate to another search provider. Will support extend to fixing performance/scaling issues that may manifest themselves on older implementations? Given the nature of business on the web, having an agile deployment strategy is a must.
I had assumed that “support” included the same enhancements to the product that will be ongoing in ESP on the Windows platform. What about new FAST customers? Has Microsoft stopped selling ESP to new Unix/Linux customers or still selling ESP even in the knowledge that these products will get minimal enhancements moving forward? If I had invested in a license of FAST on Unix/Linux I would have expected to get future enhancements in step with ESP on Windows as part of my support and maintenance agreement. If John is right, this appears not to be the case.
Your observation around 64 bit “engineering” is also well made. The expectation that 64 bit technology will in itself somehow make an older enterprise search engine fit for today’s information management demands is like giving an unfit person (me) an extra five gears on my bicycle and expecting me to transform into Lance Armstrong. If only it were true.
thanks for thi nice post
For those who noticed Steve’s adventure in a future life, the conversation took place at Internet Librarian International (www.internet-librarian.com) on 15 October 2009.