Performance Fireworks: Microsoft Fast Fizzles, Google Explodes
July 4, 2009
I was sitting in an airport, and I clicked on a link for Microsoft Fast ESP. A video ran and presented me with a couple of professional fellows talking about Microsoft Fast search. The video was interesting, but I went back and snagged one screen frame from the presentation because it struck me as a way to explain the distance between the performance of Microsoft Fast and the performance of Google’s system. Now performance data for search systems is a murky area. I don’t want to get into a squabble about something being five times faster. The difference here makes a point, and I will leave it to Googlers and Microsofties to post corrected performance data in the Comments section of this Web log, assuming those companies’ professionals have time to read the thoughts of the addled goose.
First, the Microsoft data. Here’s the screenshot, and I want you to notice that the performance that is presented is five to 20 queries per second. That is pretty modest for a performance threshold even for a Microsoft team in Charlotte, North Carolina, where I have heard the pace of life is on par with Harrod’s Creek.
Source: http://www.youtube.com/ watch?v=kTbcCNby8xE
I ask you to click here to look at the performance data I calculated for Google. The key point is that if the Google data are reasonably accurate, the Google is cranking along about about 1,700 queries per second. Even Yahoo appears to perform better than Microsoft Fast. See my write up here.
That’s a big gap. Assume the Google data are off by a factor of four. The Google is handling 400 queries per second. If we boost the Microsoft Fast performance by a factor of four to 20 queries per second to 80 queries per second, the Google appears to be the speed demon.
If you want performance fireworks, my thought is that the Google is the fire cracker if the data are correct.
Stephen Arnold, July 4, 2009
Comments
4 Responses to “Performance Fireworks: Microsoft Fast Fizzles, Google Explodes”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
[…] Performance Fireworks: Microsoft Fast Fizzles, Google Explodes … By Stephen E. Arnold News and Information from ArnoldIT.com about Search and Content Processing. Beyond Search – http://arnoldit.com/wordpress/ […]
This might be the case of geese and too much power point. The product sheet for Goog appliance http://googleenterprise.blogspot.com/2009/02/google-search-appliance-providing.html list 300 (++ for higher end) queries per minute. Assuming even distribution that gives you 50 queries/sec. Even if this topic is strictly about product sheet/ppt acrobatics based query benchmarking assume any system (from lucene to google to fast and autonomy) can demonstrate triple digit qps performance.
I suspect this whole thing is an misunderstanding. I’m an Enterprise Search Consultant for a Fast / MS Partner and have worked on a varierty of ESP Projects. We’ve thrown over 100 QPS to a 4 node install during performance testing so 20 QPS is not a problem.
What I think the presentation means is If you need a guaranteed 20 QPS with a sub second response time that Fast ESP is a better solution than the standard Sharepoint Search.
QPS for all vendors varies on hardware, data and complexity of the query etc.
In fact here’s a write up which mentions 1500 qps. http://www.espen.com/archives/2009/02/fast_forward_20.html.
Regards
Pete
FAST QPS pertains to Enterprise Search whereas the Google Search QPS you cite pertains to, presumably, Google.Com, which is internet search. You ought to know better than to compare these two types of searches as if they are the same. Also, you should know better that any performance comparisons of just some QPS numbers is meaningless without ensuring that the hardware, workload and dataset used are comparable.
In my view, you have lost all credibility as a search analyst. Please stop misleading folks.