Prepping for Google’s Udi Manber Keynote: Another Datawocky Scoop

June 11, 2008

At the Gilbane Conference in San Francisco next week, the keynote speaker is Udi Manber, top wizard of search at Google. I think this post by Anand Rajaraman in Datawocky is not serendipitous. Once again, Googlers find that Datawocky is a trusted conduit of first class information. My view is that if you’re smart enough to know about Datawocky, then you have earned the right to get some juicy Googlebits before the rest of the world. Mr. Rajaraman delivers I believe a sneak preview of a Google revelation. Mr. Rajaraman, who is best buds with a number of Googlers, spoke with Peter Norvig, who is on leave in order to update his book about artificial intelligence. Dr. Norvig struck me as awfully chatty and well informed for someone on leave and writing a book. But we are in the midst of 40 days and 40 nights of information raining from Google’s senior managers. These folks instant message, MOMA post, and email 24×7, so it’s not surprising that high-powered Googlers are reading from the same page.

Mr. Rajaraman’s “How Google Measures Search Quality” is a very important essay, and I think it is a prime beef summary of what Mr. Manber will touch upon in his keynote at the Gilbane Conference on June 18th. Click here and read Mr. Rajaraman’s post. This link will also provide you with a link to the first part of Mr. Ramanathan’s write up about search quality and artificial intelligence at Google.

The key point in the Datawocky essay for me is this statement:

Google does not use such real usage data to tune their search ranking algorithm. What they really use is a blast from the past. They employ armies of “raters” who rate search results for randomly selected “panels” of queries using different ranking algorithms. These manual ratings form the gold-standard against which ranking algorithms are measured — and eventually released into service.

Google uses humans!

Let’s think about this. Google has legions of wizards. Google has a heck of a super computer. Google has more fancy math than a dozen universities. Yet Google relies on humans. I don’t know much about Google, but I know what human indexers can do; for example:

  1. Make judgments that algorithms at this time cannot match. Most companies have chopped humans out of the indexing and search analysis loop. Google is putting them in. Big news for me.
  2. Google’s use of artificial intelligence is useful but it may be paying off in areas other than search and advertising. Where does the AI deliver a hefty payload? That’s a question that warrants investigation.
  3. Ask, Microsoft, Yahoo and the other Web search engines are falling behind the GOOG in market share. If humans are Google’s secret sauce, the razzle dazzle technology from these three companies will not be able to close the gap. There and other competitors will have to have technology and the money to hire expensive, inefficient humans to make the search results better for actual users. Can these three companies invest in humans? I don’t know. But it’s clear that none of these three is able to slow down Googzilla on its march to search dominance.

Kudos to Mr. Rajaraman for getting another Google scoop. Now I won’t have to attend Mr. Manber’s lecture. I think I know what he’ll be saying. Google has a tendency to create talks and then have its top dogs “run the game plan.”

A happy quack to Techmeme.com for the link to Datawocky. I owe you one.

Stephen Arnold, June 12, 2008

Comments

3 Responses to “Prepping for Google’s Udi Manber Keynote: Another Datawocky Scoop”

  1. Daniel Tunkelang on June 12th, 2008 10:24 am

    Actually, what’s interesting is not that Google uses humans, but which humans it chooses to use. By ignoring the usage data and instead relying on how their “raters” assess benchmark queries, Google, by Norvig’s own admission, is taking a very retro approach. Unless this is just misinformation. It’s hard for me to believe that anyone with such a vast audience of real users wouldn’t make better use of it.

  2. Stephen E. Arnold on June 12th, 2008 11:33 am

    Yes, and I have heard that there are real “humans” working on this quality angle as well. In fact, the interview I posted with Mats Bjore from Silobreaker makes a very similar point about the role of humans. I posted a link to a Microsoft autonomous computing paper. I think that software’s ability to do some jobs will create some work opportunities for humans.

    Again… thanks for taking the time to post.

    Stephen Arnold, June 12, 2008 12 33 pm

  3. SearchCap: The Day In Search, June 12, 2008 on June 12th, 2008 5:31 pm

    […] Prepping for Google’s Udi Manber Keynote: Another Datawocky Scoop :, Beyond Search […]

  • Archives

  • Recent Posts

  • Meta