Microsoft, the US Treasury, and Search
December 9, 2010
The new Microsoft-based Treasury.gov Web site works pretty well. Pictures flash, the links work, and the lay out is reasonably clear. There is the normal challenge of government jargon. So “Help, I am going to lose my home” becomes “Homeowner’s HOPE Hotline”.
I am interested in search and retrieval. I wanted to run through my preliminary impressions of the search interface, system responsiveness, and the relevance of the queries. I look at public facing search services differently from most people’s angle of attack. Spare me direct complaints via email. Just put your criticisms, cautions, and comments in the form provided at the foot of this Web page.
Search Interface
The basic search box is in the top right hand corner of the splash page. No problem, and when I navigate to other pages in the Web site, the search box stays put. However, when I click on some links I am whisked outside of the Treasury.gov site and the shift is problematic. No search box on some pages. Here’s an example: http://www.makinghomeaffordable.gov/index.html. Remember my example from the HOPE Hotline reference? Well, that query did not surface content gold on Treasury.gov. I went somewhere else, and I was confused. This probably is a problem peculiar to me, but I found it disconcerting.
Other queries I ran a query for “Treasury Hunt,” a service that allows me to determine if a former Arnold left money or “issues” for me. Here’s the result screen for the query “Treasury Hunt”:
The first hit in the result list points to this page:
The problem is that the hot link from this page points to this Web site, which I could not locate in the results list.
Several observations:
First, the response time for the system was sluggish, probably two seconds, which was longer than Google’s response time. No big deal, just saying “slower.”
Second, the results list did not return the expected hit. For most people, this makes zero difference. For me, I found the lack of matching hits to explicit links interesting. In fact, I assumed that the results list would have the TreasuryDirect hit at the top of the results list. Not wrong, just not what I expected.
Third, the search results interface has lots of buttons and check boxes. I know that user experience is a huge deal in Microsoft’s world. For me, I prefer a more streamlined presentation. Who knows what the date limit should be. That data point emerges once I have reviewed some results. Also the result type is interesting, and the number of documents by file type is interesting, but an advanced search feature like file type is probably going to be ignored by most users. The idea of having an “advanced” button for the two or three percent on online users who want more than the default list is obviously not what the contractors implemented. The check box for different collections or corpuses was interesting, but I have zero idea what the difference is between a resource center and a bureau. Why not list in a drop down “related sites”.
Microsoft’s implementation of search is likely to become widespread. I don’t think it will help users locate the information either better or worse than a Google-style interface. More problematic is the relevance ranking of the results. As I said, the explicit link to TreasuryDirect was not in the results list. That link appeared in the footer to each page.
Enough said. Work to be done on response time, relevance, and hit boosting. If the new Dialog interface implements this type of relevance ranking and functionality, I think there may be some push back from professional online researchers. Advanced is not sufficiently advanced and basic is not delivering what most users want: The “I’m feeling lucky type of one click and good enough” experience.
System Responsiveness
In my experience, US government Web sites are not really high traffic destinations. At tax time, the IRS gets traffic but once the forms have been downloaded and the filings are in, traffic slumps. I noted that basic query response was slower than Google’s Uncle Sam service. Try Google’s government index yourself at http://www.google.com/unclesam.
More problematic for me was the sluggish page display when I wanted to click through pages 2 and following of the results. The system kept me waiting about 10 seconds. Work is needed to improve results display right now, not at the next budget cycle. The statement of work should have a metric, and I trust that the SOW writers stipulated a response time comparable to Google’s or Exalead’s.
There is one hassle in system responsiveness related to user interface. Instead of having the “next page of results” function at the bottom of the results list, the next links are at the top of the results page. This adds time and a burr under my saddle. I expect the next links at the bottom of the results page because I scan results from the top down, not from the bottom up.
Maybe Microsoft has this backwards as it does with some of the odd functions in Microsoft Word’s auto format service.
My concern is that Fast Search & Transfer’s Web search service AllTheWeb.com used to be quite snappy. The Microsoft implementation is demonstrating what I would characterize as performance issues. That’s a factor that needs to be monitored on the Treasury.gov Web site.
Relevance
I want to focus on relevance, digging more deeply than in my earlier reference to the “expected hit”. What happens when I don’t know exactly what I want? I ran a number of queries, and I noted several interesting quirks in the system. I am going to focus on one query and then highlight the points I noted. Keep in mind that your search mileage may vary as you zoom down the Treasury.gov speedway.
Treasury has quite a few units. I wanted information on the damage $100 bills that are stored in a warehouse until the wrinkled and non wrinkled bills can be sorted out. I ran this query from the splash page at www.treasury.gov. Here is my test query: “defective $100 bill”. Here’s the result list I saw on December 8, 2010:
The first hit dates from 2004 and does not address the bound phrase. I could live with a null set. The defective bill story made headlines a few days ago. See “Billion $100 Bills Being Checked for Creases before Being Issued.” The second hit points to the splash page at www.treasury.gov. None of the hits is relevant to the query.
Now for a student looking for information about this issue for a sixth grade homework assignment, the relevance of the Microsoft Fast system is going to produce a frustrated child.
What happens if I run the query on Google’s Uncle Sam service? The results list generates information about defective bills, but there is zero information in the index about the billion defective $100 bills. This seems to be a problem, not with Google’s relevance, but with the US government’s making information about a timely issue available.
What happens if I search for a historical matter. My second query was “great depression.” The system chugged away for five seconds and displayed this result set:
The hits did include references to current documents that included the phrase “great depression”. But none of the documents were about the great depression. Years ago, I poked around several US Treasury libraries and I recall seeing some interesting studies about this difficult time in US economic history. Not in the Treasury index if my search results were accurate. I did not grind through the 190 results. As I loaded increased deep sets of results, the page display time became an issue. I needed faster access.
My conclusions about relevance:
- Work is needed on indexing
- Expected content is not in the digital corpus. Perhaps a statement of what’s indexed would be helpful
- The relevance problems noted elsewhere in this write up were not unique. Relevance is an issue.
Wrap Up
I don’t think the Treasury staff working on this project are at fault. I think the issue is exactly the same one that created some push back for Fast Search & Transfer prior to the Microsoft acquisition of the company.
There’s been some cosmetic work done, but not enough effort has been put into plumbing. Just my opinion.
Stephen E Arnold, December 9, 2010
Freebie
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
