Bitpipe: A New Direction For Indexing

January 2003 Technology from Harrods Creek

OURCs on the Isis are a mystery to information professionals who tap keyboards and push mouse buttons. To former Oxford rowers, the Charles River is inspiration. "I do my best thinking whilst tugging at the oars," said Nick Copley, cofounder of Bitpipe.

"The States is a good place to do business," he added, "but I am concerned about getting a mouthful of the Charles' water when I row. But after a good hour on the water, I see nothing but opportunities for Bitpipe. Information technology is a quite large and dynamic subject. We have, it seems, just pushed our Cull into the water and the current is sweeping us along."

Mr. Copley paused and stared toward the Charles River, "Much less time for rowing. Bit of a mixed blessing I guess. I much prefer the Isis, of course. With the growth of Bitpipe, but I have precious little time to jet back to England. I have little time to risk the Charles either."

Bitpipe is one of the most successful of what is becoming a new segment of the online information business-a vertical content provider. (For a list of other vertical search engines, click here.) With the steady increase in the number of Web pages indexed by Google (more than three billion) and FAST Search & Retrieval (more than two billion), a one or two word query returns hundreds of potentially relevant hits.

AltaVista recently said that it would return to search, abandoning its America Online approach to information. Yahoo! purchased Inktomi's Web searching business. With these two actions, people looking for information on the Web will have even more pages to peruse.

As the large online indexing services shift to the mass market, much specialized content becomes lost. The more Overture, eSpotting, and Google emphasize links that generate revenues for these companies, the excellent document posted by a small engineering firm may never appear on the first page of hits in a major search engine. Some documents may never be indexed simply because the papers are located three or four levels deep in a public Web site.

"The surge in popularity of a few big search utilities was one factor in our creating our service," said Nick Copley, an Oxford graduate and veteran of The Thomson Corporation's electronic publishing digital mills. "A few of my colleagues had learned that white papers and special reports from large information technology consultancies were quite useful sources of information. We decided that creating a service that focused on these materials would fill a niche in the search business."

Information overload is a tired concept, and most professionals have struggled to cope with the avalanche of services that deliver the equivalent of Constance Winchell's now out of print Guide to Reference Books.

"We saw the expansion of general Web indexes as an opportunity to provide a useful service to a group of information systems and programming professionals," said Nick Copley. "We found it quite hard to get a handle on the write ups about mainstream software and the reports prepared by specialists in the information technology field. The idea came to me whilst rowing on the Charles. When an opportunity to set up a company to create a specialized collection came, I helped found Bitpipe. The growth rate and industry acceptance have been surprising to me. What's even more astounding to our accountant and solicitor is that we are profitable."

Bitpipe's database presently contains more than 40,000 documents from value-added vendor and analyst research content from over 3,500 IT vendors and over 60 leading IT analyst firms. Bitpipe built its service as an XML-based indexing and database engine that allows Bitpipe to render the white paper content in the exact look and feel of their partners' Web site. A visitor to the IT Net Central Web sites in the Philippines uses Bitpipe as though it was a service provided by IT Net Central. Visitors to the sites where IT buyers are actively conducting research access Bitpipe documents.

The reasons Bitpipe has climbed to the top of the technology vertical search engine segment comes from the tangible benefits Bitpipe's partners receive. First, Bitpipe does the work of obtaining white papers, indexing and abstracting them, and maintaining the content service. In effect, partners get a completely outsourced service with all design, implementation and maintenance handled by Bitpipe. Second, Bitpipe's support team helps sites offering its content to add context links to white papers related to the partner's users. A site focusing on Portals such as Portals Magazine offers white papers on topics of interest to those looking for information about Portals. Added Mr. Copley, "The full Bitpipe database is also available, but the customized view of the Bitpipe content adds value to our partners' Web site."

Bitpipe provides the names and contact information for anyone wanting to download a white paper. In addition, Bitpipe has electronic commerce services. These "buy online" services permit a visitor to Bitpipe or its partners' Web sites to purchase market research reports with one click. "We took a hard look at how to offer for-fee, high-value reports. We have found a formula that seems to please our users, the providers of these documents, and our accountants. We get a commission on every report that we sell. With reports from IDC, Gartner and other respected research firms, we have found a way to enrich the white paper content without alienating users, partners, or authors of no-charge documents.," noted Mr. Copley.

Mr. Copley and four former Thomson employees started Bitpipe in 1998, and the company has grown in the past 18 months to more than 40 people. Bitpipe is one of the success stories in what is called "vertical search." Vertical search is not a new phenomenon. Mr. Copley and his colleagues have rediscovered what the first index and abstract products learned decades ago. Specialized collections of information have considerable value because the organized content focuses on a specific area of interest.

"We developed our own information technology thesaurus, and we apply these keywords to each of the documents that we include in Bitpipe. Our users, which now number in the hundreds of thousands per week, tell us that when they search for information about customer relationship management or XML databases, the results are spot on," said Mr. Copley. "We originally wanted to provide a collection of technical information for systems professionals, but we have learned that our customer base includes consultants, financial professionals, students, even owners of small business looking for information about how technology can help their business."

A visit to Bitpipe's offices is a shock to the visitor expecting a go-go Dot Com operation. Bitpipe- located in downtown Boston near the waterfront-resembles the indexing and abstracting operations used to produce Engineering Information's Compendex, Thomson's Investext, and the Courier Journal's ABI/ INFORM database.

"We operate like a somewhat traditional-but certainly not an old fashioned-publisher. Each white paper and research report submitted is reviewed by a team of editors. I think we have more editors and librarians than we do managers and programmers. We ask that each white paper and report have an author-written abstract," added Mr. Copley.

"Our professionals read the white papers and reports, reject or accept them as judgment dictates, and then we revise the abstract. Once this is completed we index the document by adding four to six of our controlled vocabulary terms. We now have about 5,600 terms, and we are adding and doing the type of linking that's necessary when customer support morphs into customer relationship management and other sorts of tricky things that language does in the technology business. We have learned that our indexing is another one of our strengths. As fancy as the search engines are these days, none can duplicate the type of thought that our editorial team brings to bear. For that reason, a hit on Bitpipe is likely to be very close to the subject of the researcher's question," said Mr. Copley.

Bitpipe in the span of 18 months has emerged as the primary collection point for white papers and information technology related consultant reports. "We discovered about two years ago an interesting new business. We began getting inquiries from publishers and information companies who wanted to tap into our database of white papers. We started with one or two partners, and we have now expanded the number of affiliates offering our white papers to more than 75. If one visits the InformationWeek Web site, for example, the white papers on offer are served by us. Our other partners include EE Times, Euronet, Oracle, Telecom Asia, and many other specialized services."

The payoff for a company or expert contributing a white paper is sales leads. Bitpipe's publishing infrastructure relies upon open source software, the Ultraseek search engine now owned by Inktomi, Inc., and Extensible Markup Language. Said Mr. Copley, "What we are able to do is provide a partner or visitor to our site with a search box. A query returns a relevance-ranked list of white paper titles. A click on a title displays what we think of as one of our greatest values to the information professional-an abstract of the white paper. Scanning an abstract allows the researcher to determine if a white paper is germane to that person's specific query. Once a white paper of interest has been identified, a click displays a Web form that asks the researcher to enter his name and electronic mail address plus a few other details. This Web form is provided directly to the white paper's author. We have heard from many white paper contributors that the leads generated from white papers are among the most useful to the authors."

A tour of Bitpipe can take quite a bit of time. Each page reveals more content than the phrase white papers brings to mind. Bitpipe includes product literature from information technology companies. Said Mr. Copley, "The product specifications for software from BEA Systems, Microsoft Corporation, and IBM that perform enterprise services are quite rich. We found it logical to offer our publishing platform to these and other companies so the product information was a click away from the white papers. We are now adding training materials, and we are finding that no one has collected, indexed, and classified these materials. We have not really done anything other than listen to the customers and the producers of these documents."

The first-time visitor to the site may be wary of providing contact information for the privilege of downloading a white paper. Bitpipe protects the identity of visitors carefully, providing the name of a person viewing a paper only to the author of that paper. "We have made a firm decision to respect the privacy of the professionals visiting our site. That policy has eliminated some revenue streams, but we believe that our approach is one that will suit most systems professionals."

Bitpipe does not store the documents on its own servers. The white papers, the research materials, and the product information reside on the servers of the companies who created the information. If a company shutters its Web site, the link in Bitpipe does not work, but the abstract of the white paper is available. The Bitpipe policy ensures that technical material is available from companies that are in business. An industry analyst might find the papers of historical interest, or an attorney may find older papers useful in patent litigation. The Bitpipe site is an excellent information resource, and its formula for monetizing content will be a useful model for many online professionals.

What's the outlook for vertical search engines in an online world dominated by a handful of indexing utilities? Significant. The need for trusted sources of information is rising. Hundreds of thousands of hits and boasting about billions of Web pages satisfies one sector of the information community. For professionals, the use of editorial policy, subject experts, and focused indexing provides a way to get on point information quickly. Is the future more Googles and Fasts? Is the future most vertical and niche search tools? Bitpipe has bet on the latter, which certainly is safer than gulping mouthfuls of Charles River water.

www.bitpipe.com

www.arnoldit.com/lists/verticalsearch.html

Stephen E. Arnold
President, Arnold Information Technology

Postal Box 320
Harrod's Creek, Kentucky 40027
Voice: 502 228 1966
E mail: sa@arnoldit.com
Web site: www.arnoldit.com