Search Waves: Are We Living through Periodicity?

March 26, 2008

I’m fascinated with cyclical phenomena. When working on my graduate degree, I accepted a grant from Duquesne University in 1967. Located in Pittsburgh, Pennsylvania, this Jesuit university was little-c catholic. All faiths were acceptable. One of my professors and friends, Dr. Richard Oehling told me, “Where else could an orthodox Jew teach the Protestant reformation to a group of Jesuits?” Such was Duquesne.

In one course, I confronted phenomenological existentialism, then a hot concept in philosophy. Although I was busy indexing sermons in Latin using ancient mainframes, the fuzzy-wuzzy world of existential philosophy caught my attention. I had zero clue about epistemology, heuristics, and other concepts that whipped serious students of different beliefs and backgrounds into a frenzy. This philosophical banter was better at stirring up emotions than a break down on the Squirrel Hill bus.

So, what’s this got to do with search?

I’m not sure, but there’s a thesis-antithesis-synthesis dialectic rippling the fabric of acquisitions, start ups, and “old wine in new bottles” innovations I read about in news releases. Just this morning, my Google Alert service informed me of “A New Wave of Enterprise Search”. The essay appeared in the CMSwatch Trendwatch blog. The key sentence to me was, “There’s a growing movement afoot to de-throne the old guard; talk of replacing FAST and Autonomy seemed to be uttered by every vendor that wasn’t a household name.”

Let’s step back. In 2003, I delivered a talk to some addled soul who paid me to brief their Board of Directors. I fail miserably at these events. I don’t golf. I don’t follow sports. I don’t drink. I don’t do “herd thinking”, which seems to be the nature of these sheep-like affairs.

I did prepare a chart, which appears below. I apologize in advance for the sorry state of the graphic, but I had only a hard copy version which I scanned for this essay.

threestagesinsearch

Let’s walk through this five-year-old analysis. Keep in mind the Board of Directors had zero idea what I was talking about. I tried to put training wheels on my discussion, but I failed. If you don’t follow my analytic line, no problem. You qualify for a senior management position and will soon be on the Board of Directors of a $500 million information company. I am a goose, right? Look at the logo to remind yourself.

On the y-axis or what I called the up and down line for the Board of Directors, I listed five characteristics of our information environment. You are probably familiar with each of these concepts. In order to find this essay, you have to know something about online, which is–I might add–not a pre requisite for occupying a compensated seat on a public company’s Board of Directors.

Then across the x-axis or what I called the left-right line, are three time slices. When I’m giving a for-fee lecture, I use a $5 word like epochs. For our purposes, these are intervals.. We will now look at what’s happing in each of these 20-year slices.

Big Iron

In the first bucket, we have the mainframe era. Computing worked pretty well in the mainframe era in my opinion. In order to “do” any computing, you had to know things like programming in such mysteries as COBOL, and everyone’s favorite FORTRAN, among others. Life was good if you worked in a computer center. In college, you could get a date for coffee by trading a priority process for a person of interest. The world of computing was split in them and us.

Believe it or not, search in the mainframe environment worked quite well. There were three reasons:

  1. Data were structured. Queries were parametric or Boolean. As long as you got the syntax right, you got results. Zero hits meant there was no information in the system matching your query. To find information, you then had to create it after double checking your search string.
  2. Users knew the ropes. Today, online has become the Wal*Mart of research. Not in the 1964-1984 period. Even when PCs poked their nose from the hot house, early PC adopters were willing to figure out how to make these gizmos work. Because there weren’t many users, you could get actual technical support. Imagine calling a company today and getting the president on the phone and having a 90 percent chance of getting accurate technical information. Today, you get a voice mail and the president is an attorney or an accountant. Technical professionals may reside in another country where schools still teach math and science to motivated individuals.
  3. Systems, though constrained, worked. Say what you will about the overall clue-less-ness of IBM. But in the mainframe hay day, those behemoths were reliable once set up and properly resourced. Today, if you click on a link on a Web 2.0 site you can have some interesting experiences. These range from losing control of your cursor to visiting a site that would make a lumberjack blush.

The PC Era

I admit that I have squeezed the PC and the client-server era together. I see the client-server revolution as opening the door to the PC. Then, when the PCs got enough horsepower, the PCs usurped the mini-computers used for client-server computing. (Keep in mind that we are looking at periodicity here. This is a sub cycle of the form client-server > PC > PC-based client server.

You know this era quite well. Most people have computer experience. Grade schools in affluent suburbs had computer labs in the mid-1980s. Today, pre-schools in Louisville, Kentucky, may have a PC tucked in a corner. The PC era fueled user’s discovery of the Internet. The Internet is decades old, but it was known to computer nerds and researchers. When more people gained access to the PC, the Internet become a “new” service to millions of people. The “information superhighway” was invented by parvenus who saw a way to deliver new services and maybe make money.

The key point is that the PC era started out as a product for the wealthy and highly educated. By 2004, convicts had access to PCs so they could work on their appeals. Now that’s progress–shifting pro bono work to people who have time on their hands and a PC.

What about search? Do you remember Gopher. A company called Microlytics had a blazing fast file search system? Mike Weiner, the president, is in a different business, but he deserves a pat on the back for showing Google and Microsoft how a desktop search utility should work–fast, unobtrusively, and with a miniscule memory footprint. Gopher was a pygmy compared to the NFL-scale systems today.

PCs could also “dial into” a timesharing network and use the new fangled bulletin board systems. A tip of the hat to Ward Christensen, who arguably invented the BBS. You could also connect via Tymnet (anyone remember that service?) and access mainframe search systems. There were vendors who would sell you a behind-the-firewall system for your IBM mainframe too. IBM offered STAIRS. Bibliographic Retrieval Services licensed BRS Search. SDC Corporation offered the ORBIT system, and I recall the first system to offer forward truncation. I believe it was InQuire, not to be confused with InQuira, an amalgam of three separate search systems.

These systems worked, but the first signs of the challenges search posed were evident in this time slice. First, a person looking for information could not run one query on one system and get results from multiple content repositories. Today, a user can “sorta”–this is a technical term–do this. But federated search, while useful, operates within some constraints. Second, search was expensive. A person trying to install STAIRS had to meet the first requirement; you needed a mainframe, a staff, and a big air conditioning system. Then you move on to the more interesting parts of deploying a mainframe system; namely, figuring out how to get the moving parts working harmoniously. Third, user expectations soared. In the early days of time slice one, it was a miracle to get useful results from a specialized information file like ABI/INFORM. In the PC era, I was told by one executive, “I want to walk in my office and have my PC tell me what I need to know today.” Nice thought, but this MBA confused Star Trek effects with the PC.

The Network Slice

The third time slice is our era: pervasive connectivity if you have money. I don’t need to tell you much about this. Several examples will remind you of our present computing environment.

  1. Mainframes are still with us. Think air traffic control systems, financial institutions and their wonderful online banking features, and changing a ticket at the airport. Obviously big iron thrives, and it can help or hinder our lives just as it did in the key punch programming era.
  2. The “crackberry”. Psychiatrists are recommending that MBAs go on vacation without their “push” smartphones. Yeah, right. The reason the psychiatrists know about wacko MBAs and their addition to persistent connectivity is MBAs ability to pay psychiatrists to advise them on relaxing.
  3. A 10-year-old with a mobile phone. Mom says, “I know how to reach her.” When I was 10 my parents knew how to reach me by opening the back door and looking in the yard or yelling, “Stephen, where are you?”

Search is particularly interesting in this time slice representing 2004 to 2024. This is where I lost touch with the Board of Directors paying me to share my thoughts about the future. Search is pretty screwed up. Most “free” Web search systems work well from the average user’s point of view. A professional informationalist can talk for hours about how lousy these systems are. The behind-the-firewall search sector is a mess as well. There are many vendors. I include in my eContent keynote a list of 150, and I put this list in an update of my 1999-2000 convergence papers for Searcher Magazine to publish in April or May 2008. Desktop search is an exercise in frustration. Email, attachments, and document versions befuddle my father who is in his 80s. I have a heck of a time finding the exact PowerPoint deck I want because I recycle the same old ideas over the decades.

That’s why I make it clear that search and retrieval is in a crisis. A shake out is coming. I think search will, like Nero, let a servant, kill it. We’re entering a new era, and it isn’t going to be the search of time slice one. Sure, key word and Boolean systems will remain. Clay tablets are available in flea markets today, and a Babylonian teleported to our era would recognize them as documents straight away. A new era is taking shape, and I talk about this in my new study Beyond Search at some length.

The Periodicity

The periodicity is that the era of pervasive connectivity and the network model brings us back to the mainframe approach to computing. There are differences, of course. These are:

  1. Today’s mainframe is chopped up and distributed. In the good old days, the online system was in one place, and it was for all practical purposes impossible to dis aggregated, synchronized the data, and pay the communications bills. Centralization has returned but in a distributed, massively parallel variant.
  2. Users don’t want a big, hot, expensive computing device in their pocket. Forget the Apple Air. Forget the fat candy bar Treos. The computer is one that is small, elegant, and smart. The iPhone is a precursor to the next generation of computing devices.
  3. Data and applications are in the cloud. Big companies are slow to change, so don’t look for the addled Ford Motor Co. to shift to Amazon’s S3 and EC2 services. But individuals and start ups are willing to use applications that are “out there.” The “out there”, of course, are on today’s version of the mainframe.

In summary, we have many computing paradigms available. We are watching the emergence of a new type of centralized computing. I don’t have a snappy phrase yet, so I am going to use the awkward term of dis-aggregated mainframe to describe the computing era now being built for you and me.

Displacement is a feature of the broader sweep of the information environment. Indeed, there is no stasis. Change is what makes computing work. The larger issue, then, is the awareness that we are living in a period of considerable innovation and tension in terms of computing and information retrieval.

What’s Next?

If periodicity exists in computing, will a variant of the PC era emerge? I think it will. I’m watching devices that auto sync with one another. I also think that young people with studs inserted in their eyes and noses will be among the first to have a PC built into their bodies. These will connect with other devices so that the storage, flexibility, and freedom of the PC becomes available.

What type of search-and-retrieval systems do we need for this new computing and information environment? I’m not sure a lower cost old-line search system will do the job. An expensive search system may be displaced by a cheaper, similarly provisioned system. But this type of change is almost irrelevant to the opportunity awaiting innovators who engineer for this third time slice. One approach is putting search into amber. The more interesting approach is to invent a better way.

Does computing periodicity exist? Let me know.

Stephen Arnold, March 25, 2008

Comments

Comments are closed.

  • Archives

  • Recent Posts

  • Meta