Browse > Home / Analytics, Feature, Text processing / Autumn Approaches: Time for Realism about Search

Autumn Approaches: Time for Realism about Search

September 1, 2014

Last week I had a conversation with a publisher who has a keen interest in software that “knows” what content means. Armed with that knowledge, a system can then answer questions.

The conversation was interesting. I mentioned my presentations for law enforcement and intelligence professionals about the limitations of modern and computationally expensive systems.

Several points crystallized in my mind. One of these is addressed, in part, in a diagram created by a person interested in machine learning methods. Here’s the diagram created by SciKit:

image

The diagram is designed to help a developer select from different methods of performing estimation operations. The author states:

Often the hardest part of solving a machine learning problem can be finding the right estimator for the job. Different estimators are better suited for different types of data and different problems. The flowchart below is designed to give users a bit of a rough guide on how to approach problems with regard to which estimators to try on your data.

First, notice that there is a selection process for choosing a particular numerical recipe. Now who determines which recipe is the right one? The answer is the coding chef. A human exercises judgment about a particular sequence of operation that will be used to fuel machine learning. Is that sequence of actions the best one, the expedient one, or the one that seems to work for the test data? The answer to these questions determines a key threshold for the resulting “learning system.” Stated another way, “Does the person licensing the system know if the numerical recipe is the most appropriate for the licensee’s data?” Nah. Does a mid tier consulting firm like Gartner, IDC, or Forrester dig into this plumbing? Nah. Does it matter? Oh, yeah. As I point out in my lectures, the “accuracy” of a system’s output depends on this type of plumbing decision. Unlike a backed up drain, flaws in smart systems may never be discerned. For certain operational decisions, financial shortfalls or the loss of an operation team in a war theater can be attributed to one of many variables. As decision makers chase the Silver Bullet of smart, thinking software, who really questions the output in a slick graphic? In my experience, darned few people. That includes cheerleaders for smart software, azure chip consultants, and former middle school teachers looking for a job as a search consultant.

Second, notice the reference to a “rough guide.” The real guide is understanding of how specific numerical recipes work on a set of data that allegedly represents what the system will process when operational. Furthermore, there are plenty of mathematical methods available. The problem is that some of the more interesting procedures lead to increased computational cost. In a worst case, the more interesting procedures cannot be computed on available resources. Some developers know about N=NP and Big O. Others know to use the same nine or ten mathematical procedures taught in computer science classes. After all, why worry about math based on mereology if the machine resources cannot handle the computations within time and budget parameters? This means that most modern systems are based on a set of procedures that are computationally affordable, familiar, and convenient. Does this similar of procedures matter? Yep. The generally squirrely outputs from many very popular systems are perceived as completely reliable. Unfortunately, the systems are performing within a narrow range of statistical confidence. Stated in a more harsh way, the outputs are just not particularly helpful.

In my conversation with the publisher, I asked several questions:

  1. Is there a smart system like Watson that you would rely upon to treat your teenaged daughter’s cancer? Or, would you prefer the human specialist at the Mayo Clinic or comparable institution?
  2. Is there a smart system that you want directing your only son in an operational mission in a conflict in a city under ISIS control? Or, would you prefer the human-guided decision near the theater about the mission?
  3. Is there a smart system you want managing your retirement funds in today’s uncertain economy? Or, would you prefer the recommendations of a certified financial planner relying on a variety of inputs, including analyses from specialists in whom your analyst has confidence?

When I asked these questions, the publisher looked uncomfortable. The reason is that the massive hyperbole and marketing craziness about fancy new systems creates what I call the Star Trek phenomenon. People watch Captain Kirk talking to devices, transporting himself from danger, and traveling between far flung galaxies. Because a mobile phone performs some of the functions of the fictional communicator, it sure seems as if many other flashy sci-fi services should be available.

Well, this Star Trek phenomenon does help direct some research. But in terms of products that can be used in high risk environments, the sci-fi remains a fiction.

Believing and expecting are different from working with products that are limited by computational resources, expertise, and informed understanding of key factors.

Humans, particularly those who need money to pay the mortgage, ignore reality. The objective is to close a deal. When it comes to information retrieval and content processing, today’s systems are marginally better than those available five or ten years ago. In some cases, today’s systems are less useful.

Examples of gaps in content processing are easy—almost too easy—to enumerate. What is your view of these items? Do these issues exist or am I lost in the mine drainage pond in Harrod’s Creek, Kentucky:

  1. Image and video search. How easy is it to locate a specific video and the item of information in that video? Example: the maintenance environment for an SU 27 fighter? The answer is on YouTube. Now, try to find it.
  2. Outputs from a predictive system that deliver consistent stock market payoffs? Example: which of the predictive systems are able to generate big payoffs from their “intelligent software”? If these fancy math solutions worked, why are the vendors of the systems selling software. Isn’t life easier just sitting back and watching the bank account grow from winning stock bets?
  3. Medical diagnoses. Which of the many smart systems crunching Medline and other scientific-technical-medical data have cracked the Ebola variants’ with a new compound? Example: if you contracted a variant of Ebola, do you want today’s smart “answer machines” treating you?

Net net: Incremental advances in information and content processing do take place. In my experience, the rate of change is glacial. Systems marketed in the 1980s like TeraText and ISYS Search are still in use and performing as well as fancy new solutions from Palantir and SRCH2. Why? There are computational barriers and knowledge barriers that cannot be surmounted with the tools now available. A fancy new chip is useless until there are debugging and programming tools. The new system built with the chips require more software and engineering tools. Eventually a body of experience emerges that makes some portions of cutting edge research usable in commercial systems.

Star Trek is entertainment, not a window into today’s computers, software, or applications. Autumn may bring the withering of some outfits selling dreams and promises. Reality just is.

Stephen E Arnold, September 1, 2014

Written by Stephen E. Arnold · Filed Under Analytics, Feature, Text processing 

Comments

Comments are closed.

  • Archives

  • Recent Posts

  • Meta