Why Search Does Not Change Too Much: Tech Debt Is a Partial Answer
August 12, 2016
I read “The Human Cost of Tech Debt.” The write up picks up the theme about the amount of money needed to remediate engineering mistakes, bugs, and short cuts. The cost of keeping an original system in step with newer market entrants’ products adds another burden.
The write up is interesting and includes some original art. Even though the art is good, the information presented is better; for example:
For a manager, a code base high in technical debt means that feature delivery slows to a crawl, which creates a lot of frustration and awkward moments in conversation about business capability. For a developer, this frustration is even more acute. Nobody likes working with a significant handicap and being unproductive day after day, and that is exactly what this sort of codebase means for developers. Each day they go to the office knowing that it’s going to take the better part of a day to do something simple like add a checkbox to a form. They know that they’re going to have to manufacture endless explanations for why seemingly simple things take them a long time. When new developers are hired or consultants brought in, they know that they’re going to have to face confused looks, followed by those newbies trying to hide mild contempt.
My interest is search and content processing. I asked myself, “Why are search and retrieval systems better than they were in 1975. When I queried the RECON system, I was able to find specific documents which contained information matching the terms in my query. Four decades ago, I could generate a useful result set. The bummer was that the information appeared on weird thermal printer paper. But I usually found the answer to my question in a fraction of the time required for me to run a query on my Windows machine or my Mac.
What’s up?
My view is that search and retrieval tends to be a recycling business. The same basic systems and methods are used again and again. The innovations are wrappers. But to make search more user friendly, add ons look at a user’s query history and behind the scenes filter the results to match the history.
The shift to mobile has been translated to providing results that other people have found useful. Want a pizza? You can find one, but if you want Cuban food in Washington, DC, you may find that the mapping service does not include a popular restaurant for reasons which may be related to advertising expenditures.
We ran a series of queries across five Dark Web search and retrieval systems. None of the systems delivered high precision and high recall results. In order to find certain large sites, manual review and one-at-a-time clicking and review were needed to locate what we were querying.
Regular Web or Dark Web. Online search has discarded useful AND, OR, NOT functions, date and time stamps, and any concern about revealing editorial or filtering postures to a user.
Technological debt explains that most search outfits lack the money to deliver a Class A solution. What about the outfits with oodles of dough and plenty of programmers? The desire and need to improve search is not a management priority.
Some vendors mobile search operates from a vendor’s copy of the indexed sites. Easy, computationally less expensive, and good enough.
Tech debt is a partial explanation for the sad state of online search at this time.
Stephen E Arnold, August 12, 2016
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
