Technical Debt: Paying Down Despite Disaster Waiting in the Wings
June 7, 2021
Some interesting ideas appear in “10 Ways to Prevent and Manage Technical Debt—Tips from Developers.” The listicle is not particularized on a specific application or service. Let me convert a few of the points in the article to the challenges that vendors of information retrieval software have to meet in a successful manner.
I am not tracking innovations in search the way I did when I wrote the first three editions of the Enterprise Search Report many years ago. Search technology, despite the hooting of marketers and “innovators” who don’t know much about the 50 year plus history of finding, search has not made much progress. In fact, if I were still giving talks at search-related events, I would present data showing that “findability” has regressed. Now to the matter at hand.
I am not sure most people understand what technical debt is in general and even fewer apply the concept to search and retrieval. To keep it simple, technical debt is not repairing and servicing your auto. You do just enough to keep the Nash Rambler on the road. Then it dies. You find that parts are tough to find and expensive to get. If you want to do the job “right,” you will find that specialists are on hand to make that hunk of junk gleam. Get out your checkbook and write small. You will be filling in some big numbers. Search is that Nash Rambler but you have a couple of Metropolitans and a junker of a 1951 Nash Ambassador sitting in your data center. You can get stuff from A to B, but each trip becomes more agonizing. Then you have to spend.
Technical debt is the amount you have to spend to get back up and running plus the lost revenue or estimated opportunity cost. These numbers are the cost of the hardware, software, knick knacks, and humans who sort of know what to do.
What about search? Let’s take three of the items identified in the article and consider them in terms of what is often incorrectly described as “enterprise search.” My work over the years has documented the fact that there is no enterprise search. Shocking? Think about it. Employees cannot find the video of that Zoom meeting or the transcript automagically prepared this morning. And that sales presentation with the new pricing? Oh, right, that’s on the VP’s laptop and it won’t connect to the cloud archiving system because the wizard executive has trouble opening a hotel room with the keycard. Like I said, “Wizard.”
Item number 2 in the article is “Embed technical debt management into the company culture.” Ho ho ho. The present state of play is to get something up and running, dump on features, and generate revenue, some revenue, any revenue. In many organizations, the pressure to move the needle trumps any weird ideas to go back and fix the plumbing. How often is the core of Google’s search and retrieval reworked? Yeah, not often and every year the job becomes less and less desirable. The legions of Xooglers who worked on the system are unlikely to return to the digital Disneyland to do this work even for dump trucks filled with Ethereum.
Item number 5 is “Make technical debt a priority in open source culture.” Okay, let’s think about open source search. Have you through about Sphinx recently. What about Xapian? The big dogs are under intense pressure from the real champions of open source like Amazon and everyone’s favorite security company Microsoft. The individuals who do the bulk of the work struggle to make the darned thing work on the latest and greatest platforms and operating systems. The more outfits like Amazon pressure Elastic, the less likely the humans who work on Lucene and Solr will be able to fend off complete commercialization. Hey, there’s always consulting work or a job at IBM, another cheerleader for open source. So priority? Right.
Now item number 6 in the article. It is “Choose a flexible architecture.” What does this mean for search and retrieval. Most search and search centric applications like policeware and intelware are mashups of open source, legacy code left over from another project, and intern-infused scripts. The “architecture” is whatever was easiest and most financially acceptable. Once those initial decisions are made or simply allowed to happen because someone knew someone, the systems are unlikely to change. Fixing up something that sort of works is similar to the stars of VanWives repairing their ageing vehicle while driving in the rain. Ain’t gonna happen.
Net net: Technical debt for most organizations is what will bring down the company. Innovations slows to a crawl and becomes a series of add ons, wrappers, and strapping tape patches. Then boom. A competitor has blown the doors off the incumbent, customers just cancel contracts for enterprise search systems, or the once valued function becomes a feature for a more important application. Technical debt, like a college grad’s student loan, is a stress inducer. Stress can shorten one’s life and kill. The enterprise search market is littered with the corpses of outfits terminated from technical debt denial syndrome.
Stephen E Arnold, June 7, 2021
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
