Autonomy: Leading the Push Beyond Enterprise Search
January 30, 2015
In “CyberOSINT: Next Generation Information Access,” I describe Autonomy’s math-first approach to content processing. The reason is that after the veil of secrecy was lifted with regard to the signal processing`methods used for British intelligence tasks, Cambridge University became one of the hot beds for the use of Bayesian, LaPlacian, and Markov methods. These numerical recipes proved to be both important and controversial. Instead of relying on manual methods, humans selected training sets, tuned the thresholds, and then turned the smart software loose. Math is not required to understand what Autonomy packaged for commercial use: Signal processing separated noise in a channel and allowed software to process the important bits. Thank you, Claude Shannon and the good Reverend Bayes.
What did Autonomy receive for this breakthrough? Not much but the company did generate more than $600 million in revenues about 10 years after opening for business. As far as I know, no other content processing vendor has reached this revenue target. Endeca, for the sake of comparison, flat lined at about $130 million in the year that Oracle bought the Guided Navigation outfit for about $1.0 billion.
For one thing the British company BAE (British Aerospace Engineering) licensed the Autonomy system and began to refine its automated collection, analysis, and report systems. So what? The UK became by the late 1990s the de facto leader in automated content activities. Was BAE the only smart outfit in the late 1990s? Nope, there were other outfits who realized the value of the Autonomy approach. Examples range from US government entities to little known outfits like the Wynyard Group.
In the CyberOSINT volume, you can get more detail about why Autonomy was important in the late 1990s, including the name of the university8 professor who encouraged Mike Lynch to make contributions that have had a profound impact on intelligence activities. For color, let me mention an anecdote that is not in the 176 page volume. Please, keep in mind that Autonomy was, like i2 (another Cambridge University spawned outfit) a client prior to my retirement.) IBM owns i2 and i2 is profiled in CyberOSINT in Chapter 5, “CyberOSINT Vendors.” I would point out that more than two thirds of the monograph contains information that is either not widely available or not available via a routine Bing, Google, or Yandex query. For example, Autonomy does not make publicly available a list of its patent documents. These contain specific information about how to think about cyber OSINT and moving beyond keyword search.
Some Color: A Conversation with a Faux Expert
In 2003 I had a conversation with a fellow who was an “expert” in content management, a discipline that is essentially a step child of database technology. I want to mention this person by name, but I will avoid the inevitable letter from his attorney rattling a saber over my head. This person publishes reports, engages in litigation with his partners, kowtows to various faux trade groups, and tries to keep secret his history as a webmaster with some Stone Age skills.
Not surprisingly this canny individual had little good to say about Autonomy. The information I provided about the Lynch technology, its applications, and its importance in next generation search were dismissed with a comment I will not forget, “Autonomy is a pile of crap.”
Okay, that’s an informed opinion for a clueless person pumping baloney about the value of content management as a separate technical field. Yikes.
In terms of enterprise search, Autonomy’s competitors criticized Lynch’s approach. Instead of a keyword search utility that was supposed to “unlock” content, Autonomy delivered a framework. The framework operated in an automated manner and could deliver keyword search, point and click access like the Endeca system, and more sophisticated operations associated with today’s most robust cyber OSINT solutions. Enterprise search remains stuck in the STAIRS III and RECON era. Autonomy was the embodiment of the leap from putting the burden of finding on humans to shifting the load to smart software.
A diagram from Autonomy’s patents filed in 2001. What’s interesting is that this patent cites an invention by Dr. Liz Liddy with whom the ArnoldIT team worked in the late 1990s. A number of content experts understood the value of automated methods, but Autonomy was the company able to commercialize and build a business on technology that was not widely known 15 years ago. Some universities did not teach Bayesian and related methods because these were tainted by humans who used judgments to set certain thresholds. See US 6,668,256. There are more than 100 Autonomy patent documents. How many of the experts at IDC, Forrester, Gartner, et al have actually located the documents, downloaded them, and reviewed the systems, methods, and claims? I would suggest a tiny percentage of the “experts.” Patent documents are not what English majors are expected to read.”
That’s important and little appreciated by the mid tier outfits’ experts working for IDC (yo, Dave Schubmehl, are you ramping up to recycle the NGIA angle yet?) Forrester (one of whose search experts told me at a MarkLogic event that new hires for search were told to read the information on my ArnoldIT.com Web site like that was a good thing for me), Gartner Group (the conference and content marketing outfit), Ovum (the UK counterpart to Gartner), and dozens of other outfits who understand search in terms of selling received wisdom, not insight or hands on facts.
Autonomy has been the outfit that has been misunderstood for decades. The sad state of enterprise search is in part due to the misapprehension of what Dr. Lynch did, packaged, and sold.
The Importance of Autonomy
My view of Autonomy is that it was the first company to make automated collection, automated content processing, and automated outputs available to governments and commercial entities. In my book, that is a first that is as important as Ron Sacks-Davis’ groundbreaking work on what today is the TeraText system or Dr. Jon Kleinberg’s CLEVER system which underpins some of the conceptual plumbing of Google’s PageRank.
Yep, Autonomy is a big deal.
In CyberOSINT, I explain what Autonomy “invented,” patented, and made available as a product. If you think Autonomy is a knock off enterprise search system, you will be surprised to learn that your perception is essentially wrong. Autonomy’s “system” foreshadowed the NGIA systems profiled in the CyberOSINT monograph.
In the monograph, I simply ignore the thrashing of Hewlett Packard. HP is not the outfit to extract value from the Lynch inventions. HP engineers can use the system, but Autonomy requires the guidance of Dr. Lynch. I am aware of HP’s desire to sue Dr. Lynch, but my view is that HP would have a better off finding a way to leverage Dr. Lynch and his inventions.
Not surprisingly, Autonomy may not be the player it was prior to its acquisition by HP. That is a loss to the intelligence community. No doubt the content management expert will point to the embarrassing situation HP has triggered as proof that Autonomy was a loser.
Wrong. The losers are in this order: HP, the CMS failed webmaster, and the US and its allies intelligence communities. Hopefully DarkTrace will fill the gap.
If you find this story interesting and germane to your intelligence requirements, you will want to get your copy of CyberOSINT and curl up with Chapter 3: Autonomy: A New Trail.
Based on the research performed by me and my team, Autonomy can be considered the catalyst for the efflorescence of cyber OSINT.
Stephen E Arnold, January 30, 2015
Comments
2 Responses to “Autonomy: Leading the Push Beyond Enterprise Search”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
What interests me about the Autonomy patent collection is the number of patents in which it is a co-applicant, such as those from Iron Mountain with Phil Notick (now at D&B) as a co-inventor.
I think many people may have missed ( I don’t see it in your coverage) that last year Michael Lynch was elected a Fellow of the Royal Society, the most prestigious scientific society in the world. He is one of very few business men to be elected FRS and it does show the respect the research community has for the work that he carried out and successfully commercialised. https://royalsociety.org/people/fellowship/2014/michael-lynch/
you’re embarrassing yourself with shilling of Autonomy so give it a freaking rest cowboy