Interview with Stephen E Arnold, Reveals Insights about Content Processing
March 22, 2016
Nikola Danaylov of the Singularity Weblog interviewed technology and financial analyst Stephen E. Arnold on the latest episode of his podcast, Singularity 1 on 1. The interview, Stephen E. Arnold on Search Engines and Intelligence Gathering, offers thought-provoking ideas on important topics related to sectors — such as intelligence, enterprise search, and financial — which use indexing and content processing methods Arnold has worked with for over 50 years.
Arnold attributes the origins of his interest in technology to a programming challenge he sought and accepted from a computer science professor, outside of the realm of his college major of English. His focus on creating actionable software and his affinity for problem-solving of any nature led him to leave PhD work for a job with Halliburton Nuclear. His career includes employment at Booz, Allen & Hamilton, the Courier Journal & Louisville Times, and Ziff Communications, before starting ArnoldIT.com strategic information services in 1991. He co-founded and sold a search system to Lycos, Inc., worked with numerous organizations including several intelligence and enforcement organizations such as US Senate Police and General Services Administration, and authored seven books and monographs on search related topics.
With a continued emphasis on search technologies, Arnold began his blog, Beyond Search, in 2008 aiming to provide an independent source of “information about what I think are problems or misstatements related to online search and content processing.” Speaking to the relevance of the blog to his current interest in the intelligence sector of search, he asserts:
“Finding information is the core of the intelligence process. It’s absolutely essential to understand answering questions on point and so someone can do the job and that’s been the theme of Beyond Search.”
As Danaylov notes, the concept of search encompasses several areas where information discovery is key for one audience or another, whether counter-terrorism, commercial, or other purposes. Arnold agrees,
“It’s exactly the same as what the professor wanted to do in 1962. He had a collection of Latin sermons. The only way to find anything was to look at sermons on microfilm. Whether it is cell phone intercepts, geospatial data, processing YouTube videos uploaded from a specific IP address– exactly the same problem and process. The difficulty that exists is that today we need to process data in a range of file types and at much higher speeds than ever anticipated, but the processes remain the same.”
Arnold explains the iterative nature of his work:
“The proof of the value of the legacy is I don’t really do anything new, I just keep following these themes. The Dark Web Notebook is very logical. This is a new content domain. And if you’re an intelligence or information professional, you want to know, how do you make headway in that space.”
Describing his most recent book, Dark Web Notebook, Arnold calls it “a cookbook for an investigator to access information on the Dark Web.” This monograph includes profiles of little-known firms which perform high-value Dark Web indexing and follows a book he authored in 2015 called CYBEROSINT: Next Generation Information Access.
HP Enterprise: Is Haven Autonomy IDOL after a Project Runway Touch Up?
March 16, 2016
Short honk: I read “HPE Launches Machine-Learning-As-a-Service on Microsoft Azure.” The hook for me was the pricing for a new cloud search and content processing service. I did not understand the approach; for example, what the heck is an “API unit”?
But what caused me to jot down this note was this list of HPE Haven OnDemand functions. Here’s the list I circled:
- Advanced Text Analysis, which pulls concepts and sentiment from text.
- Format conversion, which converts data wherever it lives.
- Search tools across on-premises or cloud data.
- Image recognition and face detection.
- Knowledge graph analysis.
- Pattern and speech recognition.
Based on my sketchy knowledge about Autonomy IDOL, this list seems to be a summary of Autonomy’s integrated data operating features. Most of these were added to the IDOL platform in the years before HP paid $11 billion for the 1998 system which, to be fair, had been upgraded in the intervening years.
The list also reminded me of some of the functions I associated with “augmented intelligence,” a niche currently occupied by outfits like Palantir and IBM i2.
In terms of pricing, the Palantir Hobbits charge for a license, training, support, and some other goodies. But the pricing is not variable. The IBM i2 folks deliver a collection of options and each option has a price tag.
HPE’s pricing is a bit of a mystery. How many API units fit on the head of Big Data project? Whittling down that $11 billion investment suggests that the API units may be more expensive than the monthly fees suggest; for example, the introductory offer offers 50,000 API units and 15 Resource Units for [the] first three months for all paid plans.” What’s a “Resource Unit”?
The write up raises more questions than it answers in my opinion. I wonder how Autonomy IDOL will look in fall fashions?
Stephen E Arnold, March 16, 2016
Tech Unicorns May Soon Disappear as Fast as They Appeared
March 15, 2016
Silicon Valley “unicorns”, private companies valued at one billion or more, may not see the magic last. The article Palantir co-founder Lonsdale calls LinkedIn plunge a bad sign for unicorns from Airline Industry Today questions the future for companies like LinkedIn whose true value has yet to result in ever-increasing profits. After disappointing Wall Street with lower earnings and revenue, investors devalued LinkedIn by about $10 billion. Joe Lonsdale, the Formation 8 venture investor who co-founded Palantir Technologies is quoted stating,
“A lot of LinkedIn’s value, according to how many of us think about it, is tied to what it will achieve in the next five to 10 years,” Lonsdale said in an appearance on CNBC’s “Squawk Alley” on Friday. “It is very similar to a unicorn in that way. Yes, it is making a few billion in revenue and it’s a public company but it has these really big long-term plans as well and is very similar to how you see these other companies.” He added a lot of people who have been willing to suspend disbelief aren’t doing that anymore. “At this point, people are asking, ‘Are you actually going to be able to keep growing?’ And they’re punishing the unicorns and punishing the public companies the same way.”
Lonsdale understands why many private companies postpone an IPO for as long as possible, given these circumstances. Regardless of the pros and cons of when a company should go public, the LinkedIn devaluation seems as if it will send a message. Whether that message is one that fearmongers similar companies into staying private for longer or one that changes profitability norms for younger tech companies remains to be seen.
Megan Feil, March 15, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Search Vendors Will Thank Forrester for Its Views on Customer Support
March 11, 2016
I read “Your Customers Don’t Want To Call You For Support.” This is a free marketing write up from the good folks at the mid tier consulting outfit Forrester.
The write up is one answer to the struggle some search vendors have had. As you may know, selling proprietary search and retrieval systems is a slow go these days. Why not use an open source system as plumbing? That’s what IBM and Palantir have done. Shift the costs of the utility function’s maintenance and bug fixing to the “community.” Shift those resources from search to something which sells. For Palantir, Gotham and Metropolitan are moving. For IBM, well, that may be a poor example. The only “moving” at IBM involves the individuals terminated.
Moving on…
The Forrester write up makes clear that “your customers” don’t want to call you on the telephone. No kidding? Has anyone at Forrester tried to call Forrester without a number linked to a specific individual?
The search vendors are struggling to find a market which really needs their search system. The candidate many search firms are chasing is the person in charge of customer support. The reason is that no one in customer support wants to talk to customers.
Put the information on the Web and let the customers “search” for answers. Everyone will be happy. At least, that’s the pitch.
Forrester thinks that self service is the “low friction” way to deal with customers. Right. If there is no human who struggles to speak in an intelligible manner about a subject germane to the called, the support person will not experience some verbal excitement.
Forrester likes the chat thing. That’s a service which opens a box, introduces a delay, and then a message appears, “Hello, I am Ted. How may I help you?” My reaction is to click the close button. Sorry, Ted.
My hunch is that search vendors will print out copies of the Forrester article and use them as proof that a better search engine will create many happy customers.
If only life were that simple.
Stephen E Arnold, March 11, 2016
Watson Weakly: Jargon and Resource Allocations
March 9, 2016
In case you missed the news, IBM seems to be trimming its workforce. Does anyone remember Robert X. Cringely’s “IBM Is So Screwed?” I do. I would wager that Mr. Cringely remembers IBM’s suggestion that Mr. Cringely was off base with his analysis.
Perhaps Mr. Cringely is vindicated. I read “IBM Job Cuts: US Tech Giant Begins Mass Firing One Third of Workforce.” Hmmm. One third of a workforce having an opportunity to find its future elsewhere? That sounds like a swell way to greet spring 2016. March in like a lion and march out like a lamb. Is the lamb heading to the local meat packers?
Against this cheerful seasonal background, I want to mention “Moving from Enterprise Search to Cognitive Exploration.” This is a recycling of an earlier white paper for which one must register in order to read or download the document. Please, note that you will have to jump through some hoops to get this March 2016 publication. Do not complain to me about the link, the involvement of a middleman, and the need to provide details about your interest in enterprise search. Take it up with IBM; that is, if someone will take your call or answer your email. Hey, good luck with that.
What’s notable about this white paper is this word pair: Cognitive Exploration. Original? Nah. The phrase turns up in the title of a collection of essays called Cognitive Exploratioin of Language and Linguistics in 1999. The phrase is some of the jingoism from the super reliable psychology linguistics disciplines. IBM has dallied with the phrase for a number of years but in the RA world, the phrase is getting a jump start. An example of IBM’s arguement is that no one no longer runs a search across a customer service database. Nope, one cognitively explores that customer database.
Cognitive Exploration. It flows trippingly on the tongue does it not. IBM does not fire people; IBM RA’s them. (RA. Resource allocation or termination or reduction in force.)
What is Cognitive Exploration? Well, it is Lucene search plus some home brew code and a dollop of acquired technology. IBM’s original commercial enterprise search system (STAIRS) is just not up to the task of cognitively exploring one’s information assets it seems.
The white paper is a tribute to the search buzzwords that have been used by marketers in the past. I just love Cognitive Exploration.
What is it? For the full answer, you will need to read the 13 pages of explanation. Here’s a sampling of the facts in the write up:
Analysts expect the total data created and copied to reach 44 ZB by the year 2020 (Analyst firm IDC). After all, there are more than 204,000,000 emails launched every minute every day (Mashable.com). How do you manage, search, and process that data and turn it into usable information?
Yep, that’s a lot of information. How is an organization going to deal with “all” those zeros and ones? I suppose I would begin by using a system designed to manipulate large data flows. How about Palantir, BAE Systems, Leidos for starters. What no IBM? Bummer.
The IBM argument advances:
To meet today’s expectations, a search system must be able to access all of your important data sources and filter results based on a user’s access permissions within the organization.
I love the “all”. IBM obviously has nailed video, audio, binaries of various types, disparate file types, and dynamic content flows from intercepts, social media, and interesting sources from the Dark Web. I love “all” type solutions. Too bad these are science fiction based on my experience.
The fix is Cognitive Exploration. Thank you, IBM. A new buzzword to explain what search and retrieval has flubbed for — what? — 50 years” IBM explains:
Cognitive exploration is the combination of search, content analytics, and cognitive computing. Not only can cognitive exploration accelerate the rate at which users can find and navigate information; by leveraging advanced technologies such as content analytics, machine learning, and reasoning it has the potential to augment human expertise.
I don’t want to be a party pooper, but this is perilously close to Palantir’s “augmented intelligence” jargon. Attivio, BA Insight, and even the French folks at Sinequa use similar lingo. Me-too’ism at its finest? Nah, this is IBM, the outfit taking Groupon (a discount coupong business) to court for allegedly infringing on Prodigy patents. Prodigy? Remember that online service?
After snoozing through the white paper’s three pillars of Cognitive Exploration, I raced to the the finish line.
Cognitive Exploration involves the i2 type of relationship analysis, some good old fashioned cuddling between search and cognitive computing (think Watson, gentle reader), and a unified view or what a popular novelist calls “God’s eye” view. Please note that IBM offers some examples, but get the numbering wrong. Where is number one? Watson, Watson, can you assist me? Guess not. IBM’s cognitive exploration essay begins counting with number 2. I am okay with zero. I am okay with one. But I am not okay with an enumerated list beginning with the number two. Careless typo? Indifference? Rushing to the RA meeting? Don’t know. Cognitive Watson counts two, three, four, not one, two, three.
At the end of this remarkable description of Cognitive Exploration I learned:
The cognitive capabilities that can be leveraged by Watson Explorer are provided by the IBM Watson platform.
Isn’t this a recycling of some of the early 1990s marketing material from i2 Group Limited, which IBM bought. Isn’t this lingo influenced by Palantir’s explanations of its Gotham platform?
Omitted from the “all” I assume is the seamless interchange of Gotham files with i2 Analyst Notebook and i2 Analyst Notebook with Gotham. The users and customers have to learn that “all,” like Mr. Clinton’s “is” may not be exactly congruent with one’s understanding of “federation” and “unified.”
Enough already. Go for the close:
IBM Watson Explorer unlocks the value within your data, utilizing that information to help employees make well-informed decisions, provide better support, and identify more customers and business opportunities. By reaching across multiple silos of information within your enterprise, search results will include information never previously integrated into single solutions. Users will benefit from search results from all the data in your company, structured and unstructured, and include data from outside as well. Rather than trying to make good decisions with limited insight, cognitive exploration users can now extract and understand all of the valuable information at their fingertips.
With such a wonderful tool at IBM’s disposal, why is IBM’s management unable to generate revenues? Perhaps the silliness of the marketing explanation of Cognitive Exploration does not deliver the results that obviously someone at IBM believes.
I am stuck on that error in numbering, the recycling of Palantir’s marketing lingo, and the somewhat silly phrase “Cognitive Exploration.”
I won’t sail my Nina, Pinta, and Santa Maria to that digital shore. I will use Google Earth and tools which I know sort of work.
Stephen E Arnold, March 9, 2016
Enterprise Search Revisionism: Can One Change What Happened
March 9, 2016
I read “The Search Continues: A History of Search’s Unsatisfactory Progress.” I noted some points which, in my opinion, underscore why enterprise search has been problematic and why the menagerie of experts and marketers have put search and retrieval on the path to enterprise irrelevance. The word that came to mind when I read the article was “revisionism” for the millennials among us.
The write up ignores the fact that enterprise search dates back to the early 1970s. One can argue that IBM’s Storage and Information Retrieval System (STAIRS) was the first significant enterprise search system. The point is that enterprise search as a productized service has a history of over promising and under delivering of more than 40 years.
Enterprise search with a touch of Stalinist revisionism.
Customers said they wanted to “find” information. What those individuals meant was have access to information that provided the relevant facts, documents, and data needed to deal with a problem.
Because providing on point information was and remains a very, very difficult problem, the vendors interpreted “find” to mean a list of indexed documents that contained the users’ search terms. But there was a problem. Users were not skilled in crafting queries which were essentially computer instructions between words the index actually contained.
After STAIRS came other systems, many other systems which have been documented reasonably well in Bourne and Bellardo-Hahn’s A History of Online information Services 1963-1976. (The period prior to 1970 describes for-fee research centric online systems. STAIRS was among the most well known early enterprise information retrieval system.) I provided some history in the first three editions of the Enterprise Search Report, published from 2003 to 2007. I have continued to document enterprise search in the Xenky profiles and in this blog.
The history makes painful reading for those who invested in many search and retrieval companies and for the executives who experienced the crushing of their dreams and sometimes career under the buzz saw of reality.
In a nutshell, enterprise search vendors heard what prospects, workers overwhelmed with digital and print information, and unhappy users of those early systems were saying.
The disconnect was that enterprise search vendors parroted back marketing pitches that assured enterprise procurement teams of these functions:
- Easy to use
- “All” information instantly available
- Answers to business questions
- Faster decision making
- Access to the organization’s knowledge.
The result was a steady stream of enterprise search product launches. Some of these were funded by US government money like Verity. Sure, the company struggled with the cost of infrastructure the Verity system required. The work arounds were okay as long as the infrastructure could keep pace with the new and changed word-centric documents. Toss in other types of digital information, make the system perform ever faster indexing, and keep the Verity system responding quickly was another kettle of fish.
Research oriented information retrieval experts looked at the Verity type system and concluded, “We can do more. We can use better algorithms. We can use smart software to eliminate some of the costs and indexing delays. We can [ fill in the blank ].
The cycle of describing what an enterprise search system could actually deliver was disconnected from the promises the vendors made. As one moves through the decades from 1973 to the present, the failures of search vendors made it clear that:
- Companies and government agencies would buy a system, discover it did not do the job users needed, and buy another system.
- New search vendors picked up the methods taught at Cornell, Stanford, and other search-centric research centers and wrap on additional functions like semantics. The core of most modern enterprise search systems is unchanged from what STAIRS implemented.
- Search vendors came like Convera, failed, and went away. Some hit revenue ceilings and sold to larger companies looking for a search utility. The acquisitions hit a high water mark with the sale of Autonomy (a 1990s system) to HP for $11 billion.
What about Oracle, as a representative outfit. Oracle database has included search as a core system function since the day Larry Ellison envisioned becoming a big dog in enterprise software. The search language was Oracle’s version of the structured query language. But people found that difficult to use. Oracle purchased Artificial Linguistics in order to make finding information more intuitive. Oracle continued to try to crack the find information problem through the acquisitions of Triple Hop, its in-house Secure Enterprise Search, and some other odds and ends until it bought in rapid succession InQuira (a company formed from the failure of two search vendors), RightNow (technology from a Dutch outfit RightNow acquired), and Endeca. Where is search at Oracle today? Essentially search is a utility and it is available in Oracle applications: customer support, ecommerce, and business intelligence. In short, search has shifted from the “solution” to a component used to get started with an application that allows the user to find the answer to business questions.
I mention the Oracle story because it illustrates the consistent pattern of companies which are actually trying to deliver information that the u9ser of a search system needs to answer a business or technical question.
I don’t want to highlight the inaccuracies of “The Search Continues.” Instead I want to point out the problem buzzwords create when trying to understand why search has consistently been a problem and why today’s most promising solutions may relegate search to a permanent role of necessary evil.
In the write up, the notion of answering questions, analytics, federation (that is, running a single query across multiple collections of content and file types), the cloud, and system performance are the conclusion of the write up.
Wrong.
The use of open source search systems means that good enough is the foundation of many modern systems. Palantir-type outfits, essential an enterprise search vendors describing themselves as “intelligence” providing systems,, uses open source technology in order to reduce costs, shift bug chasing to a community, The good enough core is wrapped with subsystems that deal with the pesky problems of video, audio, data streams from sensors or similar sources. Attivio, formed by professionals who worked at the infamous Fast Search & Transfer company, delivers active intelligence but uses open source to handle the STAIRS-type functions. These companies have figured out that open source search is a good foundation. Available resources can be invested in visualizations, generating reports instead of results lists, and graphical interfaces which involve the user in performing tasks smart software at this time cannot perform.
For a low cost enterprise search system, one can download Lucene, Solr, SphinxSearch, or any one of a number of open source systems. There are low cost (keep in mind that costs of search can be tricky to nail down) appliances from vendors like Maxxcat and Thunderstone. One can make do with the craziness of the search included with Microsoft SharePoint.
For a serious application, enterprises have many choices. Some of these are highly specialized like BAE NetReveal and Palantir Metropolitan. Others are more generic like the Elastic offering. Some are free like the Effective File Search system.
The point is that enterprise search is not what users wanted in the 1970s when IBM pitched the mainframe centric STAIRS system, in the 1980s when Verity pitched its system, in the 1990s when Excalibur (later Convera) sold its system, in the 2000s when Fast Search shifted from Web search to enterprise search and put the company on the road to improper financial behavior, and in the efflorescence of search sell offs (Dassault bought Exalead, IBM bought iPhrase and other search vendors), and Lexmark bought Brainware and ISYS Search Software.
Where are we today?
Users still want on point information. The solutions on offer today are application and use case centric, not the silly one-size-fits-all approach of the period from 2001 to 2011 when Autonomy sold to HP.
Open source search has helped create an opportunity for vendors to deliver information access in interesting ways. There are cloud solutions. There are open source solutions. There are small company solutions. There are more ways to find information than at any other time in the history of search as I know it.
Unfortunately, the same problems remain. These are:
- As the volume of digital information goes up, so does the cost of indexing and accessing the sources in the corpus
- Multimedia remains a significant challenge for which there is no particularly good solution
- Federation of content requires considerable investment in data grooming and normalizing
- Multi-lingual corpuses require humans to deal with certain synonyms and entity names
- Graphical interfaces still are stupid and need more intelligence behind the icons and links
- Visualizations have to be “accurate” because a bad decision can have significant real world consequences
- Intelligent systems are creeping forward but crazy Watson-like marketing raises expectations and exacerbates the credibility of enterprise search’s capabilities.
I am okay with history. I am not okay with analyses that ignore some very real and painful lessons. I sure would like some of the experts today to know a bit more about the facts behind the implosions of Convera, Delphis, Entopia, and many other companies.
I also would like investors in search start ups to know a bit more about the risks associated with search and content processing.
In short, for a history of search, one needs more than 900 words mixing up what happened with what is.
Stephen E Arnold, March 9, 2016
Alphabet Google to Advise the US Department of Defense
March 8, 2016
There is a delicious irony in “Former Google CEO Schmidt to Head New Pentagon Innovation Board.” Alphabet Google’s core business is based on the experiences of some search predecessors. I understand the shoulders of giants thing. But in Google’s case, there are some specific folks to thank for the efficacy of the pre-2006 Google search system. Say what, Ghemawat? How mean, Mr. Dean? And for the revenue model, there is always the outfit one can “GoTo.” But no more Clever.
I recall that Mr. Schmidt had a seat on the Apple board. I wonder how some of the members of the Apple board liked their Android powered Samsung phones?
The point is that Google does some innovative things, but these are often built on top of other ideas, concepts, and implementations. Did you forget the pre IPO settlement of Yahoo’s issue with the Google ad system? Harvard did and lots of others folks ignore that hiccup as well.
The write up reports:
Eric Schmidt, the former chief executive officer of Google, will head a new Pentagon advisory board aimed at bringing Silicon Valley innovation and best practices to the U.S. military… Modeled on the Defense Business Board, which provides advice on best business practices from the private sector, the new panel is intended to help the Pentagon become more innovative and adaptive in developing technology and doing business.
One imagines that Palantir’s executives will be eager to join the new group.
As both Alphabet Google and Palantir turn to buying companies to acquire innovative people and technology, the Department of Defense may rekindle its love for In-Q-Tel-type deals. Make no mistake. Any outfit with a seat on the board has some Tesla-like spark.
Stephen E Arnold, March 8, 2016
Italian Firm Adds to the Buzzword Blizzard in an Expert Way
March 7, 2016
I don’t pay too much attention to lists of functions an information intelligence system must have. The needs are many because federation, normalization of disparate data, and real time content processing are not ready for prime time. Don’t believe me? Ask the US Army which is struggling with the challenges of DCGS-A, Palantir, and other vendors’ next generation systems in actual use in a battle zone. (See this presentation for one example.)
I read “No Time to Waste! 5 Essential Features for Your Information Intelligence Solution.” I like the idea of a company (Expert System) which was founded a quarter century ago, urging speedy action.
You can work through the well worn checklist of entity extraction, links and relationships, classification, and sticking info in a “knowledge base.” I want to focus on one point which introduces a nifty bit of jargon which I had not seen in use since I was in college decades ago.
The word is anaphora.
There you go. An anaphora, as I recall, is repetition or word substitution. Not clear? Here are a couple of examples:
Rhetorical:
For want of revenue the investors were lost.
For want of a product credibility was lost.
For want of an application the market was lost.
Grammatical:
The marketing cacophony increased and that drove off the potential customers.
Now you can work these points into your presentation when the users want actionable information which fuses available information into a meaningful output.
Because modern systems are essentially works in progress, buzzwords like anaphora take the place of dealing with real world information problems.
But marketing by thought leaders is so much more fun. That may trouble some. Parse that, gentle reader. What can one make in the midst of a blizzard of buzzwords? One hopes revenue which keeps the stock out of penny territory.
Expert System SpA, if Google Finance is accurate, about $2 a share. Roger, anaphora that.
Stephen E Arnold, March 7, 2016
Attivio: Dines on Data Dexterity
March 7, 2016
Attivio was founded by some former Fast Search & Transfer executives. Attivio also had a brush with a board member who found himself in a sticky wicket. Quite a pedigree.
I read “Enterprise Search Takes Its Place at the Big Data Table.” The write up is built upon an interview with the chief executive officer of Attivio. Nice looking fellow who had a degree in music and marketing and an MBA from Wharton, the institution which helped educate Donald Trump.
What caught my attention were these points in the write up. My observations are in italics:
- Enterprise search has been around for two decades. [Nah, enterprise search is closing in on 50 years of fun and delight.]
- Enterprise search “finds unstructured content housed in file shares like SharePoint and other content management systems, in email archives, and in the content repositories of applications like customer relationship management. [Yep, and that is part of the problem with enterprise search. The bulk of the systems I have examined do not handle video, audio, binaries, and odd ball file types like those in ANB format very well or not at all. Plus users expect comprehensive results updated in near real time presented in a form which allows instant use.]
- Enterprise search does analytics and accelerated data discovery. [Yep, if the customer licenses a system like BAE NetReveal, the Palantir platform, or another industrial-strength fusion vendor.]
What I found interesting was the phrase “reducing the time to insight.” There is a suggestion from Attivio and from other vendors that their systems process digital content in a super fast mode.
In our testing, we have found that throughput for new content can require considerable investment in engineering and processing capability. Furthermore, dealing with flows from intercepts or other high volume content sources, most enterprise search systems cannot handle:
- Processing large flows of content in a matter of minutes. Hours or days is a more suitable time unit
- Updating the index or indexes
- Integrating real time data into search results, reports, and visualizations in a dynamic manner.
That’s why outfits who are emulating Palantir-style information access use open source search and then invest hundreds of millions in specialized engineering, interfaces, and fusion technologies.
Enterprise search vendors chasing Palantir-type systems are delivering what marketers find quite easy to describe. Here’s an example:
Not only that, but many enterprises can only “see” 10 percent of their data. The other ninety percent remains hidden—dark data. Data is often locked in silos, and it’s just too time-consuming to get it out. And making connections across structured, semi-structured, and unstructured information to serve to a BI tool is a completely manual, slow process – although highly valuable for developing strategic insights. Organizations that can cross this chasm will be poised to transform productivity, mitigate risks, and seize market opportunities.
The only hitch in the git along is that systems which handle “dark data” are available now. There are outfits able to handle “dark” data today. True, these are not based on enterprise search concepts because the core of a utility function is not a solid foundation for next generation information access. There are platforms which deliver actionable outputs. Even more interesting is that the US government is funding research to develop next generation systems designed to leap frog Palantir, i2, DCGS-A, and many other solutions.
Why?
Marketing is one thing. Delivering a system which works reliably, exhibits consistency, and integrates with work flows is a work in progress.
The notion that a Fast-type system can deliver what a Palantir-type system does is something I believe is wordsmithing. Watson does wordsmithing; others deliver next generation information access. Has Attivio hit a home run with its new positioning? Is the Attivio solution a starter for the Hickory Crawdads? My hunch the folks investing $70 million in Attivio want to start for the Boston Red Sox this year. Play ball.
Stephen E Arnold, March 7, 2016
Alphabet Google Gets into the Corporate Storytelling Game
March 6, 2016
I read “Google’s New Site Lets Engineers Tell the Backstory of Some of Its Best Products.” Organizations crank out stories, but most of them are kept in the buildings or within the walled gardens of the minds eager to keep getting a paycheck.
According to the write up, Google has created barfoo where “engineers can tell their stories.” Okay, I expect the unvarnished truth, no editorial shaping. Well, that’s crazy. No secretive, paranoid outfit like Alphabet Google is going to do a Jerry Spring program for products and services that “emerge,” get bought and reinvented, or just me-too’ed.
The write up says:
The site currently covers four topics: collaboration in Docs, smart composing in Gmail, voice search recognition and how Google built a faster YouTube. The site also has a section for open jobs at Google, should you want to work there yourself. The topics are pretty in-depth, too. Not only does Google tell you the inspiration behind some of its products, it dives into the process of delivering them to users.
I am not sure the story of Google’s online advertising system will be revealed. There are some other interesting products and services which are likely to put on a lower priority track too.
But that GoTo/Overture/Yahoo “innovation” would be a story I would read. I would skim anything to do with the Glass, Parviz, and staff interaction activities as well. Yep, very low priority.
Palantir has its Tolkien and comic book “myths.” Google is going to do a reality show with post production I assume.
Stephen E Arnold, March 6, 2016
-
- Subscribe to Beyond Search
Feature archive
News archive
-
