The Search Wars: When Open Starts to Close
March 12, 2019
Compass Search. The precursor. The result? Elasticsearch. No proprietary code. Free and open source. The world of enterprise search shifted.
As a result of Shay Bannon’s efforts, an alternative to proprietary search and interesting financial maneuvers, an individual or organization could download code and set up a functional enterprise search system.
There are proprietary search systems available like Coveo. But most of the offerings are sort of open sourcey. It is a marketing ploy. The forward leaning companies do not use the word search to market their products because zippier functionality is what brings tire kickers and some buyers.
The landscape of search seems to be doing its Hawaii volcano act. No real eruption buts shakes, hot gas, and cracks have begun to appear. The lava flows will come soon enough.
The path is clear to the intrepid developer.
The tip off is Amazon’s announcement that it now offers an open distro for Elasticsearch. Why is Amazon taking this step? The company explains:
Elasticsearch has become an essential technology for log analytics and search, fueled by the freedom open source provides to developers and organizations. Our goal is to ensure that open source innovation continues to thrive by providing a fully featured, 100% open source, community-driven distribution that makes it easy for everyone to use, collaborate, and contribute.
DarkCyber’s briefings about Amazon’s policeware initiative suggest that the online bookstore is adding another component to its robust intelligence system and services.
The move involves or will involve:
- Entrepreneurs who will see Amazon as creating low friction for new products and services
- Partners because implementing search can be a consulting gold mine
- Users
- Developers who will use an Amazon “off the shelf” solutions
- Competitors who may find the “other open source” Elasticsearch lagging behind the Amazon “house brand”.
The move is not much of a surprise. Amazon seeks to implement its version of IBM’s 1960s style vendor lock in. Open source is open source, isn’t it? A version of the popular Elasticsearch system which has utility in commercial products to add ons which help make log files more mine-able. Plus search snaps into the DNA of the Amazon jungle of services, functions, features, and services. Where there is confusion, there are opportunities to make money.
Adding a house brand to its ecosystem is a basic tactic in the Amazon playbook. Those T shirts with the great price are Amazon’s, not the expensive stuff with a fancy brand name. T shirts and search? Who cares?
What’s the play mean for over extended proprietary search systems which may never generate a pay day for investors? A lot of explaining seems likely.
What the play mean for Elastic, the company which now operates the son of Compass Search? Some long off site meetings may be ahead and maybe some chats with legal eagles.
What’s the play mean for vendors using Amazon as back end plumbing for their enterprise or policeware services? A swap out of the Elasticsearch system for the Amazon version could be in the cards. Amazon Elasticsearch will probably deliver fewer headaches and lost weekends than using the Banon-Elastic version. Who wants headaches in an already complex, expensive implementation?
The Register quotes an evangelist from AWS as saying:
“We will continue to send our contributions and patches upstream to advance these projects.”
DarkCyber interprets this action and Amazon’s explanations from the perspective and context of a high school football coach:
“Front line, listen up, fork that QB. I want that guy put down. Hard. Let’s go.”
Amazon. The best defense is a good offense, right?
The coach shouts:
“Let’s hit those Sheep hard. Arrrgh.”
Stephen E Arnold, March 12, 2019
Beyond Search for 2019
January 1, 2019
I started Beyond Search to focus on new developments in enterprise search. That was in 2008. After 10 years of focusing on search, I have decided to retain the url but shift Beyond Search to cover the hidden Internet and lesser known Internet services. The blog will undergo a modest redesign and be called “DarkCyber Annex.”
Why an annex?
The modified Beyond Search blog will include information which supplements my weekly DarkCyber video. DarkCyber has been in production for one year, and it is—as far as I know—the only weekly video news program reporting about intelware, hidden Internet sites, and cyber crime.
To keep the videos in the 10 minute range, my team and I have to prune stories and content.
DarkCyber Annex, therefore, will be the online location for some of our additional content. We will continue to include links in the weekly videos, but now a version of the video story will appear in the DarkCyber Annex and include hyperlinks to source documents.
The flow of stories to Beyond Search will go down, and those assisting me in creating content for DarkCyber Annex will increase the flow of stories on the themes I have identified.
I plan to leave the Beyond Search content online. The 16,899 stories will be searchable but frozen. Looking back, enterprise search companies often described a fantastical world in which instant access was both marketed and sold.
That contributed to the implosion of the enterprise search sector. Today, if one wants search, many choose Lucene / Solr. Vendors of old school proprietary information retrieval systems will still market aggressively, pay consulting firms to sing the praises of the systems, and hold conferences which recycle words and concepts which are decades old.
Enough. Stale conferences. Endless repetition of hard-to-believe claims. Weird Eisenhower / BCG charts comprised of subjective silliness. Flaccid essays in online blogs and news services about “content management.” Yada yada yada.
For me, the subject is not just uninteresting. Enterprise search is a case study in what is likely to happen to other technologies in search of a solution informed by watching Star Trek. Explaining enterprise search in terms of “governance” in our Facebook world is shallow.
In 2019, I will try to make clear that intelware, not search, is where information access is today. Banging in key words still works, but the innovators are pushing into function spaces that deliver on some of the wild and crazy claims made in the salad days of Autonomy, Convera, Endeca, Fast Search & Transfer, and the dozens upon dozens of other companies I tracked in my career.
Enterprise search has fallen on its sword. New solutions have become available, but so far enterprises remain unaware of some of the most promising vendors.
DarkCyber videos and DarkCyber Annex will try to fill the information void. After all, we know traditional search is not too useful, right?
Stephen E Arnold, January 1, 2018
Elastic Bounces and Rolls Away from Other Search Vendors
October 6, 2018
Please, do not confuse what Bing and Google deliver as “search” with the type of information access system which is available from Elastic. The founder of Compass Search (remember that?) has emerged as the big dog in the information access world. At a time when direct competitors like Attivio, Coveo, and Funnelback are working overtime to become something other than information access providers, Elastic and its Elasticsearch ecosystem have pulled off a digital kudzu play.
The evidence is not the raucous Elastic developer conferences. The proof is not the fact that most policeware vendors use Elastic as the plumbing for their systems. The hard facts are dollars.
I learned that Elastic pulled off its IPO and closed up 94.4 percent. Talk about happy investors. Those believers in the Shay Bannon approach must be turning cartwheels. For more financial insights, navigate to “Search Company Elastic Nearly Doubles on First Trading Day.” The write up states:
The debut rally is all the more pronounced because it comes on a down day for the broader market, particularly the tech sector.
Elastic, it seems, represents a bright spot.
Congrats to Mr. Bannon and the Elastic team.
There are some outfits likely to take a hard look at their “search” business. Among them will be the vendors of proprietary search systems like the companies I mentioned above. Most of these outfits continue to find a way to make their investors happy. Attivio bounces between business intelligence and search. Coveo roves from search to customer support. Funnelback, well, Funnelback chugs along because one of their management team told me that the company is not open source. I wonder if that wizard wishes it were playing open source canasta.
The more interesting company to consider in the context of the Elastic solid triple in the search big leagues is LucidWorks. This company played its open source card. The company flipped CEOs, changed its focus, and emulated the polymorphic approach to search that the proprietary vendors followed. LucidWorks then found itself facing the Amazon search system staffed helpfully with a LucidWorks’ veteran or two. LucidWorks has consumed more than $100 million in investment capital, pushed founder Marc Krellenstein down the memory hole, and watched as the Elastic outfit blasted past LucidWorks and into the lushness of the IPO. Both companies had similar business models. Both companies leveraged the open source development community. Both companies followed similar marketing scripts.
But there was a difference.
Shay Bannon provided vision and he figured out that he needed a strong supporting cast. The result is that Elastic moved forward, added capabilities, made prudent decisions about supplemental modules, and offered reasonable for fee option to those who tried out the open source version of the search system and then moved to pay for service and other goodies available from Elastic.
The result?
The future for LucidWorks now looks a bit different. The company has to find a way to pay back its investors. The firm’s Elastic like business model may have to be reevaluated. Heck, the product line up may be require a refurbishing comparable to those performed on automobile programs which take an interesting vehicle and turn it into a winner.
Unfortunately fixing up search vendors is not as easy to do in real life. A TV show has the benefit of post production and maybe some color and sound experts to spiff up the automobile.
/arc-anglerfish-arc2-prod-sltrib.s3.amazonaws.com/public/NGG3Q5HFNVFIVKF2EKJQAOBSPY.jpg)
Competitors like LucidWorks will have to spiff up their 1956 automobiles in order to catch customers’ eyes as Elastic rolls rapidly into the future.
Search doesn’t work that way.
The question becomes, “What will LucidWorks do?”
Even those of us in Harrod’s Creek know what Elastic will do. The company will chug along and become the go to way to provide utility search, log analysis, and other basic functions to outfits which appear to be independent high tech search wizards.
Stephen E Arnold, October 6, 2018
Wake Up Time: IBM Watson and Real Journalists
August 11, 2018
I read “IBM Has a Watson Dilemma.” I am not sure the word “dilemma” embraces the mindless hyperbole about Vivisimo, home brew code, and open source search technology. The WSJ ran the Watson ads which presented this Lego collection of code parts one with a happy face. You can check out the Watson Dilemma in your dead tree edition of the WSJ on page B1 or pay for online access to the story at www.wsj.com.
The needle point of the story is that IBM Watson’s push to cure cancer ran into the mushy wall composed of cancerous cells. In short, the system did not deliver. In fact, the system created some exciting moments for those trying to handcraft rules to make Dr. Watson work like the TV show and its post production procedures. Why not put patients in jeopardy? That sounds like a great idea. Put experts in a room, write rules, gather training data, and keep it update. No problem, or so the received wisdom chants.
The WSJ reports in a “real” news way:
…Watson’s recommendations can be wrong.
Yep, hitting 85 percent accuracy may be wide of the mark for some cognitive applications.
From a practical standpoint, numerical recipes can perform some tasks to spin money. Google ads work this magic without too much human fiddling. (No, I won’t say how much is “too much.”)
But IBM believed librarians, uninformed consultants who get their expertise via a Gerson Lehrman phone session, and from search engine optimization wizards. IBM management did not look at what search centric systems can deliver in terms of revenue.
Over the last 10 years, I have pointed out case examples of spectacular search flops. Yet somehow IBM was going to be different.
Sorry, search is more difficult to convert to sustainable revenues than many people believe. I wonder if those firms which have pumped significant dollars into the next best things in information access look at the Watson case and ask themselves, “Do you think we will get our money back?”
My hunch is that the answer is, “No.”
For me, I will stick to humanoid doctors. Asking Watson for advice is not something I want to do.
But if you have cancer, why not give IBM Watson a whirl. Let me know how that works out.
Stephen E Arnold, August 11, 2018
Thoughtspot: Confused in Kentucky over AI for BI Plus Search Plus Analytics
August 3, 2018
i read “Nutanix Co-Founder Lures Away Its President to Be New CEO at ThoughtSpot.” The headline is a speed bump. But what puzzled me was this passage:
ThoughtSpot Inc. has hired Nutanix Inc.’s president as its new CEO. Sudheesh Nair joins ThoughtSpot about three months after the Palo Alto enterprise search business raised $145 million in a funding round that valued the company at more than $1 billion.
I added the emphasis on the phrase “enterprise search business.”
Search is not exactly the hottest buzzword around these days. After shock from the FAST Search & Transfer and IBM Watson adventures I hypothesize.
Now here’s the pothole: The ThoughtSpot Web site states:
Search & AI Driven analytics.
I noted the phrase “next generation analytics for the enterprise.” Plus, ThoughtSpot is a platform.
But what about artificial intelligence? Well, that’s part of the offering as well.
Remarkable: A Swiss Army knife. Many functions which may work in a pinch and certainly better than no knife at all.
But what’s the company do? Gartner suggests the firm has vision.
That helps. The first time around with FAST ESP and IBM Watson-like marketing the slow curves went right by the batters and the buyers. The billion dollar valuation is juicy as well. Another Autonomy? Worth watching.
Stephen E Arnold, August 3, 2018
Enterprise Search: Long Documents Work, Short Documents, Not So Much
July 16, 2018
Enterprise Search goals are notoriously wordy and complex. Is this just a symptom of a complicated system that cannot be explained any other way? Probably not, and it’s all one venture capitalist’s idea, according to Business Insider’s recent story: “One Simple Management Trick to Improve Performance, According to John Doer.”
According to the story which is about Doerr’s book “Measure What Matters.”
“[It] explains the thinking behind the Objectives and Key Results (OKR) goal-setting process famously used by companies like Google, MyFitness Pal, and Intel…. “The theory explains that hard goals “drive performance more effectively than easy goals,” and that “specific hard goals ‘produce a higher level of output’ than vaguely worded ones.”
According to Skyword, there are things you and your vendors can do to reverse this trend (Some may not want to reverse. Hey, it’s your world.). Mainly, it deals with understanding your audience and giving them what they crave.
However, short documents often make sense in context; that is, metadata, information about the sender / author and reader / person looking for information, category tags, and other useful information. Enterprise search, despite the wide availability of low cost or no cost solutions, struggle to make sense of short messages like:
“Doesn’t work.”
Videos, encrypted messages, audio, compound documents—Enterprise search systems struggle and often fail. More OPUD.
Patrick Roland, July 16, 2018
The Future of Enterprise Search: The View of a WeWork World
July 13, 2018
I read “The Future of Enterprise Search: Visual, Voice & Vertical.” My reaction was, “This approach to enterprise search describes a WeWork world.” My view of enterprise search is much, much different. Let me point out that I am okay with voice interfaces, but I am struggling to come to grips with the idea of visual search in an enterprise as one of the three pillars of enterprise search as i understand the function. The “vertical” angle is another way of saying, “Enterprise search does not work as a one size fits all solution. Therefore, let’s embrace a search engine for the legal unit, one for the computational chemists, one for marketers, and so on.
The write up points out that organizations, needs, and marketing are in flux. Uncertainty is the name of the game. That’s why there are employees who aren’t really full time equivalents working in Starbuck’s and WeWork offices. Who has a desk, assistants, and a regular nine to five job? Darned few people today. If we recognize the medieval set up of most organizations, the traditional definition of a job has more in common with the world of the Willy Loman (low man on totem pole, get it?). Life today has kings, court staff, and peasants. The difference is that the staff and peasants have mobile phones; otherwise, we’re back in the 7th century CE.
Skipping over the copy and paste of an Economist chart, the guts of the expository essay explains visual, voice, and vertical. The Vs reminded me of IBM’s alleged insight about Big Data’s volume, velocity, and variety. A mnemonic with alliteration. Okay, just not enterprise search as I have defined it in a number of my books; for example, The New Landscape of Search, published by Panda, years ago.
Enterprise search makes it possible for an employee to obtain the information needed to complete a business task. I pointed out that an employee cannot perform some work without locating digital information and data needed to answer a question. My examples included locating the most recent version of a CEO’s PowerPoint presentation, a list of the suppliers for a particular component in a product, information about an alleged personnel matter which violated the terms of an agreement with a customer, and the lab notes relative to a new compound developed by a chemical engineer with a structure diagram.
Now it would be wonderful if I could speak to a mobile device and have the data for any one of these enterprise search tasks delivered to me. But there are a couple of problems; namely, the screen size and capabilities of most mobile devices. For these information tasks, I personally prefer a multi monitor set up, a printer, plus old fashioned paper and pencil for notes.
The visual search angle is useful when looking for engineering drawings or chemical structures. But the visual component is only a part of the information I needed. That lab notebook is important, particularly if the product is going to be commercialized or patented or used as a bargaining chip in a deal with a potential partner or acquisition.
The vertical part is, as I have said, the reason that the typical organization has dozens of information access systems. A decade ago, according to our research, a Fortune 500 company licensed most of the available enterprise search systems, one or more legal search systems, and the specialist tools for those working with engineering drawings, specifications, vendor profiles, etc.
I don’t want to suggest that the discussion of visual, voice, and vertical search in the Search Engine Journal is with value. The information is simply not on point for today’s organization information access requirements.
For those in the top tier of workers — that is, those with senior positions and staff — the tools needed are more diverse and must be more robust. For those laboring away in WeWork offices, voice and visual search may be the go to ways to get information. The vertical search systems are useful, but for many workers, the expertise required to make a chemical structure search system deliver useful outputs is outside those workers’ skill set without some work and midnight oil.
To sum up, enterprise search is a difficult concept. Simplifying it to the three Vs understates the challenge. Explaining enterprise search in terms of semantic technology, natural language processing, and the other difficult to define jargon sprints to the far end of the complexity spectrum.
That’s why enterprise search is problematic. The vendors hope for a buyer and then head for the beach or a new job. The customers end up like Robinson Caruso, stranded and alone with tools that usually fall into disrepair quickly. Enterprise search itself is jargon, but it is jargon which has been marginalized by systems which over promised and under delivered.
That’s a mnemonic and acronym for you: OPUD.
Stephen E Arnold, July 13, 2018
Sinequa Review: Questions Go Unanswered
July 12, 2018
I read “Sinequa Review” by an outfit called Finances Online. I think the idea is an interesting one. Navigate to a Web page and get a snapshot about a product. In this case, the vendor is Sinequa, and its product is described as “an integrated search platform that can extract information from your interconnected applications and environments. Aside from letting you draw data rapidly, it affords you actionable and intelligent insights that it gains through its deep content analyses.”
That suggests that Sinequa is more than a search engine. In my book CyberOSINT, I pointed out that next generation information access systems represent the path forward for making sense of digital information. I did not include Sinequa is that book’s profiles of vendors to watch.
Like many vendors of keyword search, Sinequa has been working hard to find a way to describe basic search in terms of higher value functions. The Finances Online write up about Sinequa illustrates the difficulty a company like Sinequa has in describing its various functions; for example, “extract information from your interconnected applications and environments.” I am not sure what that means.
The listing of benefits strikes me as different from what I identified in CyberOSINT. In that monograph, I focused on a system’s ability to identify high value or potentially high interest data automatically, interfaces which move beyond Google style lists of results which create more work for the analyst because relevance and a document’s inclusion of a specific item of data are impossible without directly reading a document, and analytic functions designed to present data in the context of the user.
Contrast CyberOSINT’s key features with Sinequa’s:
- In depth analytics
- Connectors
- Mobile strategy
We have congruence with analytics; however, misses on the other features.
The features of Sinequa are a listing of buzzwords. In CyberOSINT, the idea is that next generation information access systems emphasize outputs, not the mechanisms “under the hood.”
The cost of an enterprise search or NGIA system is a difficult issue. The big expenses for enterprise search or NGIA systems are planning, administration, training, set up, optimization, customization, content preparation, and remediation (most of these systems don’t work “out of the box.”)
Here’s how pricing of Sinequa is explained:
Sinequa is a platform that transforms your organization into an information driven one. If you are interested in powering your company or institution with the solution, you can request custom enterprise pricing from the sales team by phone, email, web form, or chat.
I think that inclusion of downsides about a product is important. Perhaps Finances Online will include:
-
- Pricing information from verified customers; for example, last year, the total cost of ownership was …
- Details about issues with the Sinequa system which may date from 2002
- A more informed listing of competitors
- Less jargon and fewer undefined buzzwords; for example, “deep content analytics” and “semantics”.
The trajectory of old school search vendors has been similar to the approach taken to extend the life and usefulness of DEC 10s and 20s. Wrappers of software can keep somethings youthful—at least at first glance. Don’t get me wrong. Quick looks can be useful.
Stephen E Arnold, July 12, 2018
Will Cognitive Search (Whatever That Is) Change Because of Squrro?
July 2, 2018
We are not exactly sure what cognitive search is. That’s a plus. Changing cognitive search should be easy. Tweak technology, do some wordsmithing, and the landscape is different. I think this works for some search wizards in today’s fluid environment for information access.,
Cognitive search is ready to undergo a major shift and a few companies seem to be leading the pack. The way that we not only process search, but the questions we ask, could see a drastic change quite soon. We were alerted to this move from a recent product called Squrro, with their post “Add Context, Accuracy, and Speed to Your Enterprise Searches.”
According to the piece:
“Cognitive search is the new generation of enterprise search that uses artificial intelligence (AI) to return results that are more relevant to the user or embedded in an application issuing the search query. The benefits of understanding and extracting more insights from content changes search from, “This document has the keyword in the title so it must be more relevant” to understanding user intent…”
According to reports, a wide variety of businesses are responding to this more detailed ability of cognitive search to serve customers. For example, one Swiss insurance company recently adopted Squrro technology to provide customers with better search options. The company, Helevetia, claims its five million customers are benefiting from this change.
We will have to wait until industry fonts of wisdom like LinkedIn and CMSWatch provide their views. LinkedIn is, however, a Microsoft entity and there is that Fast Search ESP stub. CMSWatch often views the world from the perspective of content management or CMS. (Is there such a thing as cognitive CMS?)
Clarity will be forthcoming in step with the anticipated Elastic IPO. That financial event makes enterprise search more interesting again. Frankly making IBM Watson and Coveo equivalents did not do it for our research team.
Here in Harrod’s Creek, we want our old fashioned Boolean queries to return relevant results. We like the old school precision and recall approach.
Patrick Roland, July 2, 2018
What Has Happened to Enterprise Search?
June 28, 2018
I read “Enterprise Search in 2018: What a Long Strange Trip It’s Been.” I found the information presented interesting. The thesis is that enterprise search has been on a journey almost like a “Wizard of Oz” experience.
The idea of consolidation, from my point of view, boils down to executives who want to cash in, get out, and move on. The reasons are not far to seek: Over promising and under delivering, financial manipulations, and positioning a nuts and bolts utility as something that solves information problems.
Some, maybe many, licensees of proprietary enterprise search systems may have viewed their investment as an opportunity that delivered an unexpected but inevitable outcome. Where is that lush scenery? Where’s the beach?
The reality is that enterprise search vendors were aced by Shay Banon. His Act II of Compass: A Finding Story was Elasticsearch and the company Elastic. Why not use free and open source software. At least the code gets some bugs fixed unlike old school proprietary enterprise search systems. Bug fixes? Yep, good luck with your Fast Search & Retrieval ESP platform idiosyncrasies.
The landscape today is a bit like the volcanic transformation of Hawaii’s Vacationland. Real estate agents will be explaining that the lava flows have created new beach views, promising that eruptions are a low probability event.
The write up does not highlight one simple fact: Enterprise search has given way to “roll up” services or what I call “meta-plays.” The idea is that search is tucked inside systems like Diffeo, Palantir Gotham, and other “intelligence” platforms.
Why aren’t these enterprise grade solutions sold as “enterprise search” or “enterprise business intelligence and discovery solutions”?
The answer is that the information retrieval nest has been marginalized by the actions of vendors stretching back to the Smart system and to the present with “proprietary” solutions which actually include open source technology. These systems are anchored in the past.
Consider Diffeo?
Why offer enterprise search when one can provide a solution that delivers information in context, provides collaboration tools, and displays information in different ways with a single mouse click?
-
- Subscribe to Beyond Search
Feature archive
News archive
-
