Enterprise Search: Not Even a Bridesmaid Now
June 1, 2018
By this point in our history, you would think that most enterprise search would be created equally. However, there are still some stalwarts that are only now stepping into the golden age of search, as it were. One odd entry into this club was discovered after reading a white paper by Hyland on “Enterprise Search”.
According to the paper on Hyland’s new product:
“Enterprise Search allows you to provide access to the right information even if users don’t know exactly where to find it. Enable anyone to build powerful queries without expertise using intuitive, menu-assisted and natural language search options. You can also find answers when there are misspelled search terms, inexact queries, substandard data, document errors and other faults.”
Seems a little elementary, right? We suspect this has a little to do with playing catchup in the field. Hyland recently acquired Lexmark’s enterprise software arm, which itself was never exactly leading the field. (Lexmark bought ISYS Search Software, an outfit with technology from the mid 1980s if the founder spoke the truth when I interviewed him in 2008.) This will be interesting to see where Hyland takes this technology, though. The company has been around for a while and seems intent on wedging itself into the enterprise data conversation. However, it’ll have to make some light speed advances in order to go toe-to-toe with the big dogs in this battle.
Patrick Roland, June 1, 2018
Attivio Continues to Move Its Technology Forward
May 25, 2018
Conceived by former Fast Search & Transfer executives, Attivio has moved from a system able to analyze baseball statistics to enterprise search to business intelligence and probably several other market spaces. Enterprise search vendors do that these days.
Now the newest version of Attivio is here, we learn from the company’s blog post, “Attivio Product News: Version 5.5.1 Available Today!” The write-up describes improvements in several areas. With the updated software development kit (SDK), one can test code before deploying it to the platform. As for security, we’re told Attivio has migrated to a stronger algorithm and upgraded libraries to their latest versions. Text extraction has also been improved and now works with over 600 formats. Furthermore, access to recent modules is also included; the post promises:
“Finally, we’ve made the latest modules part of the install. This includes the WebCrawler module, which enables you to ingest web pages, as well as newly released Search Analytics and Search UI Toolkit. As we’ve written about previously, Search Analytics gives you insight into the performance of your search platform in real time. And SUIT, Attivio’s Search UI Toolkit, is a framework for quickly building search applications from the simplest to the most complex. It’s an open source application that can be downloaded from GitHub, and enhanced by the community. It not only works with the Attivio platform, but also with Elasticsearch and Solr.”
How Fast like is Attivio? A faint imprint of the genetic code is there, but Attivio has, like other search vendors, adopted proprietary and open source technology. The trick is the marketing today. Attivio is chugging along but it faces enterprise search challengers fueled by venture funding. What’s interesting is that money continues to flow into what I would describe as “traditional” enterprise search plays; for example, Coveo. The hurdle, of course, is to convert investors’ money and support into sustainable, growing, profit spinning revenue. And that’s a challenge from my point of view.
See the Attivio post for more details on each of the above improvements. Founded in 2007 (shortly before Fast Search’s implosion and the sale of the Fast property to Microsoft and the legal dust up about Fast Search’s “fast” math). Attivio’s seems to be hiring. That’s encouraging.
Cynthia Murrell, May 25, 2018
Search Is a Problem: Still a Clumsy Song and Dance Routine
May 24, 2018
Enterprise search has been around for decades. Hundreds of consultants have asserted patterns, models, methods, and MBA infused strategies to “fix” enterprise search.
Why?
Wherever there is an organization with one or more enterprise search systems, I have found these characteristics:
- Unhappy users
- Unhappy senior manager
- Unhappy bean counters
- Unhappy vendors
- Usually happy consultants if they are paid.
I am biased, old, and hard nosed. After writing the first three editions of the Enterprise Search Report, the New Landscape of Search, adding a word or two to that astounding guru Martin White’s book about Successful Enterprise Search Management, talking with dozens of PhD candidates whose dissertatioins about search and retrieval would change the world, and meeting with vendors large and small for decades—I am amused by the arm waving enterprise search engenders.
Don’t get me wrong. There are very good information access systems. But these vendors license solutions which usually focus on solving a specific problem. Case in point: Blackdot, Terbium Labs, and Verint, and many others.
From the point of view of flailing content management experts, “enterprise search” means finding information in a usually flawed, Rube Goldberg construct called a CMS or content management system.
Against this wallpaper with my scrawled biases, I read “Diagnosing Enterprise Search Failures.” The pivot point for the story is another report that almost two thirds of enterprise search users are not satisfied with the retrieval system.
Like a reprise of a vaudeville act from the 1920s at a rap concert, the music and the footwork are stale, out of touch, and worn.
Enterprise search had its decade in the sun. The period between 1995 and 2005 was the golden age of search. Then the sun imploded. Over-promising and under-delivering made it clear to those licensing enterprise search systems that finding information was not a solution to digital information woes.
In fact, an enterprise search system exacerbated the problems employees encountered when trying to locate specific information. Fast Search & Transfer, Convera, Delphes, Entopia (remember that outfit), and other aggressively marketed companies found out that companies would license technology and then balk at the on going costs.
One by one the big names in enterprise search went out of business or found themselves owned by larger firms with a belief that their managers could make search a winner.
How did that work out? Chase down someone at Lexmark and ask about their experience with ISYS Search Software. Repeat the process at Dassault Systèmes? Do the same thing for products ranging from Artificial Linguistics to Vivisimo.
The result is that the universe of companies offering search solutions has changed since 2008. The legal dust up between HP and Autonomy continues. Search did not make HP happy.
Surveys are fine, but the data reveal nothing new. Enterprise search is not a solution to information problems in an enterprise. Companies are embracing free or low cost solutions based on open source technology. Specialist systems which address specific information access problems are thriving. One may not think of Diffeo and Palantir Technologies as enterprise search systems, but they are information access solutions and not designed to solve a panoply of retrieval and information management issues.
The reason enterprise search fails to please users boils down to the disconnect between what the user wants and what an enterprise wide system can deliver. The vendors promise more than technology can provide.
Checklists, MBA rah rah, and misplaced confidence in technology will not solve these specific challenges:
- The cost of maintaining, upgrading, and tuning an enterprise search system to the needs of specific users is significant
- Users have a keen desire to rely on the software to do the thinking for them. When a system requires the user to think or formulate a query or perform downstream analysis, the search system becomes a problem
- Procurement teams often lack the discipline and clout to lay out tight requirements and select a vendor to do that job. The pattern is to create a wish list, sign a deal, and leave the baggage of failure behind.
- The systems provided do not match what the marketers demo, suggest, or assert the software will actually do in an affordable, reliable, understandable manner.
As a former rental at a reasonably competent management consulting firm, a method for figuring out how to solve a problem has one objective: Sell billable work. I understand that.
Do not confuse a consultant’s report with solving the problem of enterprise search. If enterprise search worked, there would be little appetite for methodologies to figure out failure.
Why such hostility to enterprise search? I think clueless large and medium sized companies want to buy a silver bullet. Even better, the bullet must kill the content vampire with a single, low cost, easy to use, accurate shot.
That’s not going to happen… ever.
The problem is that individuals looking for information need tools to solve quite specific business tasks. In enterprise search, there are numerous points of failure; for example:
- Management support is weak
- Organizational infighting triggers departments to get their own search solution
- The technology does not work
- Results do not meet user needs
- Funding is insufficient
- Technical staff find that fixes are not easy or possible
- Content known to be in the system cannot be found
- Vendors change direction from search to customer support and leave search customers dangling
- The people involved are focused on their careers, lunch, or finding a new job, not the nitty gritty of designing a solution for a specific group of workers with an information need.
And there are other issues related to over-promising and under-delivering. I wrote about this years ago and talked about falling off the cliff of high expectations. Enterprise search users inevitably crash into the reality of the system. Thus, the significant percentage of dissatisfaction with enterprise search.
I know of no enterprise search system which delivers on these points. Furthermore, as venture funding flows into Coveo and LucidWorks, as IBM falls farther and farther behind its revenue goals for Watson search (OmniFind, Vivisimo, et al), and as Microsoft buys more and more search start ups in the hopes of finding a silver bullet to its search mess—It is clear that stakeholders, customers, and users are going to become increasingly annoyed at the problem of enterprise search.
Why did Google bow out of enterprise search? Why has Elastic emerged as the go-to solution for many enterprise search applications? Why are companies like Funnelback, Sophia, Exorbyte, and dozens of others scrambling?
Enterprise search looked like a solution to some important problems. Today not so much. Open source search software is fine. However, how many of the open source vendors are going to be able to generate a return for their investors with what amounts to free software.
Enterprise search is the wrong label for today’s solutions. Even proprietary systems in hock for $100 million have longer odds than a nag entered in the Kentucky Derby.
Therefore, thrashing.
Stephen E Arnold, May 24, 2018
Algolia: Doing What Exalead Failed to Do
May 7, 2018
I read “How Algolia Built Their Real-time Search as a Service Product.” Reading between the lines and doing a bit of thinking, I arrived a hypothesis. The story begins with the Exalead search system. (You can get some information from the original three editions of “The Enterprise Search Report” which I wrote between 2004 and 2008. I also have a for fee profile of Exalead which you can order by writing benkent2020 @ yahoo dot com. The report is $40 payable via PayPal.)
The developers of Algolia focused on the shortcomings of Exalead, which has not changed significantly since its purchase by Dassault Systèmes. A number of Exalead professionals have left the company and had an impact on a number of companies. That may be the case at Algolia, or the founders of Algolia identified the weakness of other French systems and moved forward. Does anyone think about Antidot, Datop, Pertimm, Sinequa, and other French centric search systems?
Crunchbase reports that Algolia says:
Algolia is the most reliable platform for building search. Our hosted search API supplies the building blocks for creating great search to connect your users with what matters most to them. Our hosted search API powers billions of queries for thousands of websites & mobile applications every month, delivering relevant results in an as-you-type search experience in under 50ms anywhere in the world. Algolia’s full-stack solution takes the pain out of building search; we maintain the infrastructure & the engine, and we provide extensive documentations to our dozens of up-to-date API clients and SDKs with all the latest search features, so you can focus on delighting your users.
The write up explains that the complexity of other search systems, the lack of a hosted cloud-based platform, and the failure to swap out proprietary code for open source alternatives have differentiated Algolia from other enterprise search systems.
Some reviews of the system are available on Stackshare. Among the strengths of the system are its speed, its ease of implementation, and its distributed search network. No negatives jumped out at me. Algolia seems to in a good place at this time.
The system is also available for free for “community projects.”
Several observations:
- Large companies purchasing search systems often find that change and improvement is difficult, if not impossible. Too bad for Exalead.’
- The open source orientation of Algolia may put some pressure on Elastic. I would include Lucidworks, but that company continues to borrow or chase venture funds because the home run swing is not yet butter smooth. But Algolia has ingested $74 million, and like Lucidworks, that money has to make money; otherwise, exciting events occur.
- French vendors have had some difficulty penetrating certain markets; for example, the US government. Perhaps Algolia will succeed where other French companies have fallen short.
For more information about Algolia, navigate to www.algolia.com.
I would point out that the European experts and the US SEO crowd have not paid much attention to Algolia. Quite a few dead horses are being whipped while Elastic romps forward. In the US, search means SEO, and that band of merry wizards remains convinced that Google will put their clients’ Web pages at the top of the results list without buying Google ads.
Yeah, and I believe in the tooth fairy.
Stephen E Arnold, May 7, 2018
Former Autonomy Executive Found Culpable
May 1, 2018
I read “U.S. Jury Convicts Former Autonomy Executive of Fraud over HP Deal.” The jury found Sushovan Hussain guilty of wire fraud. Reuters said that in 2009 Mr. Hussain began to “deceive Autonomy’s investors and HP( about the company’s financial condition and prospects for growth.”
An ArnoldIT profile about Autonomy is available without charge at this link.
The HP purchase of Autonomy was one of the major turning points in enterprise search. The $11 billion deal made clear that enterprise search was an application space which would have provided rocket fuel for HP’s growth.
HP discovered that enterprise search was a challenging technology to use as a way to generates billions of dollars in revenue quickly and easily.
Beyond Search’s view of this deal—as well as the sale of Exalead to Dassault Systèmes, Vivisimo’s sale to IBM, and the manic repositioning of the vendors pitching proprietary search solutions—is that may companies found that enterprise search was a tough nut to crack. Marketing is easier than generating sustainable revenues. In my experience, enterprise search requires specialized expertise.
What’s interesting is that I heard that HP executives reviewed the Autonomy financial data, knew about the Qatalyst 2011 report about Autonomy, and decided to purchase the company. The deal seemed to unfold quickly and then implode almost as quickly. HP paid more for Autonomy than any other search acquisition of which I was aware. HP emerged as the proud owner of the firm which brought Bayesian methods to the enterprise. (I want to mention that the mathematical procedures implemented by Autonomy are now incorporated into most of the next generation information access systems I discussed in CyberOSINT, my book on what’s beyond search.
A page from the 2011 Qatalyst report about Autonomy. The full document, once available on the Oracle Web site, is now difficult to locate via open source methods.
This legal dust up may gather momentum. Possibly appeals and more people accused by HP making their way to the court room. I will stay tuned for developments reported by “real news” outfits like Thomson Reuters.
Stephen E Arnold, May 1, 2018
Attivio and MC+A Combine Forces
April 7, 2018
Over the years, Attivio positioned itself as more than search. That type of shift has characterized many vendors anchored in search and retrieval. We noted that Attivio has “partnered” with MC+A, a search centric company. MC+A also forged a relationship with Coveo, another search and retrieval vendor with a history of repositioning.
We learned from “Attivio and MC+A Announce Partnership to Deliver Next-Generation Cognitive Search Solutions” at Markets Insider that:
“MC+A will resell Attivio’s platform, seamlessly integrate their enterprise-grade connectors into it, and provide SI services in the US market. ‘Partnering with MC+A extends our ability to address organizations’ needs for making all information available to employees and customers at the moment they need it,’ said Stephen Baker, CEO at Attivio. ‘This is particularly critical for companies looking to upgrade legacy search applications onto a modern, machine-learning based search and insight platform.’ …
The story added:
“By combining self-learning technologies, such as natural language processing, machine learning, and information indexing, the Attivio platform is helping Fortune 500 enterprises leverage customer insight, surface upsell opportunities, and improve compliance productivity. MC+A has over 15 years of experience innovating with search and delivering customized search-based applications solutions to enterprises. MC+A has also developed a connector bridge solution that allows customers to leverage existing infrastructure to simplify the transition to the Attivio platform.”
Attivio was founded in 2007, and is headquartered in Newton, Massachusetts. The company’s client roster includes prominent organizations like UBS, Cisco, Citi, and DARPA. Attivio in its early days was similar in some ways to the Fast Search & Transfer technology once cleverly dubbed ESP. No, not extra sensory perception. ESP was the enterprise search platform.
Based in Chicago and founded in 2004, MC+A specializes in implementations of cognitive search and insight engine technology. A couple of years ago, MC+A was involved with Yippy, the former Vivisimo metasearch system. When IBM bought Vivisimio, the metasearch technology morphed into a Big Data component of Watson.
If this walk down memory lane suggests that vendors of proprietary systems have been working to find purchase on revenue mountain, there may be a reason. The big money, based on information available to Beyond Search, comes from integrating open source solutions like Lucene into comprehensive analytic systems.
In a nutshell, the rise of Lucene and Elastic have created opportunities for some companies which can deliver more comprehensive solutions than search and retrieval anchored in old-school solutions.
More than repositioning, jargon, and partnerships may be needed in today’s market place where “answers”, not laundry lists are in demand. For mini profiles of vendors which are redefining information access and answering questions, follow the news stories in our new video news program DarkCyber. There’s a new program each week. Plus, you can get a sense of the new directions in information access by reading my 2015 book (still timely and very relevant) CyberOSINT: Next Generation Information Access.
Stephen E Arnold,
Stephen E Arnold, April 7, 2018
Speeding Up Search: The Challenge of Multiple Bottlenecks
March 29, 2018
I read “Search at Scale Shows ~30,000X Speed Up.” I have been down this asphalt road before, many times in fact. The problem with search and retrieval is that numerous bottlenecks exist; for example, dealing with exceptions (content which the content processing system cannot manipulate).
Those who want relevant information or those who prefer superficial descriptions of search speed focus on a nice, easy-to-grasp metric; for example, how quickly do results display.
May I suggest you read the source document, work through the rat’s nest of acronyms, and swing your mental machete against the “metrics” in the write up?
Once you have taken these necessary steps, consider this statement from the write up:
These results suggest that we could use the high-quality matches of the RWMD to query — in sub-second time — at least 100 million documents using only a modest computational infrastructure.
The path to responsive search and retrieval is littered with multiple speed bumps. Hit any one when going to fast can break the search low rider.
I wish to list some of the speed bumps which the write does not adequately address or, in some cases, acknowledge:
- Content flows are often in the terabit or petabit range for certain filtering and query operations., One hundred million won’t ring the bell.
- This is the transform in ETL operations. Normalizing content takes some time, particularly when the historical on disc content from multiple outputs and real-time flows from systems ranging from Cisco Systems intercept devices are large. Please, think in terms of gigabytes per second and petabytes of archived data parked on servers in some countries’ government storage systems.
- Populating an index structure with new items also consumes time. If an object is not in an index of some sort, it is tough to find.
- Shaping the data set over time. Content has a weird property. It evolves. Lowly chat messages can contain a wide range of objects. Jump to today’s big light bulb which illuminates some blockchains’ ability house executables, videos, off color images, etc.
- Because IBM inevitably drags Watson to the party, keep in mind that Watson still requires humans to perform gorilla style grooming before it’s show time at the circus. Questions have to be considered. Content sources selected. The training wheels bolted to the bus. Then trials have to be launched. What good is a system which returns off point answers?
I think you get the idea.
Hulbee: Enterprise Intranet Search System
February 26, 2018
I associated the Hulbee brand with the Web search system Swisscows. Like Exalead and Qwant, Swisscows provides a user friendly search system. Key in some words, and Swisscows delivers the results. I ran a query for the UK smart software system SherlockML and received these results:
The distinctive features of the system struck me as:
- Privacy-centric. The queries are not retained by Swisscows.
- The system filters to eliminate violent and pornographic videos. Overt queries return no results. Certain queries return results which might raise some users’ hackles.
- Search results appear to come from Microsoft Bing, a “partnership.”
- Tile search which are clickable rectangular blocks of related content. which are similar to the Ben Shneiderman inspired visualizations for presenting search results, but Swisscows cleans up and makes more usable the visuals
- An icon which sends the user to the page in the results list. Most search engines display a hyperlink, which can be difficult to top accurately on some mobile device display screens
- The key search term presented in a white block with “closeness” of other concepts and terms shown by proximity to the white block; for example, ASI Data is the developer of SherlockML and the company is based in the UK. However, ASI does not appear in the blocks. The idea is a useful one in my idea, but some refinement may be warranted.
I learned from Telecompaper that Hulbee also offers an enterprise search system for Intranet content. The idea is that Hulbee, like Yippy and other search vendors, can be a replacement for the more than 55,000 orphaned Google Search Appliance customers. I often wonder how many of these GSAs are still in use because Google has never provided oodles of data about its misguided, overpriced, and odd ball “one size fits all” approach to what is a highly particularized problem.
Telecompaper reports:
Enterprise Search is flexible and scalable; in addition to internal use, it can also be used on the company’s website and external online shop. The advantage for companies is that they can tailor the search tool to suit their needs, without any external advertising included in the results. Customers can also choose Enterprise Search as a hosted service at Swisscom data centers or an on-premise service on their own servers.
One of company’s promotional videos features — wait for it — Swiss cows, although I am not able to differentiate among cow nationalities:
It seems that “enterprise search for an Intranet” has bundled a number of other search and retrieval functions; for example, Web site search and eCommerce. In my experience, some enterprise search vendors have offered “Swiss Army knife solutions” in the past. The reality of commercial enterprises is that search and retrieval needs are idiosyncratic; for example, lawyers require systems that can be used for eDiscovery, engineers have to locate drawings and their associated products, marketers want to pinpoint versions of PowerPoints, marketing collateral, and email, etc.
If you want more information about Swisscows, navigate to this link. You can check out the personal appeal for a donation from the company’s founder at this Web page.
Give the system a look, please.
Stephen E Arnold, February
Everyone Should Know the Term Cognitive Computing
December 19, 2017
Cognitive computing is a term everyone in the AI world should already be familiar with. If not, it’s time for a crash course. This is the DNA of machine learning and it is a fascinating field, as we learned from a recent Information Age story, “RIP Enterprise Search –AI-Based Cognitive Insight is the Future.”
According to the story:
The future of search is linked directly to the emergence of cognitive computing, which will provide the framework for a new era of cognitive search. This recognizes intent and interest and provides structure to the content, capturing more accurately what is contained within the text.
Context is king, and the four key (NOTE: We only included the most important two) elements of context detection are as follows:
Who – which user is looking for information? What have they looked for previously and what are they likely to be interested in finding in future? Who the individual is key as to what results are delivered to them.
What – the nature of the information is also highly important. Search has moved on from structured or even unstructured text within documents and web pages. Users may be looking for information in any number of different forms, from data within databases and in formats ranging from video and audio, to images and data collected from the internet-of-things (IOT).
Who and what is incredibly important, but that might be putting the cart before the horse. First, we must convince CEOs how important AI is to their business…any business. Thankfully, folks like Huffington Post are already ahead of us and rallying the troops.
Patrick Roland, December 19, 2017
The Future Is Search. Hmmm
December 17, 2017
I read an unusual chunk of content marketing. Navigate to “In the Rush to Big Data, We Forgot about Search.” Who’s the “we”? I think the “we” are customers who are migrating next generation information access systems. Lawyers have relativity. Manufacturers have SAP and Dassault solutions. Folks without much faith in commercial search vendors have Elasticsearch or low-cost systems which deliver a list of results which match a query. The “we”, therefore, seems to refer to the Lucid Imagination outfit now doing business as Lucidworks.
The write up explains that “we need to look at search to be the glue that lets us find the data and analyze it together no matter where it lives.”
That sounds super.
I think there are companies delivering this type of service as they have been for a number of years.
The reason is that vendors who are anchored in search and retrieval like Lucidworks have been bypassed.
In Dark Cyber I write about a stealthy outfit called Blackdot. The company complements the Relativity eDiscovery platform. Sure, there’s a search function, but Relativity does analytics, clustering, and functions which fit the needs of those engaged in eDiscovery. Search is part of the game, which for big cases, involves big data.
Blackdot enhances Relativity. You can learn about some of the functions of this company in the December 26, 2017, Dark Cyber video program.
So what?
The so what is that the services provided by Relativity and Blackspot deliver high value outputs that provide outputs which are immediately useful to analysts, investigators, lawyers, and others who use the integrated systems to solve problems.
A company which wants to deliver this type of service is likely wade into high water and thrash for purchase. The reason is that building a solution from open source tools and home brew scripts is a tough job.
Specialists have been using open source and proprietary code to roll out information access solutions. Relativity is just one example. By the way, Relativity has been plugging away for more than a decade.
A column which makes a case for a customer to let a vendor of open source search build from ground zero a next generation information access solution is going to be a vendor with a smile. However, once the solution fails to meet expectations, those smiles will turn to frowns.
Maybe that’s why Lucidworks has burned through one original founder, several presidents, and $59 million?
Search is a utility. It is not a headliner. Search works when it complements higher value functionality such as those delivered by Relativity and Blackdot or any of the other firms we track for our CyberOSINT research.
Search had its fling, but the glory days faded. When we look at the landscape of enterprise search or Big Data for that matter, we see winners. From our vantage point in Harrod’s Creek, the company leading the much smaller search parade is Elastic. Yep, it’s Lucene, but it has a following.
Guess who one of the followers is. Give up. Lucidworks. The technology is based on Lucene.
Selling consulting services is one thing. Selling search is another.
Today’s forward looking companies want next generation access, and they can get it from dozens of vendors. No starting from scratch. Sign a deal and begin processing data (big or small).
I highlighted this statement from the write up:
So if you move some of your data to SaaS solutions, move some of your data to PaaS solutions, move some of your data to IaaS solutions and across multiple vendors’ cloud platforms while maintaining some of your data behind the firewall—yeah, no one is going to find anything!
Sure. Solve problems. Don’t create them. One can search for solutions using a search engine. Let me know how that works out for your next big decision which you have to make in 10 seconds or less.
Stephen E Arnold, December 17, 2017
-
- Subscribe to Beyond Search
Feature archive
News archive
-
