Calling Out Search: Too Little, Too Late

January 20, 2020

The write up’s title is going to be censored in DarkCyber. We are not shrinking violets, but we think that stop word lists do exist. Problem? Buzz your favorite ad supported search vendor and voice your complaints.

The write is “How Is Search So #%&! Bad? A ‘Case Study’.” The author appears to be frustrated with the outputs of ad supported and probably other types of seemingly “free” search systems providing links to Web content. This is what some people call “open source intelligence online”. There are other information resources available, but most of the consumer oriented, eyeball hungry vendors ignore i2p, forums with minimal traffic, what some experts call the Dark Web, and even some government information services. How many people pay any attention to the US National Archives? Be honest in your assessment.

Here’s a passage we noted:

Google Search is ridiculously, utterly bad.

This seems clear.

The write up provides some examples, but I anticipate that some other people have found that the connection between a user’s query and the Google search outputs is tenuous at best. One criticism DarkCyber has of the write up is that it mentions Google, shifts to Reddit, and then to metadata. The key point for us was the focus on time.

Now time is an interesting issue in indexing. Years ago I did a research project on the “meaning” of “real time” in online services. I think my research team identified five or six different types of time. I will skip the nuances we identified and focus only on the data or freshness of an item in a results list.

Let’s by sympathetic to the indexing company. Here’s why:

First, many documents do not provide an explicit date in the text of the article. In Beyond Search and DarkCyber, you will notice that we provide the author’s name and a day and data at which the article was posted. Many write ups on the open Web don’t bother. In fact, there will be no easy way to date the time the author posted the story within the content displayed in a browser. Don’t you love news releases which do not include a date, time, and time zone?

Second, many write ups include dates and times in the text of an article. For example, the reference to Day 2 of the recent CES trade show may include the explicit date January 8, 2020, for a product announcement. The approach is similar to using CES without spelling out “Consumer Electronics Show.” Buy, hey, these folks are busy, and everyone in the know understands the what and when, right?

Third, auto-assigned dates by operating systems may be “correct” when a file or content object is created. But what happens when a file or drive is restored? The original dates and metadata may be replaced with the time stamp of the restore. What about date last accessed or date last changed? Too much detail. Yada yada.

Fourth, time sorting is possible. Google invested in Recorded Future (now part of Insight). I had heard that someone at the GOOG thought Recorded Future’s time functions were nifty. Guess not. Google did not implement more sophisticated time functions in any service other than those related to advertising. For the great unwashed masses of those who don’t work at Google, tough luck I supposed.

Fifth, when was the content first indexed. More significantly, when was the content last updated. Important? May be, gentle reader. May be.

There are several other conditions as well. For the purposes of a blog post, I want to make clear: The person who is annoyed with search should have been annoyed decades ago. These time problems are not new, and they are persistent.

The author with a penchant for tardy profanity stated:

Part of the issue in this specific case is that they’ve started ignoring settings for displaying results from specific time periods. It’s definitely not the whole issue though, and not something new or specific to phone searches. Now, I’ve always been biased towards the new – books, tech, everything, but I can’t help but feel that a lot of things which were done pretty well before are done worse today. We do have better technology, yet we somehow build inferior solutions with it all too often. Further, if they had the same bias of showing me only recent results I’ll understand it better, but that’s not even the case. And yes, I get that the incentives of users and providers don’t align perfectly, that Google isn’t your friend, etc. But what is DDG’s excuse? As for the Case Study part, and me saying this isn’t simply a rant – I lied, hence the quotation marks in the title. Don’t trust everything you read, especially the goddamn dates on your search results.

The write up omits a few other minor problems with modern search and retrieval systems. Yep, this includes Reddit, LinkedIn, and a bunch of others. Let me provide a few dot points:

  • Poorly implemented Boolean search
  • Zero information about what’s in an index
  • Zero information about what’s excluded from and index and why
  • Minimal auto linking to information about an “author” or the “source” of the content
  • No data to make a precision or recall calculation possible and reproducible
  • No data to make it possible to determine overlap among Web indexes. Analyses must be brute forced. Due to the volatility, latency, and editorial vagaries of ad supported Web search systems, data are mostly suggestive.

Why? Why are none of these dot points operative?

Answer: Too expensive, too hard, not appropriate for our customers, and “What are you talking about? We never heard of half these issues you identified.”

Net net: Years ago I wrote an article for Searcher Magazine, edited at the time by Barbara Quint, a bit of an expert in online information retrieval. She worked at RAND for a number of years as an information expert. She said, “Do you really want me to use the title ‘Search Sucks’ on your article.” I told her, use whatever title you want. But if you agree with me, go with “sucks.”  She used “sucks”. Let’s see that was a couple of decades ago.

Did anyone care? Nope. Does anyone care today? Nope. There you go.

Stephen E Arnold, January 20, 2020

DuckDuckGo Lands for European Search Users

January 14, 2020

I read “DuckDuckGo Beats Microsoft Bing In Google’s New Android Search Engine Ballot.” There have been numerous reports about this decision.

Digital Information World is a representative write up in today’s world of Google EU analysis. DarkCyber noted:

The introduction of this “choice screen” seems to be a clear response to the antitrust ruling from the European Union during last March and how Google was fined $5 billion by EU regulators. According to them, Google was playing illegally in tying up the search engine to its browser for mobile OS.

Okay. But how does a search engine get listed? We learned:

you can expect Google to not show search engines which are popular but the ones whose providers are willing to pay well.

The write up includes a run down of what search options will be displayed in each EU country. The ones we spotted are:

  • DuckDuckGo
  • GMX
  • Privacy Wall
  • Qwant
  • Yandex.

Bing is a no show as are Giburu, iSeek, Mojeek, Yippy, and others. It is worth noting that some of these outfits are metasearch engines. This means that the systems send queries to Bing, Google, and other services and aggregate the results. Dogpile and Vivisimo were metasearch engines. DuckDuckGo and Ixquick (StartPage) are metasearch engines`.  The reason metasearch is available boils down to cost. It is very expensive to index the public Web.

The DarkCyber team formulated a few hypotheses about the auction, the limitations on default search engines, and the dominance of Google search in Europe; for example, Google accounts for more than 95 percent of the search traffic in Denmark. The same situation exists in Germany and other EU countries.

Will these choices make any difference? Sure, for small outfits like DuckDuckGo any increase in traffic is good news. But will the choices alter Google’s lock on search queries from Europe?

Not a chance.

Does anyone in the EU government know? Probably not. Do these people care? Not to much.

Remember one of my Laws of Information: Online generates natural monopolies. Here’s another Law: User behavior is almost impossible to change once mental memory locks in.

So Google gets paid and keeps on trucking.

Stephen E Arnold, January 14, 2020

Enterprise Search and the AI Autumn

January 13, 2020

DarkCyber noted this BBC write up: “Researchers: Are We on the Cusp of an AI Winter?” Our interpretation of the Beeb story can be summarized this way:

“Yikes. Maybe this stuff doesn’t work very  well?”

The Beeb explains in Queen’s English based on quotes of experts:

Gary Marcus, an AI researcher at New York University, said: “By the end of the decade there was a growing realization that current techniques can only carry us so far.”

He [Gary Marcus and AI wizard at NYU] thinks the industry needs some “real innovation” to go further. “There is a general feeling of plateau,” said Verena Rieser, a professor in conversational AI at Edinburgh’s Herriot Watt University. One AI researcher who wishes to remain anonymous said we’re entering a period where we are especially skeptical about AGI.

Well, maybe.

But the enterprise search cheerleaders have not gotten the memo. The current crop of “tap your existing unstructured information” companies assert that artificial intelligence infuses their often decades old systems with zip.

The story is being believed by venture outfits. The search for the next big thing is leading to making sense of unstructured text. After all, the world is awash in unstructured text. Companies have to solve this problem or red ink and extinction are just around the corner.

Net net: AI is a collection of tools, some useful, some not too useful. Enterprise search vendors are looking for a way to make sales to executives who don’t know or don’t care about past failures to index unstructured text on a company wide basis with a single system.

Stephen E Arnold, January 13, 2020

Search Your Computer

January 13, 2020

On January 10, 2020, one of the DarkCyber team needed to locate a file on a Windows 10 machine. Windows 10 search was okay, but it generated false drops and took too long.

DarkCyber tried to get its copy of ISYS Desktop Search 8 to work, but that was a non starter. We had given up on Copernic a couple of versions ago. The DTSearch trial had expired as had a couple of New Age search systems vendors had provided to us to test; for example, X1, Vound and Perfect Search, among others. Elastic was overkill. Yikes.

We then checked our files for “desktop search” and located links to these articles:

We found a couple of these programs useful. In fact, the Everything software, version 1.4 did the trick for us.

We wanted to thank Martin Brinkmann for his articles which provided useful links and helpful information to us. Good job!

Stephen E Arnold, January 13, 2020

Lucidworks: Beyond Search for Sure

January 9, 2020

Lucid Imagination experienced what DarkCyber recalls as a bit of turmoil. From the git go, there was tension in the open sourcey ranks. One of the founders was unceremoniously given an opportunity to find his future elsewhere. Then there was the game of Revolving Door Presidents. Next was the defection of some lucid thinkers to Amazon, not in Seattle but just up the 101 to some non descript buildings. Like a law of nature another round of presidential revolving doors. Along the way, more investors wrote checks for what was an open source play based on Lucene/Solr. (I know that writing the two “names” together does not capture the grandiosity of the conception of community supported search and the privately held companies efforts to create a huge, billion dollar information access business. Sigh.

Now Lucidworks (which I automatically interpret as the phrase “Lucidworks. Really?”) has acquired an eCommerce vendor. Hello, what’s happening Magento, Mercado, Shopify, and Amazon. Yep, Amazon. But doesn’t Amazon have search too? Trivial point. Lucidworks is going to turn the $200 million in investment capital, an interface scripting engine, open source software, and Cirrus10 (an ecommerce service provider) into billions. Yes, billions!

According to “Lucidworks Acquires Cirrus10, Global Ecommerce Service Provider, to Deepen Domain Expertise and Become a Leader in Digital Commerce Solutions” states:

 Lucidworks, leader in AI-powered search, acquires Cirrus10, ecommerce solutions expert with more than 100 ecommerce customers. Lucidworks and Cirrus10 have worked together as partners for the past two years and now combine their domain expertise to provide more targeted solutions for different domains in the fast-moving ecommerce market.

The Yahoo news story points out that Lucidworks’ secret sauce is a system:

produces relevant results, recommends products that meet customer goals, and predicts shopper intent to create a more engaging experience.

And don’t forget artificial intelligence. AI! Obviously.

But whose AI? The answer appears to be AI from Cirrus10. DarkCyber noted this statement from a co founder of the ecommerce service provider:

“Fusion is the world’s only platform for extensible AI-driven search. Fusion elevated our service offerings by giving us a framework for exploring machine learning with our customers, and using it, we can build personalized and scalable relevancy models without a black box or army of data scientists. By combining Lucidworks search and AI expertise with our deep experience in the ecommerce space we can cement our role as digital commerce solution leaders.”—Peter Curran, Cirrus10

What appears to be the business strategy for Lucidworks is to get something that generates sustainable revenue, allows the company to upsell Cirrus10’s customers, and differentiate Lucidworks from the competitors in plain old search.

There are competitors; for example, outfits with venture capital backers demanding results (Algolia, Coveo). Also, open sourcey solutions (Drupal Commerce, Magento Community Edition) and small, feisty outfits like SLI Systems and EasyAsk). Note: This is a partial list. I almost forget companies like Amazon, eBay, and Google.

DarkCyber interprets the “beyond search” phrase as an attempt to make a 12 year old company into a revenue and profit machine.

DarkCyber, which is an annex to our blog Beyond Search, wishes the clear thinkers a great 2020. The question “Lucidworks. Really?” could be answered as long as AI, NLP, machine learning, open source, and synergy produce a winner, not a horse designed by a committee.

Stephen E Arnold, January 9, 2019

Are Media Worthless? Matt Taibbi Says Yes

January 3, 2020

Robert Steele, a former US spy whom I know, and also the top reviewer for non-fiction books in English, has published Review: Hate Inc. Why Today’s Media Makes Us Despise One Another by Matt Taibbi and given the book five stars, calling it “”totally brilliant.”

I was drawn to this statement in Steele’s review:

There will come a time, guaranteed, when Americans pine for a powerful neither-party-aligned news network, to help make sense of things.

Steele’s review appears to provide a concise summary of the book that those who worry about accuracy, data integrity, ethics, and the concept of social value should find interesting. Steele concludes the review by noting:

The same is true of the intelligence community, and the academy, of non-profits and governments. Keep the money moving, never mind the facts.

Facts? Are facts irrelevant? Steele and Taibbi appear to agree that facts remain important. Dissenters: Possibly the “media?”

Stephen E Arnold, January 3, 2020

Cognitive Search: A Silver Bullet?

January 2, 2020

Search is a basic function that requires tinkering to make it intuitive and a useful tool for enterprise systems. In the past, most out of the box search solutions stink and require augmentations from the IT department to work. Enterprise search, however, has dramatically improved and that makes a slow news day for search experts. Most headlines based enterprise search include the latest buzz topics, like, “Significance Of Cognitive Search In The Enterprise” posted on Analytics Insight.

Cognitive search is apparently the newest thing. It is basically enterprise search injected with machine learning/artificial intelligence steroids. An undeniable truth is that enterprise systems are pulling their data across many systems, on site and in the cloud. A good search tool will crawl each dataset and return the most accurate results. Cognitive search uses AI to make search smarter aka “more cognitive,” which basically means the search tool learns from search queries, make search suggestions, and offer predictions. The official jargon sounds smarter:

“Cognitive search is associated with the concept of machine learning, where a computer system processes new insights and convert the way it reacts based on the newly gained data. By using the form of AI, it provides more in-depth search outcomes based on local information, previous search history and other variables. It also brings more specific results to an end-user as the cognitive system learns how an individual or system acts these searches.

This makes the cognitive search method a variable implementation into an enterprise’s network search capability.”

In other words, based off the latest technology craze enterprise search is going to become smarter and more intuitive for users! Blah, blah, semantic search, blah, blah, search engines, blah, blah, algorithms. It is the same “new and improved” spiel that comes every year.

Whitney Grace, January 2, 2020

PubMed: Some Tweaks

December 27, 2019 is an old school online information service. The user types in one or more terms, and the system generates a list of results. Controlled terms work better than “free text” guesses.

According to “Announcing the New PubMed”:

The National Library of Medicine (NLM) is replacing the current version of the PubMed database with a newly re-designed version. The new version is now live and can be found at

The appearance of the site has been updated. To one of the DarkCyber team members, the logo was influenced by PayPal’s design motif. Clicking for pages of results has been supplanted by the infinite scroll. Personally, I prefer to know how many pages of results have been found for a particular query. But, just tell me, “Hey, boomer, you are stupid.” I get it.

The write up does not comment upon backlog, changes in editorial policy, and cleaning citations to weed out those which are essentially marketing write ups or articles with non reproducible results, wonky statistics, or findings unrelated to the main job of medicine. But you can use the service on a mobile phone.

Stephen E Arnold, December 27, 2019

Expanding the User Base: Voice Search As the Three Rs Are Deprecated

December 24, 2019

How does a company deal with declining reading and writing skills? Easy. Shift to voice search. The technology opens the door to individuals who may not be able to read or write. Voice search sidesteps barriers to obtaining information online. “You’ll Soon Be Able Search YouTube Web by Using Your Voice” definitely makes clear that creating a way for those who are unable or unwilling to thumb type is going to be a reality. The write up itself is an example of what happens when basic skills are not up to snuff.

We noted this passage:

As per renowned reverse engineer Jane Manchun Wong, YouTube is currently working on voice search for the web, meaning it will let users use their voice to perform searches.

DarkCyber is not sure what a “per renowned reverse engineer” is, but the message is clear. If a person cannot read, spell, or write—voice search will make it possible to watch YouTube videos.


Advertising revenue is one possible answer. Even those lacking certain skills buy stuff equipped with voice recognition. The reading and writing? Unnecessary for some consumers.

Stephen E Arnold, December 24, 2019

Do Four Peas Make a Useful Digital Pod?

December 24, 2019

The Four P’s of Information

This has the problem with data since at least the turn of this century—Forbes posts a “Reality Check: Still Spending More Time Gathering Instead of Analyzing.” Writer and Keeeb CTO Sid Probstein reminds us:

“Numerous studies of ‘knowledge worker’ productivity have shown that we spend too much time gathering information instead of analyzing it. In 2001, IDC published its venerable white paper, ‘The High Cost of Not Finding Information,’ noting that knowledge workers were spending two and a half hours a day searching for information. Since then, we have seen the rise of the cloud, ubiquitous computing, connectivity and everything else that was science fiction when we were kids becoming a reality — including the imminent emergence of AI. Yet in 2012, a decade after the IDC report, a study conducted by McKinsey found that knowledge workers still spend 19% of their time searching for and gathering information, and a 2018 IDC study found that ‘data professionals are losing 50% of their time every week’ — 30% searching for, governing and preparing data plus 20% duplicating work. Clearly, all the technology advances have not flipped the productivity paradigm; it seems like we still spend more time searching for information that exists rather than analyzing and creating new knowledge.”

Probstein believes much of the problem lies in data silos. There are four subsets of the data silo issue, we’re told, but most proposed solutions fail to address all of them. They are the “four P’s” of information: Public Data (info that is searchable across the World Wide Web), Private Data (information behind login pages or firewalls), Paid Data (like industry research, datasets, and professional information), and Personal Data (our own notes, bookmarks, and saved references). See the article for more about each of these areas. Bridging these silos remains a challenge for knowledge workers, but it seems businesses may be taking the issue more seriously. Will we soon be making better use of all that data? Do four peas make a pod? Not yet.

Cynthia Murrell, December 24, 2019

Next Page »

  • Archives

  • Recent Posts

  • Meta