NoSQL DBMS: A Surprising Inclusion
February 12, 2020
“Top Databases Used in Machine Learning Project” is a listicle. The information in the write up is similar to the lists of “best” products whipped up by Silicon Valley type publications, mid tier consulting firms (a shade off the blue chip outfits like McKinsey, Booz, and BCG), and 20 somethings fresh from university.
The interesting inclusion in the list of DBMS is?
If you said, Elasticsearch you would be correct. Elasticsearch is an open source play doing business as Elastic. The open source version is at its core a search and retrieval system. (Does this mean the index is the data and the database?)
DarkCyber is not going to get into a discussion of whether an enterprise search system can be a database management system. Both sides in the battle are less interested in resolving the fuzzy language than making sales.
Maybe Elasticsearch is just doing what other enterprise search systems have done since the 1980s? Vendors describe search and retrieval as the solution to the world’s data management Wu Flu.
Net net: Without boundaries, why make distinctions? Just close the deal. Distinctions are irrelevant for some business tasks.
Stephen E Arnold, February 12, 2020
Founder of Autonomy: Extradition Action
February 5, 2020
DarkCyber noted this CBR Online story: “Mike Lynch Submits Himself for Arrest.” The write up states:
Former Autonomy CEO Dr Mike Lynch has submitted himself for arrest this morning, in what his legal team described as a formality required as part of an extradition process initiated by the US Department of Justice. Lynch is still contesting extradition.
The story about the founder of Autonomy and DarkTrace continues. A free profile about Autonomy is available at this link. (Note: this document is a rough draft prepared for a client before the Hewlett Packard purchase of the company. Also, Autonomy was a client of mine before I retired in 2013.)
Stephen E Arnold, February 5, 2020
Paris Museums: More Art Online. Search Means Old Fashioned Hunting Around
February 5, 2020
Oh, boy—it is a collection of art from the many Paris Museums available online at Paris Musées Collections. This artist’s daughter is delighted!
Unfortunately, the site’s search functionality disappoints. Unless your goal is either to find a specific work or to aimlessly browse the 150,213 public domain images, it is another almost unusable collection. I suppose trusting to serendipity has its place, but most of us are looking for something a bit more specific, even if we don’t have a particular title or artist in mind.
There is a section titled “Thematic Discovering,” which might be useful to some. They have put together 11 preconfigured themes that span museums, like “Sport, Jeux Olympiqes et Paris” (Sports, Olympic Games, and Paris) or “Elements: Air, Terre, Feu, Eau” (Elements: Air, Earth, Fire, Water). They do make for interesting guided tours. There are also a highlighted Virtual Exhibition and a few suggested works at the bottom of the page.
I was excited to find this resource—it really is a valuable collection to have at our fingertips. If only it were easier to navigate. Check it out if you feel persistent.
And for those who think search is really great. None of the visual art collections feature a search which delivers what most users seek.
Cynthia Murrell, February 5, 2020
Search Vendor Comparison
January 30, 2020
The Finland-based AddSearch published a comparison of its search and retrieval service with two competitors: Algolia and Swiftype. Each of these is a for-fee solution. The write up appeared in 2019, but DarkCyber wanted to call attention to the article because it does a good job of outlining some of the main characteristics of commercial search solutions. You can locate the article by Anna Pogrebniak in the AddSearch Blog.
Kenny Toth, January 30, 2020
Verizon Tries Its Hand at… Search
January 29, 2020
Verizon is hopping on the privacy bandwagon but how do we know we can trust it? ArsTechnica reports, “Verizon Offers No-Tracking Search Engine, Promises to Protect Your Privacy.” The company’s new platform is called OneSearch. It gets its results from Bing, but imposes privacy features on them. While still running contextual advertising, of course, but ones that rely on keywords, not tracking. This search service is in addition to, not replacing, the existing Yahoo search engine. (Verizon bought Yahoo in 2017.) Yahoo search also gets its results from Bing, but makes no promises about privacy.
Here are the promises we get from OneSearch: no cookie tracking, retargeting, or personal profiling; no sharing of personal data with advertisers; and no storing search histories. When users click on the “Advanced Privacy Mode” button, the platform encrypts their search terms and URL and sets the results link to expire in an hour. However, writer Jon Brodkin reports:
“The Verizon search engine homepage says, ‘OneSearch doesn’t use cookies. Period.’ Chrome detected that OneSearch did set one cookie on my computer, so that statement seems to be exaggerated. The EFF’s Privacy Badger detected a potential tracker that’s tied to the u.yimg.com domain, indicating a connection between OneSearch and Yahoo’s image service. What Verizon apparently means is that it doesn’t use cookies to build ad-targeting profiles. Verizon uses your IP address to determine your ‘general location,’ helping it deliver location-specific search results. Verizon said that ‘We only ever infer location data up to the city level of specificity for search localization purposes.’”
The write-up also lists in more detail the steps OneSearch performs for each query. We are also reminded of less-than-stellar performance of Verizon’s media division thus far. See the article for those details.
So, can we trust Verizon to deliver on its privacy vows? Brodkin notes several events that may give us pause: In 2016, the company paid a $1.35 million fine and agreed to change their ways over “supercookies;” it, along with T-Mobile, Sprint, and AT&T, was caught selling mobile customers’ location data to third-party brokers; and Verizon regularly opposes government regulations that would require carriers to protect customer privacy. The write-up suggests DuckDuckGo and Startpage as alternatives for anyone hesitant to take Verizon at its word.
Cynthia Murrell, January 29, 2020
Calling Out Search: Too Little, Too Late
January 20, 2020
The write up’s title is going to be censored in DarkCyber. We are not shrinking violets, but we think that stop word lists do exist. Problem? Buzz your favorite ad supported search vendor and voice your complaints.
The write is “How Is Search So #%&! Bad? A ‘Case Study’.” The author appears to be frustrated with the outputs of ad supported and probably other types of seemingly “free” search systems providing links to Web content. This is what some people call “open source intelligence online”. There are other information resources available, but most of the consumer oriented, eyeball hungry vendors ignore i2p, forums with minimal traffic, what some experts call the Dark Web, and even some government information services. How many people pay any attention to the US National Archives? Be honest in your assessment.
Here’s a passage we noted:
Google Search is ridiculously, utterly bad.
This seems clear.
The write up provides some examples, but I anticipate that some other people have found that the connection between a user’s query and the Google search outputs is tenuous at best. One criticism DarkCyber has of the write up is that it mentions Google, shifts to Reddit, and then to metadata. The key point for us was the focus on time.
Now time is an interesting issue in indexing. Years ago I did a research project on the “meaning” of “real time” in online services. I think my research team identified five or six different types of time. I will skip the nuances we identified and focus only on the data or freshness of an item in a results list.
Let’s by sympathetic to the indexing company. Here’s why:
First, many documents do not provide an explicit date in the text of the article. In Beyond Search and DarkCyber, you will notice that we provide the author’s name and a day and data at which the article was posted. Many write ups on the open Web don’t bother. In fact, there will be no easy way to date the time the author posted the story within the content displayed in a browser. Don’t you love news releases which do not include a date, time, and time zone?
Second, many write ups include dates and times in the text of an article. For example, the reference to Day 2 of the recent CES trade show may include the explicit date January 8, 2020, for a product announcement. The approach is similar to using CES without spelling out “Consumer Electronics Show.” Buy, hey, these folks are busy, and everyone in the know understands the what and when, right?
Third, auto-assigned dates by operating systems may be “correct” when a file or content object is created. But what happens when a file or drive is restored? The original dates and metadata may be replaced with the time stamp of the restore. What about date last accessed or date last changed? Too much detail. Yada yada.
Fourth, time sorting is possible. Google invested in Recorded Future (now part of Insight). I had heard that someone at the GOOG thought Recorded Future’s time functions were nifty. Guess not. Google did not implement more sophisticated time functions in any service other than those related to advertising. For the great unwashed masses of those who don’t work at Google, tough luck I supposed.
Fifth, when was the content first indexed. More significantly, when was the content last updated. Important? May be, gentle reader. May be.
There are several other conditions as well. For the purposes of a blog post, I want to make clear: The person who is annoyed with search should have been annoyed decades ago. These time problems are not new, and they are persistent.
The author with a penchant for tardy profanity stated:
Part of the issue in this specific case is that they’ve started ignoring settings for displaying results from specific time periods. It’s definitely not the whole issue though, and not something new or specific to phone searches. Now, I’ve always been biased towards the new – books, tech, everything, but I can’t help but feel that a lot of things which were done pretty well before are done worse today. We do have better technology, yet we somehow build inferior solutions with it all too often. Further, if they had the same bias of showing me only recent results I’ll understand it better, but that’s not even the case. And yes, I get that the incentives of users and providers don’t align perfectly, that Google isn’t your friend, etc. But what is DDG’s excuse? As for the Case Study part, and me saying this isn’t simply a rant – I lied, hence the quotation marks in the title. Don’t trust everything you read, especially the goddamn dates on your search results.
The write up omits a few other minor problems with modern search and retrieval systems. Yep, this includes Reddit, LinkedIn, and a bunch of others. Let me provide a few dot points:
- Poorly implemented Boolean search
- Zero information about what’s in an index
- Zero information about what’s excluded from and index and why
- Minimal auto linking to information about an “author” or the “source” of the content
- No data to make a precision or recall calculation possible and reproducible
- No data to make it possible to determine overlap among Web indexes. Analyses must be brute forced. Due to the volatility, latency, and editorial vagaries of ad supported Web search systems, data are mostly suggestive.
Why? Why are none of these dot points operative?
Answer: Too expensive, too hard, not appropriate for our customers, and “What are you talking about? We never heard of half these issues you identified.”
Net net: Years ago I wrote an article for Searcher Magazine, edited at the time by Barbara Quint, a bit of an expert in online information retrieval. She worked at RAND for a number of years as an information expert. She said, “Do you really want me to use the title ‘Search Sucks’ on your article.” I told her, use whatever title you want. But if you agree with me, go with “sucks.” She used “sucks”. Let’s see that was a couple of decades ago.
Did anyone care? Nope. Does anyone care today? Nope. There you go.
Stephen E Arnold, January 20, 2020
DuckDuckGo Lands for European Search Users
January 14, 2020
I read “DuckDuckGo Beats Microsoft Bing In Google’s New Android Search Engine Ballot.” There have been numerous reports about this decision.
Digital Information World is a representative write up in today’s world of Google EU analysis. DarkCyber noted:
The introduction of this “choice screen” seems to be a clear response to the antitrust ruling from the European Union during last March and how Google was fined $5 billion by EU regulators. According to them, Google was playing illegally in tying up the search engine to its browser for mobile OS.
Okay. But how does a search engine get listed? We learned:
you can expect Google to not show search engines which are popular but the ones whose providers are willing to pay well.
The write up includes a run down of what search options will be displayed in each EU country. The ones we spotted are:
- DuckDuckGo
- GMX
- Info.com
- Privacy Wall
- Qwant
- Yandex.
Bing is a no show as are Giburu, iSeek, Mojeek, Yippy, and others. It is worth noting that some of these outfits are metasearch engines. This means that the systems send queries to Bing, Google, and other services and aggregate the results. Dogpile and Vivisimo were metasearch engines. DuckDuckGo and Ixquick (StartPage) are metasearch engines`. The reason metasearch is available boils down to cost. It is very expensive to index the public Web.
The DarkCyber team formulated a few hypotheses about the auction, the limitations on default search engines, and the dominance of Google search in Europe; for example, Google accounts for more than 95 percent of the search traffic in Denmark. The same situation exists in Germany and other EU countries.
Will these choices make any difference? Sure, for small outfits like DuckDuckGo any increase in traffic is good news. But will the choices alter Google’s lock on search queries from Europe?
Not a chance.
Does anyone in the EU government know? Probably not. Do these people care? Not to much.
Remember one of my Laws of Information: Online generates natural monopolies. Here’s another Law: User behavior is almost impossible to change once mental memory locks in.
So Google gets paid and keeps on trucking.
Stephen E Arnold, January 14, 2020
Enterprise Search and the AI Autumn
January 13, 2020
DarkCyber noted this BBC write up: “Researchers: Are We on the Cusp of an AI Winter?” Our interpretation of the Beeb story can be summarized this way:
“Yikes. Maybe this stuff doesn’t work very well?”
The Beeb explains in Queen’s English based on quotes of experts:
Gary Marcus, an AI researcher at New York University, said: “By the end of the decade there was a growing realization that current techniques can only carry us so far.”
He [Gary Marcus and AI wizard at NYU] thinks the industry needs some “real innovation” to go further. “There is a general feeling of plateau,” said Verena Rieser, a professor in conversational AI at Edinburgh’s Herriot Watt University. One AI researcher who wishes to remain anonymous said we’re entering a period where we are especially skeptical about AGI.
Well, maybe.
But the enterprise search cheerleaders have not gotten the memo. The current crop of “tap your existing unstructured information” companies assert that artificial intelligence infuses their often decades old systems with zip.
The story is being believed by venture outfits. The search for the next big thing is leading to making sense of unstructured text. After all, the world is awash in unstructured text. Companies have to solve this problem or red ink and extinction are just around the corner.
Net net: AI is a collection of tools, some useful, some not too useful. Enterprise search vendors are looking for a way to make sales to executives who don’t know or don’t care about past failures to index unstructured text on a company wide basis with a single system.
Stephen E Arnold, January 13, 2020
Search Your Computer
January 13, 2020
On January 10, 2020, one of the DarkCyber team needed to locate a file on a Windows 10 machine. Windows 10 search was okay, but it generated false drops and took too long.
DarkCyber tried to get its copy of ISYS Desktop Search 8 to work, but that was a non starter. We had given up on Copernic a couple of versions ago. The DTSearch trial had expired as had a couple of New Age search systems vendors had provided to us to test; for example, X1, Vound and Perfect Search, among others. Elastic was overkill. Yikes.
We then checked our files for “desktop search” and located links to these articles:
- Microsoft Windows 10 Search Indexer Diagnostics
- WizFile Is an Ultra-Fast Windows Search Tool
- UltraSearch, Fast Windows File Finder
- Everything Desktop Search
- FileSearchy Is a Fast Windows Search Alternative
- Windows Search Replacement Fileseek
- SearchMyFiles, A Versatile Desktop Search for Windows
We found a couple of these programs useful. In fact, the Everything software, version 1.4 did the trick for us.
We wanted to thank Martin Brinkmann for his articles which provided useful links and helpful information to us. Good job!
Stephen E Arnold, January 13, 2020
Lucidworks: Beyond Search for Sure
January 9, 2020
Lucid Imagination experienced what DarkCyber recalls as a bit of turmoil. From the git go, there was tension in the open sourcey ranks. One of the founders was unceremoniously given an opportunity to find his future elsewhere. Then there was the game of Revolving Door Presidents. Next was the defection of some lucid thinkers to Amazon, not in Seattle but just up the 101 to some non descript buildings. Like a law of nature another round of presidential revolving doors. Along the way, more investors wrote checks for what was an open source play based on Lucene/Solr. (I know that writing the two “names” together does not capture the grandiosity of the conception of community supported search and the privately held companies efforts to create a huge, billion dollar information access business. Sigh.
Now Lucidworks (which I automatically interpret as the phrase “Lucidworks. Really?”) has acquired an eCommerce vendor. Hello, what’s happening Magento, Mercado, Shopify, and Amazon. Yep, Amazon. But doesn’t Amazon have search too? Trivial point. Lucidworks is going to turn the $200 million in investment capital, an interface scripting engine, open source software, and Cirrus10 (an ecommerce service provider) into billions. Yes, billions!
According to “Lucidworks Acquires Cirrus10, Global Ecommerce Service Provider, to Deepen Domain Expertise and Become a Leader in Digital Commerce Solutions” states:
Lucidworks, leader in AI-powered search, acquires Cirrus10, ecommerce solutions expert with more than 100 ecommerce customers. Lucidworks and Cirrus10 have worked together as partners for the past two years and now combine their domain expertise to provide more targeted solutions for different domains in the fast-moving ecommerce market.
The Yahoo news story points out that Lucidworks’ secret sauce is a system:
produces relevant results, recommends products that meet customer goals, and predicts shopper intent to create a more engaging experience.
And don’t forget artificial intelligence. AI! Obviously.
But whose AI? The answer appears to be AI from Cirrus10. DarkCyber noted this statement from a co founder of the ecommerce service provider:
“Fusion is the world’s only platform for extensible AI-driven search. Fusion elevated our service offerings by giving us a framework for exploring machine learning with our customers, and using it, we can build personalized and scalable relevancy models without a black box or army of data scientists. By combining Lucidworks search and AI expertise with our deep experience in the ecommerce space we can cement our role as digital commerce solution leaders.”—Peter Curran, Cirrus10
What appears to be the business strategy for Lucidworks is to get something that generates sustainable revenue, allows the company to upsell Cirrus10’s customers, and differentiate Lucidworks from the competitors in plain old search.
There are competitors; for example, outfits with venture capital backers demanding results (Algolia, Coveo). Also, open sourcey solutions (Drupal Commerce, Magento Community Edition) and small, feisty outfits like SLI Systems and EasyAsk). Note: This is a partial list. I almost forget companies like Amazon, eBay, and Google.
DarkCyber interprets the “beyond search” phrase as an attempt to make a 12 year old company into a revenue and profit machine.
DarkCyber, which is an annex to our blog Beyond Search, wishes the clear thinkers a great 2020. The question “Lucidworks. Really?” could be answered as long as AI, NLP, machine learning, open source, and synergy produce a winner, not a horse designed by a committee.
Stephen E Arnold, January 9, 2019
-
- Subscribe to Beyond Search
Feature archive
News archive
-
