Amazon: Elasticsearch Bounced and Squished
October 14, 2019
DarkCyber noted “AWS Elasticsearch: A Fundamentally-Flawed Offering.” The write up criticizes Amazon’s implementation of Elasticsearch. Amazon hired some folks from Lucidworks a few years ago. But under the covers, Lucene thrums along within Amazon and a large number of other search-and-retrieval companies, including those which present themselves as policeware. There are many reasons: [a] good enough, [b] no one company fixes the bugs, [c] good enough, [d] comparatively cheap, [e] good enough. Oh, one other point: Not under the control of one company like those good, old fashioned solutions like STAIRS III, Fulcrum (remember that?), or Delphes (the francophone folks).
This particular write up is unlikely to earn a gold star from Amazon’s internal team. The Spun.io essay states:
I’m currently working on a large logging project that was initially implemented using AWS Elasticsearch. Having worked with large-scale mainline Elasticsearch clusters for several years, I’m absolutely stunned at how poor Amazon’s implementation is and I can’t fathom why they’re unable to fix or at least improve it.
I think the tip off is the phrase “how poor Amazon’s implementation is…”
The section Amazon Elasticsearch Operation provides some color to make vivid the author’s viewpoint; for example:
On Amazon, if a single node in your Elasticsearch cluster runs out of space, the entire cluster stops ingesting data, full stop. Amazon’s solution to this is to have users go through a nightmare process of periodically changing the shard counts in their index templates and then reindexing their existing data into new indices, deleting the previous indices, and then reindexing the data again to the previous index name if necessary. This should be wholly unnecessary, is computationally expensive, and requires that a raw copy of the ingested data be stored along with the parsed record because the raw copy will need to be parsed again to be reindexed. Of course, this also doubles the storage required for “normal” operation on AWS. [Emphasis in the original essay.]
The wrap up for the essay is clear from this passage:
I cannot fathom how Amazon decided to ship something so broken, and how they haven’t been able to improve the situation after over two years.
DarkCyber’s team formulated several observations. Let’s look at these in the form of questions and trust that some young sprites will answer them:
- Will Amazon make its version of Elasticsearch proprietary?
- Are these changes designed to “pull” developers deeper into the AWS platform, making departure more difficult or impossible for some implementations?
- Are the components the author of the essay finds objectionable designed to generate more revenue for Amazon?
Stephen E Arnold, October 14, 2019
The Chernobyl Control Rooms of the Digital Era
October 11, 2019
A minor error. Chernobyl melted down. Radiation galore. According to Red Ferret (great name!), a motivated individual can now tour the control room of that nuclear plant. For details, navigate to “Chernobyl Control Room – You Can Now Go Inside the Infamous Site.” [Note: This story has an interesting url. If the link doesn’t work, I am not sure a Bing, Google, or Yandex query will point to the source.]
My thought is, “Will those in the future visit the offices in which major companies took decisions that are Chernobyl-like in their impact?”
Here are a handful of examples of future tour destinations which might become vacations of the future:
- The office in which Tim Cook at Apple decided to explain that Apple was making an independent, objective decision about an app. This app informed iPhone users of Hong Kong police movements. For details, see the Guardian.
- The office at Google where executives made the decision that the HKmap.live would endanger lives. See the paywalled Wall Street Journal here.
- The work area of the person who “secretly” recorded Mark Zuckerberg explaining that he would go to the mats to fight the breakup of Facebook. See Bloomberg’s story here.
A new way to boost revenue for the whiz bang tech companies?
Could be.
Stephen E Arnold, October 11, 2019
Cyber Security: Hand Waving Instead of Results?
October 9, 2019
Beta News published what DarkCyber views as a bit of an exposé. “Security Professionals Struggle to Measure Success within the Business” recycles information which appears to come from a services firm called Thycotic. (DarkCyber has not been able to locate the referenced report.)
Among the statements in the write up, DarkCyber noted these as particularly thought provoking:
- “Nearly half (44 percent) [of those in the Thycotic sample] say their organization struggles to align security initiatives with the business’s overall goals”
- “More [than] 35 percent aren’t clear what the business goals are”
- “The most commonly used metric is to count the number of security breaches (56 percent) followed by the time taken to resolve a breach (51 percent). It appears, however, that these criteria may not be terribly useful.”
- “Around two in five (39 percent) say they have no way of measuring what difference past security initiatives have made to the business.”
- “36 percent agree it’s not a priority for them to measure security success once initiatives have been rolled out.”
These are interesting results. If an information unit cannot demonstrate that their security efforts are useful, budgets will be cut or staff rotated. Vendors will be sucked into this negative atmosphere.
Are cyber security vendors delivering solutions which work? Are customers able to use these products? Will executives lose confidence in their staff and vendors because security challenges continue to bedevil the organization?
The big question, however, remains:
Do the hundreds of vendors have solutions that are useful?
Paying invoices for hand waving can be an issue in some organizations. Well funded cyber security start ups might run into choppy waters after several years of smooth sailing and the support of investors who believe that nothing can derail new cyber security solutions.
Stephen E Arnold, October 9, 2019
Will the Real Disintermediating Entity Step Forward?
October 3, 2019
Big Microsoft day. It’s back in the mobile phone business. Sometime next year, probably coincident with a delayed Win 10 update, the Microsoft Surface Dual Screen Folding Android Phone becomes available. You can get the scoop and one view of Microsoft’s “we’re in phones again strategy” in “Microsoft’s Future Is Built on Google Code.” Do I agree? Of course not, that’s my method: Find other ways to look at an announcement.
The write up posits:
Google underpins Microsoft’s browser and mobile OS now.
I noted this statement as well:
… it could come as quite a shock that the CEO of Microsoft doesn’t care that much about operating systems. But there it is, in black and white. Microsoft obviously isn’t abandoning Windows — it announced a new version of it today — but it matters much more to Microsoft that you use its services like Office. That’s where the money is, after all.
Money. A phone that is not here?
But there’s another side to Microsoft. Amazon, the evil enemy, makes it possible run Microsoft on the AWS platform.
Now who is going to disintermediate whom?
Will Google get frisky and nuke Microsoft’s Android love?
Will Amazon just push MSFT SQLServer and other Microsoft innovations off the AWS platform and suck up the MSFT business.
Will Microsoft find that loving two enemies is more a management hassle than getting a Windows 10 server out the door?
Will Amazon and Google escalate their skirmishes and take actions that miss one enemy and plug the Redmond frenemy?
The stakes are high. Microsoft has done a pivot with an double backflip.
Perfect 10 or broken foot? Enron tried something like Microsoft’s approach. The landing was bumpy. The cloud may not cushion a lousy landing.
Stephen E Arnold, October 3, 2109
Amazon Policeware: The Path to IBM-Style Lock In on Steroids
September 27, 2019
Quite a bit of Amazon news has flowed through the DarkCyber system. The problem is that most of the information is oblivious to Amazon’s policeware initiative. DarkCyber’s research suggests that Amazon is building a surveillance system. One DarkCyber team member said, “Amazon is building what China has been working on for several years.” Is this DarkCyber researcher correct? Who knows?
I do want to provide a diagram from our Amazon webinar which puts Amazon’s activities into a context for enforcement. The scope of Amazon’s business strategy extends beyond local law enforcement and the Ring video doorbell activities, beyond the cloud services for several US government agencies, and beyond the company’s online businesses.
Amazon may be positioning itself to provide:
- IRS-related services associated with tax investigations
- Drug enforcement actions related to physicians who allegedly overprescribe or entities which obtain certain compounds using obfuscation methods
- SEC-related services to determine entity interaction, expenditures, and related financial activities
- Credit verification, including other financial analyses, for government and retail financial activities.
Other “extensions” are possible. What’s interesting is that few have noticed and even fewer pay much attention beyond hand waving about Alexa. There’s more than Alexa, which is a low level gateway service.
Here’s the diagram, which is copyrighted by Stephen E Arnold, operator of DarkCyber, and author of the forthcoming monograph, Dark Edge: Amazon’s Policeware Initiative.
© Stephen E Arnold, 2019.
How do you use this diagram? Just map Amazon’s most recent product announcements into the grid.
The DarkCyber Amazon policeware webinar walks through the tactics and the strategy for this “in plain sight” play. Analysts, journalists, policeware vendors paying Amazon to host their systems, and Microsoft-type outfits are oblivious to what is now the end game for a 12 year push by Amazon to make IBM-style lock in seem as quaint as a Model T Ford.
For those who recycle my information and claim it as your own creative output, why not be somewhat ethical and provide attribution. You know. Old-fashioned stuff like a footnote. Yep, that includes a real journalist who writes for the New York Times and the Epstein linked MIT publication, among others.
Stephen E Arnold, September 27, 2019
Google Cannot Patch Up Its Local News Service
September 27, 2019
A Xoogler had an idea for local news. Patch flopped. “Google Shutters Bulletin, Its Hyperlocal News Experiment” reports:
In a letter to users (obtained by Android Police), Google said in two weeks the Bulletin app will no longer be accessible. Users who shared content on the app will be able to download their posts until November 22nd. One reason Bulletin never took off was that it wasn’t highly publicized, so chances are few people are going to miss this.
With the erosion of daily and weekly newspapers, how does a person get information about a particular city?
Not easily and maybe not at all.
What value were those old fashioned information services? Well, those old fashioned outfits provided advertising opportunities to zippy outfits like Google.
Who cares? Probably not too many Silicon Valley types. Anyway Google tried.
Stephen E Arnold, September 27, 2019
Google Maps: Complex and Tricky for Some Users
September 12, 2019
Google Maps has become the one stop map tool due to its reliability, ease of use, accuracy, and wealth of information. The map app, however, is not as accurate as you think says Media Street in the article, “You Can’t Trust Google Maps To Find It All-Fake Businesses Are Everywhere.” The Wall Street Journal discovered that nearly eleven million businesses listed on Google Maps are fake. Other companies create the listings to boost their own business info ahead of the competition and others are scams.
In 2018, Google removed more than three million fake listings and more than 90% were removed before a user saw them. Users reported 250,000 fake profiles, while Google’s own system flagged 85% of the removals. Google encourages users to report anything suspicious or appears fraudulent.
Google does its best to track down the fake businesses:
“Google typically verifies if a business is legit by calling, mailing a postcard, or emailing a numerical code that is then entered on the website. It’s a pretty easy process for savvy scammers who likely use fake addresses and businesses for their listings anyway. Knowing this, the company says that they are constantly developing new ways to weed out fake listings, but can’t elaborate on what they are due to the sensitive nature.
Every month Maps is used by more than a billion people around the world, and every day we and our users work as a community to improve the map for each other,’ Google Maps’ product director, Ethan Russell, wrote in the blog post. ‘We know that a small minority will continue trying to scam others, so there will always be work to do and we’re committed to keep doing better.’”
There are ways to be wise to scams. You can avoid businesses that have names that included “dependable” or “emergency,” screen your phone calls, do not trust all the reviews, and also do your own research. See if the business has a Web site, check other review sites, view social media accounts, etc. Never forget to trust your gut instinct either.
Whitney Grace, September 12, 2019
Read Two Google Ads and Call Me in the Morning
September 10, 2019
We know Google has been branching out every which way it can, largely through divisions like X Lab, Sidewalk Labs, Jigsaw, and Deep Mind, among others. Now Health Impact News reports that “Google Joins the Pharmaceutical Industry” through its healthcare divisions Verily Life Sciences and Calico Labs. Writer Kate Raines seems suspicious of Google’s motives and its ties to “Big Pharma.” She describes some projects that Verily is working on with partners in the pharmaceutical industry. (That company started out pursuing miniaturization tech and machine learning for projects like smart lenses.) What we find interesting here is Google’s heavy push into the healthcare arena—are they chasing Amazon?
We learn:
“Verily now partners with a number of pharmaceutical companies that develop vaccines on projects ranging from smart lenses with Alcon (a subsidiary of Novartis) and surgical robotics with Johnson & Johnson to early identification and intervention in chronic diseases with Merck Sharp & Dohme and diabetes management with Sanofi. Verily is partnered with Gilead on profiling the immune system to clarify the biological mechanisms of autoimmune disease and with Verve Therapeutics on nanoparticle formulations. Verily is also partnered with GlaxoSmithKline, the world’s largest vaccine manufacturer, in the development of bioelectronic medicine. With the creation of Galvani Bioelectronics in collaboration with GlaxoSmithKline, Verily now has its own pharmaceutical company that is working to ‘enable the research, development and commercialization of bioelectronic medicines,’ which aim to treat disease using miniaturized implanted devices. Another of Verily’s projects is the development of the ‘sterile insect technique’ to manipulate mosquito populations by releasing sterile male mosquitoes that will reduce the populations of insects carrying such diseases as dengue, Zika, chikungunya and yellow fever.
The company has also entered the clinical study arena, first with its own study called Baseline, which seeks to connect potential study participants with clinical research groups.”
That is indeed a lot of cooperation; whether that is a good or a bad thing we leave our gentle readers to decide. Raines spends less time describing Calico, which is focused on increasing the human lifespan. It was established by the former CEO of vaccine developer Genentech and now employs a Head of Drug Development who was once a VP at vaccine developer Amgen. Very suspicious.
Raines concludes by noting that Google’s search algorithm specifically makes it difficult to find certain information about vaccines that runs afoul of the “government and pharmaceutical industry.” I think others call that addressing the scourge of fake news, specifically anti-vaxer propaganda in this case.
Cynthia Murrell, September 10, 2019
Amazon and Its Dark Edge
September 5, 2019
Later this year, I will make available a free essay about Amazon. Those who want the document will have to provide an email address and some other information. The essay is titled “Dark Edge.”
I am mentioning this because Information Age published “Is the Cloud and AI Becoming Two Sides of the Same Coin?” Despite the wonky subject-verb in the title, I noted a couple of points in the write up which raise issues discussed in “Dark Edge.”
First, the basic idea is that if a company embraces the cloud, smart software comes as part of the deal. That’s a good point, but it does not go far enough. In fact, most of the cloud vendors don’t think beyond generating more revenue than they did the previous quarter and figuring out how to take advantage of the dazed and confused companies trying to “reinvent” themselves. We noted this statement:
perhaps it’s time we realized that the AI and cloud computing industries are not mutually exclusive.
Yes, time. Just past time.
Second, we circled these two statements:
“Cloud adoption is motivating enterprises to undertake more proofs of concept in their firms with AI because it’s easier than ever before to get started,” said David Schatsky, managing director at Deloitte LLP.
According to Schatsky, this path is also becoming more attractive to enterprises as cloud providers continue developing AI offerings to business functions, without big upfront costs.
Easy. Cheap.
Key points.
There are a number of questions which the write up sidesteps; for example:
- What constitutes a win for a cloud platform? More revenue? Market share? More developers?
- What are the downsides for a digital environment which reflects the “winner take all” trajectory of outfits like Facebook, Google, Netflix, etc.?
- How quickly will one the integration of the cloud and smart software become the norm for computing, apps, and enterprise solutions?
Dark Edge will address these. Watch for the essay later this year, which assumes I will have some time to assemble our research when I am tromp around a Neanderthal disco.
Stephen E Arnold, September 5, 2019
Can a Well Worn Compass Help Enterprise Search Thrive?
September 4, 2019
In the early 1990s, Scotland Yard (which never existed although there is a New Scotland Yard) wanted a way to make sense of the data available to investigators in the law enforcement sector.
A start up in Cambridge, England, landed a contract. To cut a multi year story short, i2 Ltd. created Analyst’s Notebook. The product is now more than a quarter century old, and the Analyst’s Notebook is owned by IBM. In the span of five or six years, specialist vendors reacted to the Analyst’s Notebook functionalities. Even though the early versions were clunky, the software performed some functions that may be familiar to anyone who has tried to locate, analyze, and make sense of data within an organization. I am using “organization” in a broad sense, not just UK law enforcement, regulatory enforcement, and intelligence entities.
What were some of the key functions of Analyst’s Notebook, a product which most people in the search game know little about? Let me highly a handful, and then flash forward to what enterprise search vendors are trying to pull off in an environment which is very different from what the i2 experts tackled 25 years ago. Hint: Focus was the key to Analyst’s Notebook’s success and to the me-too products which are widely available to LE and intel professionals. Enterprise search lacks this singular advantage, and, as a result, is likely to flounder as it has for decades.
The Analyst’s Notebook delivered:
- Machine assistance to investigators implemented in software which generally followed established UK police procedures. Forget the AI stuff. The investigator or a team of investigators focused on a case provided most of the brain power.
- Software which could identify entities. An entity is a person, place, thing, phone number, credit card, event, or similar indexable item.
- Once identified, the software — influenced by the Cambridge curriculum in physics — could display a relationship “map” or what today looks like a social graph.
- Visual cues allowed investigators to see that a person who received lots of phone calls from another person were connected. To make the relationship explicit, a heavy dark line connected the two phone callers.
- Ability to print out on a big sheet of paper these relationship maps and other items of interest either identified by an investigator or an item surfaced using maths which could identify entities within a cluster or an anomaly and its date and time.
Over the years, other functions were added. Today’s version offers a range of advanced functions that make it easy to share data, collaborate, acquire and add to the investigative teams’ content store (on premises, hybrid, or in the cloud), automate some functions using IBM technology (no, I won’t use the Watson word), and workflow. Imagery is supported. Drill down makes it easy to see “where the data came from.” An auditor can retrace an investigator’s action in order to verify a process. If you want more about i2, just run a Bing, Google, or Yandex query.
Why am I writing about decades old software?
The reason is that is read an item from my files as my team was updating my comments about Amazon’s policeware for the October TechnoSecurity & Digital Forensics Conference. The item I viewed is titled “Thomson Reuters Partners with Squirro to Combine Artificial Intelligence Technology and Data to Unlock Customer Intelligence.” I had written about Squirro in “Will Cognitive Search (Whatever That Is) Change Because of Squirro?”
I took a look at the current Squirro Web site and learned that the company is the leader in “context intelligence.” That seemed similar to what i2 delivered in the 1990s version of Analyst’s Notebook. The software was designed to fit the context of a specific country’s principal police investigators. No marketing functions, no legal information, no engineering product data — just case related information like telephone records, credit card receipts, officer reports, arrest data, etc.
Squirro, founded in 2012 or 2013 (there are conflicting dates online) states that the software delivers
a personalized, real-time contextual stream from the sea of information directly to your workplace. It’s based on Squirro’s digital fingerprint technology connecting personal interests and workflows while learning and refining as user interactions increase.
I also noted this statement:
Squirro combines all the different tools you need to work with unstructured data and enables you to curate a self-learning 360° context radar natural to use in any enterprise system. ‘So What?’ Achieving this reduces searching time by 90%, significantly cutting costs and allows for better, more effective decision-making. The highly skilled Swiss team of search experts has been working together for over 10 years to create a precise context intelligence solution. Squirro: Your Data in Context.
Well, 2013 to the present is six years, seven if I accept the 2012 date.
The company states that it offers “A.I.-driven actionable Insights,” adding:
Squirro is a leading AI-platform – a self-learning system keeping you in the know and recommending what’s next.
I’m okay with marketing lingo. But to my way of thinking, Squirro is edging toward the i2 Analyst’s Notebook type of functionality. The difference is that Squirro wants to serve the enterprise. Yep, enterprise search with wrappers for smart software, reports, etc.
I don’t want to make a big deal of this similarity, but there is one important point to keep in mind. Delivering an enterprise solution to a commercial outfit means that different sectors of the business will have different needs. The different needs manifest themselves in workflows and data particular to their roles in the organization. Furthermore, most commercial employees are not trained like police and intelligence operatives; that is, employees looking for information have diverse backgrounds and different educational experiences. For better or worse, law enforcement intelligence professionals go to some type of training. In the US, the job is handled by numerous entities, but a touchstone is FLETC. Each country has its equivalent. Therefore, there is a shared base of information, a shared context if you will.
Modern companies are a bit like snowflakes. There’s a difference, however, the snowflakes may no longer work together in person. In fact, interactions are intermediated in numerous ways. This is not a negative, but it is somewhat different from how a team of investigators worked on a case in London in the 1990s.
What is the “search” inside the Squirro information retrieval system? The answer is open source search. The features are implemented via software add ons, wrappers, and micro services plus other 2019 methods.
This is neither good nor bad. Using open source reduces some costs. On the other hand, the resulting system will have a number of moving parts. As complexity grows with new features, some unexpected events will occur. These have to be chased down and fixed.
New features and functions can be snapped in. The trajectory of this modern approach is to create a system which offers many marketing hooks and opportunities to make a sale to an organization looking for a solution to the ever present “information problem.”
My hypothesis is that i2 Analyst’s Notebook succeeded an information access, analysis, and reporting system because it focused on solving a rather specific use case. A modern system such as a search and retrieval solution that tries to solve multiple problems is likely to hit a wall.
The digital wall is the same one that pushed Fast Search & Transfer and many other enterprise search systems to the sidelines or the scrap heap.
Net net: Focus, not jargon, may be valuable, not just for Squirro, but for other enterprise search vendors trying to attain sustainable revenues and a way to keep their sources of funding, their customers, their employees, and their stakeholders happy.
Stephen E Arnold, September 4, 2019
- Subscribe to Beyond Search
Feature archive
News archive
-
