Amazon: Specialist in Complexity

October 22, 2019

The word “complexification” is tailor made for Amazon. A couple of examples might be helpful, right?

  • Third party sellers provide expired food. Something’s wrong it seems. Complexification of the vendor vetting, product vetting, and warehouse vetting processes might be a reason. (I am setting aside “profit at all costs” because who wants to rain on the Amazon bulldozer.
  • AWS services. Really, who can name the different types of Amazon databases. There’s an Oracle killer, an unstructured data killer, there’s an Amazon blockchain solution that’s just perfect for Dubai. Can’t keep ‘em straight? Take a cheap course in how to speak Amazon, you dynamo, you.
  • Return authorizations. Use Opera? Well, the labels don’t print correctly. Call a human? It is helpful to speak two or three languages other than English. English as she is spoken at Amazon is — well, let’s think about it this way — may not be what talking heads on CNBC speak.

But the most interesting complexity problem concerns Twitch. Twitch may be a problem for YouTube and — get this, gentle reader — Facebook.

The hitch in the git along was summarized this way by Verge’s interview with Emmett Shear, the big Twitcher. Here’s the passage I noted:

The changes are coming, Shear said, because the company didn’t think it was doing well enough when it talked to streamers about moderating their channels. There were streamers with teams that had everything working, but there were also streamers who felt overwhelmed and like they couldn’t figure out how to use all of Twitch’s moderation tools. “It popped as a problem,” Shear said. “We decided we had to do better. And I think it’s a big step in the right direction.” Twitch’s moderation philosophy, in general, comprises two parts: enforcement works on the level of the individual and on the level of the platform.

Okay, complexity, two tier moderation, and a lack of “transparency.” Transparency is an interesting word because it suggests making stuff clear. A lack of transparency means stuff is not clear.



In my recent lecture at the TechnoSecurity & Digital Forensics Conference I offered a few examples of Twitch’s challenges:

  1. Streaming gambling with links to donate money to the gamblers and tips for getting an advantage
  2. SweetSaltyPeach’s soft excitement morphing into RachelKay’s really dull doing nothing but providing a momentary glimpse of the old formula for success
  3. A first run movie available via a stream.

Net net: Amazon’s fatal flaw may be its burgeoning complexity. Not even Bezos billions can make some things simple, clear, and easy to understand.

If Twitchers can’t figure out what to do, what will lesser mortals in government agencies achieve? Let’s watch Dubai for clues.

Stephen E Arnold, October 21, 2019

Oracle Not Performant: The Gloves Are Off

October 16, 2019

I read “Migration Complete – Amazon’s Consumer Business Just Turned off its Final Oracle Database.” The world’s largest online bookstore is free from the Oracle handcuffs. No more proprietary databases. It took Amazon decades to reach this point.

According to the write up, Oracle is not “performant.” I think that means “not as good as Amazon’s data management technology.” In other words, loser or more accurately “Fair game.”

Oracle does databases, and it also is in the data licensing business. Amazon may have designs on that sector as well, but with a distinct Amazon flavor: Amazon will focus on stream data and have its own proprietary data goodies to license and use in its proprietary data management systems.


Amazon data management solutions in their various forms. Question: Where does Amazon’s blockchain data management fit in this semi-helpful, mostly opaque diagram? Answer:  DynamoDB.

The write up states:

Over the years we realized that we were spending too much time managing and scaling thousands of legacy Oracle databases. Instead of focusing on high-value differentiated work, our database administrators (DBAs) spent a lot of time simply keeping the lights on while transaction rates climbed and the overall amount of stored data mounted. This included time spent dealing with complex & inefficient hardware provisioning, license management, and many other issues that are now best handled by modern, managed database services.

One might infer that this litany of woes are not part of the Amazon data management services. DarkCyber thinks the passage is more than a catalog of Oracle problems; it is the list of Amazon benefits generated with a bit of quick editing; for example, “keeping the lights on” is an Oracle problem. Amazon delivers “lights on operation”.

When will the Amazon Oracle challenger début. Look to the United Arab Emirates and maybe suburban Virginia.

Performant. What a word.

Stephen E Arnold, October 16, 2019

Amazon: Elasticsearch Bounced and Squished

October 14, 2019

DarkCyber noted “AWS Elasticsearch: A Fundamentally-Flawed Offering.” The write up criticizes Amazon’s implementation of Elasticsearch. Amazon hired some folks from Lucidworks a few years ago. But under the covers, Lucene thrums along within Amazon and a large number of other search-and-retrieval companies, including those which present themselves as policeware. There are many reasons: [a] good enough, [b] no one company fixes the bugs, [c] good enough, [d] comparatively cheap, [e] good enough. Oh, one other point: Not under the control of one company like those good, old fashioned solutions like STAIRS III, Fulcrum (remember that?), or Delphes (the francophone folks).

This particular write up is unlikely to earn a gold star from Amazon’s internal team. The essay states:

I’m currently working on a large logging project that was initially implemented using AWS Elasticsearch. Having worked with large-scale mainline Elasticsearch clusters for several years, I’m absolutely stunned at how poor Amazon’s implementation is and I can’t fathom why they’re unable to fix or at least improve it.

I think the tip off is the phrase “how poor Amazon’s implementation is…”

The section Amazon Elasticsearch Operation provides some color to make vivid the author’s viewpoint; for example:

On Amazon, if a single node in your Elasticsearch cluster runs out of space, the entire cluster stops ingesting data, full stop. Amazon’s solution to this is to have users go through a nightmare process of periodically changing the shard counts in their index templates and then reindexing their existing data into new indices, deleting the previous indices, and then reindexing the data again to the previous index name if necessary. This should be wholly unnecessary, is computationally expensive, and requires that a raw copy of the ingested data be stored along with the parsed record because the raw copy will need to be parsed again to be reindexed. Of course, this also doubles the storage required for “normal” operation on AWS. [Emphasis in the original essay.]

The wrap up for the essay is clear from this passage:

I cannot fathom how Amazon decided to ship something so broken, and how they haven’t been able to improve the situation after over two years.

DarkCyber’s team formulated several observations. Let’s look at these in the form of questions and trust that some young sprites will answer them:

  1. Will Amazon make its version of Elasticsearch proprietary?
  2. Are these changes designed to “pull” developers deeper into the AWS platform, making departure more difficult or impossible for some implementations?
  3. Are the components the author of the essay finds objectionable designed to generate more revenue for Amazon?

Stephen E Arnold, October 14, 2019

Amazon Twitch: Some Thinking and Work to Do

October 10, 2019

I assume that this Verge story is accurate: “An Anti-Semitic Shooting in Germany Was Live-Streamed on Twitch.” Twitch allegedly said:

We are shocked and saddened by the tragedy.

Okay, but it is time for:

  • Time delays in Twitch streams
  • More aggressive content takedowns for soft porn, transmission of commercial television shows, and interesting online gambling sessions, among others
  • Elimination of a banned user under one name (SweetSaltyPeach) now streaming as RachelKay.

The Verge reports:

Today’s attack echoed the March mass shooting of Muslims in Christchurch, New Zealand — which was streamed on Facebook Live. In today’s roughly 35-minute video, a man is seen shooting two people and attempting unsuccessfully to break into the synagogue. He also gives a brief speech into the camera, railing against Jews and denying that the Holocaust happened. Two people have been confirmed dead in today’s attack, and German law enforcement has raised the possibility that multiple attackers were involved. Only one perpetrator appears in this video.

Were young kids and young adults watching murder in real time? The Verge dances around the point:

It’s unclear how many people watched the initial stream or how many copies may have been archived at Twitch — which is owned by Amazon — or on other sites. Extremism researcher Megan Squire reported that the video was also spread through the encrypted platform Telegram, with clips being viewed by around 15,600 accounts. The Christchurch shooting was viewed live by only a few people, but reuploaded roughly 1.5 million times after the attack — so dealing with the aftermath will be a real concern. Complicating this is the fact that video of the attack — from people besides the perpetrator — is newsworthy footage. But as all social networks continue to fight hate content, live videos of shootings are a uniquely sensitive issue for live-streaming platforms.

Amazon wants to be a player in the policeware market. Amazon Twitch streaming crime is one thing. I might even believe it if the driver of the Bezos bulldozer opined, “Well, that’s a lot of video to screen.”

I think streaming murder just may be more important because what advertiser wants a pre-roll before a series of killings?

Does a live stream encourage illegal activity?

DarkCyber opines that the answer is, “Yes.”

The good old days are dead just like those who were killed on the Twitch stream.

Responsibility, not arrogance may be useful.

Stephen E Arnold, October 10, 2019

Cloudera Bids to Be the Next Gen Anti Financial Crime Platform

October 10, 2019

DarkCyber read “Moving Towards the Next Gen Financial Crimes Platform.” The essay, which is two parts information and three parts marketing collateral, presents a diagram of the Cloudera anti financial crime platform. The phrase “financial crime platform” could be interpreted as the airfield for dispatching a range of malware attacks, a position in which some cloud vendors find themselves either wittingly or unwittingly. In this DarkCyber article, I will refer to the Cloudera vision as an anti financial crimes platform, hopefully to make clear that the cloud vendor is not a bad actor.

In DarkCyber’s view, there are three main points about Cloudera’s enterprise focused solution. Silos of information are a problem, and Cloudera will sweep across organizational data silos, at least that’s the idea. Here are points DarkCyber noted:

  1. The focus is on the enterprise, not on a wider scope; for example, a bank, not a number of FBI field offices, each of which operates more or less autonomously
  2. Smart software (artificial intelligence, machine learning, et al) are used at the edge to provide necessary signals about activity warranting further analysis by more numerical recipes
  3. The solution can accommodate innovations either from Cloudera or from partners.

Cloudera includes a diagram of what the solution’s broad outlines are. Here’s the illustration from the Cloudera article:


Working from right to left, data are ingested by Cloudera. The content goes into an enterprise data store. A suite of financial crime “applications” operate on the data in the Enterprise Data Store and its modules. At the right hand of the diagram analytical tools (maybe like Tibco SpotFire?), business intelligence systems, and Cloudera’s Data Science Workbench allow authorized users to interact with the system.

Cloudera’s article includes this statement:

With CDP as the foundation, intelligence gaps are mitigated by a holistic enterprise view of all customer and financial crime-related data (holistic KYC), systems, models and processes.  You will also be able to tighten the loop between detecting and responding to new fraud patterns. CDP also supports open-source advances to ensure that your teams are able to experiment with and adopt the latest technologies and methods, which helps to mitigate technology and vendor lock-in.  The diagram below illustrates the Cloudera Data Platform and its various components for enterprise management. [Emphasis in the original source]

Several observations are warranted:

  1. Vendor lock is an organic consequence of putting one’s egg in one cloud-centric basket. Although it is possible to envision a system which accepts enhancements, the write and the diagram do not include a provision for this type of extension. DarkCyber posits that restrictions will apply.
  2. The diagram has “financial crime applications” without providing much “color” or detail about these policeware components. One key question is, “Will these policeware applications run “on Cloudera” or on some other system; for example, IBM cloud which delivers Analyst Notebook functions?”
  3. The write up does not provide information about restrictions on data; for example, streaming data from telephone intercept systems.
  4. Information about functional components, application programming interfaces, and programmatic methods for the platform are not provided. DarkCyber understands the need for economy in writing, but a table or a list of suggested links would be helpful.

Why is Cloudera making this play?

DarkCyber hypothesizes that Cloudera realizes Amazon’s “as is” capabilities pose a substantial threat. Cloudera wants to stake out some territory before the Bezos bulldozer rolls through the policeware market.

Stephen E Arnold, October 9, 2019

Amazon Policeware: Getting Visible in Spite of Amazon

October 9, 2019

An enterprising reporter included some information from my Amazon research. You can find these open source factoids in “Meet America’s Newest Military Giant: Amazon.” Like good recipients of Jeffrey Epstein love, the publication will enjoin you to pay to read the recycled version of my research. Hey, that’s capitalism in action.

The write up does veer from “military giant” into policeware, a term I coined to make clear that there are platforms, applications, and tools purpose-built to support law enforcement, analysts, and investigators.


© Stephen E Arnold, 2016

You may want to read the article and take a look at the information I have published in this blog and on YouTube and Vimeo. The search systems struggle to highlight this content, but that’s the way life is in the world of ad-supported search. (Tip: To locate the information, use the search box on this Web site or you can explore these short videos at these links:

October 30, 2018

November 6, 2018

November 13, 2018

November 20, 2018

Another peek at Amazon’s activities is provided in a side mirror attached to a speeding Chevrolet Volt. “Ring’s Police Partnerships Must End, Say More Than 30 Civil Rights Groups” is an “open letter.” That document, according to CNet, “urges local lawmakers to cancel all existing police deals with Amazon’s video doorbell company.”

Good luck with that.

The CNet write up adds:

Ring has more than 500 police partnerships across the US, and a coalition of civil rights groups are calling for local governments to cancel them all. On Tuesday, tech-focused nonprofit Fight For the Future published an open letter to elected officials raising concerns about Ring’s police partnerships and its impacts on privacy and surveillance.  The letter is signed by more than 30 civil rights groups, including the Center for Human Rights and Privacy, Color of Change and the Constitutional Alliance. Along with asking mayors and city councils to cancel existing Ring partnerships, the letter also asks for surveillance oversight ordinances to prevent police departments from making these deals in the future, and also requested members of Congress to investigate Ring’s practices.

Read more

Open Source: Everything New Is Old Again

October 7, 2019

The Andreessen Horowitz open source info blitz contains some good stuff. You will want to read the essay “Open Source: From Community to Commercialization” and, if you qualify, download the pdf of lecture notes. We noted this statement from the essay about the SaaS open source business model:

In a SaaS model, you provide a complete hosted offering of the software. If your value and competitive edge is in the operational excellence of the software, then SaaS is a good choice. However, since SaaS is usually based around cloud hosting, there is the potential risk that public clouds will choose to take your open source code and compete.


We noted this statement at the end of the article:

I [Peter Levine / Jennifer Li?] believe Open Source 3.0 will expand how we think of and define open source businesses. Open source will no longer be RedHat, Elastic, Databricks, and Cloudera; it will be – at least in part – Facebook, Airbnb, Google, and any other business that has open source as a key part of its stack. When we look at open source this way, then the renaissance underway may only be in its infancy. The market and possibilities for open source software are far greater than we have yet realized.


Years ago, the DarkCyber team undertook a study of a dozen open source software vendors specializing in search and retrieval. Today, most of those vendors have embraced “artificial intelligence”, “predictive analytics”, and “natural language processing”. That’s because search is a utility and the developers and vendors of general purpose open source software have to differentiate themselves. In the course of that research, DarkCyber noted several things.

  1. Big companies in 2008 were among the most enthusiastic testers and eventually users of open source software. Why? Our data suggested that open source allowed users of commercial proprietary software more freedom to make changes. Bug fixes would often arrive in a more timely way. Plus, the IBM- and Oracle-style license fees did not come along for the ride. That is probably true in some cases today.
  2. Open source was a free lunch. The developers often contributed for the common good; others created and made available open source software as a way to demonstrate and prove their capabilities. Translation, as one person told one of my researchers, “A job, man. Big bucks.”
  3. Monetization was mostly “little plays”; that is use our free stuff and then pay for support or proprietary extensions.

Flash forward to today. Some of these three decade old findings may still be in play, but the context is now very different.

What’s changed?

For the first time, meta plays are possible. Forget the investment, merger, and acquisition angles that motivate venture capital firms. Think in terms of just using Amazon and paying for what you need.

Start ups no longer just use Microsoft because it is available and works. Start ups use Amazon because it appears to be open source, cheap or subsidized, and available globally.

The challenge this presents to open source is significant. DarkCyber is not convinced that open source developers, users of open source software, analysts, and other professionals recognize what Amazon’s meta play and strategy is doing; that is, creating a new context of open source.

Want to learn more about Amazon’s meta play for open source? Write seaky2000 at yahoo dot com and inquire about our Amazon strategy webinar. Note: It’s not a freebie.

Everthing new is old again, including vendor lock in.

Stephen E Arnold, October 7, 2019


Amazon AWS, DHS Tie Up: Meaningful or Really Meaningful?

October 7, 2019

In my two lectures at the TechnoSecurity & Digital Forensics conference in San Antonio last week, my observations about Amazon AWS and the US government generated puzzled faces. Let’s face it. Amazon means a shopping service for golf shirts and gym wear.

I would like to mention — very, very briefly because interest in Amazon’s non shopping activities is low among some market sectors — “DHS to Deploy AWS-Based Biometrics System.” The deal is for Homeland Security:

to deploy a cloud-based system that will process millions of biometrics data and support the department’s efforts to modernize its facial recognition and related software.

The system will run on the AWS GovCloud platform. Amazon snagged this deal from the incumbent Northrop Grumman. AWS takes over the program in 2021. DarkCyber estimates that the contract will be north of $80 million, excluding ECOs and scope changes.

This is not a new biometrics system. Its been up and running since the mid 1990s. What’s interesting is that the seller of golf shirts displaced one of the old line vendors upon which the US government has traditionally relied.

DarkCyber finds this suggestive which is a step toward really meaningful. Watch for “Dark Edge: Amazon Policeware”. It will be available in the next few months.

Stephen E Arnold, October 7, 2019

Will the Real Disintermediating Entity Step Forward?

October 3, 2019

Big Microsoft day. It’s back in the mobile phone business. Sometime next year, probably coincident with a delayed Win 10 update, the Microsoft Surface Dual Screen Folding Android Phone becomes available. You can get the scoop and one view of Microsoft’s “we’re in phones again strategy” in “Microsoft’s Future Is Built on Google Code.” Do I agree? Of course not, that’s my method: Find other ways to look at an announcement.

The write up posits:

Google underpins Microsoft’s browser and mobile OS now.

I noted this statement as well:

… it could come as quite a shock that the CEO of Microsoft doesn’t care that much about operating systems. But there it is, in black and white. Microsoft obviously isn’t abandoning Windows — it announced a new version of it today — but it matters much more to Microsoft that you use its services like Office. That’s where the money is, after all.

Money. A phone that is not here?

But there’s another side to Microsoft. Amazon, the evil enemy, makes it possible run Microsoft on the AWS platform.

Now who is going to disintermediate whom?

Will Google get frisky and nuke Microsoft’s Android love?

Will Amazon just push MSFT SQLServer and other Microsoft innovations off the AWS platform and suck up the MSFT business.

Will Microsoft find that loving two enemies is more a management hassle than getting a Windows 10 server out the door?

Will Amazon and Google escalate their skirmishes and take actions that miss one enemy and plug the Redmond frenemy?

The stakes are high. Microsoft has done a pivot with an double backflip.

Perfect 10 or broken foot? Enron tried something like Microsoft’s approach. The landing was bumpy. The cloud may not cushion a lousy landing.

Stephen E Arnold, October 3, 2109

Smart Dubai: An Amazon AWS Connection

October 2, 2019

Amazon AWS provides the plumbing for the Amazon owned Amazon has a “region” and “edge location” in the United Arab Emirates. Amazon is supporting an educational push to infuse those with an interest in computer science in the ways and lingo of AWS. There was an Amazon summit in the UAE as well.

I thought about these Amazon actions when I read “Smart Dubai to Have a Marketplace for Sharing and Exchanging Data by 2021.” I learned:

Smart Dubai is building a data marketplace in a bid to monetise data through centralised and decentralised platforms, a top official said.

This initiative may be a glimpse of the smart data system disclosed in US 9947043. The Dubai activity may be the testing ground for a service which may be rolled out in the US as Amazon edges toward broader investigative services for the US government’s enforcement agencies, the IRS, and the SEC, among others.

Worth monitoring or you can express your interest in DarkCyber’s AWS policeware webinar by writing benkent2020 at yahoo dot com.

Stephen E Arnold, October 2, 2019

Next Page »

  • Archives

  • Recent Posts

  • Meta