Analytics: From Predictions to Prescriptions

October 19, 2018

I read an interesting essay originating at SAP. The article’s title: “The Path from Predictive to Prescriptive Analytics.” The idea is that outputs from a system can be used to understand data. Outputs can also be used to make “predictions”; that is, guesses or bets on likely outcomes in the future. Prescriptive analytics means that the systems tell or wire actions into an output. Now the output can be read by a human, but I think the key use case will be taking the prescriptive outputs and feeding them into other software systems. In short, the system decides and does. No humans really need be involved.

The write up states:

There is a natural progression towards advanced analytics – it is a journey that does not have to be on separate deployments. In fact, it is enhanced by having it on the same deployment, and embedding it in a platform that brings together data visualization, planning, insight, and steering/oversight functions.

What is the optimal way to manage systems which are dictating actions or just automatically taking actions?

The answer is, quite surprisingly, a bit of MBA consultantese: Governance.

The most obvious challenge with regards to prescriptive analytics is governance.

Several observations:

  • Governance is unlikely to provide the controls which prescriptive systems warrant. Evidence is that “governance” in some high technology outfits is in short supply.
  • Enhanced automation will pull prescriptive analytics into wide use. The reasons are one you have heard before: Better, faster, cheaper.
  • Outfits like the Google and In-Q-Tel funded Recorded Future and DarkTrace may have to prepare for new competition; for example, firms which specialize in prescription, not prediction.

To sum up, interesting write up. perhaps SAP will be the go to player in plugging prescriptive functions into their software systems?

Stephen E Arnold, October 19, 2018

Google: Online to Brick and Mortar Cross Correlation

August 31, 2018

Our research suggests that Amazon may have a slight edge in the cross correlation of user data. Google, whether pulling a me too or simply going its own way, has decided to link online and brick and mortar data.

The effort was revealed in “Google and MasterCard Cut a Secret Ad Deal to Track Retail Sales.” Amazon has access to some data which makes it possible for those with appropriate system access to perform analyses of Amazon customers’ buying behavior.

According to the write up:

For the past year, select Google advertisers have had access to a potent new tool to track whether the ads they ran online led to a sale at a physical store in the U.S. That insight came thanks in part to a stockpile of MasterCard transactions that Google paid for. But most of the two billion MasterCard holders aren’t aware of this behind-the-scenes tracking. That’s because the companies never told the public about the arrangement.

To be fair, I am not sure any of the financial services and broker dealer firms provide much output about the data in their possession, who has access to these data, and what use cases are applicable to these data.

From my vantage point in Harrod’s Creek, Kentucky, Google can find its own use cases for Mastercard data.

One question: Does Mastercard pay Amazon to process its data, or does Amazon pay Mastercard?

Google, if the information in the real news article is accurate, is paying for data.

I will address Amazon’s real time streaming data marketplace in my upcoming lecture in Washington, DC. If the information in the US government document I cite in my talk in correct, Google has to shift into high gear with regard to cross correlation of shopper data.

Stephen E Arnold, August 31, 2018

The Obvious: Business Intelligence Tools May Need Clarity

August 14, 2018

Artificial intelligence and business have been a natural pair since the moment we began speculating about this technology. However, we are currently in a sort of golden age of AI for business (or drowning in a swamp of it, depending who you ask) and we could all use a little help sorting through the options. That’s why a recent Data Science Central story “A Comparative Analysis of Top 6 BI and Data Visualization Tools in 2018” seemed so relevant.

According to the story:

“It is often hard to separate the facts from fiction when evaluating various business intelligence (BI) tools, as every BI vendor markets their product as the only “best” solution, often flooding the Internet with biased reviews. If you want to understand the functional product value, avoid the hype and useless clicking through endless pages of partial reviews, you’ve come to the right place.”

This is a very important breakdown and it goes over some really compelling programs, depending on your needs. This seems to be a trend in the industry as we become awash in BI choices. Recently, we also discovered a valuable contrast looking at augmented analytics versus business intelligence tools. What seems obvious is that developers are trying to provide point and click math insight and expertise to individuals who may lack a firm foundation in evaluating data quality, statistics, and other disciplines. No, majoring in medieval literature is not what is needed to make sense of data. To be fair, some find art in proofs.

Insight from slick interfaces? Maybe.

Patrick Roland, August 14, 2018

Thoughtspot: Confused in Kentucky over AI for BI Plus Search Plus Analytics

August 3, 2018

i read “Nutanix Co-Founder Lures Away Its President to Be New CEO at ThoughtSpot.” The headline is a speed bump. But what puzzled me was this passage:

ThoughtSpot Inc. has hired Nutanix Inc.’s president as its new CEO. Sudheesh Nair joins ThoughtSpot about three months after the Palo Alto enterprise search business raised $145 million in a funding round that valued the company at more than $1 billion.

I added the emphasis on the phrase “enterprise search business.”

Search is not exactly the hottest buzzword around these days. After shock from the FAST Search & Transfer and IBM Watson adventures I hypothesize.

Now here’s the pothole: The ThoughtSpot Web site states:

Search & AI Driven analytics.

I noted the phrase “next generation analytics for the enterprise.” Plus, ThoughtSpot is a platform.

But what about artificial intelligence? Well, that’s part of the offering as well.

Remarkable: A Swiss Army knife. Many functions which may work in a pinch and certainly better than no knife at all.

But what’s the company do? Gartner suggests the firm has vision.

That helps. The first time around with FAST ESP and IBM Watson-like marketing the slow curves went right by the batters and the buyers. The billion dollar valuation is juicy as well. Another Autonomy? Worth watching.

Stephen E Arnold, August 3, 2018

Data Wizard: School or Short Cut?

August 2, 2018

With the increase focus on data analytics and search, the role of data scientists has changed drastically over the last decade, or heck, even over the last twelve months. With that increasing dependence on their skills and the continual flexibility of their world, higher ed has been responding. Turns out, these number crunchers are becoming increasingly educated, according to a fascinating article from Kaggle, “The State of ML and Data Science in 2017.”

  • The survey spoke with thousands of machine learning and data science experts and found a variety of insights, like how 41.8% have a Master’s degree, but only 15% have doctorals.
  • “What is your highest level of formal education?
  • “So, should you get that next degree? In general, the highest percentage of people in working data science, obtained a Master’s degree. But those people in the highest salary ranges ($150K – $200K and $200k+) are just as likely to have a doctoral degree.”

Many schools are beginning to offer data science programs for undergrads and grad students, however, universities are now struggling to define what this fluid field exactly, “is”. The University of Houston had to grapple with just such an issue and the results were vague at best. But, we’d say these baby steps are in the right direction.

Beyond Search believes that some “data experts” just tweak their LinkedIn profiles. Easy. Quick. Marketing.

Patrick Roland, August 2, 2018

Amazon Intelligence Gets a New Data Stream

June 28, 2018

I read “Amazon’s New Blue Crew.” The idea is that Amazon can disintermediate FedEx, UPS (the outfit with the double parking brown trucks), and the US Postal Service.

On the surface, the idea makes sense. Push down delivery to small outfits. Subsidize them indirectly and directly. Reduce costs and eliminate intermediaries not directly linked to Amazon.

FedEx, UPS, and the USPS are not the most nimble outfits around. I used to get FedEx envelopes every day or two. I haven’t seen one of those for months. Shipping vis UPS is a hassle. I fill out forms and have to manage odd slips of paper with arcane codes on them. The US Postal Services works well for letters, but I have noticed some returns for “addresses not found.” One was an address in the city in which I live. I put the letter in the recipient’s mailbox. That worked.

The write up reports:

The new program lets anyone run their own package delivery fleet of up to 40 vehicles with up to 100 employees. Amazon works with the entrepreneurs — referred to as “Delivery Service Partners” — and pays them to deliver packages while providing discounts on vehicles, uniforms, fuel, insurance, and more. They operate their own businesses and hire their own employees, though Amazon requires them to offer health care, paid time off, and competitive wages. Amazon said entrepreneurs can get started with as low as $10,000 and earn up to $300,000 annually in profit.

Now what’s the connection to Amazon streaming data services and the company’s intelligence efforts? Several hypotheses come to mind:

  • Amazon obtains fine grained detail about purchases and delivery locations. These are data which no longer can be captured in a non Amazon delivery service system
  • The data can be cross correlated; for example, purchasers of a Kindle title with the delivery of a particular product; for example, hydrogen peroxide
  • Amazon’s delivery data make it possible to capture metadata about delivery time, whether a person accepted the package or if the package was left at the door and other location details such as a blocked entrance, for instance.

A few people dropping off packages is not particularly useful. Scale up the service across Amazon operations in the continental states or a broader swatch of territory and the delivery service becomes a useful source of high value information.

FedEx and UPS are ripe for disruption. But so is the streaming intelligence sector. Worth monitoring this ostensible common sense delivery play.

Stephen E Arnold, June 28, 2018

DarkCyber for June 5, 2018: Amazon and Its LE and Intelligence Services

June 5, 2018

The DarkCyber for June 5, 2018, is now available at or on Vimeo at

This week’s DarkCyber presents an extract from Stephen E Arnold’s lectures at the Prague Telestrategies ISS conference. The conference is designed for security, intelligence, and law enforcement professionals in Europe.

Stephen’s two lectures provided attendees with a snapshot of the services Amazon’s streaming data marketplace offer to customers, developers, and entrepreneurs.

Stephen said:

The Amazon platform is positioned to provide a robust, innovative way to anonymize digital currency transactions and perform the type of analyses needed to deal with bad actors and the activities.

The information was gleaned from Amazon conference lectures, Amazon’s Web logs and documentation, and open source documents.

For example, one public document stated:

“… A law enforcement agency may be a customer and may desire to receive global Bitcoin transactions, correlated by country, with USP data to determine source IP addresses and shipping addresses that correlate to Bitcoin addresses.”

Coupled with Amazon’s facial recognition service “Rekognition” and Amazon’s wide array of technical capabilities, Amazon is able to provide specialized content processing and data services.

Stephen stated:

Instead of learning how to use many different specialized systems, the Amazon approach offers a unified capability available with a Kindle-style interface. This is a potential game changer for LE, intel, and security service providers.

In this week’s DarkCyber video, Stephen provides an eight minute summary of his research, including the mechanisms by which new functions can be added to or integrated with the system.

A for fee lecture about what Stephen calls “Amazon’s intelligence services” is available on a for fee basis. For information, write darkcyber333 at yandex dot com.

Kenny Toth, June 5, 2018

DarkCyber, May 29, 2018, Now Available

May 29, 2018

Stephen E Arnold’s DarkCyber video news program for Tuesday, May 29, 2018, is now available.

This week’s story line up is:

  • The “personality” of a good Web hacker
  • Why lists are replacing free Dark Web search services
  • Where to find a directory of OSINT software
  • A new Dark Web index from a commercial vendor.

You can find this week’s program at either or on Vimeo at

On June 5, 2018, Stephen will be giving two lectures at the Telestrategies ISS conference in Prague. The audiences will consist of intelligence, law enforcement, and security professionals from Europe. A handful of attendees from other countries will be among the attendees.

On Tuesday, June 5, 2018, Stephen will reveal one finding from our analysis of Amazon’s law enforcement, war fighting, and intelligence services initiative.

Because his books have been reused (in several cases without permission) by other analysts, the information about Amazon is available via online or in person presentations.

The DarkCyber team has prepared short video highlighting one research finding. He will include some of the DarkCyber research information in his Prague lectures.

The Amazon-centric video will be available on Tuesday, June 5, 2018. After viewing the video, if you want the details of his for fee lecture, write him at darkcyber333@yandex dot com. Please, put “Amazon” in the subject line.

Several on the DarkCyber team believe that most people will dismiss Stephen’s analysis of Amazon. The reason is that people buy T shirts, books, and videos from the company. However, the DarkCyber research team has identified facts which suggest a major new revenue play from the one time bookseller.

Just as Stephen’s analyses of Google in 2006 altered how some Wall Street professionals viewed Google, his work on Amazon is equally significant. Remember those rumors about Alexa recording what it “hears”? Now think of Amazon’s services/products as pieces in a mosaic.

The picture is fascinating and it has significant financial implications as well.

Enjoy today’s program at this link.

Kenny Toth, May 29, 2018

One View of the Amazon Game Plan

May 27, 2018

I read “Invisible Asymptotes.” Job One for me was trying to match the meaning of “asymptote” with the research my DarkCyber team has conducted into one slice of Amazon’s business roll outs in the last three years.

As you know, an “asymptote” is a mathy way of saying “you can’t get from here to there.” According to Wolfram Mathword:

An asymptote is a line or curve that approaches a given curve arbitrarily closely.

Here’s a diagram. No equations, I promise.

Image result for asymptote

This diagram suggests a business angle to the “asymptote” reference: No matter what you do, it requires effort and a commitment to “quality”. The good news is that although one can quantify time, one cannot quantify “quality” or “perfection.” Okay, gerbil, run in that Ferris wheel gizmo in your cage.

The write up points out:

We focus so much on product-market fit, but once companies have achieved some semblance of it, most should spend much more time on the problem of product-market unfit.

I am not exactly sure what “unfit” means. The author provides a hint:

For me, in strategic planning, the question in building my forecast was to flush out what I call the invisible asymptote: a ceiling that our growth curve would bump its head against if we continued down our current path.

Okay, the idea seems to be that if Amazon enters a new market, the “invisible asymptote” is what slows growth or stops it completely. (Is this the Amazon phone’s and the slowing sales of Alexa in the face of competition from the Google Home device?)

The reason Amazon cannot grow ever larger is because of an “invisible asymptote”; that is, a factor which prevents Amazon from becoming a company that Vanderbilt, JP Morgan, and John D. Rockefeller would have wished they had.

The write up does not discuss Amazon’s semi-new entrance into the law enforcement and intelligence market. That’s a push I am exploring in my lecture at the Telestrategies ISS conference in early June.

The focus shifts to a more mundane and increasingly problematic aspect of Amazon’s business: Shipping fees. Fiat, law, and the costs of fuel are just a few of the challenges Amazon faces. I am not sure these are “invisible”, but let’s trudge forward.

Twitter becomes that foundation for social media. I noted this passage:

No company owes it to others to allow people to build direct competitors to their own product.

If Amazon wants to make law enforcement and intelligence services into a major revenue stream, I think the first evidence of this intent will be cutting off the vendors using Amazon’s infrastructure to serve their clients now. (Keep in mind that most of the specialist vendors in the LE and intel space use Amazon as plumbing. To cite one example, Marinus, the anti human trafficking group, follows this approach.

The author brings up Snapchat and other social media companies. I find this example important. Amazon’s facial recognition capabilities hit out radar when my team was assembling “CyberOSINT: Next Generation Information Access”, written in 2014 and published in 2015.

We did not include Amazon in my review of LE and intel tools because I had only references in some Amazon conference videos, a few patent applications which were particularly vague about applications in the Background and Claims sections of the documents, and chatter at meetings I attended.

The American Civil Liberties Union has made a bit of noise about Amazon’s facial recognition system. Recognition is spelled “rekognition”, presumably to make it easy to locate in the wonky world of Bing and Google search. The reason is that Amazon’s facial recognition system can identify individuals and cross tabulate that piece of information with other data available to the Amazon system.

Instant bubblegum card.

The write up “Invisible Asymptote” talks about social content and social rich media without offering any comment about the importance of these types of data to Amazon’s intelligence services or its marketplace.

The conclusion of the 10,000 word essay is more “invisible asymptote”. Is this Amazon’s the secret sauce:

Lastly, though I hesitate to share this, it is possible to avoid invisible asymptotes through sheer genius of product intuition.

Here’s a diagram from the essay which looks quite a bit like the self help diagram I included at the top of this Beyond Search post:


Several observations:

  1. The write up makes clear that if anyone thinks Amazon’s platform is neutral, think again.
  2. Strategists at Amazon are not able to “see” and “explain” the nuts and bolts of the “we may be a monopoly but” approach of the Big Dog of the Amazon
  3. The long, long essay does not stray very far from selling stuff to consumers who love free shipping.

Taken as a group of three perceptions, what does this say about Amazon?

For me, I think companies using Amazon’s plumbing will want to do a bit of strategizing using “What if” questions to spark discussion.

For companies behind or beneath the curve, there will be a ceiling, and it will not be easy to break through.

Amazon, on the other hand, may have break through and then replace the old ceiling with a nifty new one made of sterner stuff.

For information about our lectures about Amazon’s Next Big Thing: Intelligence Services, write me at Put Amazon Streaming Marketplace in the subject line, please.

We now offer for fee webinars and on site consulting sessions. On June 5, 2015, coincident with my two lectures in Prague before an audience of LE and intel professionals, I will release a nine minute DarkCyber video exploring some of the inventions Amazon disclosed in an April document not widely reported in the media. Watch this blog for a link.

Stephen E Arnold, May 27, 2018

You Know You Are in Deep Doo Doo When…

May 7, 2018

I flipped through the Overflight news feeds and noted several stories. Remember when you were a wee thing, and you did something wrong. Your friends knew. Your friends’ mom knew. Your mom knew. Then your father or significant parental other (SPO) knew. That may be the feeling of some of the Cambridge Analytica wunderkind.

An example is warranted:

That excellent hire Christopher Wylie has allegedly shared more information about turning clicks into votes. The good hearted wizard told the Guardian about data, target variables, and profiling. There’s even a reference to a patent (absent the patent number, the assignee, and other data which allows one to locate the referenced patent). The kimono is open and the sight does not strike me as one I would describe as attractive.

Will declaring bankruptcy allow the Cambridge Analytica “owner” to avoid further scrutiny? That seems unlikely.

Will an expert step forward and suggest that Cambridge Analytica may have precipitated the Brexit anguish? That seems unlikely.

Nevertheless, I would hypothesize that moms.

PS. Include patent identifiers when you quote patents, dear Guardian editors, please. Perhaps you too are engaging in some data shaping just on a tiny scale?

Stephen E Arnold, May 7,l 2018

Next Page »

  • Archives

  • Recent Posts

  • Meta