The Obvious: Business Intelligence Tools May Need Clarity

August 14, 2018

Artificial intelligence and business have been a natural pair since the moment we began speculating about this technology. However, we are currently in a sort of golden age of AI for business (or drowning in a swamp of it, depending who you ask) and we could all use a little help sorting through the options. That’s why a recent Data Science Central story “A Comparative Analysis of Top 6 BI and Data Visualization Tools in 2018” seemed so relevant.

According to the story:

“It is often hard to separate the facts from fiction when evaluating various business intelligence (BI) tools, as every BI vendor markets their product as the only “best” solution, often flooding the Internet with biased reviews. If you want to understand the functional product value, avoid the hype and useless clicking through endless pages of partial reviews, you’ve come to the right place.”

This is a very important breakdown and it goes over some really compelling programs, depending on your needs. This seems to be a trend in the industry as we become awash in BI choices. Recently, we also discovered a valuable contrast looking at augmented analytics versus business intelligence tools. What seems obvious is that developers are trying to provide point and click math insight and expertise to individuals who may lack a firm foundation in evaluating data quality, statistics, and other disciplines. No, majoring in medieval literature is not what is needed to make sense of data. To be fair, some find art in proofs.

Insight from slick interfaces? Maybe.

Patrick Roland, August 14, 2018

Thoughtspot: Confused in Kentucky over AI for BI Plus Search Plus Analytics

August 3, 2018

i read “Nutanix Co-Founder Lures Away Its President to Be New CEO at ThoughtSpot.” The headline is a speed bump. But what puzzled me was this passage:

ThoughtSpot Inc. has hired Nutanix Inc.’s president as its new CEO. Sudheesh Nair joins ThoughtSpot about three months after the Palo Alto enterprise search business raised $145 million in a funding round that valued the company at more than $1 billion.

I added the emphasis on the phrase “enterprise search business.”

Search is not exactly the hottest buzzword around these days. After shock from the FAST Search & Transfer and IBM Watson adventures I hypothesize.

Now here’s the pothole: The ThoughtSpot Web site states:

Search & AI Driven analytics.

I noted the phrase “next generation analytics for the enterprise.” Plus, ThoughtSpot is a platform.

But what about artificial intelligence? Well, that’s part of the offering as well.

Remarkable: A Swiss Army knife. Many functions which may work in a pinch and certainly better than no knife at all.

But what’s the company do? Gartner suggests the firm has vision.

That helps. The first time around with FAST ESP and IBM Watson-like marketing the slow curves went right by the batters and the buyers. The billion dollar valuation is juicy as well. Another Autonomy? Worth watching.

Stephen E Arnold, August 3, 2018

Data Wizard: School or Short Cut?

August 2, 2018

With the increase focus on data analytics and search, the role of data scientists has changed drastically over the last decade, or heck, even over the last twelve months. With that increasing dependence on their skills and the continual flexibility of their world, higher ed has been responding. Turns out, these number crunchers are becoming increasingly educated, according to a fascinating article from Kaggle, “The State of ML and Data Science in 2017.”

  • The survey spoke with thousands of machine learning and data science experts and found a variety of insights, like how 41.8% have a Master’s degree, but only 15% have doctorals.
  • “What is your highest level of formal education?
  • “So, should you get that next degree? In general, the highest percentage of people in working data science, obtained a Master’s degree. But those people in the highest salary ranges ($150K – $200K and $200k+) are just as likely to have a doctoral degree.”

Many schools are beginning to offer data science programs for undergrads and grad students, however, universities are now struggling to define what this fluid field exactly, “is”. The University of Houston had to grapple with just such an issue and the results were vague at best. But, we’d say these baby steps are in the right direction.

Beyond Search believes that some “data experts” just tweak their LinkedIn profiles. Easy. Quick. Marketing.

Patrick Roland, August 2, 2018

Amazon Intelligence Gets a New Data Stream

June 28, 2018

I read “Amazon’s New Blue Crew.” The idea is that Amazon can disintermediate FedEx, UPS (the outfit with the double parking brown trucks), and the US Postal Service.

On the surface, the idea makes sense. Push down delivery to small outfits. Subsidize them indirectly and directly. Reduce costs and eliminate intermediaries not directly linked to Amazon.

FedEx, UPS, and the USPS are not the most nimble outfits around. I used to get FedEx envelopes every day or two. I haven’t seen one of those for months. Shipping vis UPS is a hassle. I fill out forms and have to manage odd slips of paper with arcane codes on them. The US Postal Services works well for letters, but I have noticed some returns for “addresses not found.” One was an address in the city in which I live. I put the letter in the recipient’s mailbox. That worked.

The write up reports:

The new program lets anyone run their own package delivery fleet of up to 40 vehicles with up to 100 employees. Amazon works with the entrepreneurs — referred to as “Delivery Service Partners” — and pays them to deliver packages while providing discounts on vehicles, uniforms, fuel, insurance, and more. They operate their own businesses and hire their own employees, though Amazon requires them to offer health care, paid time off, and competitive wages. Amazon said entrepreneurs can get started with as low as $10,000 and earn up to $300,000 annually in profit.

Now what’s the connection to Amazon streaming data services and the company’s intelligence efforts? Several hypotheses come to mind:

  • Amazon obtains fine grained detail about purchases and delivery locations. These are data which no longer can be captured in a non Amazon delivery service system
  • The data can be cross correlated; for example, purchasers of a Kindle title with the delivery of a particular product; for example, hydrogen peroxide
  • Amazon’s delivery data make it possible to capture metadata about delivery time, whether a person accepted the package or if the package was left at the door and other location details such as a blocked entrance, for instance.

A few people dropping off packages is not particularly useful. Scale up the service across Amazon operations in the continental states or a broader swatch of territory and the delivery service becomes a useful source of high value information.

FedEx and UPS are ripe for disruption. But so is the streaming intelligence sector. Worth monitoring this ostensible common sense delivery play.

Stephen E Arnold, June 28, 2018

DarkCyber for June 5, 2018: Amazon and Its LE and Intelligence Services

June 5, 2018

The DarkCyber for June 5, 2018, is now available at or on Vimeo at

This week’s DarkCyber presents an extract from Stephen E Arnold’s lectures at the Prague Telestrategies ISS conference. The conference is designed for security, intelligence, and law enforcement professionals in Europe.

Stephen’s two lectures provided attendees with a snapshot of the services Amazon’s streaming data marketplace offer to customers, developers, and entrepreneurs.

Stephen said:

The Amazon platform is positioned to provide a robust, innovative way to anonymize digital currency transactions and perform the type of analyses needed to deal with bad actors and the activities.

The information was gleaned from Amazon conference lectures, Amazon’s Web logs and documentation, and open source documents.

For example, one public document stated:

“… A law enforcement agency may be a customer and may desire to receive global Bitcoin transactions, correlated by country, with USP data to determine source IP addresses and shipping addresses that correlate to Bitcoin addresses.”

Coupled with Amazon’s facial recognition service “Rekognition” and Amazon’s wide array of technical capabilities, Amazon is able to provide specialized content processing and data services.

Stephen stated:

Instead of learning how to use many different specialized systems, the Amazon approach offers a unified capability available with a Kindle-style interface. This is a potential game changer for LE, intel, and security service providers.

In this week’s DarkCyber video, Stephen provides an eight minute summary of his research, including the mechanisms by which new functions can be added to or integrated with the system.

A for fee lecture about what Stephen calls “Amazon’s intelligence services” is available on a for fee basis. For information, write darkcyber333 at yandex dot com.

Kenny Toth, June 5, 2018

DarkCyber, May 29, 2018, Now Available

May 29, 2018

Stephen E Arnold’s DarkCyber video news program for Tuesday, May 29, 2018, is now available.

This week’s story line up is:

  • The “personality” of a good Web hacker
  • Why lists are replacing free Dark Web search services
  • Where to find a directory of OSINT software
  • A new Dark Web index from a commercial vendor.

You can find this week’s program at either or on Vimeo at

On June 5, 2018, Stephen will be giving two lectures at the Telestrategies ISS conference in Prague. The audiences will consist of intelligence, law enforcement, and security professionals from Europe. A handful of attendees from other countries will be among the attendees.

On Tuesday, June 5, 2018, Stephen will reveal one finding from our analysis of Amazon’s law enforcement, war fighting, and intelligence services initiative.

Because his books have been reused (in several cases without permission) by other analysts, the information about Amazon is available via online or in person presentations.

The DarkCyber team has prepared short video highlighting one research finding. He will include some of the DarkCyber research information in his Prague lectures.

The Amazon-centric video will be available on Tuesday, June 5, 2018. After viewing the video, if you want the details of his for fee lecture, write him at darkcyber333@yandex dot com. Please, put “Amazon” in the subject line.

Several on the DarkCyber team believe that most people will dismiss Stephen’s analysis of Amazon. The reason is that people buy T shirts, books, and videos from the company. However, the DarkCyber research team has identified facts which suggest a major new revenue play from the one time bookseller.

Just as Stephen’s analyses of Google in 2006 altered how some Wall Street professionals viewed Google, his work on Amazon is equally significant. Remember those rumors about Alexa recording what it “hears”? Now think of Amazon’s services/products as pieces in a mosaic.

The picture is fascinating and it has significant financial implications as well.

Enjoy today’s program at this link.

Kenny Toth, May 29, 2018

One View of the Amazon Game Plan

May 27, 2018

I read “Invisible Asymptotes.” Job One for me was trying to match the meaning of “asymptote” with the research my DarkCyber team has conducted into one slice of Amazon’s business roll outs in the last three years.

As you know, an “asymptote” is a mathy way of saying “you can’t get from here to there.” According to Wolfram Mathword:

An asymptote is a line or curve that approaches a given curve arbitrarily closely.

Here’s a diagram. No equations, I promise.

Image result for asymptote

This diagram suggests a business angle to the “asymptote” reference: No matter what you do, it requires effort and a commitment to “quality”. The good news is that although one can quantify time, one cannot quantify “quality” or “perfection.” Okay, gerbil, run in that Ferris wheel gizmo in your cage.

The write up points out:

We focus so much on product-market fit, but once companies have achieved some semblance of it, most should spend much more time on the problem of product-market unfit.

I am not exactly sure what “unfit” means. The author provides a hint:

For me, in strategic planning, the question in building my forecast was to flush out what I call the invisible asymptote: a ceiling that our growth curve would bump its head against if we continued down our current path.

Okay, the idea seems to be that if Amazon enters a new market, the “invisible asymptote” is what slows growth or stops it completely. (Is this the Amazon phone’s and the slowing sales of Alexa in the face of competition from the Google Home device?)

The reason Amazon cannot grow ever larger is because of an “invisible asymptote”; that is, a factor which prevents Amazon from becoming a company that Vanderbilt, JP Morgan, and John D. Rockefeller would have wished they had.

The write up does not discuss Amazon’s semi-new entrance into the law enforcement and intelligence market. That’s a push I am exploring in my lecture at the Telestrategies ISS conference in early June.

The focus shifts to a more mundane and increasingly problematic aspect of Amazon’s business: Shipping fees. Fiat, law, and the costs of fuel are just a few of the challenges Amazon faces. I am not sure these are “invisible”, but let’s trudge forward.

Twitter becomes that foundation for social media. I noted this passage:

No company owes it to others to allow people to build direct competitors to their own product.

If Amazon wants to make law enforcement and intelligence services into a major revenue stream, I think the first evidence of this intent will be cutting off the vendors using Amazon’s infrastructure to serve their clients now. (Keep in mind that most of the specialist vendors in the LE and intel space use Amazon as plumbing. To cite one example, Marinus, the anti human trafficking group, follows this approach.

The author brings up Snapchat and other social media companies. I find this example important. Amazon’s facial recognition capabilities hit out radar when my team was assembling “CyberOSINT: Next Generation Information Access”, written in 2014 and published in 2015.

We did not include Amazon in my review of LE and intel tools because I had only references in some Amazon conference videos, a few patent applications which were particularly vague about applications in the Background and Claims sections of the documents, and chatter at meetings I attended.

The American Civil Liberties Union has made a bit of noise about Amazon’s facial recognition system. Recognition is spelled “rekognition”, presumably to make it easy to locate in the wonky world of Bing and Google search. The reason is that Amazon’s facial recognition system can identify individuals and cross tabulate that piece of information with other data available to the Amazon system.

Instant bubblegum card.

The write up “Invisible Asymptote” talks about social content and social rich media without offering any comment about the importance of these types of data to Amazon’s intelligence services or its marketplace.

The conclusion of the 10,000 word essay is more “invisible asymptote”. Is this Amazon’s the secret sauce:

Lastly, though I hesitate to share this, it is possible to avoid invisible asymptotes through sheer genius of product intuition.

Here’s a diagram from the essay which looks quite a bit like the self help diagram I included at the top of this Beyond Search post:


Several observations:

  1. The write up makes clear that if anyone thinks Amazon’s platform is neutral, think again.
  2. Strategists at Amazon are not able to “see” and “explain” the nuts and bolts of the “we may be a monopoly but” approach of the Big Dog of the Amazon
  3. The long, long essay does not stray very far from selling stuff to consumers who love free shipping.

Taken as a group of three perceptions, what does this say about Amazon?

For me, I think companies using Amazon’s plumbing will want to do a bit of strategizing using “What if” questions to spark discussion.

For companies behind or beneath the curve, there will be a ceiling, and it will not be easy to break through.

Amazon, on the other hand, may have break through and then replace the old ceiling with a nifty new one made of sterner stuff.

For information about our lectures about Amazon’s Next Big Thing: Intelligence Services, write me at Put Amazon Streaming Marketplace in the subject line, please.

We now offer for fee webinars and on site consulting sessions. On June 5, 2015, coincident with my two lectures in Prague before an audience of LE and intel professionals, I will release a nine minute DarkCyber video exploring some of the inventions Amazon disclosed in an April document not widely reported in the media. Watch this blog for a link.

Stephen E Arnold, May 27, 2018

You Know You Are in Deep Doo Doo When…

May 7, 2018

I flipped through the Overflight news feeds and noted several stories. Remember when you were a wee thing, and you did something wrong. Your friends knew. Your friends’ mom knew. Your mom knew. Then your father or significant parental other (SPO) knew. That may be the feeling of some of the Cambridge Analytica wunderkind.

An example is warranted:

That excellent hire Christopher Wylie has allegedly shared more information about turning clicks into votes. The good hearted wizard told the Guardian about data, target variables, and profiling. There’s even a reference to a patent (absent the patent number, the assignee, and other data which allows one to locate the referenced patent). The kimono is open and the sight does not strike me as one I would describe as attractive.

Will declaring bankruptcy allow the Cambridge Analytica “owner” to avoid further scrutiny? That seems unlikely.

Will an expert step forward and suggest that Cambridge Analytica may have precipitated the Brexit anguish? That seems unlikely.

Nevertheless, I would hypothesize that moms.

PS. Include patent identifiers when you quote patents, dear Guardian editors, please. Perhaps you too are engaging in some data shaping just on a tiny scale?

Stephen E Arnold, May 7,l 2018

Metadata Collection Spike: Is There a Reason?

May 6, 2018

I read “NSA Triples Metadata Collection Numbers Sucking Up over 500 Million Call Records in 2017.” Interesting report, but it raised several questions here in Harrod’s Creek. But first, let’s look at the “angle” of the story.

I noted this statement:

The National Security Agency revealed a huge increase in the amount of call metadata collected, from about 151 million call records in 2016 to more than 530 million last year — despite having fewer targets.

The write up pointed out that penetration testing and trace and tap orders declined. That’s interesting as well.

The write up focused on what’s called “call detail records.” These, the write up explained, are:

things like which numbers were called and when, the duration of the call, and so on…

The write up then reminds the reader that “one target can yield hundreds or thousands of sub-targets.”

The article ends without any information about why. My impression of the write up is that the government agency is doing something that’s not quite square.

My initial reaction to the data in the write up was, “That does not seem like such a big number.” A crawl of the Dark Web, which is a pretty tiny digital space, often generates quite a bit of metadata. Stuffing the tiny bit of Dark Web data into a robust system operated by companies from Australia to the United States can produce terabytes of data. In fact, one Israeli company uploads new data in zipped block to its customers multiple times a day. The firm of which I am thinking performs this work for outfits engaged in marketing consumer products. In comparison, the NSA effort strikes me as modest.

My first question, “Why so little data?” Message, call, image, and video data are going up. The corresponding volume of metadata is going up. Toss in link analysis pointers, and that’s a lot of data. In short, the increase reported seems modest.

The second question is, “What factors contributed to the increase?” Based on our research, we think that some of the analytic systems are bogged down due to the wider use of message encryption technology. I will be describing one of these systems in my June 2018 Telestrategies ISS lecture related to encrypted chat. I wonder if the change in the volume reported in the write up is related to encryption.

My third question is, “Is government analysis of message content new or different?” Based on the information I have stumbled upon here in rural Kentucky, my thought is that message traffic analysis has been chugging along for decades. I heard an anecdote when I worked at a blue chip consulting firm. It went something like this:

In the days of telegrams, the telegraph companies put paper records in a bag, took them to the train station in Manhattan, and sent them to Washington, DC.

Is the anecdote true or false? My hunch is that it is mostly true.

My final question triggered by this article is, “Why does the government collect date?” I suppose the reasons are nosiness, but my perception is that the data are analyzed in order to get a sense of who is doing what which might harm the US financial system or the country itself.

My point is that numbers without context are often not helpful. In this case, the 2010 Pew Data reported that the average adult with a mobile makes five calls per day. Text message volume is higher. With 300 million people in the US in 2010 and assuming 30 percent mobile phone penetration, the number of calls eight years ago works out to about 1.5 billion calls. Flash forward to the present. The “number” cited in the article seems low.

Perhaps the author of the article could provide more context, do a bit of digging to figure out why the number is what it is, and explain why these data are needed in the first place.

One can criticize the US government. But I want to know a bit more.

Net net: It seems that the NSA is showing quite a bit of focus or restraint in its collection activities. In the May 16, DarkCyber, I report the names of some of the companies manufacturing cell site simulators. These gizmos are an interesting approach to data collection. Some of the devices seem robust. To me, capturing 500 million calls seems well within the specifications of these devices.

But what do I know? I can see the vapor from a mine drainage ditch from my back window. Ah, Kentucky.

Stephen E Arnold, May 6, 2018

Cambridge Analytica: Those Greek Tragedians Understood Bad Decisions

May 2, 2018

Here in Harrod’s Creek, the echoes of the khoros are sometimes audible. For example, we heard that Cambridge Analytica, the zippy data outfit has folded its tents. The firm’s offices in the US and elsewhere are shuttered or in the process of turning out the lights and unplugging from the interwebs.

We noted this statement in Gizmodo:

The news was announced during a conference call led by Julian Wheatland, the current chairman of the SCL Group who was reportedly tapped to take over as Cambridge Analytica’s next CEO. Both companies will now close their doors. During the call, Wheatland said that the board determined that rebranding the company’s current offerings in the current environment is “futile.”

In the online information the Beyond Search team has been reviewing, there was no reference to this statement, allegedly crafted by Sophocles:

I would prefer even to fail with honor than win by cheating.

CNBC reported:

The firm is shuttering in part due to mounting legal fees associated with its investigation into whether there had been any wrongdoing with regard to Facebook data, according to the Wall Street Journal who first reported the shuttering.

That “real” news report did not include this statement, allegedly penned by Euripides:

Cleverness is not wisdom.

Will other clever individuals bump into Greek truisms?

The odds in Harrod’s Creek that trouble will be coming to some social media outfits are the same as those on Justified, the favorite in the Kentucky Derby.

Race tracks do need workers to clean their Augean stables. The work is less sporty than crunching data of mysterious origins, but it pays because horses are often more reliable than some humans. Plus there is a party for workers after the race on Saturday.

Stephen E Arnold, May 2, 2018

Next Page »

  • Archives

  • Recent Posts

  • Meta