Business Intelligence, Expectations, and Data Fog

February 10, 2021

Business intelligence and government intelligence software promises real time data intake, analysis and sense making, and outputs with a mouse click. Have you heard the phrase, “I have the Brooklyn Bridge for sale”? Sure, sure, I know I don’t want to own the Brooklyn Bridge, but that super spiffy intelligence software (what I call intelware), count me in.

The marketing pitch for business intelligence and general intelligence software has not changed significantly over the years. In my experience, a couple of nifty outputs like a relationship diagram and a series of buttons set up to spit out “actionable intelligence” often close the deal. The users of the software usually discover three points not making up a large part of the demos, the discussions, and the final contract for the customer’s requirements.

I read “The Age Of Continuous Business Intelligence.” The idea is appealing. Lots of information and no time to read, review, digest, analyze, and discuss the available information. In my opinion, the attitude now is “I don’t have time.”

Yep, time.

The write up asserts:

we [an outfit called KX] know that shortening the time it takes to ingest, store, process, and analyze historic and real-time data is a game changer for businesses in all sectors. Our customers in finance, manufacturing, automotive, telecommunications and utilities tell us that when processes and systems are continuously fed by real-time data that is enriched by the context of historic data, they can automate critical business decisions resulting in significant operational and commercial benefits.

The write up contains a diagram which lays bare “continuous business intelligence.”


The write up concludes:

As the research clearly shows, real-time data analytics is a critical area of investment for many firms. To ensure maximum value is derived from these investments, it is imperative that organizations – regardless of size and sector – challenge their understanding of what real-time means. By implementing a strategy of continuous business intelligence, firms can dramatically reduce the time it takes to uncover and act on insights that can materially change the game in terms of growth, efficiency and profitability.

I love that “research clearly shows.” The challenges for the continuous thing include:

  • Defining real time. (According to research my team did for a project years ago, there are numerous definitions of real time, and there is a Grand Canyon sized gap among these.)
  • Making clear the computational short cuts necessary to process “fire hoses”. (Yep, these compromises have a significant impact on costs, validity of system outputs, and the mechanisms for issuing meaningful outputs from sense making.)
  • Managing the costs. (Normalizing, verifying, processing, storing, and moving data require human and machine resources. Right, those things.)

Net net: Software whether for business or government applications in intelligence work only if the focus is narrow and the expectations of a wild and crazy MBA are kept within a reality corral. Otherwise, business intelligence will shoot blanks, not silver bullets.

Oh, KX is hooked up with a mid tier consulting firm. What’s that mean? A sudden fog has rolled in, and it is an expensive fog.

Stephen E Arnold, February 10, 2021

Dashboards Evil? Worth a Thought. Nah, Just Take What Is Output

February 1, 2021

Business intelligence (BI) is jargon for the technologies and strategies used to manage business data analytics. It is a fancy term for standard operating procedures and looks good on a resume, but one IT CEO wants to make it obsolete. Diginomica discusses how BI could head to the recycling bin in the article: “ThoughtSpot CEO-‘I Want To Kill BI And I Want All Dashboards To Die.’”

COVID-19 has changed global business practices and technology experts spent 2020 investigating ways to aggregate business data. ThoughtSpot CEO Sudheesh Nair explained in the article that traditional aggregation patterns do not apply anymore and companies need to change in order to maintain their customers. Nair believes his ThoughtSpot platform, described as ‘Google for numbers,’ will deliver key insights the same way Google provides information.

Nair pointed out that opinions are easily accessible via a Google search, but facts are harder to find in the endless search results. Nair wants his ThoughSpot platform to make facts as easily accessible as opinions. ThoughtSpot combines hardened facts with a NLP interface to make finding facts easier, think Windows 95 versus the old command land interface:

“ThoughtSpot does this by allowing users to search enterprise data with hyper-personalized questions using natural language processing. It aims to not only give a result for the question you ask, but then also uses AI to offer up alternative questions and results that may be helpful. This is very different to traditional BI, which typically offers you a template for which to present historical, aggregate data.”

In other words, Nair wants to sift the information noise from facts. Today’s BI dashboards offer a plethora of information, but lack personalization notes that could win new customers and retain older ones.

ThoughtSpot will supposedly combine old data with new data to push out BI dashboards and create a new data analytics space for businesses. Nair’s description of ThoughtSpot is an interesting pitch, but it sounds more like a new way to search information. Instead of explaining how ThoughtSpot works it would be better to offer demonstrations of its capabilities.

But thinking? Not the core competency of the thumb typing generations.

Whitney Grace, February 1, 2021

Twitter and the Fire Hose for Academics

January 29, 2021

I read “Enabling the Future of Academic Research with the Twitter API.” According to the official Twitter statement:

Our developer platform hasn’t always made it easy for researchers to access the data they need, and many have had to rely on their own resourcefulness to find the right information.

Understatement, of course.

The post continues:

We’ve also made improvements to help academic researchers use Twitter data to advance their disciplines, answer urgent questions during crises, and even help us improve Twitter.

Help is sometimes — well — helpful. But self help is often a positive step; for example, verifying the actual identity of a person who uses the tweeter thing. There are some software robots chugging along I believe.

Also, charging a subscription fee. The amount is probably less important than obtaining verifiable bank information. Sure, some software robots have accounts at outstanding institutions like Credit Suisse and HSBC, but whatever account data are available might be helpful under certain circumstances.

But academics? How many academics work for non governmental or governmental entities as experts, analysts, and advisors? Will the tweeter thing’s new initiative take such affiliations into account before and during usage of Twitter data?

I assume that a tweeter senior manager will offer an oracular comment like, “For sure.”

There are three hoops through which the agile academic must jump, and I quote:

  1. You are either a master’s student, doctoral candidate, post-doc, faculty, or research-focused employee at an academic institution or university.
  2. You have a clearly defined research objective, and you have specific plans for how you intend to use, analyze, and share Twitter data from your research…
  3. You will use this product track for non-commercial purposes….

Sounds like a plan which will make some nation states’ academics wriggle with anticipative joy.

My view is that this new initiative may unfold in interesting ways. But I am sure the high school science club managers have considered such possibilities. Why who would hire a graduate student to access tweeter outputs to obtain actionable information for use by a country’s intelligence professionals? The answer in the twitterverse is, “Who would risk losing the trust of Twitter by doing that?” Certainly not an academic funded by an intelligence or law enforcement entity.

Right, no one. Misuse the tweeter? Inconceivable.

Stephen E Arnold, January 29, 2021

Tweet This! Real News Discovers the Concept of Hidden in Plain Sight

December 31, 2020

Remember the Purloined Letter? No, that’s okay. Thumbtypers don’t either. I read “Just How Bad Was This Year? These Professors Found Answers on Twitter.” I noted this passage:

Since 2008, the duo [professors at a school in Vermont] has taken a random 10 percent of everything tweeted each day, seeking truths hidden in plain sight. (Whileacknowledging, as Danforth put it, that “Twitter is a nonuniform subsample of utterances made by a nonuniform subsample of humans whoare on the Internet.”) They’ve used it, for example, to explore fame, finding that DonaldTrump and K-pop band BTS are mentioned as commonly as some regular words (think: “after,” “would.”). As Dodds put it, “The word‘Trump’ has been in the top 300 words all year this year, which he’s never done before. That’s more common than the word ‘God.’ ”

The sampling is done by the Hedonometer, possible a reference to either a town in England or a unit of pleasure used to theoretically weigh people’s happiness. I like the latter candidate, split infinitive, and the weird idea of “weighing” happiness. I often say to the grocery clerk in Harrod’s Creek, Kentucky, “I will take a pound of happiness and a half pound of ricotta, please.”

The big find seems to be:

Some trends have emerged through the years. All else being equal,Saturday is the week’s happiest day on Twitter, Tuesday the saddest.National holidays cause huge spikes in happiness, with Christmas beingthe most cheerful. Major sporting events and birthdays of pop stars,particularly K-pop stars, tend to make for gleeful days. On the flipside, natural disasters and mass shootings tend to spark more unhappydays.

What’s the analysis reveal?

“In the last five years, we’ve seen the usual weekly cycle justget busted,” Dodds added. “It’s sort of all over the place now.Events are happening any day of the week. It’s much more what Iwould call emotional turbulence.”

Remarkable in a way, a modest way.

Stephen E Arnold, December 31, 2020

Saddle Up, Statistical Analytics Fans. Place Your Bets on AI

December 7, 2020

The Best Way to Win a Horse Race? Mathematicians May Have the Answer” is an interesting example of a snappy headline not supported by the write up’s text. For gamblers, the promise of finding a way to predict which mighty steed will cross the finish line first is catnip. (Sorry for the mixed animal metaphor, but I could not resist.) The problem is that the summary of the study includes lots of references to data collection and number crunching. Then the killer statement:

Various other scientific attempts to explain performance over the past 4 decades “haven’t been particularly successful,” he says—and not just because horses vary so much in body size and aerobic capacity: The models cannot account for the horse’s own behaviors. For example, a horse might give up when another horse passes it, because it doesn’t understand that it’s supposed to win. Until researchers can get inside the horse’s head and account for psychological variables, Knight says, “we can’t truly model performance.”

Net net: If one can’t model an equine, what’s that suggest for figuring out what human will cross an innovation finish line, crack a tough problem, or write a headline which does mislead the pony player?

Stephen E Arnold, December 7, 2020

Clarity: A Better Name Than Pluton. Pluton?

November 20, 2020

After two years, Clarity has finally made it out of Beta, we learn from “Microsoft Clarity Debuts as Free Analytics Tool with Heat Maps” at Search & Performance Marketing Daily. The free tool uses heat maps to analyze the behavior of visitors to one’s website. Reporter Laurie Sullivan writes:

“Clarity — designed to have a low impact on page-load times and there are no caps on traffic no matter what the number of visitors to the website — helps give marketers a deeper understanding of why at website performs one way and not another. It also provides anonymized heat maps and data that show where site visitors clicked and scrolled, and enables marketers to analyze use behavior on the website exactly as it happened through a job description code. Some of the data includes the name of the browser, and whether they are using a PC, tablet or mobile phone to access the site. Heat maps provide a visual way to examine large numbers of site visitor interactions. Microsoft built two types: click maps and scroll maps. While the heat maps tell marketers which pages get the most clicks, the click maps tell marketers what website page content visitors interact with the most. Areas in the map marked in red have the highest frequency of clicks and are usually centered on focal points.”

The heat maps let marketers know whether visitors are clicking where they want them to. It also reports certain behaviors—excessive scrolling, dead clicks, and rage clicks. The last term describes users clicking several times on a spot they believe should be a hyperlink but is not—one would want to either fix an intended link or tweak the graphics on those spots. The tool also supplies a dashboard that presents metrics of the overall traffic patterns, time spent on the site, and even concurrent JavaScript errors. Microsoft pledges Clarity complies with the EU’s General Data Protection Regulation.

But Pluton, Microsoft’s mystery processor? Pluton?

Cynthia Murrell, November 20, 2020

Surveys: These Marketing Devices Are Accurate, Right?

November 10, 2020

There’s nothing like a sample, a statistical sample, that is. What’s interesting is that the US polls seem to have been reflecting some interesting but marketing-type trends. The bastion of “real journalism”— the UK Daily Mail — published “…We Did a Good Job: Defiant Pollster Nate Silver Rushes to Defend His Profession after Another Systematic Failure of Polls in the Build-Up to an Election.” Bibliophiles will note that I have omitted the tasteful obscenity. I like to avoid using words likely to irritate the really smart software which edits blog posts.

The write up points out:

FiveThirtyEight founder and editor-in-chief Nate Silver hit back at those slamming the website for being so off with their election predictions.

Let’s think about why FiveThirtyEight and other polls seem to have predicted a reality different from the one generated by humanoids marking ballots.

First, there is the sample. Picking people at random is dependent on a number of factors: Sources, selection bias, humanoids who don’t respond, etc.

Second, there are the humanoids themselves. Some people plug in the “answers” which get the poll over with really fast. I lose interest at the first hint of dark patterns which make it tough to know how may questions I have to answer to get the coupon, pat on the head, or the free shopping sack.

Third, there is counting. Yep, humans or machine things can happen.

Fourth, there is analysis. It is remarkable what one can do when counting or doing “analytics.”

The Daily Mail quotes an expert about making polls better:

‘The polling profession needs to reshape and reorganize their questionnaires,’ Luntz [the polling expert] told ‘It’s the only way they’ll ever get it right.’

But I keep thinking about the FiveThirtyEight obscenity. Defensive? Eloquent? Subjective? Insightful?

That subjective thing.

Stephen E Arnold, November 10, 2020

Linear Math Textbook: For Class Room Use or Individual Study

October 30, 2020

Jim Hefferon’s Linear Algebra is a math textbook. You can get it for free by navigating to this page. From Mr. Hefferon’s Web page for the book, you can download a copy and access a range of supplementary materials. These include:

  • Classroom slides
  • Exercise sets
  • A “lab” manual which requires Sage
  • Video.

The book is designed for students who have completed one semester of calculus. Remember: Linear algebra is useful for poking around in search or neutralizing drones. Zaap. Highly recommended.

Stephen E Arnold, October 30, 2020

Exclusive: Interview with DataWalk’s Chief Analytics Officer Chris Westphal, Who Guides an Analytics Rocket Ship

October 21, 2020

I spoke with Chris Westphal, Chief Analytics Officer for DataWalk about the company’s string of recent contract “wins.” These range from commercial engagements to heavy lifting for the US Department of Justice.

Chris Westphal, founder of Visual Analytics (acquired by Raytheon) brings his one-click approach to advanced analytics.

The firm provides what I have described as an intelware solution. DataWalk ingests data and outputs actionable reports. The company has leap-frogged a number of investigative solutions, including IBM’s Analyst’s Notebook and the much-hyped Palantir Technologies’ Gotham products. This interview took place in a Covid compliant way. In my previous Chris Westphal interviews, we met at intelligence or law enforcement conferences. Now the experience is virtual, but as interesting and information in July 2019. In my most recent interview with Mr. Westphal, I sought to get more information on what’s causing DataWalk to make some competitors take notice of the company and its use of smart software to deliver what customers want: Results, not PowerPoint presentations and promises. We spoke on October 8, 2020.

DataWalk is an advanced analytics tool with several important innovations. On one hand, the company’s information processing system performs IBM i2 Analyst’s Notebook and Palantir Gotham type functions — just with a more sophisticated and intuitive interface. On the other hand, Westphal’s vision for advanced analytics has moved past what he accomplished with his previous venture Visual Analytics. Raytheon bought that company in 2013. Mr. Westphal has turned his attention to DataWalk. The full text of our conversation appears below.

Read more

Tickeron: The Commercial System Which Reveals What Some Intel Professionals Have Relied on for Years

October 16, 2020

Are you curious about the capabilities of intelware systems developed by specialized services firms? You can gat a good idea about the type of information available to an authorized user:

  • Without doing much more than plugging in an entity with a name
  • Without running ad hoc queries like one does on free Web search systems unless there is a specific reason to move beyond the provided output
  • Without reading a bunch of stuff and trying to figure out what’s reliable and what’s made up by a human or a text robot
  • Without having to spend time decoding a table of numbers, a crazy looking chart, or figuring out weird colored blobs which represent significant correlations.

Sound like magic?

Nope, it is the application of pattern matching and established statistical methods to streams of data.

The company delivering this system, tailored to Robinhood-types and small brokerages, has been assembled by Tickeron. There’s original software, some middleware, and some acquired technology. Data are ingested and outputs indicate what to buy or sell or to know, as a country western star crooned, “know when to hold ‘em.”

A rah rah review appeared in The Stock Dork. “Tickeron Review: An AI-Powered Trading Platform That’s Worth the Hype” provides a reasonably good overview of the system. If you want to check out the system, navigate to Tickeron’s Web site.

Here’s an example of a “card,” the basic unit of information output from the system:


The key elements are:

  • Icon to signal “think about buying” the stock
  • A chart with red and green cues
  • A hot link to text
  • A game angle with the “odds” link
  • A “more” link
  • Hashtags (just like Twitter).

Now imaging this type of data presented to an intel officer monitoring a person of interest. Sound useful? The capability has been available for more than a decade. It’s interesting to see this type of intelware finds its way to those who want to invest like the wizards at the former Bear Stearns (remember that company, the bridge players, the implosion?).

DarkCyber thinks that the high-priced solutions available from Wall Street information providers may wonder about the $15 a month fee for the Tickeron service.

Keep in mind that predictions, if right, can allow you to buy an exotic car, an island, and a nice house in a Covid-free location. If incorrect, there’s van life.

The good news is that the functionality of intelware is finally becoming more widely available.

Stephen E Arnold, October 16, 2020

Next Page »

  • Archives

  • Recent Posts

  • Meta