Big Data Answers the Question ‘Are You Happy?’
November 30, 2018
navigate to the capitalist tool and read “Mapping World Happiness 2015-2018 Through 850 Million News Articles.” Keep in mind that the write up does not explain what percentage of the “news articles” are fake news, the outputs of anti American interest groups, bots, public relations outfits like Definers, or marketing wizards chugging along in the search engine optimization game, and other interesting sources of the data. The write up is a bit of promotion for what is called the GDelt Project. The exercise reveals some of the strengths of open source intelligence. The idea is that collection and analysis can reveal trends and insights.The process involved some number crunching; for example:
Its sentiment mining system has computed more than 2.3 trillion emotional assessments across thousands of distinct emotions from “abashment” to “wrath.”
Google apparently contributed resources.
The question becomes, “Is this analysis an example of real news or is it more marketing?”
The Beyond Search goose has no strong “sentiment” either way. Just asking a simple question.
Stephen E Arnold, November 30, 2018
Making Sense of Big Data: What Is Needed Now
October 29, 2018
Picture, images, and visualization will chop Big Data down to size. SaveDelete explained this idea in depth in its recent story: “The Next Big Phase of Big Data: Simplification.”
According to the article:
“Data visualization is a growing trend, and that momentum isn’t likely to decline anytime soon. Visuals make everything simpler; complex relationships between data points can be seen at a glance, reporting is reduced to a handful of pages, and the esoteric mathematics and statistics behind variable relationships disappear when you’re communicating with someone inexperienced.”
Other ways to deal with making sense of Big Data include:
- “Approachable” software. I think this means easy to use, maybe?
- Gathering the right data. Yep, if one wants to understand terrorist attacks one does not need too much data about hamburger sales in downtown Louisville.
- Reducing insights. This is a tough one. I think the idea is similar to Admiral Craig Hosmer’s statement to me in 1973: “If you can’t get it on a 4×6 note card, I don’t want to see it.”
- Make everything simple. Homer Simpson would be proud.
Useful for math and statistics majors.
Stephen E Arnold, October 29, 2018
Cognos Gets a Rework
October 25, 2018
Cognos? Cognos?
Oh, right, that’s the Canadian analytics company founded in 1969. I think that works out to 49 years young. IBM has owned Cognos since 2008, Now after a decade of vast investment, savvy upgrades, and stellar management decisions, Cognos is going to get even better. Think of it as a US professional football player from the 1960s, suiting up and starting for the Kansas City Chiefs or the Chicago Bears. That’s a strategy that the opposing teams will find surprising.
Same with advanced analytics. Quid, Palantir, Recorded Future! Are you nervous about the new and improved Cognos revealed in “IBM Integrates Business Intelligence and Data Science with New Major Update to Cognos Analytics.”
What’s the fountain of youth?
According to the write up:
… Storytelling… allows users to create interactive narratives by assembling visualizations into a sequence and then enhancing it with media, web pages, images, shapes, and test.
And:
Smart exploration will help users be able to better understand what’s behind their results by analyzing it with machine learning and pattern detection.d then enhancing it with media, web pages, images, shapes, and test.
And:
advanced analytics that include predictive analytics, the ability to identify data patterns and variables driving a certain outcome, smart annotation, and natural language generated insights of data.
But the number one enhancement is… wait for it….
The key new features of this release are a new AI Assistant and pattern detection capability. The AI Assistant enables users to make queries and then receive results in natural language. According to IBM, this makes it easier to not only look for answers, but understand where they come in.
Ah, IBM. Making a product that is half a century young even more appealing to millennials.
Stephen E Arnold, October 25, 2018
Analytics: From Predictions to Prescriptions
October 19, 2018
I read an interesting essay originating at SAP. The article’s title: “The Path from Predictive to Prescriptive Analytics.” The idea is that outputs from a system can be used to understand data. Outputs can also be used to make “predictions”; that is, guesses or bets on likely outcomes in the future. Prescriptive analytics means that the systems tell or wire actions into an output. Now the output can be read by a human, but I think the key use case will be taking the prescriptive outputs and feeding them into other software systems. In short, the system decides and does. No humans really need be involved.
The write up states:
There is a natural progression towards advanced analytics – it is a journey that does not have to be on separate deployments. In fact, it is enhanced by having it on the same deployment, and embedding it in a platform that brings together data visualization, planning, insight, and steering/oversight functions.
What is the optimal way to manage systems which are dictating actions or just automatically taking actions?
The answer is, quite surprisingly, a bit of MBA consultantese: Governance.
The most obvious challenge with regards to prescriptive analytics is governance.
Several observations:
- Governance is unlikely to provide the controls which prescriptive systems warrant. Evidence is that “governance” in some high technology outfits is in short supply.
- Enhanced automation will pull prescriptive analytics into wide use. The reasons are one you have heard before: Better, faster, cheaper.
- Outfits like the Google and In-Q-Tel funded Recorded Future and DarkTrace may have to prepare for new competition; for example, firms which specialize in prescription, not prediction.
To sum up, interesting write up. perhaps SAP will be the go to player in plugging prescriptive functions into their software systems?
Stephen E Arnold, October 19, 2018
Google: Online to Brick and Mortar Cross Correlation
August 31, 2018
Our research suggests that Amazon may have a slight edge in the cross correlation of user data. Google, whether pulling a me too or simply going its own way, has decided to link online and brick and mortar data.
The effort was revealed in “Google and MasterCard Cut a Secret Ad Deal to Track Retail Sales.” Amazon has access to some data which makes it possible for those with appropriate system access to perform analyses of Amazon customers’ buying behavior.
According to the write up:
For the past year, select Google advertisers have had access to a potent new tool to track whether the ads they ran online led to a sale at a physical store in the U.S. That insight came thanks in part to a stockpile of MasterCard transactions that Google paid for. But most of the two billion MasterCard holders aren’t aware of this behind-the-scenes tracking. That’s because the companies never told the public about the arrangement.
To be fair, I am not sure any of the financial services and broker dealer firms provide much output about the data in their possession, who has access to these data, and what use cases are applicable to these data.
From my vantage point in Harrod’s Creek, Kentucky, Google can find its own use cases for Mastercard data.
One question: Does Mastercard pay Amazon to process its data, or does Amazon pay Mastercard?
Google, if the information in the real news article is accurate, is paying for data.
I will address Amazon’s real time streaming data marketplace in my upcoming lecture in Washington, DC. If the information in the US government document I cite in my talk in correct, Google has to shift into high gear with regard to cross correlation of shopper data.
Stephen E Arnold, August 31, 2018
The Obvious: Business Intelligence Tools May Need Clarity
August 14, 2018
Artificial intelligence and business have been a natural pair since the moment we began speculating about this technology. However, we are currently in a sort of golden age of AI for business (or drowning in a swamp of it, depending who you ask) and we could all use a little help sorting through the options. That’s why a recent Data Science Central story “A Comparative Analysis of Top 6 BI and Data Visualization Tools in 2018” seemed so relevant.
According to the story:
“It is often hard to separate the facts from fiction when evaluating various business intelligence (BI) tools, as every BI vendor markets their product as the only “best” solution, often flooding the Internet with biased reviews. If you want to understand the functional product value, avoid the hype and useless clicking through endless pages of partial reviews, you’ve come to the right place.”
This is a very important breakdown and it goes over some really compelling programs, depending on your needs. This seems to be a trend in the industry as we become awash in BI choices. Recently, we also discovered a valuable contrast looking at augmented analytics versus business intelligence tools. What seems obvious is that developers are trying to provide point and click math insight and expertise to individuals who may lack a firm foundation in evaluating data quality, statistics, and other disciplines. No, majoring in medieval literature is not what is needed to make sense of data. To be fair, some find art in proofs.
Insight from slick interfaces? Maybe.
Patrick Roland, August 14, 2018
Thoughtspot: Confused in Kentucky over AI for BI Plus Search Plus Analytics
August 3, 2018
i read “Nutanix Co-Founder Lures Away Its President to Be New CEO at ThoughtSpot.” The headline is a speed bump. But what puzzled me was this passage:
ThoughtSpot Inc. has hired Nutanix Inc.’s president as its new CEO. Sudheesh Nair joins ThoughtSpot about three months after the Palo Alto enterprise search business raised $145 million in a funding round that valued the company at more than $1 billion.
I added the emphasis on the phrase “enterprise search business.”
Search is not exactly the hottest buzzword around these days. After shock from the FAST Search & Transfer and IBM Watson adventures I hypothesize.
Now here’s the pothole: The ThoughtSpot Web site states:
Search & AI Driven analytics.
I noted the phrase “next generation analytics for the enterprise.” Plus, ThoughtSpot is a platform.
But what about artificial intelligence? Well, that’s part of the offering as well.
Remarkable: A Swiss Army knife. Many functions which may work in a pinch and certainly better than no knife at all.
But what’s the company do? Gartner suggests the firm has vision.
That helps. The first time around with FAST ESP and IBM Watson-like marketing the slow curves went right by the batters and the buyers. The billion dollar valuation is juicy as well. Another Autonomy? Worth watching.
Stephen E Arnold, August 3, 2018
Data Wizard: School or Short Cut?
August 2, 2018
With the increase focus on data analytics and search, the role of data scientists has changed drastically over the last decade, or heck, even over the last twelve months. With that increasing dependence on their skills and the continual flexibility of their world, higher ed has been responding. Turns out, these number crunchers are becoming increasingly educated, according to a fascinating article from Kaggle, “The State of ML and Data Science in 2017.”
- The survey spoke with thousands of machine learning and data science experts and found a variety of insights, like how 41.8% have a Master’s degree, but only 15% have doctorals.
- “What is your highest level of formal education?
- “So, should you get that next degree? In general, the highest percentage of people in working data science, obtained a Master’s degree. But those people in the highest salary ranges ($150K – $200K and $200k+) are just as likely to have a doctoral degree.”
Many schools are beginning to offer data science programs for undergrads and grad students, however, universities are now struggling to define what this fluid field exactly, “is”. The University of Houston had to grapple with just such an issue and the results were vague at best. But, we’d say these baby steps are in the right direction.
Beyond Search believes that some “data experts” just tweak their LinkedIn profiles. Easy. Quick. Marketing.
Patrick Roland, August 2, 2018
Amazon Intelligence Gets a New Data Stream
June 28, 2018
I read “Amazon’s New Blue Crew.” The idea is that Amazon can disintermediate FedEx, UPS (the outfit with the double parking brown trucks), and the US Postal Service.
On the surface, the idea makes sense. Push down delivery to small outfits. Subsidize them indirectly and directly. Reduce costs and eliminate intermediaries not directly linked to Amazon.
FedEx, UPS, and the USPS are not the most nimble outfits around. I used to get FedEx envelopes every day or two. I haven’t seen one of those for months. Shipping vis UPS is a hassle. I fill out forms and have to manage odd slips of paper with arcane codes on them. The US Postal Services works well for letters, but I have noticed some returns for “addresses not found.” One was an address in the city in which I live. I put the letter in the recipient’s mailbox. That worked.
The write up reports:
The new program lets anyone run their own package delivery fleet of up to 40 vehicles with up to 100 employees. Amazon works with the entrepreneurs — referred to as “Delivery Service Partners” — and pays them to deliver packages while providing discounts on vehicles, uniforms, fuel, insurance, and more. They operate their own businesses and hire their own employees, though Amazon requires them to offer health care, paid time off, and competitive wages. Amazon said entrepreneurs can get started with as low as $10,000 and earn up to $300,000 annually in profit.
Now what’s the connection to Amazon streaming data services and the company’s intelligence efforts? Several hypotheses come to mind:
- Amazon obtains fine grained detail about purchases and delivery locations. These are data which no longer can be captured in a non Amazon delivery service system
- The data can be cross correlated; for example, purchasers of a Kindle title with the delivery of a particular product; for example, hydrogen peroxide
- Amazon’s delivery data make it possible to capture metadata about delivery time, whether a person accepted the package or if the package was left at the door and other location details such as a blocked entrance, for instance.
A few people dropping off packages is not particularly useful. Scale up the service across Amazon operations in the continental states or a broader swatch of territory and the delivery service becomes a useful source of high value information.
FedEx and UPS are ripe for disruption. But so is the streaming intelligence sector. Worth monitoring this ostensible common sense delivery play.
Stephen E Arnold, June 28, 2018
DarkCyber for June 5, 2018: Amazon and Its LE and Intelligence Services
June 5, 2018
The DarkCyber for June 5, 2018, is now available at www.arnoldit.com/wordpress or on Vimeo at https://vimeo.com/273170550.
This week’s DarkCyber presents an extract from Stephen E Arnold’s lectures at the Prague Telestrategies ISS conference. The conference is designed for security, intelligence, and law enforcement professionals in Europe.
Stephen’s two lectures provided attendees with a snapshot of the services Amazon’s streaming data marketplace offer to customers, developers, and entrepreneurs.
Stephen said:
The Amazon platform is positioned to provide a robust, innovative way to anonymize digital currency transactions and perform the type of analyses needed to deal with bad actors and the activities.
The information was gleaned from Amazon conference lectures, Amazon’s Web logs and documentation, and open source documents.
For example, one public document stated:
“… A law enforcement agency may be a customer and may desire to receive global Bitcoin transactions, correlated by country, with USP data to determine source IP addresses and shipping addresses that correlate to Bitcoin addresses.”
Coupled with Amazon’s facial recognition service “Rekognition” and Amazon’s wide array of technical capabilities, Amazon is able to provide specialized content processing and data services.
Stephen stated:
Instead of learning how to use many different specialized systems, the Amazon approach offers a unified capability available with a Kindle-style interface. This is a potential game changer for LE, intel, and security service providers.
In this week’s DarkCyber video, Stephen provides an eight minute summary of his research, including the mechanisms by which new functions can be added to or integrated with the system.
A for fee lecture about what Stephen calls “Amazon’s intelligence services” is available on a for fee basis. For information, write darkcyber333 at yandex dot com.
Kenny Toth, June 5, 2018