November 2, 2016
Today’s headline sounds like something one would read printed on a grocery store tabloid or a conspiracy Web site. Before I start making claims about the Illuminati, this is not a claim about magical powers, but rather big data and hard science…I think. Defense One shares that, “The CIA Says It Can Predict Social Unrest As Early As 3 To 5 Days Out.” While deep learning and other big data technology is used to drive commerce, science, healthcare, and other industries, law enforcement officials and organizations are using it to predict and prevent crime.
The CIA users big data to analyze data sets, discover trends, and predict events that might have national security ramifications. CIA Director John Brennan hired Andrew Hallman to be the Deputy Director for Digital Innovations within the agency. Under Hallman’s guidance, the CIA’s “anticipatory intelligence” has improved. The CIA is not only using their private data sets, but also augment them with open data sets to help predict social unrest.
The big data science allows the CIA to make more confident decisions and provide their agents with better information to assess a situation.
Hallman said analysts are “becoming more proficient in articulating” observations to policymakers derived in these new ways. What it adds up to, Hallman said, is a clearer picture of events unfolding—or about to unfold—in an increasingly unclear world.
What I wonder is how many civil unrest events have been prevented? For security reasons, some of them remain classified. While the news is mongering fear, would it not be helpful if the CIA shared some of its success stats with the news and had them make it a priority to broadcast it?
November 1, 2016
I noted “NGA Chooses 16 Orgs for Disparate Data Challenge Phase 2.” “NGA” is the acronym for the National Geospatial Intelligence Agency. The geo-analytics folks at this unit do some fascinating things. The future, however, demands that today’s good enough is not sufficient. NGA tapped 15 outfits to do some poking around in their innovation tool chests. Here are the firms:
- App Symphony
- Blue Zoo
- Meta DDC
Recognize any of these outfit? Familiarity might be a useful task.
Stephen E Arnold, November 1, 2016
October 28, 2016
The GOOG is shifting from desktop search to mobile search. The transition will take time and make life exciting for the Web marketers who have to [a] justify their budgets, [b] generate traffic, [c] keep their jobs. The search engine optimization wizards will be looking a McMansions and BMW convertibles. Business is likely to boom for the purveyors of fairy dust and jargon.
Navigate to “50+ Web Measurement KPIs – Analytics Demystified.” The write up presents four dozen ways to accomplish your objectives. The write up groups the analytics some folks view like the Rosetta Stone. The principal categories are:
- Key Performance Indicators to Measure Return on Investment
- KPIs to Measure Lead Generation Campaigns
- KPIs to Measure Intent to Purchase
- KPIs to Measure Website Engagement
I worked through the long write up, complete with mini MBA comments and screenshots of the magic data. The thought I had was that some folks are reaching for straws to build their career. The number that matters is the revenue produced by a digital marketing program.
Intent? Probably to sell consulting.
Stephen E Arnold, October 28, 2016
October 26, 2016
If it works as advertised, a new Google feature will be welcomed by many users—World News Report tells us, “Google Introduced Fact Checking Feature Intended to Help Readers See Whether News Is Actually True—Just in Time for US Elections.” The move is part of a trend for websites, who seem to have recognized that savvy readers don’t just believe everything they read. Writer Peter Woodford reports:
Through an algorithmic process from schema.org known as ClaimReview, live stories will be linked to fact checking articles and websites. This will allow readers to quickly validate or debunk stories they read online. Related fact-checking stories will appear onscreen underneath the main headline. The example Google uses shows a headline over passport checks for pregnant women, with a link to Full Fact’s analysis of the issue. Readers will be able to see if stories are fake or if claims in the headline are false or being exaggerated. Fact check will initially be available in the UK and US through the Google News site as well as the News & Weather apps for both Android and iOS. Publishers who wish to become part of the new service can apply to have their sites included.
Woodford points to Facebook’s recent trouble with the truth within its Trending Topics feature and observes that many people are concerned about the lack of honesty on display this particular election cycle. Google, wisely, did not mention any candidates, but Woodford notes that Politifact rates 71% of Trump’s statements as false (and, I would add, 27% of Secretary Clinton’s statements as false. Everything is relative.) If the trend continues, it will be prudent for all citizens to rely on (unbiased) fact-checking tools on a regular basis.
October 19, 2016
If your diet includes too much sugar, it is a good idea to cut back on the amount you consume. If also turns out if you have too much sugar in your research, the sugar industry will bribe you to hide the facts. Stat News reports that even objective academic research is not immune from corporate bribes in the article, “Sugar Industry Secretly Paid For Favorable Harvard Research.”
In the 1960s, Harvard nutritionists published two reviews in medical journals that downplayed the role sugar played in coronary heart disease. The sugar industry paid Harvard to report favorable results in scientific studies. Dr. Cristin Kearns published a paper in JAMA Internal Medicine about her research into the Harvard sugar conspiracy.
Through her research, she discovered that Harvard nutrionists Dr. Fredrick Stare and Mark Hegsted worked with the Sugar Research Foundation to write a literature review that countered early research that linked sucrose to coronary heart disease. This research would later help the sugar industry increase its market share by convincing Americans to eat a low-fat diet.
Dr. Walter Willett, who knew Hegsted and now runs the nutrition department at Harvard’s public health school, defended him as a principled scientist…‘However, by taking industry funding for the review, and having regular communications during the review with the sugar industry,’ Willett acknowledged, it ‘put him [Hegsted] in a position where his conclusions could be questioned. It is also possible that these relationships could induce some subtle bias, even if unconscious,’ he added.
In other words, corporate funded research can skew scientific data so that it favors their bottom dollar. This fiasco happened in the 1960s, have things gotten worse or better? With the big competition for funding and space in scientific journals, the answer appears to be yes.
October 18, 2016
With so many options for cloud computing, it can be confusing about which one to use for your personal or business files. Three of the most popular cloud computing options are Amazon Web Services (AWS), Google Cloud Platform, and Microsoft Azure. Beyond the pricing, the main differences range from what services they offer and what they name them. Site Point did us a favor with its article comparing the different cloud services: “A Side-By-Side Comparison Of AWS, Google Cloud, And Azure.”
Cloud computing has the great benefit of offering flexible price options, but they can often can very intricate based on how much processing power you need, how many virtual servers you deploy, where they are deployed, etc. AWS, Azure, and Google Cloud do offer canned solutions along with individual ones.
AWS has the most extensive service array, but they are also the most expensive. It is best to decide how you want to use cloud computing because prices will vary based on the usage and each service does have specializations. All three are good for scalable computing on demand, but Google is less flexible in its offering, although it is easier to understand the pricing. Amazon has the most robust storage options.
When it comes to big data:
This requires very specific technologies and programming models, one of which is MapReduce, which was developed by Google, so maybe it isn’t surprising to see Google walking forward in the big data arena by offering an array of products — such as BigQuery (managed data warehouse for large-scale data analytics), Cloud Dataflow (real-time data processing), Cloud Dataproc (managed Spark and Hadoop), Cloud Datalab (large-scale data exploration, analysis, and visualization), Cloud Pub/Sub (messaging and streaming data), and Genomics (for processing up to petabytes of genomic data). Elastic MapReduce (EMR) and HDInsight are Amazon’s and Azure’s take on big data, respectively.
Without getting too much into the nitty gritty, each of the services have their strengths and weaknesses. If one of the canned solutions do not work for you, read the fine print to learn how cloud computing can help your project.
October 14, 2016
i read “Big Data Is Useless without Visual Analytics.” (Nope, I won’t comment on the fact that “data” is a plural.) The main point of the article is that looking at a table is not as easy as looking at a graphic, preferably Hollywood style, presentation ready visualizations. If you want to see a nifty visualization, check out the Dark Trace three dimensional, real time visualizations.
The write up informed me:
Visualizations are valuable because they display a lot of data in an easy-to-understand visual format that works well for our visually-oriented minds.
Okay. A lot. Good.
I learned that “data mining is too complicated for most uses of Big Data.”
No kidding. Understanding and making justifiable decisions about data validity, math with funny symbols, and numerical recipes which make the numbers conform to the silliness taught in university statistics classes. These are difficult tasks for avid Facebook users and YouTube content consumers to deal with.
I understand. Some folks are just really busy.
The write up explains that Excel is not the proper tool for real information analysis. Never mind that Excel remains a reasonably popular chunk of software. Some Excel lovers write memos and reports in Excel. That’s software love.
So what’s the fix?
Surprisingly the write up does not provide one. But there is a link which allows me to download a report from my pals at IDC. You remember IDC, right? That is the outfit which tried to sell my content on Amazon without my permission and without having a agreement with me to publish my research. If you have forgotten what I call the “Schubmehl play”, you can get some of the details at this link.
Nice write up. Too bad it lacks useful content on the subject presented in the headline. But what else does one expect these days?
Stephen E Arnold, October 14, 2016
October 13, 2016
It’s no surprise that hackers may be any age, but that teenagers could cause 60 million pounds worth of damage to a corporation is newsworthy, regardless of age. The Telegraph published an article, From GCHQ to Google: the battle to outpace hackers in the cyber race, reporting on this. A 15-year-old boy hacked the TalkTalk computer network stole personal data, including financial information, of 157,000 customers. This comes at a time when the UK government announced plans to invest £1.9 billion in cyber security over the next five years. We also learned,
No amount of money will help overcome one of the greatest difficulties in the security industry though: the lack of skilled people. By 2019 there will be a global shortfall of 1.5 million security professionals, according to ISC Squared, a security certification and industry education body. And the numbers could in fact be significantly higher, given that there are already more than 1 million cybersecurity positions unfilled worldwide, according to a 2015 Cisco report. Heading up the government’s move to train more cyber defenders is spook agency GCHQ, which sponsors academic bursaries, runs summer camps and training days, holds competitions and has created a cyber excellence accreditation for top universities and masters programmes. The intention is to spot talent in children and nurture them through their education, with the end goal being a career in the industry.
The problem of for any rocketing industry ready to blast off always seems to boil down to people. We have seen it with big data in all of it’s forms from electronic medical records to business analytics to cyber security. It seems industry is most fertile when people and technology work best stride-by-stride.
October 11, 2016
The lawless domain just got murkier. Apart from illegal firearms, passports, drugs and hitmen, you now can procure a verifiable college degree or diploma on Dark Web.
Cyber criminals have created a digital marketplace where unscrupulous students can
purchase or gain information necessary to provide them with unfair and illegal
academic credentials and advantages.
The certificates for these academic credentials are near perfect. But what makes this cybercrime more dangerous is the fact that hackers also manipulate the institution records to make the fake credential genuine.
The article ADDS:
A flourishing market for hackers who would target universities in order to change
grades and remove academic admonishments
This means that under and completely non-performing students undertaking an educational course need not worry about low grades or absenteeism. Just pay the hackers and you have a perfectly legal degree that you can show the world. And the cost of all these? Just $500-$1000.
What makes this particular aspect of Dark Web horrifying interesting is the fact that anyone who procures such illegitimate degree can enter mainstream job market with perfect ease and no student debt.
September 28, 2016
If you wonder why a Grand Canyon exists between the people who offer point-and-click statistical analysis and the folks who “do” the math, you will want to read “Markov Chain Monte Carlo Without All…” When you read the source document, keep in mind that selecting an icon and generating a report is like using an automatic teller machine. Punching buttons delivers an output. The inner workings of the system are not visible. User friendly, embedded business intelligence systems are like the chrome trim on a door stop.
Stephen E Arnold, September 28, 2016