CyberOSINT banner

Indeed. One Can Fix Government Economic Forecasts

August 26, 2015

Big Data is magic. Big Data is revolutionary. Big Data is good consulting angle.

But Big Data is not going to fix government forecasts. I hate to rain on the parade of a distinguished academic and chief economist, but those rain drops keep a falling.

Navigate to “Economic Forecasts in the Age of Big Data.” The passage I highlighted with my sea of red ink colored marker was:

Properly used, new data sources have the potential to revolutionize economic forecasts. In the past, predictions have had to extrapolate from a few unreliable data points. In the age of Big Data, the challenge will lie in carefully filtering and analyzing large amounts of information. It will not be enough simply to gather data; in order to yield meaningful predictions, the data must be placed in an analytical framework. The Fed may have blundered in releasing its data ahead of schedule. But its mistake offers us an important opportunity. In order to improve economic predictions, economists must be encouraged to seek new sources of data and develop new forecasting models. As we learn how to harness the power of big data, our chances of predicting – and perhaps even preventing – the next recession will improve.

I am thrilled with job opening analyses in Boston, demand for rentals in San Francisco, and housing starts in Los Angeles (you know the water crisis city).

However, government economic analyses are not into reality. In Washington, DC, there is a big building adjacent the train station. It is filled with folks who do economic forecasts among other things. There are economic forecasts cranked out by lobbyists. There are super grades in Federal entities crunching numbers. The numbers get reviewed, shaped, and tweaked. Eventually the numbers emerge in a new release which may or many not be widely distributed. The government process for creating economic forecasts is institutionalized. Like an aircraft carrier, the system carries some momentum.

A person who wants to inject real time Big Data into these procedures can go through the normal process. Get involved in an allocation for an initiative. Find a way to work on a statement of work. Compete for a Big Data economic forecast project. Do the work. Have the work reviewed and taken under advisement.

End of the day: The existing system keeps on generating forecasts.

Net net: Economic forecasts from DC and other world capitals drift above real time. Rome had the same problem.

Stephen E Arnold, August 26, 2015

Search Does Not Work: Maybe, Sometimes

August 23, 2015

I read “Feds Keep Magically Finding Documents They Insisted Didn’t Previously Exist.” I noted that the US government struggles with finding content, if the article is on the money. I learned:

Gawker had sought the email communications of Hillary Clinton deputy Philippe Reines, focused on his conversations with journalists. The State Department came back with a no responsive records reply, which was clearly bullshit, since Reines was known for regularly emailing reporters. So Gawker sued and guess what just happened: the State Department just magically found 17,855 emails that are likely responsive. How about that?

Obviously the US government is not aware of the search systems which can find documents. But what if the US government has these systems. Isn’t the finding issue an indication that basic search and retrieval does  not work? Interesting thought.

Stephen E Arnold, August 23, 2015

Open Source Tools for IBM i2

August 17, 2015

IBM has made available two open source repositories for the IBM i2 intelligence platform: the Data-Acquisition-Accelerators and Intelligence-Analysis-Platform can both be found on the IBM-i2 page at GitHub. The IBM i2 suite of products includes many parts that work together to give law enforcement, intelligence organizations, and the military powerful data analysis capabilities. For an glimpse of what these products can do, we recommend checking out the videos at the IBM i2 Analyst’s Notebook page. (You may have to refresh the page before the videos will play.)

The Analyst’s Notebook is but one piece, of course. For the suite’s full description, I turned to the product page, IBM i2 Intelligence Analysis Platform V3.0.11. The Highlights summary describes:

“The IBM i2 Intelligence Analysis product portfolio comprises a suite of products specifically designed to bring clarity through the analysis of the mass of information available to complex investigations and scenarios to help enable analysts, investigators, and the wider operational team to identify, investigate, and uncover connections, patterns, and relationships hidden within high-volume, multi-source data to create and disseminate intelligence products in real time. The offerings target law enforcement, defense, government agencies, and private sector businesses to help them maximize the value of the mass of information that they collect to discover and disseminate actionable intelligence to help them in their pursuit of predicting, disrupting, and preventing criminal, terrorist, and fraudulent activities.”

The description goes on to summarize each piece, from the Intelligence Analysis Platform to the Information Exchange Visualizer. I recommend readers check out this page, and, especially, the videos mentioned above for better understanding of this software’s capabilities. It is an eye-opening experience.

Cynthia Murrell, August 18, 2015

Sponsored by, publisher of the CyberOSINT monograph

Revisionism: Hit That Delete Key for Happiness

August 14, 2015

The Jive Aces are into happy songs. Reddit has figured out how to make some folks who love revisionism happy. I wonder if the Jive Aces have a tune for removing content to create digital happiness.

Navigate to “Reddit Responds after Being Threatened, Banned and Unbanned by the Russian Government.” The main point is that one cannot find information if it is not in the index. Magic. Better and cheaper than reprinting history books in certain countries.

The write up says:

One thing that is clear is that Russia doesn’t play around though when it comes to speech encouraging drug use online. In 2013, Roskomnadzor blacklisted Wikipedia in its entirety for a single article on “Cannabis smoking.” Reddit doesn’t address concerns over restrictions of free speech from the Russian government in this statement, but instead seems to say that whatever the situation, wherever it’s posted, Reddit and Reddit alone has the final call. It sounds like a lot of redditors are perplexed by Reddit’s right to “restrict content,” but in the time being it seems that stability, rather than free speech, is Reddit’s main priority.

Cue the music:

Even when the darkest clouds are in the sky
You mustn’t sigh and you mustn’t cry
Spread a little happiness as you go by

That’s it. Reddit is spreading a little happiness. Are Russian content mavens smiling? I assume they are having a “golden shoes day.” When information is disappeared, that makes someone happy.

Stephen E Arnold, August 14, 2015

Ooooh, Darpa, Ooooh, Baby, Wail

July 29, 2015

I read “DARPA Hired a Jazz Musician to Jam with Their Artificially Intelligent Software.” I would have used the pronoun “its” but I am not artificially intelligence. DARPA brings it axes to a jam fest. The DARPA barrelhouse features some Bose bouncing riffs.


DARPA’s robot, AI infused quartet is down by law.

The write up said:

“A human musician also builds a knowledge base by practicing and by listening and by learning and studying,” Thomas [DARPA cat] said. “So the thing we’re proposing to do is analogous to the way a human learns, but eventually it will be able to do this on a much larger scale. It can scour thousands of transcriptions instead of dozens or hundreds.” Many people might not consider music a form of communication, but Paul Cohen, an AI researcher and head of the Communicating with Computers project, thinks music shares many qualities with the spoken and written word.

You can watch a video and watch these fantastic musicians keep up with a human hep cat. If Dark Web search and drone performance improves with this investment, that’s cool.

Maybe we have another US government moldy fig, dude? Robot musicians would not be persons of interest for alleged use of controlled substances unless WD-40 were reclassified.

Stephen E Arnold, July 29, 2015

Hewlett Packard: A Fashion Forward Management Moment

July 26, 2015

I read “HP Bans T Shirts at Work, and Employees Are Furious.” The write up explains:

several teams within HP’s 100,000-employee-strong Enterprise Services division were sent a confidential memo cracking down on casual dress in the workplace, because higher-ups in the company are concerned that customers visiting the offices will be put off by dressed-down developers, reports The Register.

I enjoy the management antics of HP. I recall the dust up with the board of directors. There is the MBA play of splitting the company in half in hopes of doubling the “value” for someone, maybe a banker or a senior manager. And, who can forget, HP’s purchase of Autonomy, the subsequent mea culpa, and the long flights of legal eagles. A deft touch for sure.

The write up states:

An HP spokesperson said the company does not have a global dress code but had no immediate comment on the report of the memo about the Enterprise group dress code.

Organized to a T shirt like this one:


Stephen E Arnold, July 26, 2015

Google and Disappeared Streets

July 14, 2015

If you spend any time with Google Maps (civilian edition), you will find blurred areas, gaps, and weird distortions which cause me to ask, “Where did that building go?”

If you really spend a lot of time with Google Maps, you will be able to see my two dogs, Max and Tess, in a street view scene.


And zoomed in. Aren’t the doggies wonderful?


The article “The Curious Case of Google Street View’s Missing Streets” is not interested in seeing what the wonky Google autos capture. The write up pokes at me with this line:

Many towns and cities are littered with small gaps in the Street View imagery.

The write up explains that Google recognizes that gaps are a known issue. The article gets close to something quite interesting when it states:

In extreme cases, whole countries are affected. Privacy has been a particular issue in Germany, where many people objected to the roll-out of Street View. Google now has Street View images only for big cities in Germany, like Berlin and Frankfurt, and appears to have given up on the rest of the country completely. Zoom out over Europe in Street View mode and Germany is mostly a blank island in a sea of blue.

Want to do something fun the author of the write up did not bother to do? Locate a list of military facilities in the UK. Then try to locate those on a Google Map. Next try to locate those on a map (oops, Uber map)?

Notice anything unusual? Now relate your thoughts to the article’s list of causes.

If not, enjoy the snap of Max and Tess.

Stephen E Arnold, July 14, 2015

Watson: Will It Be Able to Make Major Government Applications More Intelligent?

June 30, 2015

I recently commented on the 25 percent problem rate in government software. You can find that Beyond Search item at this link. I can relate to companies who want to improve US government software. Go for it.

IBM has a plan which apparently ignores IBM Federal Systems (an outfit which creates, upgrades, and maintains some US government software). The approach focuses on a group of student from the University of Texas at Austin.

The write up states that Lauri Saft, director of the IBM Watson Ecosystem, has this view:

“You don’t program Watson, you teach it. We gave them [the students] the empty shell of Watson and said, ‘Go and come up with ideas you feel would be valuable.’”

The training method was one of Autonomy IDOL’s most important functions. The challenge which IDOL licensees faced, as I understand the system, is that the system can drift unless training and calibration are part of the routine maintenance cycle. For some licensees, the time and cost of the training and calibration were hurdles. Has Watson moved beyond Autonomy’s approach?

I assume the answer is, “Yes.” Therefore, the use of Watson to improve government software by making that software more intelligent should be a home run.

The students in Austin created a Watson app named CallScout. The focus is not on government software as I think of market opportunities. The students are working on a social service program in Texas. The application is customer support solution. That’s good.

My thought was that IBM Federal Systems would be using Watson’s remarkably broad spectrum of capabilities to address issues at the national level, maybe the regional level for Homeland Security or the EPA. I did not expect a local app for social services in Texas.

Perhaps the IBM Federal Systems Watson home run will be announced soon. The incubator with student thing is not likely to boost IBM top line revenues in the way a major US government Watson deal would. But PR is PR, big or small. IBM needs its Federal Systems’ unit to get the Watson revenue flowing. Time is a wastin’ because there are other outfits nosing into this potentially lucrative territory.

Stephen E Arnold, June 30, 2015

How to Succeed in China: Maybe Follow the Rules?

June 27, 2015

I love articles which explain how to do something to anticipated readers who have zero chance to build a business in China. Navigate to “A New Wave of US Internet Companies Is Succeeding in China—By Giving the Government What It Wants.”

I am not sure a degree in business is required to understand this concept. In my experience, when one is in another country, common sense suggests that the government officials expect outsiders to play by the rules. Ever wonder why West Point cadets look so darned polished. Well, consider the downside associated with wearing an Iron Maiden T shirt, soccer trunks, and flip flops?

The write up points out:

“If you want to develop an internet business in Chinese now, you have to be willing to work with the Chinese government, even if that means censoring content or sharing access to your data,” Ben Cavender, principal at the China Market Research Group, told Quartz.

Outfits who have learned this simple lesson, according to the write up, are LinkedIn, Uber, and Evernote. Outfits who have not figured out the calculus of the West Point approach to order include Facebook, Google, and Twitter. Hey, Facebook is trying. I saw a news item revealing that the Facebook top Facebooker learned sort of Chinese. Yippy.

So which companies have “better” managers? Those in the big market or those looking at the big market?

How does this related to search and content processing? I don’t know of too many information access companies dominating the Chinese market. When it comes to cyberOSINT, there is Knowlesys which sort of operates in Hong Kong and does have an office in China.

Class dismissed. Oh, you with the flip flops, may I have a word with you?

Stephen E Arnold, June 27, 2015

You Cannot Search It If It Is Not There: The Wayback in Russia

June 26, 2015

I know that many people believe that a search reveals “all” and “everything” about a topic. Nothing is further from the truth. There are forces at work which wish to ensure that only certain information is available to a person with an Internet connection.

Navigate to “Russia Bans the Internet Archive’s ‘Wayback Machine’.” The Wayback Machine, which once had tie ups with outfits as different as the Library of Congress and Amazon. I found it useful when working as an expert witness to be able refresh my memory on certain Web sites’ presentation of information. I am confident there are other uses of “old information.”

According the write up, Russia is not to keen on the notion of old information. Kenneth Waggner, one of my high school teachers, had Russian language textbooks from the Stalin era. He had marked passages included in one book and excluded from another. If he were correct, the tradition of filtering has a reasonable track record in Russia. Keep in mind that other countries and company and individuals have the same goal: Present only what a smarter, more informed person thinks I should be able to access.

The article states:

By banning access to the Internet Archive, the government is denying Russian Internet users a powerful tool—one that is particularly useful in an environment where websites often disappear behind a state-operated blacklist, as is increasingly true in Russia today.

Governments are like horse races. No one is sure of the winner unless the race is rigged.

Stephen E Arnold, June 26, 2015

Next Page »