Crime Prediction: Not a New Intelligence Analysis Function

March 16, 2018

We noted “New Orleans Ends Its Palantir Predictive Policing Program.” The interest in this Palantir Technologies’ project surprised us from our log cabin with a view of the mine drainage run off pond. The predictive angle is neither new nor particularly stealthy. Many years ago when I worked for one of the outfits developing intelligence analysis systems, the “predictive” function was a routine function.

Here’s how it works:

  • Identify an entity of interest (person, event, organization, etc.)
  • Search for other items including the entity
  • Generate near matches. (We called this “fuzzification” because we wanted hits which were “near” the entity in which we had an interest. Plus, the process worked reasonably well in reverse too.)
  • Punch the analyze function.

Once one repeats the process several times, the system dutifully generates reports which make it easy to spot:

  • Exact matches; for example, a “name” has a telephone number and a dossier
  • Close matches; for example, a partial name or organization is associated with the telephone number of the identity
  • Predicted matches; for example, based on available “knowns”, the system can generate a list of highly likely matches.

The particular systems with which I am familiar allow the analyst, investigator, or intelligence professional to explore the relationships among these pieces of information. Timeline functions make it trivial to plot when events took place and retrieve from the analytics module highly likely locations for future actions. If an “organization” held a meeting with several “entities” at a particular location, the geographic component can plot the actual meetings and highlight suggestions for future meetings. In short, prediction functions work in a manner similar to Excel’s filling in items in a number series.

heat map with histogram

What would you predict as a “hot spot” based on this map? The red areas, the yellow areas, the orange areas, or the areas without an overlay? Prediction is facilitated with some outputs from intelligence analysis software. (Source: Palantir via Google Image search)

Read more

Is Change Coming to High Tech Lobbying in Washington, DC?

March 14, 2018

The received wisdom in Washington, DC is that when it comes to politics, money talks.

The idea is simple: Donate money to a politician’s campaign or a politician’s favorite “cause” and get your email and phone calls answered.

The Independent explains that, “Google Outspends All Rival Washington Lobbyists For First Time In 2017.”

In 2017, Google spent $18 million to lobby Congress on a slew of issues ranging from immigration, tax reform, antitrust, and online advertising. Tech companies have big bucks and the power to take on Congress on governmental policies. Lawmakers, on the other hand, fire back with pot shots like allowing Russian operatives to share content and how their software and other technology allows tech companies to abuse their power.

Google’s Washington operation proposed legislation that would require Web companies to collaborate on a public database of political as that run on their platforms. The idea is that the database would prevent foreign nations from exploiting online platforms. Other companies like Amazon and Facebook have ramped up their lobbying spending too.

Despite the power tech companies wield, their roles in society are changing and there is some fear associated with it:

“‘These are companies that are touching so many parts of the economy, they are touching so many parts of our geography. So it’s inevitable that they are going to engage in a host of political and policy issues,’ said Julie Samuels, the executive director of Tech: NYC, a group that represents New York-based tech firms. Samuels added that Silicon Valley has also had to adjust to a new political order, under a Republican administration. ‘Many tech companies had only been real players during the Obama administration. They had a lot to learn.’”

Now the received wisdom may have to modified. Beyond Search noted that Palantir has landed a chunk of a US government contract to create a DCGS which meets the needs of the US Army.

We think that Google will continue to support lobbying, but it will seek more deals like its tie up with the US government’s push for artificial intelligence. What may emerge is a new approach to influencing procurement decisions and legislation in Washington.

Whitney Grace, March 14, 2018

Short Honk: Palantir Technologies and DCGS

March 10, 2018

I don’t know if the information in “Army Taps Raytheon, Palantir for Potential $876M Ground Intell system Support Contract.” The Beyond Search and Dark Cyber teams will monitor the subject. The GovConWire stated on March 9, 2018:

Raytheon and Palantir Technologies have won spots on a potential 10-year, $876 million contract to help the U.S. Army address technology requirements for the service branch’s Distributed Common Ground System.

If on the money, this is big news. Our perception was that Palantir was not in the DCGS winner’s circle. Looks like IBM and its technology partners have to adapt.

Stephen E Arnold, March 10, 2018

Palantir: Accused of Hegelian Contradictions

January 29, 2018

I bet you have not thought about Hegel since you took that required philosophy course in college. Well, Hegel and his “contradictions” are central to “WEF 2018: Davos, Data, Palantir and the Future of the Internet.”

I highlighted this passage from the essay:

Data is the route to security. Data is the route to oppression. Data is the route to individual ideation. Data is the route to the hive mind. Data is the route to civic wealth. Data is the route to civic collapse.

Thesis, antitheses, synthesis in action I surmise.

The near term objective is synthesis. I assume this is the “connecting the dots” approach to finding what one needs to know.

I learned:

The stakes for big data couldn’t be bigger.

Okay, a categorical in our fast changing, diverse economic and political climate. Be afraid seems to be the message.

Palantir’s point of operations in Davos is described in the write up as “a pimped up liquor store.” Helpful and highly suggestive too.

The conclusion of the essay warranted a big red circle:

So next time you hear the names Palantir or Alex Karp, stop what you’re doing and pay attention. The future – your future – is under discussion. Under construction. This little first draft of history of which you’ve made it to the end (congratulations and thanks) – the history of data – is of a future that will in time come to be seen for what it is: digital that truly matters.

Several observations:

  • The author wants me to believe that Palantir is not a pal.
  • The big data thing troubles the author because Palantir is one of the vendors providing next generation information access.
  • The goal of making Palantir into something unique is best accomplished by invoking Fancy Dan ideas.

I would suggest that knowledge about companies like Gamma Group FinFisher, Shoghi, Trovicor, and some other interesting non US entities might put Palantir in perspective. Palantir has an operational focus; some of the other vendors perform different information services.

Palantir is an innovator, but it is part of a landscape of data intercept and analysis organizations. I could make a case that Palantir is capable but some companies in Europe and the East are actually more technologically advanced.

But these outfits were not at Davos. Why? That’s a good question. Perhaps they were too busy with their commercial and government work. My hunch is that a few of these outfits were indeed “there”, just not noticed by the expert who checked out the liquor store.

Stephen E Arnold, January 29, 2019

Palantir and Google: Surprising Allegation from St Louis

November 16, 2017

I read “Thiel Gave Money to Missouri Attorney General Going after Google.” The article reports:

Peter Thiel, the venture capitalist who backed Donald Trump’s presidential run, gave $300,000 to a political campaign of Josh Hawley, the Missouri attorney general who opened an antitrust investigation into Google this week.

My reaction was, “Is there a connection between this donation and the investigation of Google by Josh Hawley, the Missouri attorney general?”

The article appears to make this connection. I am not so quick to seize upon this implication. From my point of view, without more factual information, the story leaves me as cold as a catfish pulled from the Crooked River.

Stephen E Arnold, November 16, 2017

Palantir Technologies: Valuation Doubts?

October 18, 2017

i read “Palantir Will Struggle to Hold On to $20 Billion Valuation, Study Says.” Interesting stuff because beating up on hapless Silicon Valley companies is becoming a mini-trend. Facebook is in the dog house because it sells ads. Google is in the kennel because Europe finds its business practices less than Euro-cool. Twitter. Poor Twitter. Its part time boss is going to improve controls on the Wild West of short messages.

Now it is Palantir, the software company which offers an alternative to the IBM Analyst Notebook system. I thought Palantir was in the cat bird seat to provide technology that would deliver certain functionality to various US government agencies, financial institutions, and other organizations wanting to make sense of data.

I learned from the Bloomberg write up:

If Palantir Technologies Inc. pursues plans for a public offering and follows through by 2019, it will need to rein in spending and woo corporate customers just to be able to hang on to a $20 billion valuation it was awarded two years ago, according to a new study. It could also be worth a lot less.

Bloomberg cites a “study” which reveals that Palantir technology needs some set up and configuration before the users can make sense of digital information processed by the system.

This apparently comes as a surprise to Bloomberg and the SharesPost research team.

The reality of next generation information access systems is different from an iPhone or Android app one downloads and uses immediately. I know this is a surprise to many “experts,” but next generation information access systems are complicated. I explain why in my 2015 CyberOSINT: Next Generation Information Access Systems.

What’s interesting is that instead of putting the Palantir systems in a meaningful context, the report and apparently Bloomberg want to make another Silicon Valley outfit look like a bent penny.

Valuation is in the eye of the beholder and the Excels generated by whiz kids who want to buy a new Porsche.

Bloomberg quotes the report as a way to wrap up the news story with a stomp on Palantir’s foot; to wit:

Palantir “is currently valued much higher than its peers in the big data and analytics space,” Kulkarni wrote, adding that he believes Palantir will maintain the rich valuation if it keeps adding corporate clients and expedites cost cutting. He wrote that Palantir remains an attractive acquisition target – Oracle weighed the option last year but demurred – and estimated Palantir’s low-end value in 2019 at $13.8 billion.

Is there another view of Palantir? Guess not.

Stephen E Arnold, October 18, 2017

China Transwarp: Can This Be a Palantir Challenger?

July 24, 2017

One of my sources provided me with a link to a write up which may be translated as “Yujialong star ring technology common to build China Palantir” or “Yu Jialong together star ring technology together to build China’s Palantir.” The link to the original article is here. “Yu Jialong” is a subsidiary of Boone Group, which may no longer be in operation. The point of the write up is that a group of Chinese wizards is working to create a “Chinese Palantir. The group is hoodek up with Six Ring Technology. TenCent is providing some financing.


This may be the experts who are tackling the Palantir like system.

There is the challenge of seamlessly importing the file formats used by developers of cyINT eDiscovery systems. I have added it to mist of companies engaged in moving beyond Analyst’s Notebook and Gotham systems.

Stephen E Arnold, July 24,2017

Palantir Technologies: The Buzzfeed Beat

July 3, 2017

I read “There’s a Fight Brewing between the NYPD and Silicon Valley’s Palantir.” Two points about this story. Palantir Technologies, a vendor profiled in my CyberOSINT and Dark Web Notebook reports is probably going to keep its eye on the real journalistic outfit Buzzfeed. I don’t know much about “real” journalism, but my hunch is that if Palantir’s stakeholders find the Buzzfeed write up coverage interesting, some of those folks might spill their Philz coffee.

The other point is that the New York Police Department may find questions about its contractual dealings a bit of distraction from the quotidian tasks the force faces each day. I would not characterize “real” journalists asking questions “annoying,” but I would hazard the phrase “time consuming” or the word “distracting.”


“You want me to believe that?” asks Max, a skeptical show dog who knows that some owners will do anything to win.

The point of the “Fight Brewing” write up strikes me as a story designed to suggest that Palantir Technologies may be showing some signs of stress. When I read the story, I thought of the news which swirled around some of the defunct enterprise search companies when one of their client engagements went south. Vendors hit with these situations can do little but ride out the storm.

Hey, enterprise search was routinely oversold. When a system was up and running, the results were usually similar to the results generated by the previous “solution to all your information problems.” The search engineers who coded the systems knew that overpromising and under delivering were highly probable once the on switch was flipped. But the sales professional were going to say what was necessary to close the deal. In fact, most of the fancy promises about an enterprise search system set the company up for failure.

Is that what’s going on in the NYPD-Palantir “showdown”? To wit:

Palantir explained the system’s functions and outputs. The NYPD signed on. Then when the system was installed, additional work was needed to make the Palantir system meet the expectations set by the Palantir sales engineers.

The “Fight Brewing” story says:

The NYPD quietly began work last summer on its replacement data system, and in February it announced internally that it would cancel its Palantir contract and switch to the new system by the beginning of July, according to three people familiar with the matter. The new system, named Cobalt, is a group of IBM products tied together with NYPD-created software. The police department believes Cobalt is cheaper and more intuitive than Palantir, and prizes the greater degree of control it has over this system.

Keep in mind that I, before I retired in 2013, had been an adviser to the original i2 Group Ltd., the company which created in my opinion the analytic and visualization method which defines modern cyber eDiscovery in the 1990s.

The notion that IBM, which now owns i2’s Analyst’s Notebook, is working hard to close deals in key Palantir accounts from what I have heard in the general store in Harrod’s Creek.

I don’t have to go much farther than my own experience to get a sense that the “fight” may be a manifestation of how the world works when it comes to making sales for systems like Palantir’s Gotham or IBM’s i2. In my work career I have seen some interesting jabs and punches thrown to close a deal.

The NYPD, like any organization, wants systems which work and represent good value. Incumbent vendors have to find a way to retain a customer. Competitors have to find a way to get a licensee of one product to switch to a different product.

I noted this statement in the “Fight Brewing” story:

Palantir has struggled to expand its work with the police force, the emails show. As of March and April 2015, Palantir had had “little exposure to the top brass,” and although it wanted to add more business, “the door there clearly still remains closed given the larger political environment,” staffers wrote in emails. A staffer at one point invoked a phrase popularized by Thiel, author of Zero to One: Notes on Startups, or How to Build the Future, saying that Palantir still needed to get “from 0->1 at NYPD.”

Now how many police forces in the US can afford a comprehensive cyber eDiscovery system like Palantir Gotham or IBM Analyst’s Notebook? This is an important point because the number of potential customers is quite small. For example, after NY, LA, Chicago, Miami, and maybe three or four other cities, the sales professional runs out of viable prospects. How many counties can foot the bill for the software, the consultants, and the people required to tag and analyze the data? The number is modest. How many US states can afford the investment in high end cyber eDiscovery software? Again, the number is small, and you can count out Illinois because getting bills paid is an interesting challenge. The same market size problem exists for US government entities.

Read more

Palantir: Outside In or In In or?

June 12, 2017

I try to keep up with Palantir Technologies’ news. I love the job openings; for example, a real estate person for the firm’s New York office. I noted a more substantive item called “Palantir Goes from Pentagon Outsider to Mattis’ Inner Circle.” General James Mattis is affectionately known by some as “Mad Dog.” In 2016, a real journalist described General Mattis as a “warrior monk.” For me, I will stick with General Mattis and ignore suggestions that he would only visit a monastery if it were stocked with certain essential items. I have heard these items include beer and possibly. Well, never mind.

The point of the write up is that on General Mattis’ watch, Palantir is a technology outfit which interests him. Perhaps a more accurate statement is that the functionality of Gotham matches his A to B thinking with regard to getting actionable intelligence.

The write up tells me:

Palantir’s startup mentality has led it to shun the way business is typically done in Washington and, as a result, made some enemies in the process, including some larger, more traditional defense companies.

I wondered, “Is IBM one of those traditional companies?” What’s a little out of court settlement between friends?

The point of the write up seems to be that three people working in the General Mattis’ unit had some involvement with Palantir. The 64 dollar question is, “So what?”

The write up put my mind at ease with this statement:

Byron Callan, an analyst with Capital Alpha Partners, said he is “skeptical” that three staffers with ties to Palantir will cause the Pentagon to flip its position on the company or start doing business with it. But he said having these Silicon Valley voices on the inside could foster a continuation of some of former Defense Secretary Ash Carter’s priorities to build bridges between the Pentagon and industry in hubs of innovation. “It was debatable whether this whole push that Carter had when he was SecDef would survive,” Callan said. “When you have people at Palantir in the positions they are, you have to believe there are some voices that are not just standing in the visitors area waiting to get in and talk about it.”

My experience is that one’s work experience can be a plus. I snagged a gig while working at a well known firm with Craig Hosmer, then a Congressman and a retired admiral. Big time consulting firms and high profile government contractors flow into and out of the government in my experience.

Is Palantir now in or is Palantir sort of in? Of course, Palantir could be “in” but still on the outside? Other permutations are possible, but almost anything is possible when one catches Potomac fever even writing about employees’ work history.

Stephen E Arnold, June 12, 2017

Palantir Technologies: A Beatdown Buzz Ringing in My Ears

April 27, 2017

I have zero contacts at Palantir Technologies. The one time I valiantly contacted the company about a speaking opportunity at one of my wonky DC invitation-only conferences, a lawyer from Palantir referred my inquiry to a millennial who had a one word vocabulary, “No.”

There you go.

I have written about Palantir Technologies because I used to be an adviser to the pre-IBM incarnation of i2 and its widely used investigation tool, Analyst’s Notebook. I did write about a misadventure between i2 Group and Palantir Technologies, but no one paid much attention to my commentary.

An outfit called Buzzfeed, however, does pay attention to Palantir Technologies. My hunch is that the online real news outfit believes there is a story in the low profile, Peter Thiel-supported company. The technology Palantir has crafted is not that different from the Analyst’s Notebook, Centrifuge Systems’ solution, and quite a few other companies which provide industrial-strength software and systems to law enforcement, security firms, and the intelligence community. (I list about 15 of these companies in my forthcoming “Dark Web Notebook.” No, I won’t provide that list in this free blog. I may be retired, but I am not giving away high value information.)

So what’s caught my attention. I read the article “Palantir’s Relationship with the Intelligence Community Has Been Worse Than You Think.” The main idea is that the procurement of Palantir’s Gotham and supporting services provided by outfits specializing in Palantir systems has not been sliding on President Reagan’s type of Teflon. The story has been picked up and recycled by several “real” news outfits; for example, Brainsock. The story meshes like matryoshkas with other write ups; for example, “Inside Palantir, Silicon Valley’s Most Secretive Company” and “Palantir Struggles to Retain Clients and Staff, BuzzFeed Reports.” Palantir, it seems to me in Harrod’s Creek, is a newsy magnet.

The write up about Palantir’s lousy relationship with the intelligence community pivots on a two year old video. I learned that the Big Dog at Palantir, Alex Karp, said in a non public meeting which some clever Hobbit type videoed on a smartphone words presented this way by the real news outfit:

The private remarks, made during a staff meeting, are at odds with a carefully crafted public image that has helped Palantir secure a $20 billion valuation and win business from a long list of corporations, nonprofits, and governments around the world. “As many of you know, the SSDA’s recalcitrant,” Karp, using a Palantir codename for the CIA, said in the August 2015 meeting. “And we’ve walked away, or they walked away from us, at the NSA. Either way, I’m happy about that.” The CIA, he said, “may not like us. Well, when the whole world is using Palantir they can still not like us. They’ll have no choice.” Suggesting that the Federal Bureau of Investigation had also had friction with Palantir, he continued, “That’s de facto how we got the FBI, and every other recalcitrant place.”

Okay, I don’t know the context of the remarks. It does strike me that 2015 was more than a year ago. In the zippy doo world of Sillycon Valley, quite a bit can change in one year.

I don’t know if you recall Paul Doscher who was the CEO of Exalead USA and Lucid Imagination (before the company asserted that its technology actually “works). Mr. Doscher is a good speaker, but he delivered a talk in 2009, captured on video, during which he was interviewed by a fellow in a blue sport coat and shirt. Mr. Doscher wore a baseball cap in gangsta style, a crinkled unbuttoned shirt, and evidenced a hipster approach to discussing travel. Now if you know Mr. Doscher, he is not a manager influenced by gangsta style. My hunch is that he responded to an occasion, and he elected to approach travel with a bit of insouciance.

Could Mr. Karp, the focal point of the lousy relationship article, have been responding to an occasion? Could Mr. Karp have adopted a particular tone and style to express frustration with US government procurement? Keep in mind that a year later, Palantir sued the US Army. My hunch is that views expressed in front of a group of employees may not be news of the moment. Interesting? Sure.

What I find interesting is that the coverage of Palantir Technologies does not dig into the parts of the company which I find most significant. To illustrate: Palantir has a system and method for an authorized user to add new content to the Gotham system. The approach makes it possible to generate an audit trail to make it easy (maybe trivial) to answer these questions:

  1. What data were added?
  2. When were the data added?
  3. What person added the data?
  4. What index terms were added to the data?
  5. What entities were added to the metadata?
  6. What special terms or geographic locations were added to the data?

You get the idea. Palantir’s Gotham brings to intelligence analysis the type of audit trail I found some compelling in the Clearwell system and other legal oriented systems. Instead of a person in information technology saying in response to a question like “Where did this information come from?”, “Duh. I don’t know.”

Gotham gets me an answer.

For me, explaining the reasoning behind Palantir’s approach warrants a write up. I think quite a few people struggling with problems of data quality and what is called by the horrid term “governance” would find Palantir’s approach of some interest.

Now do I care about Palantir? Nah.

Do I care about bashing Palantir? Nah.

What I do care about is tabloidism taking precedence over substantive technical approaches. From my hollow in rural Kentucky, I see folks looking for “sort of” information.

How about more substantive information? I am fed up with podcasts which recycle old information with fake good cheer. I am weary of leaks. I want to know about Palantir’s approach to search and content processing and have its systems and methods compared to what its direct competitors purport to do.

Yeah, I know this is difficult to do. But nothing worthwhile comes easy, right?

I can hear the millennials shouting, “Wrong, you dinosaur.” Hey, no problem. I own a house. I don’t need tabloidism. I have picked out a rest home, and I own 60 cemetery plots.

Do your thing, dudes and dudettes of “real” journalism.

Stephen E Arnold, April 27, 2017

Next Page »

  • Archives

  • Recent Posts

  • Meta