Crime Prediction: Not a New Intelligence Analysis Function
March 16, 2018
We noted “New Orleans Ends Its Palantir Predictive Policing Program.” The interest in this Palantir Technologies’ project surprised us from our log cabin with a view of the mine drainage run off pond. The predictive angle is neither new nor particularly stealthy. Many years ago when I worked for one of the outfits developing intelligence analysis systems, the “predictive” function was a routine function.
Here’s how it works:
- Identify an entity of interest (person, event, organization, etc.)
- Search for other items including the entity
- Generate near matches. (We called this “fuzzification” because we wanted hits which were “near” the entity in which we had an interest. Plus, the process worked reasonably well in reverse too.)
- Punch the analyze function.
Once one repeats the process several times, the system dutifully generates reports which make it easy to spot:
- Exact matches; for example, a “name” has a telephone number and a dossier
- Close matches; for example, a partial name or organization is associated with the telephone number of the identity
- Predicted matches; for example, based on available “knowns”, the system can generate a list of highly likely matches.
The particular systems with which I am familiar allow the analyst, investigator, or intelligence professional to explore the relationships among these pieces of information. Timeline functions make it trivial to plot when events took place and retrieve from the analytics module highly likely locations for future actions. If an “organization” held a meeting with several “entities” at a particular location, the geographic component can plot the actual meetings and highlight suggestions for future meetings. In short, prediction functions work in a manner similar to Excel’s filling in items in a number series.
What would you predict as a “hot spot” based on this map? The red areas, the yellow areas, the orange areas, or the areas without an overlay? Prediction is facilitated with some outputs from intelligence analysis software. (Source: Palantir via Google Image search)
Several years ago, Recorded Future packaged numerical recipes, a workflow management systems, and a visualization system and focused on “predictive analytics.” The company leapfrogged the outfit which once paid me to slog in the corn fields. The math, in some cases, was similar. What was different was the sophistication of the software management wrapper and the visualization modules. As I have mentioned in this blog and in my lectures for Telestrategies ISS and other organizations, there are 10 algorithmic methods which are used by most companies providing analytics tools. The bulk of these mathematical procedures are statistical in nature and permit extrapolation or “prediction” of one sort or another. Plus there are open source modules which many intelligence analytics vendors use; for example, Hadoop, Maltego, and Lucene/Solr, among others.
The write up which emphasizes the end of the New Orleans’ Palantir “program” seems to suggest that something was wrong, off base, or ineffective.
Let me highlight this statement from the write up:
Yesterday, Orleans Criminal District Court Judge Camille Buras agreed to hear a motion from Kentrell Hickerson challenging his racketeering and drug conspiracy convictions. Hickerson’s attorney, Kevin Vogeltanz, filed his motion with the court on March 8th, citing the nondisclosure of any relevant intelligence from Palantir about his client’s alleged involvement in the 3NG street gang as a potential violation of Hickerson’s rights. Under the Supreme Court case Brady v. Maryland, defendants have the right to procure any and all potentially exculpatory evidence assembled against them by law enforcement.
My view from rural Kentucky consists of three observations. Make of these what you will.
Common sense. Use of next generation investigative systems and methods are usually kept close to the vest. The reasons range from a desire to avoid having to explain how smart software “works” to individuals with little background in the disciplines which underpin these analytic systems. I know from first hand experience when statistical procedures and probabilities are involved, explaining the functions to someone unfamiliar with the art is a very daunting tasks.
Tip offs. What’s the benefit of providing a bad actor with a front row seat to how certain intelligence analysis systems function, what they deliver, and their efficacy in performing such essential functions and associate identification, geofencing, timeline operations, and social graphing. Only a small percentage of bad actors are rocket scientists but there are many clever malefactors. Why provide a Cliff’s Notes or a recipe book so that evasive actions can be formulated?
Cultural norms. My team and I have worked for clients with different needs. We have learned that those engaged in security, law enforcement, and intelligence come in many flavors. But there is one quite interesting shared characteristic: These folks are not keen on talking to those not in their foxhole as one of my former clients put it. The idea is that those not in the foxhole are not going to be included in planning, operating, assessing, and making use of the outputs from intelligence systems. These norms are going to be difficult to change.
When a contract is not renewed, the reasons are often murky. Reasons range from budget issues. New Orleans may want to use one of the open source tools or systems now available. New Orleans may have had a personnel change and the new people want to use a system from another vendor. Budget issues can derail a deal. Political pressure can force a vendor out. Maybe the vendor decided the client was a bad fit for the firm’s products and services. Yada yada.
I don’t have a dog in this hunt. I don’t work for Palantir. In fact, I am retired and I don’t work for anyone at this moment. Even the training I do for law enforcement and intelligence professionals is provided at zero cost to the people who ask me to share what little knowledge I have.
I think that it is important to keep in mind that there are some systems and software which remain outside of the spotlight. Furthermore, these companies are in some ways unable to explain the facts behind a particular decision. Implying that something is amiss is not the same as understanding the constraints within which certain companies operate.
Stephen E Arnold, March 16, 2018
Comments
One Response to “Crime Prediction: Not a New Intelligence Analysis Function”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
{The web can certainly be bombarded with worthless content but that’s certainly not the case here. That is one of the decent ones. It’s actually supplied me some strategies for my new site.