DarkCyber for January 12, 2021, Now Available
January 12, 2021
DarkCyber is a twice-a-month video news program about online, the Dark Web, and cyber crime. You can view the video on Beyond Search or at this YouTube link.
The program for January 12, 2021, includes a featured interview with Mark Massop, DataWalk’s vice president. DataWalk develops investigative software which leapfrogs such solutions as IBM’s i2 Analyst Notebook and Palantir Gotham. In the interview, Mr. Massop explains how DataWalk delivers analytic reports with two or three mouse clicks, federates or brings together information from multiple sources, and slashes training time from months to several days.
Other stories include DarkCyber’s report about the trickles of information about the SolarWinds’ “misstep.” US Federal agencies, large companies, and a wide range of other entities were compromised. DarkCyber points out that Microsoft’s revelation that bad actors were able to view the company’s source code underscores the ineffectiveness of existing cyber security solutions.
DarkCyber highlights remarkable advances in smart software’s ability to create highly accurate images from poor imagery. The focus of DarkCyber’s report is not on what AI can do to create faked images. DarkCyber provides information about how and where to determine if a fake image is indeed “real.”
The final story makes clear that flying drones can be an expensive hobby. One audacious drone pilot flew in restricted air zones in Philadelphia and posted the exploits on a social media platform. And the cost of this illegal activity. Not too much. Just $182,000. The good news is that the individual appears to have avoided one of the comfortable prisons available to authorities.
One quick point: DarkCyber accepts zero advertising and no sponsored content. Some have tried, but begging for dollars and getting involved in the questionable business of sponsored content is not for the DarkCyber team.
Finally, this program begins our third series of shows. We have removed DarkCyber from Vimeo because that company insisted that DarkCyber was a commercial enterprise. Stephen E Arnold retired in 2017, and he is now 77 years old and not too keen to rejoin the GenX and Millennials in endless Zoom meetings and what he calls “blatant MBA craziness.” (At least that’s what he told me.)
Kenny Toth, January 12, 2021
LinkedIn Reveals Disinterest in Search and Retrieval
December 7, 2020
LinkedIn does quite a bit of info-ramming when either one of my team or I log in to the Microsoft social media system. Here’s the graphic displayed when we were checking to see if our automated posts from this blog were appearing:
The eight “cards” tell me about LinkedIn Groups in which I may have an interest. The little boxes reveal a small amount of information about the content access topics in which the unemployed, the consultants cruising for gigs, and the self-promoters have an interest.
The table below presents some of the data in this graphic in tabular form. No, I did not use Excel 365 connected to Teams. Sorry, Mother Microsoft. I still recall Bob. (You remember Bob, don’t you, gentle reader?)
| LinkedIn Group Name | Number of LinkedIn Followers |
| Data Science Central | 374,694 |
| Association for Intelligent Information Management | 27,861 |
| Scientific, Technical, Medical Publishing Group | 12,253 |
| Data & Text Analytics Professionals | 12,503 |
| Special Libraries Asso. | 15,191 |
| Semantic Web | 15,098 |
| Semantic Technologies Group | 3,772 |
| Enterprise Search & Discovery | 624 |
LinkedIn does not reveal the hard count for its total number of registered humans, the number of human users who log on to the system once per week, or the number of paying human users. Hence, figuring out the percentage of LinkedIn members interested in these groups is a difficult task akin to predicting the share price of Palantir Technologies on January 1, 2022.
An outfit called Oberlo reports with confidence that LinkedIn has 660 million users. Close enough for horseshoes.
The table below presents the percentage of these LinkedIn users interested in each the groups suggested to me:
| LinkedIn Group Name | Percentage of LinkedIn Members Interested in These Topics |
| Data Science Central | 0.0567718182% |
| Association for Intelligent Information Management | 0.0042213636% |
| Scientific, Technical, Media Publishing Group | 0.0018565152% |
| Data & Text Analytics Professionals | 0.0018943939% |
| Special Libraries Asso. | 0.0023016667% |
| Semantic Web | 0.0022875758% |
| Semantic Technologies Group | 0.0005715152% |
| Enterprise Search & Discovery | 0.0000945455% |
Eyeballing my math, surely there are errors. How can such a compelling subject as Enterprise Search & Discovery appeal to 0.0000945455 percent of the LinkedIn members.
What’s interesting is that an astounding 0.0042213636 percent of the LinkedIn membership are pulled to the Association for Intelligent Information Management.
And the semantic topics. Magnetic indeed.
What’s the analysis suggest? Anyone looking for a job in enterprise search may want to spin their expertise a different way.
Stephen E Arnold, December 7, 2020
Why Investigative Software Is Expensive
December 3, 2020
In a forthcoming interview, I explore industrial-strength policeware and intelware with a person who was Intelligence Officer of the Year. In that review, which will appear in a few weeks, the question of cost of policeware and intelware is addressed. Systems like those from IBM’s i2, Palantir Technologies, Verint, and similar vendors are pricey. Not only is there a six or seven figure license fee, the client has to pay for training, often months of instruction. Plus, these i2-type systems require systems and engineering support. One tip off of to the fully loaded costs is the phrase “forward deployed engineer.” The implicit message is that these i2-type systems require an outside expert to keep the digital plumbing humming along. But who is responsible for the data? The user. If the user fumbles the data bundle, bad outputs are indeed possible.
What’s the big deal? Why not download Maltego? Why not use one of the $100 to $3,000 solutions from jazzy startups by former intelligence officers? These are “good enough”, some may assert. One facet of the cost of industrial strength systems available to qualified licensees is a little appreciated function: Dealing with data.
“Keep Data Consistency During Database Migration” does a good job of explaining what has to happen in a reliable, consistent way when one of the multiple data sources contributes “new” or “fresh” data to an intelware or policeware system. The number of companies providing middleware to perform these functions is growing. Why?
Most companies wanting to get into the knowledge extraction business have to deal with the issues identified in the article. Most organizations do not handle these tasks elegantly, rapidly, or accurately.
Injecting incorrect, stale, inaccurate data into a knowledge centric process like those in industrial strength policeware causes those systems to output unreliable results.
What’s the consequence?
Investigators and analysts learn to ignore certain outputs.
Why? The outputs can be more serious than a flawed diagram whipped up by an MBA who worries only about the impression he or she makes on a group of prospects attending a Zoom meeting.
Data consistency is a big deal.
Stephen E Arnold, December 2, 2020
Amazon and the Cyber Security Industrial Complex
November 24, 2020
This is probably no big deal. Cyber security, threat intelligence, and wonky proprietary tools from startups populated by retired or RIFed intel officers are a big business. I was asked by a “real news” reporter, “How big?” I dutifully sent links to companies selling market forecasts for global cyber security revenues. How big were these numbers? Acquisition big. The hypothesis I have formulated is that when wild and crazy market size projections fly like hungry sparrows, there is a revenue problem. Specifically there are too many sparrows chasing available bugs and bread crumbs. That’s why Blackberry is in the cyber security business. Why LookingGlass stepped away from Cyveillance. That’s why Dark Web indexes of bad actors’ Crime as a Service offerings are a dime a dozen.
It is, therefore, no surprise that the write up “Trend Micro integrates with AWS Network Firewall” explains that Amazon is continuing to add to its pool of 65,000 plus partners. Many of these outfits like Palantir Technologies are in the cyber intelligence and cyber threat business. Bad actors beware.
The write up reports:
Trend Micro’s built-in IPS intelligence will inspect traffic for malicious intent so that the firewall can stop threats before they get a foothold in a virtual private cloud. Together, AWS and Trend Micro offer a simple, scalable service with reliable protection that does not require any infrastructure management.
What’s the hook? Here’s the statement I circled with an Amazon happy face:
Trend Micro’s threat intelligence will be available free with easy deployment for AWS Network Firewall customers.
What do I make of free cyber security services? No much but I hear the Bezos bulldozer pulling into the cyber intelligence and security services shopping mall. Roll up or roll over time for the cheerful orange machine with a big smile painted on the cab.
Stephen E Arnold, November 24, 2020
Court Case Hunger? Judyrecords Is Available
November 24, 2020
Unable to pay the fee for LexisNexis-type commercial search systems? You are not alone. If you want information from court records, navigate to Judyrecords. Within the last couple of months, the system has added more than 35 million cases. Aren’t these data available for free elsewhere? Sure, if you like going through hoops like verification procedures. Judyrecords lets a user plug in the names of entities and view results. I ran one of my go to queries: “Palantir IBM.” Here are the results:
This may not be important to you, but for those who have to wade through for fee legal search systems, Judyrecords is helpful. But for how long? Yes, that is a good question. For now, however, give it a whirl. Keep in mind that US court systems without online technology or special arrangements for document access prevent the system from being comprehensive. Lawyers enjoy results which must be checked by billable professionals, however.
Stephen E Arnold, November 23, 2020
Soros: Just in Time 20-20 Hindsight
November 18, 2020
Here’s an interesting quote (if it is indeed accurate):
SFM [a George Soros financial structure] made this investment [in Palantir Technologies] at a time when the negative social consequences of big data were less understood,” the firm said in a statement Tuesday. SFM would not make an investment in Palantir today.
The investment concerns Palantir Technologies. George Soros, who is 90 years young, according to “Soros Regrets Early Investment in Peter Thiel’s Palantir,” includes this statement:
Soros has sold all the shares it’s permitted to sell at this time and will keep selling, according to the statement. “SFM does not approve of Palantir’s business practices,” the firm said.
Hindsight is 20-20. Or is it?
Hindsight bias can cause memory distortion. Because the event happened like you thought it would, you go back and revise your memory of what you were thinking right before the event. You re-write history, so to speak, and revise the probability in hindsight. Going forward, you use that new, higher probability to make future decisions. When in fact, the probabilities haven’t changed at all. That leads to poor judgment.—“Innovators: Beware the Hindsight Bias”
Stephen E Arnold, November 18, 2020
Exclusive: Interview with DataWalk’s Chief Analytics Officer Chris Westphal, Who Guides an Analytics Rocket Ship
October 21, 2020
I spoke with Chris Westphal, Chief Analytics Officer for DataWalk about the company’s string of recent contract “wins.” These range from commercial engagements to heavy lifting for the US Department of Justice.
Chris Westphal, founder of Visual Analytics (acquired by Raytheon) brings his one-click approach to advanced analytics.
The firm provides what I have described as an intelware solution. DataWalk ingests data and outputs actionable reports. The company has leap-frogged a number of investigative solutions, including IBM’s Analyst’s Notebook and the much-hyped Palantir Technologies’ Gotham products. This interview took place in a Covid compliant way. In my previous Chris Westphal interviews, we met at intelligence or law enforcement conferences. Now the experience is virtual, but as interesting and information in July 2019. In my most recent interview with Mr. Westphal, I sought to get more information on what’s causing DataWalk to make some competitors take notice of the company and its use of smart software to deliver what customers want: Results, not PowerPoint presentations and promises. We spoke on October 8, 2020.
DataWalk is an advanced analytics tool with several important innovations. On one hand, the company’s information processing system performs IBM i2 Analyst’s Notebook and Palantir Gotham type functions — just with a more sophisticated and intuitive interface. On the other hand, Westphal’s vision for advanced analytics has moved past what he accomplished with his previous venture Visual Analytics. Raytheon bought that company in 2013. Mr. Westphal has turned his attention to DataWalk. The full text of our conversation appears below.
Xoogler Awakes to a Reality: How about That?
August 30, 2020
Navigate to “Eric Schmidt: China Could Be AI’s Superpower If We Don’t Act Now.” The point of the write up seems to be to overlook the obvious. Mr. Schmidt was the “adult” at Google. He is now the technical advisor to the board of Alphabet where he was formerly the executive chairman. As executive chairman, he was responsible for the external matters of all of the holding company’s businesses, including Google Inc., advising their CEOs and leadership on business and policy issues.
Based on the information in the article, he seems to be going into full pundit mode.
Google and China. Advisor to Alphabet’s Board. Yep, telling the US China could be a leader in artificial intelligence. Perfect for 2020, a year of novelties.
I want to point out that an AI hot spot in China is Tsinghua University. This means that if the information in “Tsinghua University Plans to Open AI Research Center in China, Names Google’s AI Chief as Advisor” is accurate, Mr. Schmidt may want to focus on sending his message directly to Google.
The interview is just about as 2020 as Palantir’s insistence that it is not a Silicon Valley type outfit.
Yes, 2020. A good year for insights and interesting information.
Oh, Jeff Dean? He’s one of the serious engineers at the Google. BigTable, Chubby, and more, particularly in the smart software realm. Perhaps he is advising Tsinghua University on recipes, once an interest of the person who may be the smartest Googler in the collection of wizards.
This AI thing. Is Google helping out China in its AI efforts? Good question. Maybe a Sillycon Valley journalist will do some investigative reporting? Nah, it’s 2020. Redefine reality.
Stephen E Arnold, August 30, 2020
Policeware Marketing: Medium Blog Post a Preferred Channel
May 27, 2020
Just a short note. Sintelix, a developer of policeware, published a white paper called “Sintelix – The Text Intelligence Solution.” The document is available on Medium, a popular publishing platform. The white paper / product description is about 1,500 words in length. The article contains illustrations, diagrams, and lists. One of the DarkCyber team said, “It reads like a product brochure.” I thought the information was useful. The Sintelix decision to use Medium is one additional example of the stepped up marketing policeware and intelware vendors are taking. Shadowdragon is using Twitter to promote its policeware system. Palantir Technologies’ CEO has used public forums and video to explain use cases for its investigative and analytic system. LookingGlass has moved forward with online demonstrations and webinars for potential licensees of its cyber system.
Observations:
- Tradition has dictated that vendors of specialized services developed for law enforcement and intelligence agencies market via personal contacts or restricted/classified events. The fierce competition for available government contracts may be forcing a change.
- Face-to-face events like breakfasts, brown bag presentations, and lectures for LE and intel professionals may be difficult to supplement with online-only initiatives. Plus, there is the danger of webinar fatigue or the sign up and no show problem. Security is also a consideration because some attendees may not be whom they purport to be on registration form.
- Expanded visibility like that achieved by NSO Group may have unexpected consequences. Some government procurement processes shy away from high-profile vendors. As a result, obtaining information about the activities of a Leidos, for example, is difficult. Increased visibility may repel some potential customers.
The shift in marketing, no matter how minor, is important. The downstream consequences of visibility are difficult to predict because conference organizers, procurement processes, customers who prefer face-to-face interactions, and similar pre Rona methods may no longer work as well.
Stephen E Arnold, May 27, 2020
AWS Kendra: A Somewhat Elastic Approach to Enterprise Search
May 12, 2020
Elastic, Shay Banon’s Version 2 of Compass, has a hurdle to jump over. Elasticsearch has been a success. The Lucene-centric “system” which some call ELK has become a go-to solution for many developers. Like Lucidworks (It does?) and many other “enterprise search and more” vendors, Elasticsearch delivers information retrieval without the handcuffs of options like good old STAIRS III or Autonomy’s neuro-linguistic black box.
Amazon took notice and has effectively rolled out its own version of enterprise search based on … wait for it … the open source version of Elastic’s Elasticsearch. The service has been around since Amazon hired some of the Lucidworks (It does?) engineers more than five years ago after frustration with the revolving doors at that firm became too much even by Silicon Valley standards. Talk about tension. Yebo!
Amazon has reinvented Elasticsearch. The same process the Bezos bulldozer has used for other open source software has been in process for more than 60 months. Like the system’s Playboy bunny namesake, Kendra has a few beauty lines in her AWS exterior.
A tweak here (access to Amazon’s smart software) and a tweak there (Amazon AWS pricing methods), and the “new” product is ready for prime time, ready for a beauty contest against other contestants in the most beautiful IR system in the digital world.
“Amazon Launches Cognitive Search Service Kendra in General Availability” reports:
Once configured through the AWS Console, Kendra leverages connectors to unify and index previously disparate sources of information (from file systems, websites, SharePoint, OneDrive, Salesforce, ServiceNow, Amazon Simple Storage Service, relational databases, and elsewhere).
Does this sound like federated search or the Palantir Gotham approach to content?
Well, yes.
The reason is that most enterprise search vendors like Coveo, Attivio, X1, IBM Omnifind (also built on Lucene), and dozens of other systems make the same claims.
The reality is that these systems do not have the bits and pieces available within a giant cloud platform with quite a few graduates of an Amazon AWS training program ready to plug in the AWS solution. For example, if a government agency wants the search in Palantir, no problem. Palantir deploys on AWS. But if that government agency wants to use Amazon’s policeware services and include search, there’s Kendra.
You can get a free copy of the DarkCyber Amazon policeware report’s executive summary by requesting the document at this link.
What does Amazon bring to the enterprise search party?
The company has more than 200 services, features, component, and modules on the shelf. Because enterprise search is not a “one size fits all”, the basic utility function has to fit into specific enterprise roles. For most enterprise search vendors, this need for user function customization is a deal breaker. Legal doesn’t want the same search that those clear minded home economics grads require in the marketing department. Microsoft SharePoint offers its version of “enterprise search” but paints over the cost of the Microsoft Certified professionals who have to make the search system work Fast. (Yep, that’s sort of an inside search joke.)
Amazon AWS provides the engine and the Fancy Dan components can be plugged in using the methods taught in the AWS “learn how to have a job for real” at a company your mom uses to shop during the pandemic. Amazon and Microsoft are on a collision course for the enterprise, and the Kendra thing is an important component.
The official roll out is capturing headlines, but the inclusion of Lucene-based search invites several observations:
- Despite AWS’ pricing, an Amazon enterprise search system allows the modern information technology professional to get a good enough service with arguably fewer headaches than other options except maybe the SearchBlox solution
- Enterprise search becomes what it has been for most organizations: A utility. Basic information retrieval is now an AWS component and that component can be enhanced with SageMaker, analytics, and other AWS services.
- Amazon wins even if Kendra does not win the hearts and minds of IBM Omnifind, Inbenta, and Algolia users. Why? Most of the cloud based enterprise search vendors support the AWS platform. What are the choices? The wonky HP cloud? The “maybe we will kill it” Google Cloud? Azure, from the outfit that cannot update Windows 10 without killing user computers who activate game mode? Plus, dumping Kendra for another TV star inspired search system is easy. Chances are that, like Palantir, AWS hosts and supports that competitive system too.
Net net: The fight with Microsoft is escalating. The Bezos bulldozer will run over open source outfits and probably some AWS customers. But Kendra’s turning her gaze on the bountiful revenues of Microsoft in the enterprise. Will Amazon buy a vendor of Word, PowerPoint, and Excel clones?
Exciting times, maybe not just because of enterprise search? Why did those defectors from Lucidworks (It does?) embrace Lucene and not SOLR? Maybe they did that too?
Stephen E Arnold, May 12, 2020
-
- Subscribe to Beyond Search
Feature archive
News archive
-
