Mobile and Social Media Users: Check Out the Utility of Metadata

January 15, 2021

Policeware vendors once commanded big, big bucks to match a person of interest to a location. Over the last decade prices have come down. Some useful products cost a fraction of the industrial strength, incredibly clumsy tools. If you are thinking about the hassle of manipulating data in IBM or Palantir products, you are in the murky field of prediction. I have not named the products which I think are the winners of this particular race.

image

Source: https://thepatr10t.github.io/yall-Qaeda/

The focus of this write up is the useful information derived from the deplatformed Parler social media outfit. An enterprising individual named Patri10tic performed the sort of trick which Geofeedia made semi famous. You can check the map placing specific Parler uses in particular locations based on their messages at this link. What’s the time frame? The unusual protest at the US Capitol.

The point of this short post is different. I want to highlight several points:

  1. Metadata can be more useful than the content of a particular message or voice call
  2. Metadata can be mapped through time creating a nifty path of an individual’s movements
  3. Metadata can be cross correlated with other data. (If you attended one of my Amazon policeware lectures, the cross correlation figures prominently.)
  4. Metadata can be analyzed in more than two dimensions.

To sum up, I want to remind journalists that this type of data detritus has enormous value. That is the reason third parties attempt to bundle data together and provide authorized users with access to them.

What’s this have to do with policeware? From my point of view, almost anyone can replicate what systems costing as much as seven figures a year or more from their laptop at an outdoor table near a coffee shop.

Policeware vendors want to charge a lot. The Parler analysis demonstrates that there are many uses for low or zero cost geo manipulations.

Stephen E Arnold, January 15, 2021

Open Source: Does It Mean What You Think It Means?

January 15, 2021

I spotted an article on Newswire called “Tech Giant Technology Is Open Source for the Pandemic, So Why Does It Feel So Closed?” The awkward title intrigued me. Open means, according to Dictionary.com:

not closed or barred at the time, as a doorway by a door, a window by a sash, or a gateway by a gate:to leave the windows open at night.

(of a door, gate, window sash, or the like) set so as to permit passage through the opening it can be used to close.

Pretty obvious. But open appears to mean closed. The “source” refers to software I assumed.

The write up sets me straight:

“The term ‘open source’ is being applied to the final design of an instrument – and I’m pleased to say there has been a willingness during the pandemic to share these final designs – but the design process itself also needs to be open, something it isn’t now,” explains physics researcher Dr Julian Stirling.

Okay, the “design process” has to be available. To get more insight into this open is closed issue, navigate to the original technical paper at this link. So far the paper is open, but as I have learned, open can be closed and often locked up behind a paywall.

Stephen E Arnold, January 15, 2021

Does Open Source Create Open Doors?

December 21, 2020

Here’s an interesting question I asked on a phone call on Sunday, December 20, 2020: “How many cyber security firms rely on open source software?”

Give up?

As far as my research team has been able to determine, no study is available to us to answer the question. I told the team that based on comments made in presentations, at lectures, and in booth demonstrations at law enforcement and intelligence conferences, most of the firms do. Whether it is a utility function like Elasticsearch or a component (code or library) that detects malicious traffic, open source is the go-to source.

The reasons are not far to seek and include:

  • Grabbing open source code is easy
  • Open source software is usually less costly than a proprietary commercial tool
  • Licensing allows some fancy dancing
  • Using what’s readily available and maintained by a magical community of one, two or three people is quick
  • Assuming that the open source code is “safe”; that is, not malicious.

My question was prompted after I read “How US Agencies’ Trust in Untested Software Opened the Door to Hackers.” The write up states:

The federal government conducts only cursory security inspections of the software it buys from private companies for a wide range of activities, from managing databases to operating internal chat applications.

That write up ignores the open source components commercial cyber security firms use. The reason many of the services look and function in a similar manner is due to a reliance on open source methods as well as the nine or 10 work horse algorithms taught in university engineering programs.

What’s the result? A SolarWinds type of challenge. No one knows the scope, no one knows the optimal remediation path, and no one knows how many vulnerabilities exist and are actively being exploited.

Here’s another question, “How many of the whiz kids working in US government agencies communicate the exact process for selecting, vetting, and implementing open source components directly (via 18f type projects) or from vendors of proprietary cyber security software?”

Stephen E Arnold, December 21, 2020

Fess Up: Elasticsearch Is a Threat to Proprietary Search and Retrieval

December 1, 2020

We have been poking around the world of Elasticsearch-based information retrieval systems. There are some interesting plays; that is, entrepreneurs use Elasticsearch (Shay Banon’s open source system) as a platform.

Fess provides Elasticsearch for personal use, although one can employ the system for an organization. The system is:

Fess is Elasticsearch-based search server, but knowledge/experience about Elasticsearch is NOT needed because of All-in-One Enterprise Search Server. Fess provides Administration GUI to configure the system on your browser. Fess also contains a crawler, which can crawl documents on Web/File System/DB and support many file formats, such as MS Office, pdf and zip.

Fess became available in 2019. The CEO of the N2SM, Inc. company is Masaharu Manabe. Demonstrations and links to the code are available at this link. A fee-based version of the software is provided under the name N2 Search. More information about the for fee version is here. A discussion forum is available at this link.

Observation: The Elasticsearch ecosystem is providing alternatives to the proprietary search systems. Beyond Search thinks that some vendors of proprietary search software are likely to be see Elasticsearch as digital kudzu. Good news or bad news for the Coveos, Fabasofts, and Microsoft Fast type folks? That’s a question some of these types of vendors stakeholders may be asking as they beat the bushes for deals in customer service, chatbots, business intelligence, and smart software services.

Stephen E Arnold, December 1, 2020

Elastic: The Add Value to Open Source Outfit Bounces Along

November 25, 2020

Elastic Adds New Features to Enterprise Search, Observability, and Security Solutions

Search and data-management firm Elastic has some new features to crow about. BusinessWire posts “Elastic Announces Innovations Across its Solutions to Optimize Search and Enhance Performance and Monitoring Capabilities.” One new tool is Kibana Lens, a visual data analysis tool with a drag-and-drop interface described as intuitive. There is also a beta launch of the searchable snapshots, an efficient way to manage data storage tiers with searchable snapshots. The press release tells us:

“New expanded Elastic Observability features, including user experience monitoring and synthetics, give developers new tools to test, measure, and optimize end-user website experiences. The launch of a new dedicated User Experience app in Kibana provides Elastic customers with an enhanced view and understanding of how end users experience their websites. In addition, Elastic customers can use the new user experience monitoring feature to review Core Web Vitals, helping website developers interpret digital experience signals. Elastic users can also leverage a dev preview release of synthetic monitoring in Elastic Uptime to simulate complex user flows, measure performance, and optimize new interaction paths without impact to a website’s end users. The combination of these two new observability features gives Elastic customers a deeper view of their customers’ digital experience before and after a site update is deployed.”

See the write-up for its list of specific updates and features to Elastic’s Enterprise Search, Observability, Security, Stack, and Cloud products. Built around open source software, the company prides itself on its user-friendly products that have been adopted by major organizations around the world, from Cisco to Verizon. Elastic began as Elasticsearch Inc. in 2012, simplified its name in 2015, and went public in 2018. The company is based in Mountain View, California, and maintains offices around the world.

Cynthia Murrell, November 25, 2020

Court Case Hunger? Judyrecords Is Available

November 24, 2020

Unable to pay the fee for LexisNexis-type commercial search systems? You are not alone. If you want information from court records, navigate to Judyrecords. Within the last couple of months, the system has added more than 35 million cases. Aren’t these data available for free elsewhere? Sure, if you like going through hoops like verification procedures. Judyrecords lets a user plug in the names of entities and view results. I ran one of my go to queries: “Palantir IBM.” Here are the results:

image

This may not be important to you, but for those who have to wade through for fee legal search systems, Judyrecords is helpful. But for how long? Yes, that is a good question. For now, however, give it a whirl. Keep in mind that US court systems without online technology or special arrangements for document access prevent the system from being comprehensive. Lawyers enjoy results which must be checked by billable professionals, however.

Stephen E Arnold, November 23, 2020

Open Source Kumbaya in 2020: Pay Me for Support

November 16, 2020

I read “No, Open Source Does Not Mean Includes Free Support.” The write up illustrates one small change in the open source community in the last five or six years. With more and more organizations using open source software as the engine for their “platform” or “system”, individuals who create open source software are shifting. For example, when we worked on the Lucene Revolution conference years ago, there was a lot of talk about the community, the FOSS spirit, the desire to break free of the chains proprietary software vendors locked to licensees, etc.

Compare that kumbaya approach to this statement in the write up:

Don’t get me wrong. I’m happy to help. Selling support is what keeps the lights on here (did I mention the cost of running a web server?). But coming to me under false pretense and/or expecting that I must provide free service on top of a software I gave away without charge is not going to win you any favors. It stops being free, when it starts costing me! My time is valuable. If you want a piece of it, I want money in return. Period.

This is an excellent point. In my own experience, we know that some high profile products would not exist without open source software. In fact, some vendors do not reveal the extent of their dependence on software which can be downloaded and used without providing so much as an email address, let alone a credit card.

Net net: 2020 may become the year in which open source kumbaya is replaced with a different ethos. Come by here but bring a way to pay.

Stephen E Arnold, November 16, 2020

Microsoft: The Joy of Figuring Out What Code Can Do

October 26, 2020

DarkCyber finds Microsoft in an interesting spot. On one hand, Microsoft wants to be open sourcey. The idea of community created and community supported software provides a useful source of ready-to-microwave code nuggets, hints about whom to hire, and an opportunity to reduce the maintenance cost of certain components.

On the other hand, monitoring what’s on GitHub and, more importantly, how code can be used is a sticky wicket.

“RIAA Blitz Takes Down 18 GitHub Projects Used for Downloading YouTube Videos” explains:

Microsoft-owned GitHub has removed today 18 projects from its code-hosting portal following a legal request filed by the Recording Industry Association of America (RIAA)….In a letter sent to GitHub, RIAA argued that the “clear purpose of this source code [the youtube-dl library]” was to “circumvent the technological protection measures used by authorized streaming services such as YouTube” and to allow users to “reproduce and distribute music videos and sound recordings […] without authorization.”

The issue is likely to be a thorny one. Code can be used for many things:

  • To perform a function
  • A way to learn how to do a task
  • Create software unrelated to the GitHub offering.

Microsoft has removed the “offending” software. But the problem could become the seed of a giant junk maple in the main Redmond campus green space. The article makes this point, and it is an important one:

RIAA isn’t alleging the library infringed on its rights, but that the library is illegal in itself.

Just as Microsoft wants to get open sourcey and more social, it finds itself in an interesting spot. Who or what will fertilize and water this tiny take down seed? Exactly what can code do? Exactly to what purposes can code be put? What about software which includes code which can do something a third-party defines as illegal? So many questions for the JEDI knights.

Stephen E Arnold, October 26, 2020

Open Source: A New Slogan Emerges. No Poster Art Yet

October 23, 2020

I read “Huawei’s Open Source Innovation Inspired by Of All, By All, for All.” Interesting. Microsoft is interested in open source. Amazon is semi interested in open source. Google is probably still interested in open source unless the team working on open source lost interest. But Huawei? Huawei is interested in open source. The write up reports:

Huawei has acknowledged the importance of open source and the role it plays in accelerating innovation within the software industry, stating that ecosystems such as openEuler, openGauss, openLooKeng, and MindSpore have created an ecosystem of open source basic software projects….The openEuler, openGauss, openLooKeng and MindSpore open source communities are all ‘led’ by Huawei as the company seeks to lay the groundwork for full-stack hardware and software collaboration.

Does Huawei’s support of open source fit into the strategic plan for Chinese technology?

The article provides a partial answer:

Huawei Cloud & AI Open Source business general manager Du Junping says that open source enables organizations to create innovation and value in an environment that is ‘open, fair, transparent, and secure’. Huawei says it is inspired by the mindset of fostering a sustainable, open source basic software ecosystem ‘Of All, By All, For All’.

Catchy: Of all, by all, for all. Very egalitarian and kumbaya-ish. Is it similar to “Smash the gang of four” or “Have fewer children, raise more pigs”? No, of course not.

Stephen E Arnold, October 23, 2020

Comparison of Elasticsearch, Solr, and Sphinx

October 8, 2020

Search and retrieval underpins most policeware and intelware systems. Open source search software has made life more challenging for vendors of proprietary enterprise search solutions. There are versions of an “in depth” enterprise search analysis like this available for thousands of dollars from marketers like https://www.adroitmarketresearch.com sporting this title:

Enterprise Search Market Demand Analysis and Projected huge Growth by 2025| IBM Corp, Coveo Corp., Polyspot & Sinequa Inc., Expert System Inc., HP Autonomy, Lucidworks, Esker Software Corp., Dassault Systemes Inc., Perceptive Software Inc., and Marklogic Inc.

Notice that none of the search vendors in “Elasticsearch vs. Solr vs. Sphinx: Best Open Source Search Platform Comparison” appears in the Adroit Market Research report. That’s important for one reason: Open source search has driven vendors of proprietary systems into a corner. What’s even more intriguing is that some vendors of enterprise search like Attivio and IBM Corp. use open source search technology but take pains to avoid revealing the plumbing under the house trailer.

The comparison is, for now, available without charge online, courtesy of Greenice. This firm, based in Ukraine, is what I would describe as a DevOps consulting and services company. It’s a mash up of advisory, coding, and technical deliverables.

The comparison contains some useful information; for example:

  • Inclusion of examples of the search systems’ visualization capabilities
  • Examples of organizations using each of the three systems compared
  • Presentation of the analyst’s perception of strengths and weaknesses of each system
  • References to machine learning in the context of the three systems.

What caught my attention is the disconnect between the expensive and somewhat over enthusiastic for fee study about search and this free analysis.

Many of the problems in search are a result of what may be described as “over enthusiastic marketing.” This approach to jazzing up what can be accomplished by information retrieval technology has resulted in at least one jail sentence for an enterprise search entrepreneur and may be followed by jail time for other companies’ executives who practice razzmataz sales techniques.

The principal value of the free comparison is that it does a good job of walking through basic information without the Madison Avenue hucksterism. Net net: A free write up with some helpful information.

Stephen E Arnold, October 8, 2020

Next Page »

  • Archives

  • Recent Posts

  • Meta