Open Source: Does It Mean What You Think It Means?
January 15, 2021
I spotted an article on Newswire called “Tech Giant Technology Is Open Source for the Pandemic, So Why Does It Feel So Closed?” The awkward title intrigued me. Open means, according to Dictionary.com:
not closed or barred at the time, as a doorway by a door, a window by a sash, or a gateway by a gate:to leave the windows open at night.
(of a door, gate, window sash, or the like) set so as to permit passage through the opening it can be used to close.
Pretty obvious. But open appears to mean closed. The “source” refers to software I assumed.
The write up sets me straight:
“The term ‘open source’ is being applied to the final design of an instrument – and I’m pleased to say there has been a willingness during the pandemic to share these final designs – but the design process itself also needs to be open, something it isn’t now,” explains physics researcher Dr Julian Stirling.
Okay, the “design process” has to be available. To get more insight into this open is closed issue, navigate to the original technical paper at this link. So far the paper is open, but as I have learned, open can be closed and often locked up behind a paywall.
Stephen E Arnold, January 15, 2021
Does Open Source Create Open Doors?
December 21, 2020
Here’s an interesting question I asked on a phone call on Sunday, December 20, 2020: “How many cyber security firms rely on open source software?”
Give up?
As far as my research team has been able to determine, no study is available to us to answer the question. I told the team that based on comments made in presentations, at lectures, and in booth demonstrations at law enforcement and intelligence conferences, most of the firms do. Whether it is a utility function like Elasticsearch or a component (code or library) that detects malicious traffic, open source is the go-to source.
The reasons are not far to seek and include:
- Grabbing open source code is easy
- Open source software is usually less costly than a proprietary commercial tool
- Licensing allows some fancy dancing
- Using what’s readily available and maintained by a magical community of one, two or three people is quick
- Assuming that the open source code is “safe”; that is, not malicious.
My question was prompted after I read “How US Agencies’ Trust in Untested Software Opened the Door to Hackers.” The write up states:
The federal government conducts only cursory security inspections of the software it buys from private companies for a wide range of activities, from managing databases to operating internal chat applications.
That write up ignores the open source components commercial cyber security firms use. The reason many of the services look and function in a similar manner is due to a reliance on open source methods as well as the nine or 10 work horse algorithms taught in university engineering programs.
What’s the result? A SolarWinds type of challenge. No one knows the scope, no one knows the optimal remediation path, and no one knows how many vulnerabilities exist and are actively being exploited.
Here’s another question, “How many of the whiz kids working in US government agencies communicate the exact process for selecting, vetting, and implementing open source components directly (via 18f type projects) or from vendors of proprietary cyber security software?”
Stephen E Arnold, December 21, 2020
Fess Up: Elasticsearch Is a Threat to Proprietary Search and Retrieval
December 1, 2020
We have been poking around the world of Elasticsearch-based information retrieval systems. There are some interesting plays; that is, entrepreneurs use Elasticsearch (Shay Banon’s open source system) as a platform.
Fess provides Elasticsearch for personal use, although one can employ the system for an organization. The system is:
Fess is Elasticsearch-based search server, but knowledge/experience about Elasticsearch is NOT needed because of All-in-One Enterprise Search Server. Fess provides Administration GUI to configure the system on your browser. Fess also contains a crawler, which can crawl documents on Web/File System/DB and support many file formats, such as MS Office, pdf and zip.
Fess became available in 2019. The CEO of the N2SM, Inc. company is Masaharu Manabe. Demonstrations and links to the code are available at this link. A fee-based version of the software is provided under the name N2 Search. More information about the for fee version is here. A discussion forum is available at this link.
Observation: The Elasticsearch ecosystem is providing alternatives to the proprietary search systems. Beyond Search thinks that some vendors of proprietary search software are likely to be see Elasticsearch as digital kudzu. Good news or bad news for the Coveos, Fabasofts, and Microsoft Fast type folks? That’s a question some of these types of vendors stakeholders may be asking as they beat the bushes for deals in customer service, chatbots, business intelligence, and smart software services.
Stephen E Arnold, December 1, 2020
Elastic: The Add Value to Open Source Outfit Bounces Along
November 25, 2020
Elastic Adds New Features to Enterprise Search, Observability, and Security Solutions
Search and data-management firm Elastic has some new features to crow about. BusinessWire posts “Elastic Announces Innovations Across its Solutions to Optimize Search and Enhance Performance and Monitoring Capabilities.” One new tool is Kibana Lens, a visual data analysis tool with a drag-and-drop interface described as intuitive. There is also a beta launch of the searchable snapshots, an efficient way to manage data storage tiers with searchable snapshots. The press release tells us:
“New expanded Elastic Observability features, including user experience monitoring and synthetics, give developers new tools to test, measure, and optimize end-user website experiences. The launch of a new dedicated User Experience app in Kibana provides Elastic customers with an enhanced view and understanding of how end users experience their websites. In addition, Elastic customers can use the new user experience monitoring feature to review Core Web Vitals, helping website developers interpret digital experience signals. Elastic users can also leverage a dev preview release of synthetic monitoring in Elastic Uptime to simulate complex user flows, measure performance, and optimize new interaction paths without impact to a website’s end users. The combination of these two new observability features gives Elastic customers a deeper view of their customers’ digital experience before and after a site update is deployed.”
See the write-up for its list of specific updates and features to Elastic’s Enterprise Search, Observability, Security, Stack, and Cloud products. Built around open source software, the company prides itself on its user-friendly products that have been adopted by major organizations around the world, from Cisco to Verizon. Elastic began as Elasticsearch Inc. in 2012, simplified its name in 2015, and went public in 2018. The company is based in Mountain View, California, and maintains offices around the world.
Cynthia Murrell, November 25, 2020
Court Case Hunger? Judyrecords Is Available
November 24, 2020
Unable to pay the fee for LexisNexis-type commercial search systems? You are not alone. If you want information from court records, navigate to Judyrecords. Within the last couple of months, the system has added more than 35 million cases. Aren’t these data available for free elsewhere? Sure, if you like going through hoops like verification procedures. Judyrecords lets a user plug in the names of entities and view results. I ran one of my go to queries: “Palantir IBM.” Here are the results:
This may not be important to you, but for those who have to wade through for fee legal search systems, Judyrecords is helpful. But for how long? Yes, that is a good question. For now, however, give it a whirl. Keep in mind that US court systems without online technology or special arrangements for document access prevent the system from being comprehensive. Lawyers enjoy results which must be checked by billable professionals, however.
Stephen E Arnold, November 23, 2020
Open Source Kumbaya in 2020: Pay Me for Support
November 16, 2020
I read “No, Open Source Does Not Mean Includes Free Support.” The write up illustrates one small change in the open source community in the last five or six years. With more and more organizations using open source software as the engine for their “platform” or “system”, individuals who create open source software are shifting. For example, when we worked on the Lucene Revolution conference years ago, there was a lot of talk about the community, the FOSS spirit, the desire to break free of the chains proprietary software vendors locked to licensees, etc.
Compare that kumbaya approach to this statement in the write up:
Don’t get me wrong. I’m happy to help. Selling support is what keeps the lights on here (did I mention the cost of running a web server?). But coming to me under false pretense and/or expecting that I must provide free service on top of a software I gave away without charge is not going to win you any favors. It stops being free, when it starts costing me! My time is valuable. If you want a piece of it, I want money in return. Period.
This is an excellent point. In my own experience, we know that some high profile products would not exist without open source software. In fact, some vendors do not reveal the extent of their dependence on software which can be downloaded and used without providing so much as an email address, let alone a credit card.
Net net: 2020 may become the year in which open source kumbaya is replaced with a different ethos. Come by here but bring a way to pay.
Stephen E Arnold, November 16, 2020
Microsoft: The Joy of Figuring Out What Code Can Do
October 26, 2020
DarkCyber finds Microsoft in an interesting spot. On one hand, Microsoft wants to be open sourcey. The idea of community created and community supported software provides a useful source of ready-to-microwave code nuggets, hints about whom to hire, and an opportunity to reduce the maintenance cost of certain components.
On the other hand, monitoring what’s on GitHub and, more importantly, how code can be used is a sticky wicket.
“RIAA Blitz Takes Down 18 GitHub Projects Used for Downloading YouTube Videos” explains:
Microsoft-owned GitHub has removed today 18 projects from its code-hosting portal following a legal request filed by the Recording Industry Association of America (RIAA)….In a letter sent to GitHub, RIAA argued that the “clear purpose of this source code [the youtube-dl library]” was to “circumvent the technological protection measures used by authorized streaming services such as YouTube” and to allow users to “reproduce and distribute music videos and sound recordings […] without authorization.”
The issue is likely to be a thorny one. Code can be used for many things:
- To perform a function
- A way to learn how to do a task
- Create software unrelated to the GitHub offering.
Microsoft has removed the “offending” software. But the problem could become the seed of a giant junk maple in the main Redmond campus green space. The article makes this point, and it is an important one:
RIAA isn’t alleging the library infringed on its rights, but that the library is illegal in itself.
Just as Microsoft wants to get open sourcey and more social, it finds itself in an interesting spot. Who or what will fertilize and water this tiny take down seed? Exactly what can code do? Exactly to what purposes can code be put? What about software which includes code which can do something a third-party defines as illegal? So many questions for the JEDI knights.
Stephen E Arnold, October 26, 2020
Open Source: A New Slogan Emerges. No Poster Art Yet
October 23, 2020
I read “Huawei’s Open Source Innovation Inspired by Of All, By All, for All.” Interesting. Microsoft is interested in open source. Amazon is semi interested in open source. Google is probably still interested in open source unless the team working on open source lost interest. But Huawei? Huawei is interested in open source. The write up reports:
Huawei has acknowledged the importance of open source and the role it plays in accelerating innovation within the software industry, stating that ecosystems such as openEuler, openGauss, openLooKeng, and MindSpore have created an ecosystem of open source basic software projects….The openEuler, openGauss, openLooKeng and MindSpore open source communities are all ‘led’ by Huawei as the company seeks to lay the groundwork for full-stack hardware and software collaboration.
Does Huawei’s support of open source fit into the strategic plan for Chinese technology?
The article provides a partial answer:
Huawei Cloud & AI Open Source business general manager Du Junping says that open source enables organizations to create innovation and value in an environment that is ‘open, fair, transparent, and secure’. Huawei says it is inspired by the mindset of fostering a sustainable, open source basic software ecosystem ‘Of All, By All, For All’.
Catchy: Of all, by all, for all. Very egalitarian and kumbaya-ish. Is it similar to “Smash the gang of four” or “Have fewer children, raise more pigs”? No, of course not.
Stephen E Arnold, October 23, 2020
Comparison of Elasticsearch, Solr, and Sphinx
October 8, 2020
Search and retrieval underpins most policeware and intelware systems. Open source search software has made life more challenging for vendors of proprietary enterprise search solutions. There are versions of an “in depth” enterprise search analysis like this available for thousands of dollars from marketers like https://www.adroitmarketresearch.com sporting this title:
Enterprise Search Market Demand Analysis and Projected huge Growth by 2025| IBM Corp, Coveo Corp., Polyspot & Sinequa Inc., Expert System Inc., HP Autonomy, Lucidworks, Esker Software Corp., Dassault Systemes Inc., Perceptive Software Inc., and Marklogic Inc.
Notice that none of the search vendors in “Elasticsearch vs. Solr vs. Sphinx: Best Open Source Search Platform Comparison” appears in the Adroit Market Research report. That’s important for one reason: Open source search has driven vendors of proprietary systems into a corner. What’s even more intriguing is that some vendors of enterprise search like Attivio and IBM Corp. use open source search technology but take pains to avoid revealing the plumbing under the house trailer.
The comparison is, for now, available without charge online, courtesy of Greenice. This firm, based in Ukraine, is what I would describe as a DevOps consulting and services company. It’s a mash up of advisory, coding, and technical deliverables.
The comparison contains some useful information; for example:
- Inclusion of examples of the search systems’ visualization capabilities
- Examples of organizations using each of the three systems compared
- Presentation of the analyst’s perception of strengths and weaknesses of each system
- References to machine learning in the context of the three systems.
What caught my attention is the disconnect between the expensive and somewhat over enthusiastic for fee study about search and this free analysis.
Many of the problems in search are a result of what may be described as “over enthusiastic marketing.” This approach to jazzing up what can be accomplished by information retrieval technology has resulted in at least one jail sentence for an enterprise search entrepreneur and may be followed by jail time for other companies’ executives who practice razzmataz sales techniques.
The principal value of the free comparison is that it does a good job of walking through basic information without the Madison Avenue hucksterism. Net net: A free write up with some helpful information.
Stephen E Arnold, October 8, 2020
Google and Its User Privacy: Happy Hunting
October 4, 2020
DarkCyber spotted an open source intelligence tool called GHunt. By the time an open source software becomes publicly available, DarkCyber believes that hardened systems and methods are integrated into specialized policeware and intelware systems. If you want to try to learn more about a particular Google email user, for instance, you may want to take a look at GHunt. There are screenshots and basic information available on Github. Google appears to be taking steps to address some of the “features” which the GHunt software taps. Some interesting open source software becomes available and then disappears; for example, DARPA Memex tools have evidenced this type of behavior. If you want this tool, DarkCyber suggests you move along in a sprightly manner.
Stephen E Arnold, October 4, 2020
-
- Subscribe to Beyond Search
Feature archive
News archive
-
