Open Source Software: The Community Model in 2021
January 25, 2021
I read “Why I Wouldn’t Invest in Open-Source Companies, Even Though I Ran One.” I became interested in open source search when I was assembling the first of three editions of Enterprise Search Report in the early 2000s. I debated whether to include Compass Search, the precursor to Shay Branon’s Elasticsearch reprise. Over the years, I have kept my eye on open source search and retrieval. I prepared a report for an the outfit IDC, which happily published sections of the document and offering my write ups for $3,000 on Amazon. Too bad IDC had no agreement with me, managers who made Daffy Duck look like a model for MBAs, and a keen desire to find a buyer. Ah, the book still resides on one of my back of drives, and it contains a run down of where open source was getting traction. I wrote the report in 2011 before getting the shaft-a-rama from a mid tier consulting firm. Great experience!
The report included a few nuggets which in 2011 not many experts in enterprise search recognized; for instance:
- Large companies were early and enthusiastic adopters of open source search; for example Lucene. Why? Reduce costs and get out of the crazy environment which put Fast Search & Transfer-type executives in prison for violating some rules and regulations. The phrase I heard in some of my interviews was, “We want to get out of the proprietary software handcuffs.” Plus big outfits had plenty of information technology resources to throw at balky open source software.
- Developers saw open source in general and contributing to open source information retrieval projects as a really super duper way to get hired. For example, IBM — an early enthusiast for a search system which mostly worked — used the committers as feedstock. The practice became popular among other outfits as well.
- Venture outfits stuffed with oh-so-technical MBAs realized that consulting services could be wrapped around free software. Sure, there were legal niceties in the open source licenses, but these were not a big deal when Silicon Valley super lawyers were just a text message away.
There were other findings as well, including the initiatives underway to embed open source search, content processing, and related functions into commercial products. Attivio (formed by former super star managers from Fast Search & Transfer), Lucid Works, IBM, and other bright lights adopted open source software to [a] reduce costs, [b] eliminate the R&D required to implement certain new features, and [c] develop expensive, proprietary components, training, and services.
Enterprise Search: Flexible and Stretchy. Er, No.
January 21, 2021
Enterprise search, the utility service, thrills users and information technology professionals alike. There are quite a few search and retrieval vendors chasing revenue. Frankly I have given up trying to keep track of outfits like Luigi’s Box, Yext (yes, enterprise search!), and quite a few repackagers of Lucene; e.g., IBM, Attivio, Voyager Search, and more. There are some proprietary outfits as well.
Then there is the Compass Search sibling Elastic and its Elasticsearch. Open source search makes a great deal of sense to:
- Companies wanting a no cost or low cost way to provide search and retrieval-type functionality to an application
- Penny pinchers who want “the community” to fix bugs so that cash is freed up to lease fancy cars, receive bonuses, and focus on more important software features which can be offered for a fee and a license handcuff
- Competitors who want to provide a familiar environment to those with cash to spend and wave the magic wand of open source in front of young believers who think proprietary software is a crime against humanity.
The history of Elasticsearch and Amazon reaches back to the era when Lucid Works (né Lucid Imagination) lost some staff to Amazon’s Burlingame, California, office. That was the bell which sounded when the Bezos bulldozer decided A9 was not enough. Sure, A9 works but the folks from the Lucene/Solr outfit would map the route from A9 to a more open, folksy world of open source search.
The open source version of Lucene was the beating heart of Elastic, the now public company.
Then Amazon does what Amazon does: The company shifted the bulldozer into gear and went for open source search developers who could seamlessly (sort of) move into the newly blazed path to AWS. Once inside, the fruits of the thousand plus services, features, and functions were just a click away. Policeware vendors, start ups, and some big outfits followed the Bezos bulldozer. The updated IBM slogan reads, “Nobody gets fired for buying AWS.”
Elastic was upset.
“Amazon: NOT OK – Why We Had to Change Elastic Licensing” picks up this story and explains where Elastic fits into the world crafted by the Bezos bulldozer.
The write up explains:
Our license change is aimed at preventing companies from taking our Elasticsearch and Kibana products and providing them directly as a service without collaborating with us.
Elastic’s essay notes:
We think that Amazon’s behavior is inconsistent with the norms and values that are especially important in the open source ecosystem. Our hope is to take our presence in the market and use it to stand up to this now so others don’t face these same issues in the future.
The essay concludes:
I believe in the core values of the Open Source Community: transparency, collaboration, openness. Building great products to the benefit of users across the world. Amazing things have been built and will continue to be built using Elasticsearch and Kibana. And to be clear, this change most likely has zero effect on you, our users. And no effect on our customers that engage with us either in cloud or on premises.
Several observations:
- Commercial behemoths like Amazon use open source the way my neighbor burns firewood, old carpets, and newspapers. The goal is to optimize available cash.
- Amazon’s move into Elastic’s territory began more than five years ago. Amazon does kill off loser products like health and food delivery but it keeps others in tall cotton when it pays off.
- Those completing [a] Amazon certification, [b] partner indoctrination, or [c] inputs from free or low cost Amazon training arrive ready to do the search thing Amazon’s way.
Net net: Beyond Search understands Elastic’s anguish and actions. Perhaps the license shift and the assumptions about open source are unlikely to stand up to the Bezos bulldozer? Open source Elasticsearch is a bargain. It may be tough to compete with free plus discounts for AWS goodies and other Amazon benefits. Legal or illegal, fair or unfair, open source or closed source — the bulldozer grinds forward.
Stephen E Arnold, January 21, 2021
KPMG: Ignoring the HR Block Case Example or That Will Not Happen at the Exceptional KPMG
January 19, 2021
Here’s a fact of life at allegedly blue chip consulting and service firms. Miss those billability goal, and you are invited to find your future elsewhere.
I read “KPMG’s Marisa Ferrara Boston embraces Auditing Disruption with Watson.” My immediate reaction was id the capable, dutiful Marisa Ferrara Boston overlook this article in Beyond Search: “Watson and Block: Tax Preparation and Watson.” Probably. Business analysis from rural Kentucky is not on the KPMG list of suggested readings.
The point of my write up was in early 2017:
The idea is that H&R Block paid cash money to IBM to integrate Watson into the H&R Block proprietary tax preparation system.
The problem, based on information available to me, the Block Watson service added complexity to the tax workflow.
Oh, oh.
Here’s what KPMG has in mind:
KPMG has partnered with IBM to integrate Watson Discovery and Watson Machine Learning into the auditing workflow. KPMG uses Watson as a backbone to a question-answering pipeline for auditors and risk analysts, enabling KPMG audit professionals to better review, classify, and search across documents to extract important attribute values.
Interesting idea. Replace billable humans with super smart, reliable, fast IBM software.
What could go wrong?
If the Block IBM deal went nowhere, the resistance came from the tax professionals the system was supposed to help. Block and IBM parted company.
At KPMG, the litmus test will be billability. Unless the smart software generates more billable hours (regardless of how the bean counters fiddle the calculations), the KPMG IBM deal is likely to be found wanting. Nothing creates more waves in a blue chip professional services firm than a partner responsible for a number who misses his/her bonus. Nothing.
This quote from the IBM blog misses the point for a big time consulting firm. IBM writes:
“I feel really lucky to be able to be in a position where I’m still in the fight to be able to help push these things along,” says Marisa. But deployment is only half the battle. When it comes to maintaining innovation in automation over time, “it’s never over,” she says. “These AIs are living. They need to be nurtured in an appropriate environment. They’re not just something that you create and consider the job to be done. If so, you have failed, and probably in a very expensive way.”
Notice that employee revenue is not mentioned. Cost control is not mentioned. The partner bonuses are not mentioned. The ire of an unhappy KPMG client who is “surprised” is not mentioned. What about the managing partner who learns that a baby Enron or Autonomy has been birthed by the energetic Watson? Exciting? Yep. Very.
Perhaps some KPMG wizards who will find themselves working at HR Block will be able to ask their new colleagues, “What did you think of that IBM Watson integration?”
Stephen E Arnold, January 19, 2021
Selling Technology in a Tough Market Roasting in Solar Waves
January 13, 2021
I read a post on Hacker’s News. You may be able to locate it at this link: http://solarleaks.net/. I don’t know if this is a scam or the answer to the question “Where’s the beef?” The message states:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Happy new year! Welcome to solarleaks.net (mirror: 5bpasg2kotxllmzsv6swwydbojnfuvfb7d6363pwe5wrzhjyn2ptvdqd.onion) We are putting data found during our recent adventure for sale. [Microsoft Windows (partial) source code and various Microsoft repositories] price: 600,000 USD data: msft.tgz.enc (2.6G) link: https://mega.nz/file/1ehgSSpD#nrtzQwh-qyCaUHBXo2qQ1dNbWiyVHCvg8J0As8VjrX0
The Solar Leaks’ post then provides information about the cost of the MSFT, Cisco, and FireEye, et al software. Prices begin at $50,000 for some alleged FireEye goodies and soar to $600,000 for the Microsoft crown jewels.
What’s important, however, is the post-SolarWinds’ misstep marketing environment. Sales professionals of products that provide enhanced cyber security, threat alerts, and the assorted jargon enhanced assertions have to close deals.
Just in time is a helpful write up from Entrepreneur Magazine called “8 Psychological Tricks to Increase Conversion Rates for SaaS Startups.” That’s on time and on target.
I am tempted to summarize the ideas with references to Machiavelli, Al Capone, and high school lovers promising to be together forever. But I will not. I will highlight three of the ideas, and you can pony up some cash to read the full entrepreneurial check list yourself.
Suggestion 1:
Offer fewer choices.
Okay, Amazon, Microsoft, and others offering secure cloud environments, are you listening? Fewer choices. The point of offering choices is to create an opportunity to confuse a customer and allow MBAs with spreadsheet fever to cook up pricing options guaranteed to lead to big surprises when the system is up and running. Cross that threshold and beyond the invoice! Outstanding.
Suggestion 2:
Introduce a third product.
You have to read the article to appreciate the wonderfulness of offering a print subscription, a digital subscription, and a com9bo subscription or an option that forces the “brain to focus on the two closest options.” I am confident that this is backed by an MBA-type book called “Thinking Slow and Slower.”
Suggestion 3:
Increase quantities rather than reduce the price.
Ah, yes, buy five packages of cookies and get an extra 20 percent discount. That’s okay, but I don’t have any place to put extra bags of cookies in my one bedroom trailer parked in Sunrise Acres in Bullet County, Kentucky. More, more, more. Yes, bullet proof. No pun intended.
With cyber security delivered via the cloud in the great SaaS approach, the trick to making sales is to shift from professional sales person to a street hustler offering “original” watches as tourists exit the bus from a tour of the Forbidden City.
What about clarity, factual information, and services which work, well, maybe just mostly work.
Good enough.
Stephen E Arnold, January 13, 2021
Marketing Insight or Marketing Desperation?
January 6, 2021
A couple of weeks ago, I became aware of a shift in techno babble. Here are some examples and their sources:
Fire-and-forget. Shoot a missile and smart software does the rest… when necessary. Source: War News
Hyperedge replacement graph grammars (HRGs). A baffler. Source: Something called NEURIPS
Performative. I think this means go fast or complete a task in a better way. Source: Mashable
Proceleration. The Age of Earthquakes.
Tangential content. The idea is that information does not have to be related; for example, if you write about car polish for a living, including articles about zebras is a good thing. Source: Next Web
Transition from pets to cattle. Moving from the status of a beloved poodle to a single, soon to be eaten bovine. Source: Amazon AWS
Fascinating terminology. Time for digital detox and maybe red tagging. No, I don’t know what these terms means either. I assume that vendors of smart software which can learn without human fiddling knows these terms and many more because of experience intelligence platforms.
Stephen E Arnold, January 6, 2021
SolarWinds Are Gusting and Blowing Hard
January 5, 2021
Many pundits have reacted to the New York Times’ story “As Understanding of Russian Hacking Grows, So Does Alarm.” Work through those analyses. What’s missing? Quite a lot, but in this short blog post I want to address one issue that has mostly ignored.
At one time, there was a list on the SolarWinds’ Web site of the outfits which had been compromised. That list disappeared. I posted “Sun Spotting in the Solar Wind” on December 23, 2020. In that post, I reported three outfits which had been allegedly compromised by the SolarWinds’ misstep (and some of the information I used as a source remains online):
City of Barrie (Canada)
Newton Public Schools (US)
Regina Public Schools (Canada).
The question is, “Why are outfits like a municipality known as part of the Greater Golden Horseshoe, Newton’s public schools, and the Regina public schools? (I’ve been to Regina in the winter. Unforgettable is it.)
My research team and I discussed the alleged exploits taking up residence in these organizations; that is, allegedly, of course, of course.
Here’s what my team offered:
- A launch pad for secondary attacks. The idea is that the original compromise was like a rat carrying fleas infected with the bubonic plague (arguably more problematic than the Rona)
- A mechanism for placing malicious code on the computing devices of administrators, instructors, and students. As these individuals thumb typed away, these high trust individuals were infecting others in their social circle. If the infections were activated, downloads of tertiary malware could take place.
- Institutions like these would connect to other networks. Malware could be placed in server nodes serving other institutions; for example, big outfits like Rogers Communications, a government ministry or two, and possibly the cloud customers of the beloved Rogers as well as BCE (Bell Canada’s parent) and Telus.
The odd ducks in the list of compromised organization, just might not be so odd after all.
That’s the problem, isn’t it? No one knows exactly when the misstep took place, what primary and downstream actions were triggered, and where subsequent rats with fleas infected with bubonic plague have go to.
Net net: It’s great to read so many words about a misstep and not have signals that the issue is understood, not even by the Gray Lady herself.
Stephen E Arnold, January 6, 2020
What Has Amazon Caught? The Google Syndrome?
January 4, 2021
No, Google has not launched a virus infecting Amazon Web Services (although AWS has people capable of designing such a virus). Instead Google has infected AWS with its “shiny object syndrome” says Last Week In AWS in the article: “The Google Disease Afflicting AWS.”
Last Week In AWS refers to Google’s mind state as the “launch new service” mentality, where Google perpetually launches new projects, works on them for the headlines, then abandons them. Apparently it is the only way to advance in Google’s hierarchy. AWS, on the other hand, is starting a “launch new service” mentality when those new services should be features on preexisting projects.
AWS should be helping their customers, instead they are creating new and competing services. This mentality has harmed Google and will not do any favors for AWS. The majority of people agree:
“For a company that believes its team should Be Right a Lot, this is pretty clearly the wrong path—according to customers, analysts, random passersby, and employees with a penchant for honesty. It’s not good for anyone, and I firmly believe that you don’t Earn Trust from your customers by making even the most diligent cloud-watchers feel that the cloud is accelerating away from their ability to understand and contextualize what you’ve built.”
The article wonders why no one has pointed this out to AWS top brass. We do not know if anyone actually has, but common sense states that an employee must have reported the top heavy and poor customer service model. This common sense employee was probably reprimanded or fired for speaking the truth. Remember the child who said the emperor was naked?
Whitney Grace, January 4, 2021
SolarWinds: One Interesting Message
December 28, 2020
I read “Wave of Cyberattacks Exposes the Powerlessness of IT Security Chiefs.” With all the hoohah about cyber superiority from government officials and commercial enterprises, one troubling fact is clear: If the advanced systems could not detect the attack nor could top secret security systems monitoring possible bad actors, the defensive and alerting methods are broken. The write up points out security focuses on a wide spread weak link:
Splunk, a U.S. company, publishes an annual list of “10 things that keeps CISOs up at night,” and this year’s includes the expanded “attack surface” created by the growing use of the internet of things (web-connected devices) and the growing use of cloud computing, “malicious insiders” and the “alert fatigue” resulting from so many layers of data security inside a big organization. But apart from that, Splunk notes the lack of money to ensure data security. “CISOs continue to face challenges in securing substantial budgets, largely because they have difficulty forecasting threats and achieving measurable results from security investments…
He said 66% of CISOs surveyed said they didn’t have adequate staff. Others cited increasingly onerous regulations and their lack of access to top management.
Something in the cyber security establishment enables breaches.
Stephen E Arnold, December 28, 2020
Satellites Are Upgraded Peeping Toms
December 28, 2020
Satellites have had powerful cameras for decades. Camera technology for satellites has advanced from being able to read credit card numbers in the early 2000s to peering inside buildings. Futurism shares details about the new (possible) invasion of privacy in: “A New Satellite Can Peer Inside Some Buildings, Day Or Night.”
Capella Space launched a new type of satellite with a new state of the art camera capable of taking clear radar images with precise resolution. It even has the ability to take photos inside buildings, such as airplane hangers. Capella Space assures people that while their satellite does take powerful, high resolution photos it can only “see” through lightweight structures. The new satellite camera cannot penetrate dense buildings, like residential houses and high rises.
The new satellite can also take pictures from space of Earth on either its daytime or nighttime side. Capella also released a new photo imaging platform that allows governments or private customers to request images of anything in the world. Most satellites orbiting the Earth use optical image sensors, which make it hard to take photos when its cloudy. Capella’s new system uses synthetic aperture radar that can peer through cloud cover and night skies.
The resolution for the SAR images is extraordinary:
“Another innovation, he says, is the resolution at which Capella’s satellites can collect imagery. Each pixel in one of the satellite’s images represents a 50-centimeter-by-50-centimeter square, while other SAR satellites on the market can only get down to around five meters. When it comes to actually discerning what you’re looking at from space, that makes a huge difference.
Cityscapes are particularly intriguing. Skyscrapers poke out of the Earth like ghostly, angular mushrooms — and, if you look carefully, you notice that you can see straight through some of them, though the company clarified that this is a visual distortion rather than truly seeing through the structures.”
Capella’s new satellite has a variety of uses. Governments can use it to track enemy armies, while scientists can use it to monitor fragile ecosystems like the Amazon rainforest. Capella has assured the public that its new satellite cannot spy into dense buildings, but if the technology improves maybe it is a possibility? Hopefully bad actors will not use Capella’s new satellite.
Whitney Grace, December 28, 2020
Arthur.ai Designed To Ensure Accuracy In Machine Learning Models
December 28, 2020
Most tech companies are investing their capital in designing machine learning models, but Arthur.ai decided to do something different. TechCrunch reveals Arthur.ai’s innovation in “Arthur.ai Snags $15M Series A To Grow Machine Learning Monitoring Tool.” The Arthur.ai is designed to ensure that machine learning models retain their precise accuracy over time.
Despite being fine tuned algorithms, machine learning AI needs maintenance like any other technology. Index Ventures saw the necessity of such a tool and lead a Series A round of funding with investments from Homebrew, AME Ventures, Workbench, Acrew, and Plexo Capital:
“Investor Mike Volpi from Index certainly sees the value proposition of this company. “One of the most critical aspects of the AI stack is in the area of performance monitoring and risk mitigation. Simply put, is the AI system behaving like it’s supposed to?” he wrote in a blog post announcing the funding.
Arthur.ai has doubled its employees since it was a startup and founder and CEO Adam Wenchel wants to continue expansion. AWS released a similar tool called SageMaker Clarity, but Wenchel views the potential competition as affirmation. If there are products that provide the same service that means there is a market for it. He is also not worried about larger cloud companies, because Arthur.ai will focus entirely on its maintenance tools while the larger ones are strained.
Whitney Grace, December 28, 2020
-
- Subscribe to Beyond Search
Feature archive
News archive
-
