GitHub Identifies a Sooty Pot and Does Not Offer a Fix
January 9, 2025
This is an official dinobaby post. No smart software involved in this blog post.
GitLab’s Sabrina Farmer is a sharp thinking person. Her “Three Software Development Challenges Slowing AI Progress” articulates an issue often ignored or just unknown. Specifically, according to her:
AI is becoming an increasingly critical component in software development. However, as is the case when implementing any new tool, there are potential growing pains that may make the transition to AI-powered software development more challenging.
Ms. Farmer is being kind and polite. I think she is suggesting that the nest with the AI eggs from the fund-raising golden goose has become untidy. Perhaps, I should use the word “unseemly”?
She points out three challenges which I interpret as the equivalent of one of those unsolved math problems like cracking the Riemann Hypothesis or the Poincaré Conjecture. These are:
- AI training. Yeah, marketers write about smart software. But a relatively small number of people fiddle with the knobs and dials on the training methods and the rat’s nests of computational layers that make life easy for an eighth grader writing an essay about Washington’s alleged crossing of the Delaware River whilst standing up in a boat rowed by hearty, cheerful lads. Big demand, lots of pretenders, and very few 10X coders and thinkers are available. AI Marketers? A surplus because math and physics are hard and art history and social science are somewhat less demanding on today’s thumb typers.
- Tools, lots of tools. Who has time to keep track of every “new” piece of smart software tooling? I gave up as the hyperbole got underway in early 2023. When my team needs to do something specific, they look / hunt for possibilities. Testing is required because smart software often gets things wrong. Some call this “innovation.” I call it evidence of the proliferation of flawed or cute software. One cannot machine titanium with lousy tools.
- Management measurements. Give me a break, Ms. Farmer. Managers are often evidence of the Peter Principle, an accountant, or a lawyer. How can one measure what one does not use, understand, or creates? Those chasing smart software are not making spindles for a wooden staircase. The task of creating smart software that has a shot at producing money is neither art nor science. It is a continuous process of seeing what works, fiddling, and fumbling. You want to measure this? Good luck, although blue chip consultants will gladly create a slide deck to show you the ropes and then churn out a spectacular invoice for professional services.
One question: Is GitLab part of the problem or part of the solution?
Stephen E Arnold, January 9, 2025
AI Outfit Pitches Anti Human Message
January 9, 2025
AI startup Artisan thought it could capture attention by telling companies to get rid of human workers and use its software instead. It was right. Gizmodo reports, “AI Firm’s ‘Stop Hiring Humans’ Billboard Campaign Sparks Outrage.” The firm plastered its provocative messaging across San Francisco. Writer Lucas Ropek reports:
“The company, which is backed by startup accelerator Y-Combinator, sells what it calls ‘AI Employees’ or ‘Artisans.’ What the company actually sells is software designed to assist with customer service and sales workflow. The company appears to have done an internal pow-wow and decided that the most effective way to promote its relatively mundane product was to fund an ad campaign heralding the end of the human age. Writing about the ad campaign, local outlet SFGate notes that the posters—which are strewn all over the city—include plugs like the following:
‘Artisans won’t complain about work-life balance’
‘Artisan’s Zoom cameras will never ‘not be working’ today.’
‘Hire Artisans, not humans.’
‘The era of AI employees is here.'”
The write-up points to an interview with SFGate in which CEO Jaspar Carmichael-Jack states the ad campaign was designed to “draw eyes.” Mission accomplished. (And is it just me, or does that name belong in a pirate movie?) Though Ropek acknowledges his part in drawing those eyes, he also takes this chance to vent about AI and big tech in general. He writes:
“It is Carmichael-Jackson’s admission that his billboards are ‘dystopian’—just like the product he’s selling—that gets to the heart of what is so [messed] up about the whole thing. It’s obvious that Silicon Valley’s code monkeys now embrace a fatalistic bent of history towards the Bladerunner-style hellscape their market imperatives are driving us.”
Like Artisan’s billboards, Ropek pulls no punches. Located in San Francisco, Artisan was launched in 2023. Founders hail from the likes of Stanford, Oxford, Meta, and IBM. Will the firm find a way to make its next outreach even more outrageous?
Cynthia Murrell, January 9, 2025
Be Secure Like a Journalist
January 9, 2025
This is an official dinobaby post.
If you want to be secure like a journalist, Freedom.press has a how-to for you. The write up “The 2025 Journalist’s Digital Security Checklist” provides text combined with a sort of interactive design. For example, if you want to know more about an item on a checklist, just click the plus sign and the recommendations appear.
There are several sections in the document. Each addresses a specific security vector or issue. These are:
- Asses your risk
- Set up your mobile to be “secure”
- Protect your mobile from unwanted access
- Secure your communication channels
- Guard your documents from harm
- Manage your online profile
- Protect your research whilst browsing
- Avoid getting hacked
- Set up secure tip lines.
Most of the suggestions are useful. However, I would strongly recommend that any mobile phone user download this presentation from the December 2024 Chaos Computer Club meeting held after Christmas. There are some other suggestions which may be of interest to journalists, but these regard specific software such as Google’s Chrome browser, Apple’s wonderful iCloud, and Microsoft’s oh-so-secure operating system.
The best way for a journalist to be secure is to be a “ghost.” That implies some type of zero profile identity, burner phones, and other specific operational security methods. These, however, are likely to land a “real” journalist in hot water either with an employer or an outfit like a professional organization. A clever journalist would gain access to a sock puppet control software in order to manage a number of false personas at one time. Plus, there are old chestnuts like certain Dark Web services. Are these types of procedures truly secure?
In my experience, the only secure computing device is one that is unplugged in a locked room. The only secure information is that which one knows and has not written down or shared with anyone. Every time I meet a journalist unaware of specialized tools and services for law enforcement or intelligence professionals I know I can make that person squirm if I describe one of the hundreds of services about which journalists know nothing.
For starters, watch the CCC video. Another tip: Choose the country in which certain information is published with your name identifying you as an author carefully. Very carefully.
Stephen E Arnold, January 9, 2025
FOGINT: The French Method for Communicating with Telegram Works
January 8, 2025
A post from the FOGINT team.
Direct action by French authorities has had a visible impact on Telegram. The FOGINT team noted a report in Arab News which provides some color to this observation. “Surge in Telegram User Data Passed to French Authorities.” The article reports:
Messaging service Telegram passed vastly more data on its users to French authorities in the second half of 2024 following founder Pavel Durov’s arrest in Paris, figures published by the platform showed.
The company said it handed over IP addresses or telephone numbers that Paris asked for in 210 cases in July-September and 673 in October-December.
Prior to the action by French authorities, Telegram had the reputation of ignoring legitimate requests from government authorities in the EU and elsewhere. The company explained that it stood for free speech. However, in April 2024, Telegram blocked Ukrainian government messages from Ukraine to Telegram users in Russia.
According to the article, Telegram explained:
He [Mr. Durov] and his supporters have claimed that most French and European authorities’ requests for user data were simply not being sent to the right department at the company and therefore received no response.
Several observations from the FOGINT team are warranted:
- Direct action by French authorities has been productive. Consequently Telegram has responded.
- Mr. Durov remains under observation by the French government with his legal proceedings moving at the speed of the French bureaucracy; that is, in a methodical manner.
- Mr. Durov’s releasing of user names associated with active investigations has pushed Telegram into a course change for 2025. The company is now emphasizing its crypto currency features, functions, and services.
Consequently Telegram’s technical platform and its ability to take advantage of growing interest in online gambling provide a new challenge to investigators. Its flurry of deals with crypto centric organizations in Eastern Europe and Southeast Asia pose a new challenge to investigators. Tracking financial transactions facilitated by Telegram’s global decentralized and distributed design becomes more costly and time intensive. Telegram’s smart automation allows transactions to move from crypto currency wallet to crypto currency wallet under different identities quickly. The likely use case for Telegram’s crypto services may be money laundering.
Stephen E Arnold, January 8, 2025
FOGINT: Russia Reveals How Important Telegram Is to Its Propaganda Program
January 8, 2025
This is an official dinobaby post. No smart software involved in this blog post.
Telegram that messaging service is important to Russia’s European propaganda efforts. Russia suggested that Telegram block messages from the Ukrainian government to Russians using Telegram April 2024. The filtering was big news in Ukraine; in the US, the Telegram action was lost in the cacophony of 24×7 digital information flows. This means that few Americans knew or cared about this Telegram acquiescence to the Kremlin.
A number of news outlets have reported that Telegram is more important to the Putin regime than many realized. Jurist.org reported in “Russia Threatens Retaliation Over Blocking of State Media Telegram Channels Across EU.” The write up states:
Russia’s Ministry of Foreign Affairs said that users attempting to access the Telegram channels of Russian state broadcasters, including RIA Novosti, Izvestia and RT, are being notified of limited access in the EU. Russia characterized the blockage as “political censorship” in violation of international obligations on free information access. Moscow warned that “specialized international organizations should duly evaluate these actions” and demanded a response from UN human rights mechanisms and UNESCO leadership.
The Jurist article adds:
This latest dispute over media access follows a pattern of escalating restrictions between Russia and the EU since Russia launched its full-scale invasion of Ukraine in February 2022. The war prompted the EU to impose massive and unprecedented sanctions against Russia, including restrictions on state media outlets accused of spreading propaganda. In June 2024, Russia blocked access to 81 European media websites from 25 European countries, affecting outlets like France’s Agence France-Presse (AFP), Le Monde, and Liberation. This action came after the EU banned four Russian state media outlets in May 2024. The EU accused the outlets Voice of Europe, RIA Novosti, Izvestia, and Rossiyskaya Gazeta of disseminating propaganda about the war in Ukraine.
The question the article does not address is, “What are the likely retaliatory measures?” Russia has blocked major European news Web sites, including Der Spiegel and El Pais, among others. Mr. Putin’s “threats” have been characterized as verbal assertions, not cyber attacks designed to cripple key EU countries or direct kinetic action against the United Kingdom.
Several observations are warranted:
- Telegram is a big player for Russia’s propaganda machine
- The Kremlin’s grousing makes it clear that some Telegram marketing verbiage is baloney when asserting that the organization operates without compromising “freedom of speech”
- The frantic push by Telegram in the crypto space can be interpreted as part of the Russia-supported effort to undermine the US dollar and get around sanctions imposed on Russia as a consequence of the three-year special operation.
Net net: Telegram warrants close observation in 2025.
Stephen E Arnold, January 8, 2025
Identifying Misinformation: A Task Not Yet Mastered
January 8, 2025
This is an official dinobaby post. No smart software involved in this blog post.
On New Year’s eve the US Department of Treasury issued a news release about Russian interference in the recent US presidential election. Tucked into the document “Treasury Sanctions Entities in Iran and Russia That Attempted to Interfere in the U.S. 2024 Election” was this passage:
GRU-AFFILIATED ENTITY USES ARTIFICIAL INTELLIGENCE TOOLS TO INTERFERE IN THE U.S. 2024 ELECTION
The Moscow-based Center for Geopolitical Expertise (CGE), founded by OFAC-designated [Office of Foreign Asset Control — Editor] Aleksandr Dugin, directs and subsidizes the creation and publication of deepfakes and circulated disinformation about candidates in the U.S. 2024 general election. CGE personnel work directly with a GRU unit that oversees sabotage, political interference operations, and cyberwarfare targeting the West. Since at least 2024, a GRU officer and CGE affiliate directed CGE Director Valery Mikhaylovich Korovin (Korovin) and other CGE personnel to carry out various influence operations targeting the U.S. 2024 presidential election. At the direction of, and with financial support from, the GRU, CGE and its personnel used generative AI tools to quickly create disinformation that would be distributed across a massive network of websites designed to imitate legitimate news outlets to create false corroboration between the stories, as well as to obfuscate their Russian origin. CGE built a server that hosts the generative AI tools and associated AI-created content, in order to avoid foreign web-hosting services that would block their activity. The GRU provided CGE and a network of U.S.-based facilitators with financial support to: build and maintain its AI-support server; maintain a network of at least 100 websites used in its disinformation operations; and contribute to the rent cost of the apartment where the server is housed. Korovin played a key role in coordinating financial support from the GRU to his employees and U.S.-based facilitators. In addition to using generative AI to construct and disseminate disinformation targeting the U.S. electorate in the lead up to the U.S. 2024 general election, CGE also manipulated a video it used to produce baseless accusations concerning a 2024 vice presidential candidate in an effort to sow discord amongst the U.S. electorate. Today, OFAC is designating CGE and Korovin pursuant to E.O. 13848 for having directly or indirectly engaged in, sponsored, concealed, or otherwise been complicit in foreign malign influence in the 2024 U.S. election. Additionally, OFAC is designating CGE pursuant to E.O. 13694, as amended, E.O. 14024, and section 224 of the Countering America’s Adversaries Through Sanctions Act of 2017 (CAATSA) for being owned or controlled by, or having acted or purported to act for or on behalf of, directly or indirectly, the GRU, a person whose property and interests in property are blocked pursuant to E.O. 13694, as amended, E.O. 14024, and section 224 of CAATSA. OFAC is also designating Korovin pursuant to E.O. 14024 for being or having been a leader, official, senior executive officer, or member of the board of directors of CGE, a person whose property and interests in property are blocked pursuant to E.O. 14024.
Several questions arise:
- Was the smart software open source or commercial? What model or models powered the misinformation effort?
- What functions could intermediaries / service providers add to their existing systems to identify and block the actions of an adversary’s operative? (Obviously existing software to identify “fake” content do not work particularly well.)
- What safeguard standards can be used to prevent misuse of smart software? Are safeguard standards possible or too difficult to implement in a “run fast and break things” setting?
- What procedures and specialized software are required to provide security professionals with a reliable early warning system? The fact of this interference illustrates that the much-hyped cyber alert services do not function in a way sufficiently accurate to deal with willful misinformation “factories.”
Stephen E Arnold, January 8, 2025
UK The Register Emits News of Chinese Cyber Excreta
January 8, 2025
This is an official dinobaby post. No smart software involved in this blog post.
I loved this write up from the UK’s The Register online information service: “China’s Cyber Intrusions Took a Sinister Turn in 2024.” The write up gathers together some notable cyber events and links them to the Middle Kingdom. Examples include:
- Router exploits
- Compromising infrastructure of major American cities
- The exfiltration of data from US telephony companies
The write up includes the zippy names cyber security researchers give these exploits and their perpetrators; for example, Volt Typhoon and Vanguard Panda.
Perhaps the most important statement in the article is, in my opinion:
“We cannot say with certainty that the adversary has been evicted, because we still don’t know the scope of what they’re doing,” Jeff Greene, CISA’s executive assistant director for cybersecurity, told reporters during a Salt Typhoon briefing in early December.
Several observations:
- The attacks are not confined to the estimable Microsoft software; more commercial software is providing warm, comfortable havens for attacking systems and stealing data
- The existing cyber security systems — no matter what the marketers say in sales material and at law enforcement / intelligence conferences — does not work very well
- Different cyber investigators discover novel, unknown, and possibly unique exploits unearthed and exploited by bad actors in China. Other countries enjoy the fruits of lousy security too I want to add.
So what? What happens if one shoots enough bullets at Butch Cassidy’s and the Sundance Kids’ adobe hideout? Answer: It falls down. Each exploit is a digital bullet hole. Without remediation — serious remediation — the US may suffer some structural collapses. PR, smarmy talk, and excuses won’t do the job.
Stephen E Arnold, January 8, 2025
FOGINT: Divergent Trajectories for Facebook and Telegram
January 7, 2025
A post from the FOGINT team.
The Techmeme splash page featured several Meta (Facebook, WhatsApp, etc.) stories. Here’s a mini-version of the home page with the Zuck-related stories identified:
The separate “stories” presented one theme: Free speech. Here’s a representative item from today’s Techmeme page at 9 20 am US Eastern: “Meta Is Ending Its Fact-Checking Program in Favor of a Community Notes System Similar to X.” The news item from NBC reports:
Meta CEO Mark Zuckerberg announced a series of major changes to the company’s moderation policies and practices Tuesday, citing a shifting political and social landscape and a desire to embrace free speech. Zuckerberg said that Meta will end its fact-checking program with trusted partners and replace it with a community-driven system similar to X’s Community Notes. The company is also making changes to its content moderation policies around political topics and undoing changes that reduced the amount of political content in user feeds, Zuckerberg said.
For me, this says, “Cut some costs and respond to “a shifting political and social landscape.” The direction in which Meta is moving seems to be “freer speech,” albeit within whatever Silly Putty guardrails Mr. Zuckerberg decrees.
In contrast, Telegram — which has out-innovated Meta for many years — is taking a different path through environmental changes in the datasphere. Since France required that Mr. Durov, founder and “owner” of Telegram remain in France until his company’s behavior has been dissected, Telegram is moving on a different trajectory. A few details of this charge have been reported in “Telegram Hands U.S. Authorities Data on Thousands of Users.” This exposé declares:
Telegram, the popular social network and messaging application which has also become a hotbed for all sorts of serious criminal activity, provided U.S. authorities with data on more than 2,200 users last year, according to newly released data from Telegram. The news shows a massive spike in the number of data requests fulfilled by Telegram after French authorities arrested Telegram CEO Pavel Durov in August, in part because of the company’s unwillingness to provide user data in a child abuse investigation. Between January 1 and September 30, 2024, Telegram fulfilled 14 requests “for IP addresses and/or phone numbers” from the United States, which affected a total of 108 users, according to Telegram’s Transparency Reports bot. But for the entire year of 2024, it fulfilled 900 requests from the U.S. affecting a total of 2,253 users, meaning that the number of fulfilled requests skyrocketed between October and December, according to the newly released data. “Fulfilled requests from the United States of America for IP address and/or phone number: 900,” Telegram’s Transparency Reports bot said when prompted for the latest report by 404 Media. “Affected users: 2253,” it added.
Since France’s direct action, Telegram has apparently become even more cooperative with law enforcement. Plus, Telegram agreed to participate in activities designed to identify human traffickers. On the surface, it appears that Telegram is becoming more agreeable to legitimate requests from law enforcement. Telegram has become associated with a number of interesting and possibly illegal activities in some countries. Examples range from groups (private and public) discussing terrorism and child pornography.
But that “shift” to cooperation distracts from what is a major change at Telegram and its affiliated entities like The Open Network Foundation, Ton.social, and assorted investment vehicles. Specifically, Telegram is doubling down on crypto currency. The Telegram infrastructure is being shaped and in some cases repurposed to host services, features, and distributed applications related to crypto. The idea, as the FOGINT team understands it, is to provide a hub or nexus for traditional financial services built on crypto, not the US dollar, euros, or “traditional” and regulated currencies.
A second effect of this shift at Telegram is its push to provide a home for a wide range of seemingly harmless online games. On the surface, a parent or a person as old as the producer of this blog, would glance at the display and think, “Oh, another child’s game.” Those individuals would be incorrect. Telegram “click to earn” games include addictive hooks and the upside of playing are points which can be converted to crypto currency. Gambling and the downstream financial services required by big winners or “whales” are the customers. The addictive element is just part of Telegram’s marketing activities.
Net net: Meta wants free speech or at least to appear to be lining up with the “shifting political and social landscape.” Telegram is using social as a way to speed use of crypto as an alternative to the US dollar. Social media giants are similar in some ways, but at this point in time, the two companies are on divergent trajectories.
Stephen E Arnold, January 7, 2025
Dubai: The 21st Century Crypto “Silicon Valley”
January 7, 2025
Information from the FOGINT research team.
How prescient was Telegram when it selected Dubai as headquarters of a decentralized, distributed company? After Pavel Durov bounced from Moscow to Berlin, to Singapore to San Francisco, and ended up in Dubai, United Arab Emirates, his judgment seems good. FOGINT’s view is that he listened to UAE government officials and determined that that country wanted to become the financial hub for crypto currency. The goal of both UEA and Telegram aligned: Both wanted to exploit a desire of many countries and financial entrepreneurs from the US-centric financial system to one based on crypto currency, largely unregulated crypto currency cut loose from the shackles of the US financial system. A standard other than and competitive with the US dollar promised a shift of finance from Wall Street to Sheikh Zayed Road.
The plan is not a secret. “UAE to Attract Crypto Ventures Amid EU’s Stringent MiCA Regulation: Experts” reports that regulations in Western Europe are adding a kick in the pants for some crypto-centric innovators. The regulation is Markets in Crypto-Assets Regulation (MiCA). Its purpose is to establish a legal framework — that is, uniform rules for crypto assets — across the EU. MiCA might be the booster that the United Arab Emirates and other Middle Eastern states want. A more supportive regulatory environment and a thriving crypto community exist in the United Arab Emirates.
According to the Crypto News’ report:
The MiCA regulation introduces a pan-European licensing and supervisory regime for crypto-assets, exchanges, and service providers… Among its stringent requirements, small stablecoin issuers must hold 30% of their reserves in low-risk EU-based commercial banks, while major players like Tether face a mandate to maintain 60% or more in similar institutions. While aimed at ensuring market stability, these rules are seen as increasing operational costs, potentially undermining the financial viability of many firms.
The FOGINT team wants to point out that the UAE provides a “crucible” for crypto innovation; specifically:
- A regulatory environment different from that in the US and Western Europe; for example, a Virtual Assets Regulatory Authority (VARA) in Dubai oversees the regulation, licensing, and governance of virtual assets
- Tax benefits because there is currently no direct taxation on cryptocurrencies in the UAE
- Infrastructure provides a “Silicon Valley”-type of magnetic pull situated almost equidistant from Asian financial hubs and Western European money centers
- The UAE supports the crypto industry via the Dubai Multi Commodities Centre and the Dubai International Financial Centre
The UAE has cultivated a robust ecosystem for crypto and blockchain innovation with more than 500 crypto startups are now based in Dubai’s free zones. One poster child for Dubai’s flexibility is Telegram’s choice of the city as the location for its “headquarters.” (Keep in mind that Telegram is a distributed and decentralized organization, so the “staff” in Dubai is modest in size for the company’s size.) Plus, the UAE has implemented measures to ensure investor protection and market stability with Know Your Customer (KYC) and Anti-Money Laundering (AML) requirements. Also, the Central Bank of the UAE approved a custodial insurance product to protect financial institutions and their clients from potential losses due to hacks or internal fraud.
One key question: Are there technical professionals with crypto experience in Dubai? The answer, in part, can be approached via the attendance at the November 2024 TON Foundation Gateway Conference. The conference attracted about 400 people in 2023. In November 2024, more than 2000 crypto savvy professionals participated in two day program held in Dubai. The UAE may be on the path to becoming the hot spot for crypto innovation.
Stephen E Arnold, January 7, 2025
Ground Hog Day: Smart Enterprise Search
January 7, 2025
I am a dinobaby. I also wrote the Enterprise Search Report, 1st, 2nd, and 3rd editions. I wrote The New Landscape of Search. I wrote some other books. The publishers are long gone, and I am mostly forgotten in the world of information retrieval. Read this post, and you will learn why. Oh, no AI helped me out unless I come up with an art idea. I used Stable Diffusion for the rat, er, sorry, ground hog day creature.
I think it was 2002 when the owner of a publishing company asked me if I thought there was an interest in profiles of companies offering “enterprise search solutions.” I vaguely remember the person, and I will leave it up to you to locate a copy of the 400 page books I wrote about enterprise search.
The set up for the book was simple. I identified the companies which seemed to bid on government contracts for search, companies providing search and retrieval to organizations, and outfits which had contacted me to pitch their enterprise search systems before they were exiting stealth mode. By the time the first edition appeared in 2004, the companies in the ESR were flogging their products.
The ground hog effect is a version of the Yogi Berra “Déjà vu all over again” thing. Enterprise search is just out of reach now and maybe forever.
The enterprise search market imploded. It was there and then it wasn’t. Can you describe the features and functions of these enterprise search systems from the “golden age” of information retrieval:
- Innerprise
- InQuira
- iPhrase
- Lextek Onix
- MondoSearch
- Speed of Mind
- Stratify (formerly Purple Yogi)
The end of enterprise search coincided with large commercial enterprises figuring out that “search” in a complex organization was not one thing. The problem remains today. Lawyers in a Fortune 1000 company want one type of search. Marketers want another “flavor” of search. The accountants want a search that retrieves structured and unstructured data plus images of invoices. Chemists want chemical structure search. Senior managers want absolutely zero search of their personal and privileged data unless it is lawyers dealing with litigation. In short, each unit wants a highly particularized search and each user wants access to his or her data. Access controls are essential, and they are a hassle at a time when the notion of an access control list was like learning to bake bread following a recipe in Egyptian hieroglyphics.
These problems exist today and are complicated by podcasts, video, specialized file types for 3D printing, email, encrypted messaging, unencrypted messaging, and social media. No one has cracked the problem of a senior sales person who changes a PowerPoint deck to close a deal. Where is that particular PowerPoint? Few know and the sales person may have deleted the file changed minutes before the face to face pitch. This means that baloney like “all” the information in an organization is searchable is not just stupid; it is impossible.
The key events were the legal and financial hassles over Fast Search & Transfer. Microsoft bought the company in 2008 and that was the end of a reasonably capable technology platform and — believe it or not — a genuine alternative to Google Web search. A number of enterprise search companies sold out because the cost of keeping the technology current and actually running a high-grade sales and marketing program spelled financial doom. Examples include Exalead and Vivisimo, among others. Others just went out of business: Delphes (remember that one?). The kiss of death for the type of enterprise search emphasized in the ESR was the acquisition of Autonomy by Hewlett Packard. There was a roll up play underway by OpenText which has redefined itself as a smart software company with Fulcrum and BRS Search under its wing.
What replaced enterprise search when the dust settled in 2011? From my point of view it was Shay Banon’s Elastic search and retrieval system. One might argue that Lucid Works (né Lucid Imagination) was a player. That’s okay. I am, however, to go with Elastic because it offered a version as open source and a commercial version with options for on-going engineering support. For the commercial alternatives, I would say that Microsoft became the default provider. I don’t think SharePoint search “worked” very well, but it was available. Google’s Search Appliance appeared and disappeared. There was zero upside for the Google with a product that was “inefficient” at making a big profit for the firm. So, Microsoft it was. For some government agencies, there was Oracle.
Oracle acquired Endeca and focused on that computationally wild system’s ability to power eCommerce sites. Oracle paid about $1 billion for a system which used to be an enterprise search with consulting baked in. One could buy enterprise search from Oracle and get structured query language search, what Oracle called “secure enterprise search,” and may a dollop of Triple Hop and some other search systems the company absorbed before the end of the enterprise search era. IBM talked about search but the last time I drove by IBM Government systems in Gaithersburg, Maryland, it like IBM search, had moved on. Yo, Watson.
Why did I make this dalliance on memory lane the boring introduction to a blog post? The answer is that I read “Are LLMs At Risk Of Going The Way Of Search? Expect A Duopoly.” This is a paywalled article, so you will have to pony up cash or go to a library. Here’s an abstract of the write up:
The evolution of LLMs (Large Language Models) will lead users to prefer one or two dominant models, similar to Google’s dominance in search.
Companies like Google and Meta are well-positioned to dominate generative AI due to their financial resources, massive user bases, and extensive data for training.
Enterprise use cases present a significant opportunity for specialized models.
Therefore, consumer search will become a monopoly or duopoly.
Let’s assume the Forbes analysis is accurate. Here’s what I think will happen:
First, the smart software train will slow and a number of repackagers will use what’s good enough; that is, cheap enough and keeps the client happy. Thus, a “golden age” of smart search will appear with outfits like Google, Meta, Microsoft, and a handful of others operating as utilities. The US government may standardize on Microsoft, but it will be partners who make the system meet the quite particular needs of a government entity.
Second, the trajectory of the “golden age” will end as it did for enterprise search. The costs and shortcomings become known. Years will pass, probably a decade, maybe less, until a “new” approach becomes feasible. The news will diffuse and then a seismic event will occur. For AI, it was the 2023 announcement that Microsoft and OpenAI would change how people used Microsoft products and services. This created the Google catch up and PR push. We are in the midst of this at the start of 2025.
Third, some of the problems associated with enterprise information and an employee’s finding exactly what he or she needs will be solved. However, not “all” of the problems will be solved. Why? The nature of information is that it is a bit like pushing mercury around. The task requires fresh thinking.
To sum up, the problem of search is an excellent illustration of the old Hegelian chestnut of Hegelian thesis, antithesis, and synthesis. This means the problem of search is unlikely to be “solved.” Humans want answers. Some humans want to verify answers which means that the data on the sales person’s laptop must be included. When the detail oriented human learns that the sales person’s data are missing, the end of the “search solution” has begun.
The question “Will one big company dominate?” The answer is, in my opinion, maybe in some use cases. Monopolies seem to be the natural state of social media, online advertising, and certain cloud services. For finding information, I don’t think the smart software will be able to deliver. Examples are likely to include [a] use cases in China and similar countries, [b] big multi-national organizations with information silos, [c] entities involved in two or more classified activities for a government, [d] high risk legal cases, and [e] activities related to innovation, trade secrets, and patents, among others.
The point is that search and retrieval remains an extraordinarily difficult problem to solve in many situations. LLMs contribute some useful functional options, but by themselves, these approaches are unlikely to avoid the reefs which sank the good ships Autonomy and Fast Search & Transfer, and dozens of others competing in the search space.
Maybe Yogi Berra did not say “Déjà vu all over again.” That’s okay. I will say it. Enterprise search is “Déjà vu all over again.”
Stephen E Arnold, January 7, 2025
-
- Subscribe to Beyond Search
Feature archive
News archive
-
