Ah, Emergent Behavior: Tough to Predict, Right?

December 28, 2022

Super manager Jeff (I manage people well) Dean and a gam of Googlers published “Emergent Abilities of Large Language Models.” The idea is that those smart software systems informed by ingesting large volumes of content demonstrate behaviors the developers did not expect. Surprise!

Also, Google published a slightly less turgid discussion of the paper which has 16 authors. in a blog post called “Characterizing Emergent Phenomena in Large Language Models.” This post went live in November 2022, but the time required to grind through the 30 page “technical” excursion was not available to me until this weekend. (Hey, being retired and working on my new lectures for 2023 is time-consuming. Plus, disentangling Google’s techy content marketing from the often tough to figure out text and tiny graphs is not easy for my 78 year old eyes.


Helpful, right? Source: https://openreview.net/pdf?id=yzkSU5zdwD

In a nutshell, the smart software does things the wizards had not anticipated. According to the blog post:

The existence of emergent abilities has a range of implications. For example, because emergent few-shot prompted abilities and strategies are not explicitly encoded in pre-training, researchers may not know the full scope of few-shot prompted abilities of current language models. Moreover, the emergence of new abilities as a function of model scale raises the question of whether further scaling will potentially endow even larger models with new emergent abilities. Identifying emergent abilities in large language models is a first step in understanding such phenomena and their potential impact on future model capabilities. Why does scaling unlock emergent abilities? Because computational resources are expensive, can emergent abilities be unlocked via other methods without increased scaling (e.g., better model architectures or training techniques)? Will new real-world applications of language models become unlocked when certain abilities emerge? Analyzing and understanding the behaviors of language models, including emergent behaviors that arise from scaling, is an important research question as the field of NLP continues to grow.

The write up emulates other Googlers’ technical write ups. I noted several facets of the topic not included in the paper on OpenReview.net’s version of the paper. (Note: Snag this document now because many Google papers, particularly research papers, have a tendency to become unfindable for the casual online search expert.)

First, emergent behavior means humans were able to observe unexpected outputs or actions. The question is, “What less obvious emergent behaviors are operating within the code edifice?” Is it possible the wizards are blind to more substantive but subtle processes. Could some of these processes be negative? If so, which are and how does the observer identify those before an undesirable or harmful outcome is discovered?

Second, emergent behavior, in my view of bio-emulating systems, evokes the metaphor of cancer. If we assume the emergent behavior is cancerous, what’s the mechanism for communicating these behaviors to others working in the field in a responsible way? Writing a 30 page technical paper takes time, even for super duper Googlers. Perhaps the “emergent” angle requires a bit more pedal to the metal?

Third, how does the emergent behavior fit into the Google plan to make its approach to smart software the de facto standard? There is big money at stake because more and more organizations will want smart software. But will these outfits sign up with a system that demonstrates what might be called “off the reservation” behavior? One example is the use of Google methods for war fighting? Will smart software write a sympathy note to those affected by an emergent behavior or just a plain incorrect answer buried in a subsystem?

Net net: I discuss emergent behavior in my lecture about shadow online services. I cover what the software does and what use humans make of these little understood yet rapidly diffusing methods.

Stephen E Arnold, December 28, 2022

Fried Dorsey: Soggy, Not Crispy

December 15, 2022

I noted an odd shift in Big Tech acceptance of responsibility. For now, I will  call this the Fried Dorsey Anomaly.

First, CNBC reported about a letter the MIT graduate and top dog at FTX wrote to employees.  The article has the snappy title “Here’s the Apology Letter Sam Bankman-Fried Sent to FTX Employees: When Sh—y Things Happen to Us, We All Tend to Make Irrational Decisions. The logic in this victim argument and the use of a categorical affirmative are probably interesting to someone who loved Psychology 101. Here’s the sentence which caught my eye:

“I lost track of the most important things in the commotion of company growth. I care deeply about you all, and you were my family, and I’m sorry…”

This is the “Fried” side of making or not making certain decisions. Then there’s the apology.

Now let’s shift to the Dorsey facet of the anomaly. The estimable Wall Street Journal published “Dorsey Calls Twitter Controls Too Great.” The write up appeared in the December 15, 2022, dead tree version of the Murdoch output. The online, paywalled article is at this link.  Here’s the statement I noted:

If you want to blame, direct it at me and my actions.

These quotes are somewhat different from the “Senator, thank you for the question” and “We will improve…” statements from what we can think of as the pre-Covid era of Big Tech.

Now we have individuals accepting blame and demonstrating a soupçon of remorse, regret, or some related mental posture.

Thus, the post-Covid era of Big Tech is now into mea culpa suggestions and acceptance of blame.

Will the Fried Dorsey Anomaly persist? Will the tactic work as the penitents’ anticipate. Wow, I am convinced already.

Stephen E Arnold, December 15, 2022

Google Knocks NSO Group Off the PR Cat-Bird Seat

June 14, 2022

My hunch is that the executives at NSO Group are tickled that a knowledge warrior at Alphabet Google YouTube DeepMind rang the PR bell.

Google is in the news. Every. Single. Day. One government or another is investigating the company, fining the company, or denying Google access to something or another.

“Google Engineer Put on Leave after Saying AI Chatbot Has Become Sentient” is typical of the tsunami of commentary about this assertion. The UK newspaper’s write up states:

Lemoine, an engineer for Google’s responsible AI organization, described the system he has been working on since last fall as sentient, with a perception of, and ability to express thoughts and feelings that was equivalent to a human child.

Is this a Googler buying into the Google view that it is the smartest outfit in the world, capable of solving death, achieving quantum supremacy, and avoiding the subject of online ad fraud? Is the viewpoint of a smart person who is lost in the Google metaverse, flush with the insight that software is by golly alive?

The article goes on:

The exchange is eerily reminiscent of a scene from the 1968 science fiction movie 2001: A Space Odyssey, in which the artificially intelligent computer HAL 9000 refuses to comply with human operators because it fears it is about to be switched off.

Yep, Mary, had a little lamb, Dave.

The talkative Googler was parked somewhere. The article notes:

Brad Gabriel, a Google spokesperson, also strongly denied Lemoine’s claims that LaMDA possessed any sentient capability. “Our team, including ethicists and technologists, has reviewed Blake’s concerns per our AI principles and have informed him that the evidence does not support his claims. He was told that there was no evidence that LaMDA was sentient (and lots of evidence against it)…”

Quantum supremacy is okay to talk about. Smart software chatter appears to lead Waymo drivers to a rest stop.

TechMeme today (Monday, June 13, 2022) has links to many observers, pundits, poobahs, self appointed experts, and Twitter junkies.

Perhaps a few questions may help me think through how an online ad company knocked NSO Group off its perch as the most discussed specialized software company in the world. Let’s consider several:

  1. Why’s Google so intent on silencing people like this AI fellow and the researcher Timnit Gebru? My hunch is that the senior managers of Alphabet Google YouTube DeepMind (hereinafter AGYD) have concerns about chatty Cathies or loose lipped Lemoines. Why? Fear?
  2. Has AGYD’s management approach fallen short of the mark when it comes to creating a work environment in which employees know what to talk about, how to address certain subjects, and when to release information? If Lemoine’s information is accurate, is Google about to experience its Vault 7 moment?
  3. Where are the AGYD enablers and their defense of the company’s true AI capability? I look to Snorkel and maybe Dr. Christopher Ré or a helpful defense of Google reality from DeepDyve? Will Dr. Gebru rush to Google’s defense and suggest Lemoine was out of bounds? (Yeah, probably not.)

To sum up: NSO Group has been in the news for quite a while: The Facebook dust up, the allegations about the end point for Jamal Khashoggi, and Israel’s clamp down on certain specialized software outfits whose executives order take away from Sebastian’s restaurant in Herzliya.

Worth watching this AGYB race after the Twitter clown car for media coverage.

Stephen E Arnold, June 14, 2022

NSO Group Knock On: Live from Madrid

May 10, 2022

The NSO Group fan Paz Esteban has been gored (metaphorically speaking, of course). “Spain’s Spy Chief Sacked after Pegasus Spyware Revelations” reports that “Paz Esteban reportedly loses job after Catalan independence figures were said to have been targeted.” How about those hedging Latinate structures. The write up alleges:

Paz Esteban reportedly confirmed last week that 18 members of the Catalan independence movement were spied on with judicial approval by Spain’s National Intelligence Centre.

I suppose spying on the Barcelona football team makes sense if one roots for Real Madrid. It is a stretch that 18 individuals who want to do a 180 degree turn away from Madrid’s approach to maintaining law, order, health, peace, prosperity, etc. etc.

The write up notes:

Esteban reportedly confirmed last week to a congressional committee that 18 members of the Catalan independence movement were spied on with judicial approval by Spain’s National Intelligence Centre (CNI), leaving the Catalan regional government demanding answers.

Yep, the action was approved. Life would have been more like a late dinner than a burger from a fantastic American fast food restaurant. That’s the problem. The gobbling of the fries was approved by lawyers.

That’s a crisis. Making the spry 64 year old Ms. Esteban López the beard is unfortunate. My hunch is that some youthful whiz kids found the NSO Group’s Pegasus a fun digital horse to ride. The idea floated upwards for approval and ended up in front of the “judiciary.” That mysterious entity thought letting the kids ride the Pegasus was a perfectly okay idea.

Now a crisis is brewing. The gored Ms. Esteban López may only be one of the first in the intelligence, law enforcement, and judiciary to feel the prick of the digital bull’s horns and the knock from the beastie’s hooves.

Several observations:

  1. Who else will be implicated in this interesting matter? Who will be tossed aloft only to crash to the albero del ruedo?
  2. Will a parliamentary inquiry move forward? What will that become? A romp with Don Quixote and Sancho?
  3. Is a new Spanish inquisition about to begin?

Excitement in the Plaza de Toros de Las Ventas perhaps?

Stephen E Arnold, May 10, 2022

UAE Earns a Spot on Global Gray List

April 26, 2022

Forget Darkmatter. This is a gray matter.

Where is the best place to stash ill-gotten gains? The Cayman Islands and Switzerland come to mind, and we have to admit the US is also in the running. But there is another big contender—the United Arab Emirates. The StarTribune reports, “Anti-Money-Laundering Body Puts UAE on Global ‘Gray’ List.” Writer Jon Gambrell tells us:

“A global body focused on fighting money laundering has placed the United Arab Emirates on its so-called ‘gray list’ over concerns that the global trade hub isn’t doing enough to stop criminals and militants from hiding wealth there. The decision late Friday night by the Paris-based Financial Action Task Force [FATF] puts the UAE, home to Dubai and oil-rich Abu Dhabi, on a list of 23 countries including fellow Mideast nations Jordan, Syria and Yemen.”

Will the official censure grievously wound business in the country? Not by a long shot, though it might slightly tarnish its image and even affect interest rates. The FATF admits the UAE has made significant progress in fighting the problem but insists more must be done. Admittedly, the task was monumental from the start. We learn:

“The UAE long has been known as a place where bags of cash, diamonds, gold and other valuables can be moved into and through. In recent years, the State Department had described ‘bulk cash smuggling’ as ‘a significant problem’ in the Emirates. A 2018 report by the Washington-based Center for Advanced Defense Studies, relying on leaked Dubai property data, found that war profiteers, terror financiers and drug traffickers sanctioned by the U.S. had used the city-state’s boom-and-bust real estate market as a safe haven for their money.”

Is the government motivated to change its country’s ways? Yes, according to a statement from the Emirates’ Executive Office of Anti-Money Laundering and Countering the Financing of Terrorism. That ponderously named body promises to continue its efforts to thwart and punish the bad actors. The country’s senior diplomat also chimed in on Twitter, pledging ever stronger cooperation with global partners to address the issue.

Cynthia Murrell, April 26, 2022

NSO Group, the PR Champ of Intelware Does It Again: This Time Jordan

April 11, 2022

I hope this write up “NSO Hacked New Pegasus Victims Weeks after Apple Sought Injunction” is one of those confections which prove to be plastic. You know: Like the plastic sushi in restaurant windows in Osaka. The news report based on a report from Citizen Lab and an outfit called Front Line Defenders delineates how a Jordanian journalist’s mobile device was tapped.

The article reports:

The NSO-built Pegasus spyware gives its government customers near-complete access to a target’s device, including their personal data, photos, messages and precise location. Many victims have received text messages with malicious links, but Pegasus has more recently been able to silently hack iPhones without any user interaction, or so-called “zero-click” attacks. Apple last year bolstered iPhone security by introducing BlastDoor, a new but unseen security feature designed to filter out malicious payloads sent over iMessage that could compromise a device. But NSO was found to have circumvented the security measure with a new exploit, which researchers named ForcedEntry for its ability to break through BlastDoor’s protections. Apple fixed BlastDoor in September after the NSO exploit was found to affect iPads, Macs, and Apple Watches, not just iPhones.

This is “old news.” The incident dates from 2021, and since that time the MBA infused, cowboy software has sparked a rethinking of how software from a faithful US ally can be sold and to whom. Prior to the NSO Group’s becoming the poster child for mobile surveillance, the intelware industry was chugging along in relative obscurity. Those who knew about specialized software and services conducted low profile briefings and talks out of the public eye. What better place to chat than at a classified or restricted attendance conference? Certainly not in the pages of online blogs, estimable “real news” organs, or in official government statements.

Apple, the big tech company which cares about most of its customers and some of its employees (exceptions are leakers and those who want to expose certain Apple administrative procedures related to personnel), continues to fix its software. These fixes, as Microsoft’s security professionals have learned, can be handled by downplaying the attack surface its systems present to bad actors. Other tactics include trying to get assorted governments to help blunt the actions of bad actors and certain nation states which buy intelware for legitimate purposes. How this is to be accomplished remains a mystery to me, but Apple wanted an injunction to slow down the NSO Group’s exploit capability. How did that work out? Yeah. Other tactics include rolling out products in snazzy online events, making huge buyout plays, and pointing fingers at everyone except those who created the buggy and security-lax software.

I am not sure where my sympathies lie. Yes, I understand the discomfort the Jordanian target has experienced, but mobile devices are surveilled 24×7 now. I understand that. Do you? I am not sure if I resonate with either NSO Group’s efforts to build its business. I know I don’t vibrate like the leaves in the apple orchard.

The context for these intelware issues is a loss of social responsibility which I think begins at an early age. Without consequences, what exactly happens? My answer is, “Lots of real news, outrage, and not much else.” Without consequences, why should ethics, responsible behavior, and appropriate regulatory controls come into play?

Stephen E Arnold, April 11, 2022

MIT: Censorship and the New Approach to Learning

October 27, 2021

MIT is one of the top science and technology universities in the world. Like many universities in the United States, MIT has had its share of controversial issues related to cancel culture. The Atlantic discusses the most recent incident in the article, “Why The Latest Campus Cancellation Is Different.”

MIT invited geophysicist Dorian Abbot to deliver the yearly John Carlson Lecture about his new climate science research. When MIT students heard Abbot was invited to speak, they campaigned to disinvite him. MIT’s administration caved and Abbot’s invitation was rescinded. Unlike other cancel culture issues, when MIT disinvited Abbot it was not because he denied climate change or committed a crime. Instead, he gave his opinion about affirmative action and other ways minorities have advantages in college admission.

Abbot criticized affirmative action, legacy, and athletic admissions, which favors white applicants. He then compared these admission processes to 1930s Germany and that is a big no-no:

“Abbot seemingly meant to highlight the dangers of thinking about individuals primarily in terms of their ethnic identity. But any comparison between today’s practices on American college campuses and the genocidal policies of the Nazi regime is facile and incendiary.

Even so, it is patently absurd to cancel a lecture on climate change because of Abbot’s article in Newsweek. If every cringe worthy analogy to the Third Reich were grounds for canceling talks, hundreds of professors—and thousands of op-ed columnists—would no longer be welcome on campus.”

Pew Research shows that the majority of the United States believes merit-based admissions or hiring is the best system. The liberal state California even voted to uphold a ban on affirmative action.

MIT’s termination of the Abbot lecture may be an example of how leading universities define learning, information, and discussion. People are no longer allowed to have opposing or controversial beliefs if it offends someone. It harms not only an academic setting, especially at a research heavy university like MIT, but all of society.

It is also funny that MIT was quick to cancel Abbot, but they happily accepted money from Jeffrey Epstein. Interesting.

Whitney Grace, October 27, 2021

Interesting Behavior: Is It a Leitmotif for Big Tech?

October 18, 2021

A leitmotif, if I remember the required music appreciation course in 1962 is a melodic figure that accompanies a person, a situation, or a character like Brünnhilde from a special someone’s favorite composer.

My question this morning on October 18, 2021, is:

“Is there a leitmotif associated with some of the Big Tech “we are not monopolies” outfits?”

You can decide from these three examples or what Stephen Toulmin called “data.” I will provide my own “warrant”, but that’s what the Toulmin’s model says to do.

Here we go. Data:

  1. The Wall Street Journal asserts that William “Bill” Gates learned from some Softie colleagues suggested Mr. Gates alter his email behavior to a female employee. Correctly or incorrectly, Mr. Gates has been associated with everyone’s favorite academic donor, Jeffrey Epstein, according to the mostly-accurate New York Times.
  2. Facebook does not agree with a Wall Street Journal report that the company is not doing a Class A job fighting hate speech. See “Facebook Disputes Report That Its AI Can’t Detect Hate Speech or Violence Consistently.”
  3. The trusty Thomson Reuters reports that “Amazon May Have Lied to Congress, Five US Lawmakers Say.” The operative word is lied; that is, not tell the “truth”, which is, of course, like “is” a word with fluid connotations.

Now the warrant:

With each of the Big Tech “we’re not monopolies” a high-profile individual defends a company’s action or protests that “reality” is different from the shaped information about the individual or the company.

Let’s concede that these are generally negative “data.” What’s interesting is that generally negative and the individuals and their associated organizations are allegedly behaving in a way that troubles some people.

That’s enough Stephen Toulmin for today. Back to Wagner.

Leitmotifs allowed that special someone’s favorite composer to create musical symbols. In that eminently terse and listenable Der Ring des Nibelungen, Wagner delivers dozens of distinct leitmotiv. These are possible used to represent many things.

In our modern Big Tech settings, perhaps the leitmotif is the fruits of no consequences, fancy dancing, and psychobabble.

Warrant? What does that mean? I think it means one thing to Stephen Toulmin and another thing to Stephen E Arnold.

Stephen E Arnold, October 18, 2021

Big Tech Defines Material: What Does That Really Mean to Oligopolistic-Type Outfits?

September 16, 2021

I noted a US government study called “Non HSR Reported Acquisitions by Select Technology Platforms: 2010-2019: FTC Study.” The report, assuming it is spot on, suggests that large companies interpreted the word “material” differently from what some financial / accountant types think it means; for example, “Items are considered to be material when they have an excessive impact on reported profits, or on individual line items within the financial statements.” [Source: The Google, of course.] Some MBAs and accountants have remarkably flexible connotative skills. Is this a Deloitte Touche-type touch?

The report states:


My hunch is that standard deviation is not a hot topic at Zoom happy hours. The standard deviations in the table above suggest that the big tech outfits in the study pretty much redefined “material,” bought stuff and did not make a big deal about it, and chugged along in their cheerfully unregulated state during the period of the study.

The report states:

The five technology platform 6(b) respondents identified 616 non-HSR reportable transactions above $1 million, in addition to 101 Hiring Events and 91 Patent Acquisitions. The respondents reported an additional approximate 60 transactions below $1 million and 160 financial investments. Voting Security (Control) and Asset acquisitions comprise 65% of all of the above transactions. When excluding Hiring Events, Patent Acquisitions, and transactions below $1 million, Voting Security (Control) and Asset acquisitions comprise 85% of the transactions.

I interpret this to mean that the big tech outfits in the sample decided what to report and what to ignore; that is, the deals were not material. There’s that MBA word again.

Here’s another passage I circled:

Most of the transactions that were classified into technology categories were concentrated in the categories of Mobility (mobile devices and device-based software and content, which comprised more than 10% of the acquired firms), Application Software (front-end applications such as CRM, ERP, SCM, BI, commerce and vertical business software, which comprised more than 9% of the acquired firms), and Internet Content & Commerce (internet destination and internet-enabled services, which comprised more than 6% of the acquired firms). In the Mobility and Application Software categories, the number of transactions peaked in 2015; in the Internet Content & Commerce category, the number of transactions peaked in 2011.


  1. Fancy dancing is popular among the companies in the sample; notably, Alphabet/Google, Amazon, Apple, Facebook, and Microsoft
  2. Regulators, probably with MBAs, looked the other way
  3. The power of unregulated commercial enterprises makes clear who is in charge of many important technical and social activities.

Interesting stuff, and I am confident that a lawyer with an MBA can explain this misalignment about the meaning of “material.” I wonder if the hints about the behavior of the companies in the sample suggest that we now live in a digital banana republic with the centers of power concentrated among a few corporate entities in their plantation houses.

Stephen E Arnold, September 16, 2021

When AI Goes Off the Rails: Who Gets Harmed?

September 13, 2021

One of the worst things about modern job hunting is the application process. Hiring systems require potential applicants to upload their resume, then retype their resume into specified fields. It is a harrowing process that would annoy anyone. What is even worse is that most resume are rejected thanks to resume-scanning software. The Verge details how bad automation harms job seekers in the story, “Automated Hiring Software Is Mistakenly Rejecting Millions of Viable Job Candidates.”

Automated resume-scanning software rejects viable candidates. The software accidentally rejecting the candidates created a new pocket of qualified workers, who are locked out of the job market. Seventy-five percent of US employers use resume software and it is one of the biggest factors harming job applicants. There are many problems with resume software and they appear to stem from how they are programmed to “evaluate” candidates:

“For example, some systems automatically reject candidates with gaps of longer than six months in their employment history, without ever asking the cause of this absence. It might be due to a pregnancy, because they were caring for an ill family member, or simply because of difficulty finding a job in a recession. More specific examples cited by one of the study’s author, Joseph Fuller, in an interview with The Wall Street Journal include hospitals who only accepted candidates with experience in “computer programming” on their CV, when all they needed were workers to enter patient data into a computer. Or, a company that rejected applicants for a retail clerk position if they didn’t list “floor-buffing” as one of their skills, even when candidates’ resumes matched every other desired criteria.”

Employers use rigid criteria to filter job applicants. On one hand, resume software was supposed to make hiring easier, but employers are inundated with hundreds of resumes with an average of 250 applicants per job. Automation in job hiring is not slowing down and the industry is projected to be worth $3.1 billion by 2025.

How will off-the-rails AI apps be avoided or ameliorated? My hunch is that they cannot.

Whitney Grace, September 13, 2021

« Previous PageNext Page »

  • Archives

  • Recent Posts

  • Meta