Facebook Engineering: Big Is Tricky
October 12, 2021
The unthinkable happened on October 4, 2021, when Facebook went offline. Despite all the bad press Facebook has recently gotten, the social media network remains an important communication and business tool. The Facebook Engineering blog explains what happened with the shutdown in the post: “More Details About The October 4 Outage.” The outage happened with the system that manages Facebook’s global backbone network capacity.
The backbone connects all of Facebook’s data centers through thousands of miles of fiber optic cable. The post runs down how the backbone essentially works:
“When you open one of our apps and load up your feed or messages, the app’s request for data travels from your device to the nearest facility, which then communicates directly over our backbone network to a larger data center. That’s where the information needed by your app gets retrieved and processed, and sent back over the network to your phone.
The data traffic between all these computing facilities is managed by routers, which figure out where to send all the incoming and outgoing data. And in the extensive day-to-day work of maintaining this infrastructure, our engineers often need to take part of the backbone offline for maintenance — perhaps repairing a fiber line, adding more capacity, or updating the software on the router itself.”
A routine maintenance job issued a command to assess the global backbone’s capacity. Unfortunately it contained a bug the audit system did not catch and it terminated connections between data centers and the Internet. A second problem made things worse. The DNS servers were unreachable yet still operational. Facebook would not connect to their data centers through the normal meals and loss of DNS connections broke internal tools used to repair problems.
Facebook engineers had to physically visit the backbone facility, which is armed with high levels of security. The facility is hard to enter and the systems are purposely designed to be difficult to modify. It took awhile, but Facebook diagnosed and resolved the problems. Baby Boomers were overjoyed to resume posting photos of their grandchildren and anti-vaxxers could read their misinformation feeds.
Perhaps this Facebook incident and the interesting Twitch data breach illustrate that big is tricky? Too big to fail become too big to keep working in a reliable way.
Whitney Grace, October 12, 2021
The Economist Zucks Facebook
October 8, 2021
Time to unfollow or defriend or just erase the Zuck?
“Facebook Is Nearing a Reputational Point of No Return” illustrates the turning of the capitalistic worm. The newspaper which sure looks like a magazine to me states:
Facebook is nearing a reputational point of no return.
Then this wowza:
If rational argument alone is no longer enough to get Facebook out of its hole, the company should look hard at its public face. Mark
Zuckerberg, Facebook’s all-powerful founder, made a reasoned statement after this week’s wave of anger. He was ignored or ridiculed and increasingly looks like a liability.
Facebook has been chugging along since 2004. Finger pointing, legal action, and the Winkelvossing have been stirred into Cambridge Analytica, apologies, and savories like WhatsApp as a new Dark Web.
Suddenly the Zuck is a liability.
Seventeen years and counting. Insight takes time to arrive.
Stephen E Arnold, October 8, 2021
An Onion Story: The Facebook Oversight Board Checks Out Rule Following
October 7, 2021
I read “Facebook’s Oversight Board to Review Xcheck System Following Investigation of Internal System That Exempted Certain Users.” Is this a story created for the satirical information service Onion, MAD Magazine, or the late, lamented Harvard Lampoon?
I noted this passage:
For certain elite users, Facebook’s rules don’t seem to apply.
I think this means that there is one set of rules for one group of users and another set of rules for another group of users. In short, the method replicates the tidy structure of a medieval hierarchy; to wit:
The “church” would probably represent the Zuck and fellow technical elite plus a handful of fellow travelers. The king is up for grabs now that the lean in expert has leaned out. The nobles and barons are those who get a special set of rules. The freemen can buy ads. The serfs? Well, peasants are okay for clicks but not much else.
Now the oversight board which is supposed to be overseeing will begin the overseeing process of what appears to be a discriminatory system.
Obviously the oversight board is either in the class of freemen or serfs. I wonder if this Onionesque management method is a variant of the mushroom approach; that is, keep the oversight board and users in the dark and feed them organic matter rich in indole, skatole, hydrogen sulfide, and mercaptans?
That Facebook is an Empyrean spring of excellence in ethics, management, and business processes. My hunch is that not even the outfits like the Onion can match this joke. Maybe Franz (Happy) Kafka could?
Stephen E Arnold, October 7, 2021
Facebook: Why Change?
October 6, 2021
I read “Facebook Can’t Be Saved.” The main point struck me as:
Facebook has experienced years of intense scrutiny over the exact issues that are being discussed in the wake of Haugen’s revelations, and has only succeeded in making its inherent problems worse. During the hearing, Haugen compared fixing Facebook’s issues to mandating that cars come with seat belts. But maybe Facebook doesn’t need a seat belt. Maybe it just needs to stop being given more chances.
This is an interesting analogy. I would ask this question, “Why should Facebook change?” The company has loyal users, lobbyists, and friends in high places. The available consequences are fines and enduring hearings and legal proceedings.
After watching the testimony by the whistle blower, my hunch is that Facebook will evolve. But the deep machine is chugging along.
Stephen E Arnold, October 6, 2021
Facebook Doing Its Thing with Weaponized Content?
October 1, 2021
I read “Facebook Forced Troll Farm Content on Over 40% of All Americans Each Month.” Straight away, I have problems with “all.” The reality is that “all” Americans means those who don’t use Facebook, Instagram, or WhatsApp. Hence, I am not sure how accurate the story itself is.
Let’s take a look at a couple of snippets, shall we?
Here’s one that caught my attention:
When the report was published in 2019, troll farms were reaching 100 million Americans and 360 million people worldwide every week. In any given month, Facebook was showing troll farm posts to 140 million Americans. Most of the users never followed any of the pages. Rather, Facebook’s content-recommendation algorithms had forced the content on over 100 million Americans weekly. “A big majority of their ability to reach our users comes from the structure of our platform and our ranking algorithms rather than user choice,” the report said. Sweeping internal Facebook memo: “I have blood on my hands” The troll farms appeared to single out users in the US. While globally more people saw the content by raw numbers—360 million every week by Facebook’s own accounting—troll farms were reaching over 40 percent of all Americans.
Yeah, lots of numbers, not much context, and the source of the data appears to be Facebook. Maybe on the money, maybe a bent penny? If we assume that the passage is “sort of correct”, Facebook has added to its track record for content moderation.
Here’s another snippet I circled in red:
Allen believed the problem could be fixed relatively easily by incorporating “Graph Authority,” a way to rank users and pages similar to Google’s PageRank, into the News Feed algorithm. “Adding even just some easy features like Graph Authority and pulling the dial back from pure engagement-based features would likely pay off a ton in both the integrity space and… likely in engagement as well,” he wrote. Allen [a former data scientist at Facebook,] left Facebook shortly after writing the document, MIT Technology Review reports, in part because the company “effectively ignored” his research, a source said.
Disgruntled employee? Fancy dancing with confidential information? A couple of verification items?
Net net: On the surface, Facebook continues to do what its senior management prioritizes. Without informed oversight, what’s the downside for Facebook? Answer: At this time, none.
Stephen E Arnold, October 1, 2021
Facebook Brings People Together: A Different Spin
September 29, 2021
I read “Lawmakers Ask Zuckerberg to Drop ‘Instagram for Kids’ After Report Says App Made Kids Suicidal.” The write up reports about more concern and hand wringing about the impact of social media. Finally an anonymous but brave Facebook whistleblower has awakened the somnambulant US elected officials from their summer siesta. Here’s a quote from the write up:
“Children and teens are uniquely vulnerable populations online, and these findings paint a clear and devastating picture of Instagram as an app that poses significant threats to young people’s wellbeing,” the lawmakers said.
Facebook was founded in 2004. Let’s see that works out to about eight days in the timescape of US elected officials, doesn’t it. Why rush?
Stephen E Arnold, September 29, 2021
Yay, A Facebook Friday
September 24, 2021
Three slightly intriguing factoids about the Zuckbook.
The first is a characterization of Facebook’s and the supreme leader’s time spirit:
“Shame, addiction, and dishonesty.”
Well, that’s a poster message for some innovator in the decorative arts. The original could be offered on Facebook Messenger and the cash transaction handled at night in a fast food joint’s parking lot. What could go wrong? And the source of this information? The work of the UX Collective and included in a write up with the title “Zuckerberg’s Zeitgeist: A Culture of Shame, Addiction, and Dishonesty.” What’s left out of the write up? How many UX Collective professionals have Facebook accounts? And what’s the method of remediation? A better interface. Okay. Deep.
The second is from “Facebook’s Incoming Chief Technology Officer Once Said People Being Cyberbullied to Suicide of Killed in Terror Attacks Organized on the Site Was a Price Worth Paying to Connect People.” The headline alleges that the new Facebook chief technology officer or C3PO robot emitted this statement. Another memorable phrase from the C2PO Facebooker is allegedly:
Maybe it costs a life by exposing someone to bullies. Maybe someone dies in a terrorist attack coordinated on our tools. And still we connect people.’
Snappy? Yep.
And, finally, today (September 24, 2021), that the estimable Salesforce luminary, Marc Benioff, who maybe said:
In regards to Facebook, they are not held accountable.
The write up “Tech Billionaire: Facebook Is What’s Wrong with America” contains an even more T shirtable slogan. I live in fear of Google’s duplication savvy smart software, but I want to be clear:
Facebook is what’s wrong with America
I like this statement whether from the humanoid running Salesforce or a thumbtyping PR expert with a degree in art history and a minor in business communications. Winner.
Net net: Facebook seems to be a font of news and inspiration. And, please, remember the fix: user interface changes. Yes.
Stephen E Arnold, September 24, 2021
Facebook: Not Happy
September 20, 2021
“What the Wall Street Journal Got Wrong” is interesting, and you may want to read it. My synopsis is: “We’re doing good.”
I noted this passage from the firm’s top PR dog:
Facebook understands the significant responsibility that comes with operating a global platform. We take it seriously, and we don’t shy away from scrutiny and criticism. But we fundamentally reject this mischaracterization of our work and impugning of the company’s motives.
I like this statement. It’s bold. It ignores the criticism. It sidesteps tricky issues like human trafficking. Very nice.
What makes me happy is the commitment to excellence. I do wonder where the Zuck is in this brutal rejoinder to leaked company info. Is he “leaning in”? Is his leaning out? Practicing a dose doe?
Stephen E Arnold, September 20, 2021
Facebook: A Pattern That Matches the Zuck
September 20, 2021
The laws of the United States (and most countries) are equally applied to everyone, unless you are rich and powerful. Facebook certainly follows this rule of thumb according to The Guardian article, “Facebook: Some High-Profile Users ‘Allowed To Break Platform’s Rules.’” Facebook has to sets of rules, one for high profile users and everyone else. The Wall Street Journal investigated Facebook’s special list.
Rich and powerful people’s profiles, such as politicians, journalists, and celebrities, are placed on a special list that exempts them from Facebook’s rules. The official terms for these shortlisted people are “newsworthy”, “influential or popular” or “PR risky” The special list is called the XCheck or “CrossCheck” system. Supposedly if these exempt people do post any rule breaking content, it is subject to review but that never happens. There are over 5.8 million people on the XCheck system and the list continues to grow:
The WSJ investigation details the process known as “whitelisting”, where some high-profile accounts are not subject to enforcement at all. An internal review in 2019 stated that whitelists “pose numerous legal, compliance, and legitimacy risks for the company and harm to our community”. The review found favouritism to those users to be both widespread and “not publicly defensible”.
Facebook said that the information The Wall Street Journal dug up were outdated and glosses over that the social media platform is actively working on these issues. Facebook is redesigning CrossCheck to improve the system.
Facebook is spouting nothing but cheap talk. Facebook and other social media platforms will allow rich, famous, and powerful people to do whatever they want on their platforms. It does not make sense why Facebook and other social media platforms allow this, unless money is involved.
Whitney Grace, September 20, 2021
Facebook and Social Media: How a Digital Country Perceives Its Reality
September 17, 2021
I read “Instagram Chief Faces Backlash after Awkward Comparison between Cars and Social Media Safety.” This informed senior manager at Facebook seems to have missed a book on many reading lists. The book is one I have mentioned a number of times in the last 12 years since I have been capturing items of interest to me and putting my personal “abstracts” online.
Jacques Ellul is definitely not going to get a job working on the script for the next Star Wars’ film. He won’t be doing a script for a Super Bowl commercial. Most definitely Dr. Ellul will not be founding a church called “New Technology’s Church of Baloney.”
Dr. Ellul died in 1994, and it is not clear if he knew about online or the Internet. He jabbered at the University of Bordeaux, wrote a number of books about technology, and inspired enough people to set up the International Jacques Ellul Society.
One of his books was the Technological Society or in French Le bluff technologique.
The article was sparked my thoughts about Dr. Ellul contains this statement:
“We know that more people die than would otherwise because of car accidents, but by and large, cars create way more value in the world than they destroy,” Mosseri said Wednesday on the Recode Media podcast. “And I think social media is similar.”
Dr. Ellul might have raised a question or two about Instagram’s position. Both are technology; both have had unintended consequences. On one hand, the auto created some exciting social changes which can be observed when sitting in traffic: Eating in the car, road rage, dead animals on the side of the road, etc. On the other hand, social media is sparking upticks in personal destruction of young people, some perceptual mismatches between what their biomass looks like and what an “influencer” looks like wearing clothing from Buffbunny.
Several observations:
- Facebook is influential, at least sufficiently noteworthy for China to take steps to trim the sails of the motor yacht Zucky
- Facebook’s pattern of shaping reality via its public pronouncements, testimony before legislative groups, and and on podcasts generates content that seems to be different from a growing body of evidence that Facebook facts are flexible
- Social media as shaped by the Facebook service, Instagram, and the quite interesting WhatsApp service is perhaps the most powerful information engine created. (I say this fully aware of Google’s influence and Amazon’s control of certain data channels.) Facebook is a digital Major Gérald, just with its own Légion étrangèr.
Net net: Regulation time and fines that amount to more than a few hours revenue for the firm. Also reading Le bluff technologique and writing an essay called, “How technology deconstructs social fabrics.” Blue book, handwritten, and three outside references from peer reviewed journals about human behavior. Due on Monday, please.
Stephen E Arnold, September 17, 2021
-
- Subscribe to Beyond Search
Feature archive
News archive
-
