Google: Single Point of Failure Engineering
September 18, 2015
Do you recall the lightning strike at the Alphabet Google’s data center in Belgium? Sure you do. Four lightning strikes caused the data center to lose data. See “Lightning in Belgium Disrupts Google Cloud Services.” I asked myself, “How could a redundant system, tweaked by AltaVista wizards decades ago, lose data?”
When I was assembling information for the first study in my three part Google series, I waded through many technical papers and patent documents from the GOOG (now Alphabet). These made clear to me that the GOOG was into redundancy. There were nifty methods with clever names. Chubby, anyone?
Now the Belgium “act of God” must have been an anomaly. Since 2003, the GOOG should have been improving its systems and their robustness. Well, maybe Belgium is lower on the hardened engineering list?
I found this article quite interesting: “Google Is 2 Billion Lines of Code. And It Is All in One Place.” Presumably the knowledge embodied in ones and zeros is not in one place. Nope. The code is in 10 data centers, kept in check with Piper, a home brew code management system.
But, I noted:
There are limitations this system. Potvin [Google wizard] says certain highly sensitive code—stuff akin to the Google’s PageRank search algorithm—resides in separate repositories only available to specific employees. And because they don’t run on the ‘net and are very different things, Google stores code for its two device operating systems—Android and Chrome—on separate version control systems. But for the most part, Google code is a monolith that allows for the free flow of software building blocks, ideas, and solutions.
No lightning strikes are expected. What are the odds for simultaneous lightning strikes at multiple data centers? Not worth worry about this unlikely disaster scenario. Earthquake? Nah. Sabotage? Nah.
No single point of failure for the Alphabet Google thingy. Cloud services just do not lose data most of the time. The key word is “most.”
Stephen E Arnold, September 18, 2015
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
