Facebook Runs Wide Open

July 28, 2010

Open source technology, once relegated to the furthest reaches of computer geekdom, is helping over 500 million people a day share status info, view photos and even poke a friend, they just don’t know it. Facebook, the King Kong of social media, has embraced open source tools, especially Lucene products, on several different levels so it works faster and smarter. A recent interview with Facebook’s senior open programs manager, David Recordon, for Developer.com “Inside Facebook’s Open Source Infrastructure,” revealed a surprising pile of open source applications. According to the piece, “Facebook’s open source Web serving infrastructure has a lot more than just the traditional LAMP (Linux/Apache/MySQL/PHP) stack behind it.”

The company taps Apache and Lucene. Cassandra, an Apache database project, is utilized heavily by the site. It is one of three open source databases used for storing information and helping the process run smoother. “While we store the majority of our user data inside of MySQL,” Recordon said. “We have about 150 terabytes of data inside of Cassandra, which we use for inbox search on the site and over 36 petabytes of uncompressed data in Hadoop overall.”

With such a well-planned method for storing data by using open source programs, it only makes sense the data analysis is handled in a similar fashion. Here, Apache Hive technology is utilizedin a major way.

“A large part of our infrastructure is open source and we really think that it’s important in terms of being able to allow developers that are building with the Facebook platform to scale using the same pieces of infrastructure that we use,” Recordan said.

Facebook is arguably one of the most important companies of our time. Few sites have changed the way we spend time at and away from the computer. So its warm embrace of open source technology feels like a sign of big things to come as companies like Lucene gain more recognition. What’s interesting is that Facebook has a number of Googlers pumping their DNA into Facebook. The technical decisions at Facebook are different from those made at Google. Facebook does social pretty well. Google does not, at least yet. Is there a message here? Beyond Search will do an Overflight at some point.

Stephen E Arnold, July 28, 2010

Freebie

Written by Stephen E. Arnold · Filed Under Business strategy, Facebook, News, Open source, Technology

Comments

Comments are closed.

Search the site
Subscribe to Beyond Search
Feature archive
News archive

Stephen E. Arnold monitors search, content processing, text mining and related topics from his high-tech nerve center in rural Kentucky. He tries to winnow the goose feathers from the giblets. He works with colleagues worldwide to make this Web log useful to those who want to go "beyond search". Contact him at sa [at] arnoldit.com. His Web site with additional information about search is arnoldit.com.