Google, Its Linux, and Some Open Source Angles
November 10, 2009
What a nice write up! Navigate to LWN.net and read “KS2009: How Google uses Linux”. The write up has a wealth of information. I am not sure that the Googler who presented the information summarized in the write up is spot on across the topics referenced. I do think the write up provides some insight into the chaotic, almost loose approach to certain technical processes at Google. Keep in mind that there are 19,000 people and that certain units may have more or less stringent work processes in place. Add to that, the segregation of certain Google initiatives. In short, Google is wild and crazy, but it has folks who know how to lock down certain methods and procedures.
Nevertheless, you will get quite a bit of useful information in the article. I want to point out one passage that struck me as important:
In the area of CPU scheduling, Google found the move to the completely fair scheduler to be painful. In fact, it was such a problem that they finally forward-ported the old O(1) scheduler and can run it in 2.6.26. Changes in the semantics of sched_yield() created grief, especially with the user-space locking that Google uses. High-priority threads can make a mess of load balancing, even if they run for very short periods of time. And load balancing matters: Google runs something like 5000 threads on systems with 16-32 cores.
Yep, scale.
When you read the article, be sure to scan the comments. I found this comment particularly stimulating to my thinking:
jmm82 (subscriber, #59425) [Link] I believe the reasons were outlined why they are not contributing code into the kernel. 1. They are not using kernels that are close to linus git head. 2. Some code would not be wanted in the mainline kernel. 3. Some code is not good enough to get into the mainline kernel. 4. They don’t want to have 30 people saying the code will only get in if it does this. Aka. They don’t want to make it support the features they are not using. 5. Some code is proprietary and they want to protect the IP. As was stated above as long as they are not distributing the code the changes are their property. 6. A lot of there patches are code backported from mainline, so it is already in the kernel. I think moving forward that you will see Google have a few developers working on mainline to try and influence future kernels because it will be financially cheaper to carry as few patches as possible. Also, I feel they will always have some patches that they feel are too valuable IP to gave back and will continue to maintain those outside the mainline.
Google and its open source “issues” may be an interesting topic for an investigative soul to explore.
Stephen Arnold, November 10, 2009
Comments
2 Responses to “Google, Its Linux, and Some Open Source Angles”
-
- Subscribe to Beyond Search
Feature archive
News archive
-
Regarding the last comment you quote, this seems perfectly reasonable to me. If an organisation is using open source code internally with their own patches, they will have to make the choice between the value gained from contributing the patches back to the community and retaining the IP developed in house. It’s a common myth that open source licensing forces you to show everything you have to the world.
What’s troubling is that companies, like Google, want it both ways with open source. On the one hand, they claim to be good open source citizens, while, on the other hand, picking their moments to “give back”.
I blogged about this theme some time ago (“The Open Source Red Herring” at http://www.martinsuter.net/blog/2008/01/the-open-source-red-herring.html).
Excerpt:
Under what terms does Google license Linux? How much does it “give back”? I’m suggesting that it is highly selective as to what it “gives back” to the open source community, and makes these decisions based on its own self-interest, not out of some altruistic motive. Google search algorithms are deeply guarded secrets, and its vaunted server farm architecture is “proprietary”. Is there anything wrong with this? Of course not. Just don’t pretend that you’re somehow better than the “commercial” vendors because you promote open source development opportunistically.
“This is what [Summer of Code] is really about: infecting students with the free software spirit, giving them the opportunity to grow into a community like ours.”
There’s another more subtle benefit, as DiBona explains. Thanks to the Summer of Code, “Google now knows all the people working on all these software projects, on which it depends,” he says. “That’s incredibly useful to us. Every once in a while we’ll come out with a new API and there’ll be some projects in the open source world that might be useful in either using that API or being a customer. You can just call them up and say, ‘hey guys, it’s Google, we’re you’re pal,’ and let them just check it out.” (http://redmondmag.com/features/article.asp?editorialsid=2395)
I have never heard, nor do I ever expect to hear, any allusions from Microsoft that it is anything but a profit-driven, commercial software vendor. As a shareholder, I expect nothing less.
That’s my .02!
Martin