Some notes on Google development process
Monday October 3, 2011 by Derek Young
Last week and I went to Google’s Cambridge MA office to hear Steve Vintner and Matthew Gray speak as part of the Mass TLC member roundtable. This was really a two part discussion – Steve discussed hiring at Google and Matthew discussed development practices there. I found the hiring discussion mostly vanilla information that I’ve seen about google before (“hire the best people”, “always be interviewing”…). I’ll summarize only my impressions of Matthew’s talk, which was more interesting to me.
Matthew works in the Search Quality group, specifically on image and book search. He’s a technical lead which means he writes code and also has others who write code reporting to him.
The very first point he made was that they’re set up to do small, very frequent releases, often several times a day. They rely on fast iteration to direct the product. To support this, they use lots and lots of testing. The tests are written by the engineers. They have a very advanced continuous integration environment. He said there are some test that take a very long time to run which aren’t included in the continuous integration test runs but the majority of the tests are run for every commit.
Matthew discussed how uniformity helps Google – they have a single code base and a single set of strict coding standards that all the groups follow. They only develop in 5 languages – C++, Java, Javascript, Python, and Sawzall (their log analysis language). There is no Ruby or Perl for example. One exception to this is that they left ITA’s LISP codebase mostly untouched. Having a single codebase, limited set of languages, and strict style guidelines helps with code reviews, which I discuss in a minute.
Almost all code is developed on the trunk. He said they don’t use special branches often. Google uses a heavily customized version of Perforce, which I found interesting. They have tools to easily rollback releases that cause failures.
Matthew discussed “fixits and taxes”. A fixit is a period of time where the team works on improvements. They might have a fixit day or hour where everyone works on some sort of cleanup. Taxes come when they decide the code isn’t where they want it to be for some reason and they make a requirement that every checkin is “taxed”. The checkin must do some sort of improvement to the code, whether it’s where the code change was or somewhere else.
While the code itself seemed to have strict requirements, Matthew said the process varies from group to group. He said one group might say they have a scrum for example, but an outsider might say “hey, that’s just a meeting”. Not all groups follow the same development processes, but it sounded like they weren’t radically different either.
He spent a lot of time discussing code reviews. It’s clear code reviews are very important to Google and are really used as one of the primary forms of communication. They use an internal system called Mondrian, written by Guido of Python fame. Matthew said nearly 100% of all code is reviewed. In core areas like web search results, the code might be reviewed by six or more people. Some developers have “LGTM” authority, meaning “looks good to me”. In certain areas, code is not released without LGTM sign off. He described the code reviews as deeply critical and difficult for new developers. The entire history of code reviews is available to all developers as part of Mondrian.
Overall, an interesting talk.
