Video Notes – Tools for Continuous Integration at Google Scale (GTAC2010)

Tools for Continuous Integration at Google Scale (GTAC2010)
Nathan York
29 min, 55 sec

Slide Notes:

  • Software Engineering Gap – lots of work on platforms and compilers, lots of work on apps. Middle (build systems, etc.) is often ignored (reach good enough and then move on)
  • Common Build System Issues – Incorrect, Slow, Cumbersome, Under-maintained
  • Why Build Systems Matter – Engineer Productivity, All about feedback
  • The Challenge At Google – 6000 engineers and one code base, everything built from source, development on mainline, extensive automated testing
  • Rough Developer Workflow (flow chart)
  • Better Build System Needed – optimized and tuned build languages, dependency analysis and scheduling, leverage infrastructure. Must be correct AND fast.
  • Inputs, Outputs and Actions – Content addressable storage (by digest of content), use relative paths, eliminate global state
  • Scaling Source Code Access – FUSE based file system. Most code needed for read only, on-demand syncing and caching, all source in the cloud, content digests as metadata
  • Making Builds Fast – Distributed builds in the cloud – built in arbitrary location
  • Scalable Distributed Builds – Caching key to scalable build. If inputs (from digest) and actions are same as previous, return prior result.
  • Scaling Build Outputs – FUSE based file systems, all output in the cloud, shared across builds and users
  • System View – builds appear local but are in the cloud
  • Platform for Automated Testing – Executing a test is just another build action.
  • Results – 20+ code changes per minute, 65K builds per day, 10000 CPUs, 50 TB memory, ~1PB output every 7 days, 94% cache hit rate.
  • Estimating build tool savings 2008 to 2009: Saving ~600 person years
  • Conclusion: build system is a core component of software engineering
  • Questions


29th Annual PNSQC Registration Now Open

I have written before about one of my favorite conferences – Pacific Nortwest Software Quality Conference. This year’s conference will be held October 10 – 12, 2011. The theme this year is "Delivering Quality" and the keynote speakers are Goranka Bjedov and Rob Sabourin. As usual the conference will be held at the World Trade Center Portland in Portland, OR.

Registration information can be found at


Test Heuristics Cheat Sheet

One of my team members was talking about test heuristics after attending the CAST 2011 conference. I went looking for Elisabeth Hendrickson’s excellent 2 page PDF entitled "Test Heuristics Cheat Sheet" and discovered the link I had not longer worked. A quick search revealed the current location is

This sheet is divided into the following sections:

  • Data Type Attacks
    • Paths/Files
    • Time and Date
    • Numbers
    • Strings
    • General
  • Web Tests
    • Navigation
    • Input
    • Syntax
    • Preferences
  • Testing Wisdom
  • Heuristics
    • Variable Analysis
    • Touch Points
    • Boundaries
    • Goldilocks
    • … and many more
  • Frameworks
    • Judgment
    • Observations
    • Flow
    • Requirements
    • Nouns & Verbs
    • Deming¬ís Cycle

I used to keep this pinned to wall above my desk. I should do it again. Recommended.


July 2011 mensming Twitter Posts

Follow mensming on Twitter

RT Conenza – Social Recruiting Infographic via @jobvite
26 Jul

The Top 10 Things Leaders Should Hear From Their Teammates |
26 Jul

RT AlgebraFact – ‘Algebra is the metaphysics of arithmetic.’ — John Ray
22 Jul

RT charrett – Product Manager says to CIO "If we keep performing the same tests, we will find the same bugs" she was quoting me…#MakingADifference
22 Jul

How to make Awesome Diagrams for your slides
22 Jul

Selenium Smells – Liberal use of sleep –
21 Jul

How Standards Proliferate <grin> –
20 Jul

Do you use these strategies? – Troy Hunt: The science of password selection
19 Jul

Finished reading Scott Berkun’s _The Myths of Innovation_ –
18 Jul

30 Creative 404 Error Pages | Flash User
14 Jul

Very useful – Andrew Beacock’s Blog: Pretty printing Java classpaths using Ant’s pathconvert task
13 Jul

Zynga’s Road to Riches: From startup to IPO – Tech News and Analysis
12 Jul

Finished Reading _Java Generics and Collections_ –
11 Jul

US Govt. plant USB sticks in security study, two thirds take the bait – TNW Industry
8 Jul

Developer error: The most dangerous programming mistakes | Application Development – InfoWorld
7 Jul

How Japan’s data centers survived the earthquake | ITworld
6 Jul

Finished Reading – _Eat People: And Other Unapologetic Rules for Game-Changing Entrepreneurs_ by Andy Kessler –
5 Jul

Posted: Build Speed Up After Java Upgrade(?) –
4 Jul

Quality of Apps for Android Is "Pathetically Low" Says Developer – Technology Review
1 Jul