April 2, 2011 by mensming.

Here are my notes from the the Velocity 2010 lecture "A Day in the Life of Facebook Operations"

A Day in the Life of Facebook Operations
Tom Cook, Facebook
Velocity 2010
June 22-24, 2010
(40 minutes, 48 seconds)

• Description of the size of Facebook in terms of minutes on site, pieces of new content, etc.
• User growth curve
• Server footprint growth curve
• Bay area and Virginia (and soon Oregon)

The stack:
Load Balancer -> (assigns a web server)
Web Server -> (assembles data)
Services (fast, complicated), Memcashed(fast, simple), Databases (slow, persistent)

Web server (HipHop for PHP)

• source code transformer
• converts PHP to C++, compiled with gcc

Memcached

• 300+ TB live data in RAM

MySQL

• persistent store
• lots of sharding
• facebook.com/MySQLatFacebook

Services

• news feed, search, chat, ads, media, etc.

Operations is supplying a platform for the Facebook developers to deploy
So, below the stack, we have:

Deployment, Monitoring
Systems Management
Core Operating System

Operating System

• Linux
• CentOS 5 variant with custom kernel

Systems Management

• Configuration management
  • Facebook uses CFengine
  • Update every 15 minutes, about 30 sec run on each machine
• On demand tools
  • No open source solution that meets Facebook needs (used to use DHS)
  • Wrote their own internal tool

Deployments

• Push for frontend code (web push)
  • At least once a day, frequently multiple times a day (bug fixes, etc.)
  • New features at least once a week
  • Built on top of on-demand control tools
  • Code distributed by BitTorrent (1 minute to push code to all servers)
• Backend deployments
  • Formal QA process removed, QA is responsibility of engineers
  • Engineers deploy their own code
  • No ‘commit and quit’ mentality
  • Ops ‘embedded’ into engineering teams
  • Change logging (every change, who, start time and end time)

Monitoring

• Ganglia (systems focus, graphing), (http://ganglia.sourceforge.net/)
• ODS (application focus), written by Facebook
• Nagios (ping, ssh, server up, etc.), alerting feeds into internal tools
• Aggregate alarms, drilldown capabilities

What Facebook operations deals with…

• Constant Growth
• Constant Failures

Look at network as logical units and dependencies

• Servers
• Racks
• Clusters (some thousand # of hosts)
• Data centers

Constant Communications

• IRC
• Internal news updates
• Banners on top of lots of tools with alerts as to current status
• Change logs / feeds
• Small teams

Recap

• Version control everything
• Optimize early
• Automate
• Use configuration mangement
• Plan to fail
• Instrument everything

Posted in operations, conferences | No Comments »

March 12, 2011 by mensming.

The following are my notes from this presentation at the Velocity 2010 conference:

The Top 5 Mistakes of Massive CSS
Nicole Sullivan, Consultant
Stoyan Stefanov, Google
2010 O’REILLY Velocity - Web Performance and Operations Conference
June 22 - 24, 2010

(Length 37 minutes, 54 seconds)

(Some demo isssues the first couple of minutes of the video…)

Why optimize CSS?

• It blocks progressive rendering
• Does not minimize as well as javascript

Top 1000 Alexa Sites Study

• 42% don’t gzip CSS
• 44% more than 1 CSS file
• 56% serve CSS with cookie
• 62% don’t minify
• 21% have > 100K of CSS

Object oriented CSS is a way to make CSS smaller.

• Concentrate on the selector
• Example: Grids projects, 574 butes, 14 lines

Granularity Fail + Stale Rules + Unpredicatbility + Duplication + Specificity Wars = Massive CSS

#5 Granularity Fail

• should not match underlying programming object model
• compare to legos - use CSS similarly
• How to determine if correct
• 1) Perform a visual inventory
  • Take a site object
  • Break it down into component CSS objects
  • Look for repeating patterns, for example headings
• 2) Develop library of component parts

#4 State Fails

2 types of stale:

• Truly stale - no longer in use
• Rules used after user action or on subsequent pages

• Try dust-me selectors
• Track over time

#3 Unpredictability

• Elements should be consistent across site (like lego bricks that don’t change behavior depending on where they are used…)
• How to test? Search for repeating definitions of same item (for example grep h[1-6]

#2 Specificity Wars

• Programmers making items more specific in order to prevent others from overriding their settings

How to simplify:

• Use hacks sparingly
• Avoid styling ids
• Avoid !important

What to do

• Style classes rather than elements
• Give rules the same strength

#1 Duplication

• Grep is your friend
• grep for margin set to zero - too many, may need a reset stylesheet
• grep for float - too many, may need grid system
• grep for font-size - may be heading disguised as something else
• Example from Facebook

Posted in performance, operations, web testing, conferences | No Comments »

August 28, 2010 by mensming.

I often use Microsoft’s remote desktop software to connect to various machines. My hands also use shortcuts for various actions via pure muscle memory. However, when using remote desktop, the operating system interprets many keystrokes as destined for the host operating system. This makes perfect sense but usually sends me scrambling to find the keyboard equivalent for the remote session.

The shortcuts are documented on various websites and in Microsoft’s own documentation. I will add yet another page on the web with this information purely for my own convenience.

Posted in operations, IT | No Comments »

June 12, 2010 by mensming.

Many months ago, there was Slashdot posting regarding a video discussing how Facebook runs its LAMP stack. I finally got around to watching the video and it is worthwhile to view if you have an interest in how to run a high volume web site. In the video, Aditya Agarwal - Directory of Engineering at Facebook - describes the architecture and the lessons learned from scaling the site.

Recommended - view here.

Posted in operations, social media | No Comments »