Archive for the ‘operations’ Category

Video Notes – Robert Johnson interviewed at Velocity 2010

Monday, March 19th, 2012

Robert Johnson interviewed at Velocity 2010
Robert Johnson, Director of Engineering, Facebook
2010 O’REILLY Velocity – Web Performance and Operations Conference
June 22 – 24, 2010
7 min, 4 sec
http://www.youtube.com/watch?v=wXTCPnuDGbg

My Notes:

  • User expectations of mobile in terms of optimization
    • Very little data to answer scientifically
    • Long term need desktop and mobile site performance to come together
  • Need to collect the data before setting benchmarks
  • Acceleration and Business Benefits Research
    • Huge payback for performance improvement
  • Initial thing that a company should do
    • Set benchmarch and understand how user sees the site
    • Keep alives on
    • Compression on
  • Workshop
    • Analyzed Velocity home page
    • Made worse than reality and brought it to current then improved

 

 

Velocity 2010: Tom Cook, “A Day in the Life of Facebook Operations”

Saturday, April 2nd, 2011

Here are my notes from the the Velocity 2010 lecture "A Day in the Life of Facebook Operations"

A Day in the Life of Facebook Operations
Tom Cook, Facebook
Velocity 2010
June 22-24, 2010
(40 minutes, 48 seconds)

  • Description of the size of Facebook in terms of minutes on site, pieces of new content, etc.
  • User growth curve
  • Server footprint growth curve
  • Bay area and Virginia (and soon Oregon)

The stack:
Load Balancer -> (assigns a web server)
Web Server -> (assembles data)
Services (fast, complicated), Memcashed(fast, simple), Databases (slow, persistent)

Web server (HipHop for PHP)

  • source code transformer
  • converts PHP to C++, compiled with gcc

Memcached

  • 300+ TB live data in RAM

MySQL

  • persistent store
  • lots of sharding
  • facebook.com/MySQLatFacebook

Services

  • news feed, search, chat, ads, media, etc.

Operations is supplying a platform for the Facebook developers to deploy
So, below the stack, we have:

Deployment, Monitoring
Systems Management
Core Operating System

Operating System

  • Linux
  • CentOS 5 variant with custom kernel

Systems Management

  • Configuration management
    • Facebook uses CFengine
    • Update every 15 minutes, about 30 sec run on each machine
  • On demand tools
    • No open source solution that meets Facebook needs (used to use DHS)
    • Wrote their own internal tool

Deployments

  • Push for frontend code (web push)
    • At least once a day, frequently multiple times a day (bug fixes, etc.)
    • New features at least once a week
    • Built on top of on-demand control tools
    • Code distributed by BitTorrent (1 minute to push code to all servers)
  • Backend deployments
    • Formal QA process removed, QA is responsibility of engineers
    • Engineers deploy their own code
    • No ‘commit and quit’ mentality
    • Ops ‘embedded’ into engineering teams
    • Change logging (every change, who, start time and end time)

Monitoring

  • Ganglia (systems focus, graphing), (http://ganglia.sourceforge.net/)
  • ODS (application focus), written by Facebook
  • Nagios (ping, ssh, server up, etc.), alerting feeds into internal tools
  • Aggregate alarms, drilldown capabilities

What Facebook operations deals with…

  • Constant Growth
  • Constant Failures

Look at network as logical units and dependencies

  • Servers
  • Racks
  • Clusters (some thousand # of hosts)
  • Data centers

Constant Communications

  • IRC
  • Internal news updates
  • Banners on top of lots of tools with alerts as to current status
  • Change logs / feeds
  • Small teams

Recap

  • Version control everything
  • Optimize early
  • Automate
  • Use configuration mangement
  • Plan to fail
  • Instrument everything

 

 

Velocity 2010: Nicole Sullivan, “The Top 5 Mistakes of Massive CSS”

Saturday, March 12th, 2011

The following are my notes from this presentation at the Velocity 2010 conference:

The Top 5 Mistakes of Massive CSS
Nicole Sullivan, Consultant
Stoyan Stefanov, Google
2010 O’REILLY Velocity – Web Performance and Operations Conference
June 22 – 24, 2010

(Length 37 minutes, 54 seconds)

(Some demo isssues the first couple of minutes of the video…)

Why optimize CSS?

  • It blocks progressive rendering
  • Does not minimize as well as javascript

Top 1000 Alexa Sites Study

  • 42% don’t gzip CSS
  • 44% more than 1 CSS file
  • 56% serve CSS with cookie
  • 62% don’t minify
  • 21% have > 100K of CSS

Object oriented CSS is a way to make CSS smaller.

  • Concentrate on the selector
  • Example: Grids projects, 574 butes, 14 lines

Granularity Fail + Stale Rules + Unpredicatbility + Duplication + Specificity Wars = Massive CSS

#5 Granularity Fail

  • should not match underlying programming object model
  • compare to legos – use CSS similarly
  • How to determine if correct
  • 1) Perform a visual inventory
    • Take a site object
    • Break it down into component CSS objects
    • Look for repeating patterns, for example headings
  • 2) Develop library of component parts

#4 State Fails

2 types of stale:

  • Truly stale – no longer in use
  • Rules used after user action or on subsequent pages

  • Try dust-me selectors
  • Track over time

#3 Unpredictability

  • Elements should be consistent across site (like lego bricks that don’t change behavior depending on where they are used…)
  • How to test? Search for repeating definitions of same item (for example grep h[1-6]

#2 Specificity Wars

  • Programmers making items more specific in order to prevent others from overriding their settings

How to simplify:

  • Use hacks sparingly
  • Avoid styling ids
  • Avoid !important

What to do

  • Style classes rather than elements
  • Give rules the same strength

#1 Duplication

  • Grep is your friend
  • grep for margin set to zero – too many, may need a reset stylesheet
  • grep for float – too many, may need grid system
  • grep for font-size – may be heading disguised as something else
  • Example from Facebook

 

 

Remote Desktop Keyboard Shortcuts

Saturday, August 28th, 2010

I often use Microsoft’s remote desktop software to connect to various machines. My hands also use shortcuts for various actions via pure muscle memory. However, when using remote desktop, the operating system interprets many keystrokes as destined for the host operating system. This makes perfect sense but usually sends me scrambling to find the keyboard equivalent for the remote session.

The shortcuts are documented on various websites and in Microsoft’s own documentation. I will add yet another page on the web with this information purely for my own convenience.

Display Windows Security dialog CTRL+ALT+DEL CTRL+ALT+END
Start Menu CTRL-ESC ALT+HOME
Windows Menu N/A ALT+DELETE
Action Host Remote
Switch between programs ALT+TAB ALT+PAGE UP
ALT+SHIFT+TAB ALT+PAGE DOWN
ALT+ESC ALT+INSERT
Copy snapshot of active window to clipboard PRINT SCREEN CTRL+ALT+MINUS (on keypad)
Copy snapshot of full session to clipboard ALT+PRINT SCREEN CTRL+ALT+PLUS (on keypad)

 

Facebook’s Lamp Stack

Saturday, June 12th, 2010

Many months ago, there was Slashdot posting regarding a video discussing how Facebook runs its LAMP stack. I finally got around to watching the video and it is worthwhile to view if you have an interest in how to run a high volume web site. In the video, Aditya Agarwal – Directory of Engineering at Facebook – describes the architecture and the lessons learned from scaling the site.

Recommended – view here.