Fistful of Sand: Monitoring Code Performance at MySpace.com
Every good developer loves building a bonsai-site—a fast, lean paean to minimalism and efficiency. The DOM tree is carefully pruned, lovingly decorated with tasteful CSS. The browser welcomes it with a sigh of relief. The server’s CPU barely registers, and the database brushes delicately against the SAN, no lookup un-clustered.
Inevitably, such perfection is short-lived. Once the site is deployed, the snowball effect begins. A new panel is added on the top left, resulting in an database call. A new button is prominently displayed on the top navigation, resulting in increased traffic to a very heavy report-style page. A fade-in effect is added to the splash page, resulting in a beefy Javascript download for end users. Like a snowball rolling downhill, the site acrues performance-undermining features. MySpace.com is a several-year-old snow boulder and grows faster the larger it gets. In this talk, we will go over a particular performance monitoring tool we have built. The talk will be liberally dosed with ego-deflating vignettes illustrating our periodic and spectacular mishaps.
- System Overview
Over the years, we have witnessed many performance-related disasters at MySpace. These typically resulted in much running around, yelling over red-lined graphs, execution of hastily written scripts, sloppy downings of overheated coffee, and, in the end, the inevitable question: “How can this be prevented in the future?” Usually the answer to this question was, “if only we had avoided this database call.” “If only we had cached this lookup table.” “If only we had avoided this redundant lock.”
The road to performance Hell is paved with “if only”s.
The goal of the system we built is to provide developers with useful performance-related information before their code reaches our staging and deployment process. The system mixes a few hard and fast rules (“avoid inline script tags”) with analytics (“your page allocated 120kb of memory for the last ten months—after your latest commit it allocated 500kb”) to give the developer a sense of what has changed.
Every night the site is compiled and deployed to a series of testing servers. This process is totally automated and requires no developer intervention. The profiling consists of two main parts. Client side profiling is used to gauge the performance of the HTML payload in the browser. Server side profiling is used to profile how an individual web server will interact with the MySpace ecosystem.
- Client Side Profiling
The Client-side profiling portion of our system tracks everything that happens from the moment the HTML leaves our web server to the moment the page is completely rendered on the user’s browser.
This system is a combination of 2 components; an Internet Explorer Plugin that tracks the performance and HTML events on the user’s machine, and a custom HTTP Proxy which simulates a slow connection and injects custom javascript code into the tested page.
Using this system, we track and analyze the following data points:
- Render Time Profiling o Time it took for each HTML element to be rendered on the browser o Which element takes the longest to be rendered o Automatic identification of rendering bottlenecks o Which HTML/JavaScript code was executed during rendering of each element o Which files were downloaded during the rendering of each element
- Downloaded Data o Total download size o Size of external stylesheets o Size of external Javascript o Size of linked images
- Performance Data o Processor Time – CPU Usage on the client side (Browser’s process) o Private Working Set – Memory state of the browser during page rendering
- Source Validation o A complete list of HTML guidelines/rules which we want to avoid; such as “Don’t put <link> tags in the HTML’s “<body>“
Screenshot of Client Rendering Tree
Screenshot of Client Code Validation
- Server Side Profiling
The Server Side Profiler runs at the same time as the Client Side Profiler, tracing the execution and operating system interaction of the server code as the page is rendered. Below are examples of the data point tracked:
- Number of database calls
- Number of cache calls
- Number of external http requests
- Total number of method calls
- Redirects
- Memory allocations by calls and memory totals.
- Number of locks per request
- Developer Tool Bar
The “Developer Tool Bar” is a browser toolbar that gives the developer a complete real-time analisys of his front-end code performance. Our toolbar provides a duplicate method of testing the HTML payload before a developer checks in their source. This places the importance of performance as far forward as possible in the development cycle.
With this toolbar, our developers could -
- Profile the rendering of the page under various connection speed
- Easily identify rendering bottlenecks
- Test the size of the page’s elements
- View the load order of each element
- Take a snapshot of the browser’s CPU and memory footprint
- Validate the output HTML against our coding standards
Screenshot of Toolbar Validating Results
Screenshot of Toolbar Showing Browser Rendering Statistics
- Notification System*
The Notification portion of the system allows developers who integrate with the Performance system to notify the subscribed users based on cutomizable criteria.
Examples
- Server Side Performance Alerts o Number of Database calls increase o Number of Cache call increase o Amount of memory allocated per requests increases
- Client Side Performance Alerts o Download size increases o Browser memory working set increases o Average render time increases o Source Validation failures
People planning to attend this session also want to see:
Chris Bissell
MySpace
Chris is the Chief Software Architect at MySpace.com. In other words, he comes up with overly convoluted ‘design patterns’ in order to lengthen the development process.
Jeremy Custenborder
MySpace.com
Jeremy Custenborder has spent several years in the trenches at MySpace, resolving many bottlenecks and issues.
Yadid Ramot
MySpace.com
Yadid has spent much time mentoring the MySpace development community on client site performance.
Add a comment (requires login)
Schedule views
Sponsorship Opportunities
For information on exhibition and sponsorship opportunities at the conference, contact Sharon Cordesse at scordesse@oreilly.com
Download the Velocity Sponsor/Exhibitor Prospectus
Media Partner Opportunities
Download the Media & Promotional Partner Brochure (PDF) for information on trade opportunities with O'Reilly conferences or contact mediapartners@ oreilly.com
Press and Media
For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com
Velocity Newsletter
To stay abreast of conference news and to receive email notification when registration opens, please sign up for the Velocity Conference newsletter (login required)
Contact Us
View a complete list of Velocity contacts
O'Reilly Home | Privacy Policy | Contact Us | Sitemap | conf-webmaster@oreilly.com
Event Software Powered by Expectnation
© 2009, O'Reilly Media, Inc. | (707) 827-7000 / (800) 998-9938
Comments
A great talk. I liked the things they’re doing across the board. This gives me a lot of ideas about what I can do (even if I can’t exactly use all their awesome tech).
I did think there was a little bit too much here for one talk. But I guess the MySpace guys presented a lot as it is :)