PWN Your Infrastructure: Behind Call of Duty: World at War

Jason LaPorte (Agora Games)
General
Location: Pavilion 2 - 3
Average rating: **...
(2.86, 35 ratings)

I’ve included the outline I’ve prepared for the talk below. In it, I cover two technologies (NFS and Monit) and how building our infrastructure around them from the ground up allowed us to solve a number of scalability and convenience problems (specifically, software installation, application deployment, server configuration, and server monitoring) for our several-dozen machine virtualized network.

I. Introduction

A. Who is Agora? B. A Typical Rails Infrastructure C. Why Change It?

II. Using a Shared Filesystem (NFS)

A. Local Software (Ruby, Gems, Administration Scripts) B. Application Directories (the entire app directory, not just shared/) 1. Simplified Deploys 2. Simplified Page Caching C. Configuring NFS D. Notes 1. Web Server Should be the File Server i. Efficiency Concerns ii. Redundancy Concerns 2. Servers/Clients Should Share a LAN 3. Security Concerns i. File Ownership ii. Protocol Security

III. Deploys

A. Capistrano Has Problems 1. Synchronous Tasks (Across Machines) 2. Failures Not Localized 3. Limitations of Networking Create Failures (Especially as You Scale) 4. Scaling Architecture Requires Scaling Other Services (SVN, etc.) 5. Insecure By Default, Requires Shared User Privileges B. An Alternate Solution, Using a Shared Filesystem 1. Simple Bash Deploy Script 2. No Shared User Privileges Required 3. Relies on Monitoring Daemon

IV. Monitoring

A. Nagios/ZABBIX/Cacti/Monit/God/Munin Not Good Enough 1. None do Everything You Need – Process Monitoring, Usage Graphing, Server Configuration 2. Several Consume Excessive Resources 3. Combining Solutions (Monit + Munin) Still Insufficient, and Hackish B. Overlord 1. Rails App, Configures Monit, NFS 2. Monit Controls Everything Else (Server/Process Monitoring, Alerting, etc.) 3. Reading/Reporting Status From Monit (Undocumented XML Feeds) 4. Graphing: RRDTool is Awesome

V. Conclusion, Questions

Jason LaPorte

Agora Games

Jason LaPorte is the system administrator and a software developer at Agora Games, a company that specializes in building gaming communities, in-game on on the web.

Comments on this page are now closed.

Comments

Michael Gaffney
05/05/2009 1:42pm PDT

Some good system adminstration topics for managing multiple servers. In short, Automate!

News and Coverage
co-presented by Ruby Central, Inc. O'Reilly
  • Engine Yard
  • Heroku
  • Sun Microsystems
  • Blue Box Group
  • New Relic

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at RailsConf, contact Yvonne Romaine at yromaine@oreilly.com.

Download the RailsConf Sponsor/Exhibitor Prospectus

Media Partner Opportunities

Download the Media & Promotional Partner Brochure (PDF) for information on trade opportunities with O'Reilly conferences or contact mediapartners@ oreilly.com

Program Ideas

Post your suggestions for speakers, topics, and activities on the RailsConf wiki or send an email to rails-idea@oreilly.com.

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com.

Contact Us

View a complete list of RailsConf 2009 contacts.