Scaling Heavy Concurrent Writes In Real Time

Flickr implemented a real-time collection of referrers to Photos, streams, sets and collections. Currently the data is gathered real time for every single user on Flickr. The data is then cooked and reported with a 24 hour lag for filtering out referrer SPAM. MySQL myISAM/INNODB, curl, Java are the only component used in the setup and it scales linearly. This talk is about building a model for capacity planning, scaling for triple the request rate, and scaling linearly for an intensive application.

• Disk I/O is not an issue capacity is based off of Disk consumption rate.

• Java is used to buffer the writes to INNODB and myISAM

• The data is federated across many servers

• The goal is to keep unlimited stats for the lifetime of the pro account

• Transactions are used to keep the data insync among multiple table views.

• Detail data is kept on a per day, week, month, year for every view class recorded

• Distributed locks across the cluster using mySQL’s GET_LOCK