Scaling MySQL and Java in High Write Throughput Environments

Kevin Burton (Spinn3r), Jonathan Moore (Tailrank/spinn3r)
Average rating: *****
(5.00, 1 rating)

We present the backend architecture behind Spinn3r – our scalable web and blog
crawler.

Most existing work in scaling MySQL has been around high read throughput
environments similar to web applications. In contrast, at Spinn3r we needed to
complete thousands of write transactions per second in order to index the
blogosphere at full speed.

We have achieved this through our ground up development of a fault tolerant
distributed database and compute infstructure all built on top of cheap
commodity hardware.

We’ve built out a number of technologies on top of MySQL that help enable us to
easily scale operations.

We’ve implemented an Open Source load balancing JDBC driver named lbpool.
(http://code.tailrank.com/lbpool). Lbpool allows us to loosely couple our MySQL
slaves which allow us to gracefully handle system failures. It also supports
load balancing, reprovisioning, slave lag, and other advanced features not
available in the stock MySQL JDBC driver.

We’ve also built out a sharded database similar to infrastructure built at other
companies such as Google (Adwords) and Yahoo (Flickr). Our sharded DB has a
number of interesting properties including ultra high throughput requirements
(we process 52TB per month), distributed sequence generation, and query plan
execution.

Photo of Kevin Burton

Kevin Burton

Spinn3r

Founder/CEO of Spinn3r, co-inventor of RSS, Apache contributor, and big data geek.

Jonathan Moore

Tailrank/spinn3r

Jonathan has been developing database driven application frameworks for over ten years. His work has been primarily based in perl, python, php and java. In the last decade he has been lead architect on six different frameworks that encompass applications ranging from social networks to educational simulators.

Sponsors
  • Kickfire
  • Zmanda, Inc.
  • Continuent
  • EDS
  • JasperSoft
  • Sun Microsystems
  • Symantec Anti-Virus Software
  • XAware
  • Data Direct Technologies
  • Dolphin Interconnect Solutions
  • Hewlett Packard
  • Infobright, Inc
  • Linagora
  • Microsoft
  • OpSource
  • Oracle
  • Pentaho
  • R1Soft
  • Red Hat
  • Ticketmaster
  • TechRepublic

Contact Us

View a complete list of MySQL contacts.