Sharding Using Spockproxy: A Sharding-only Version of MySQL Proxy

Frank Flynn (Spock Networks)
Architecture and Technology
Location: Ballroom E
Average rating: ***..
(3.60, 5 ratings)

Spock.com is a people search engine serving millions of queries per day. In order to grow our back-end to service these queries against a billion records (and growing) we developed Spockproxy.

Spockproxy is a modified version of MySQL proxy which does sharding inherently. It only does sharding; no Lua or anything else. It is very easy to set up; all configuration information is kept in 3 configuration tables. See http://spockproxy.sourceforge.net/ for documentation and downloads.

I propose to cover how to layout the databases shards, what is universal data as opposed to federated data and how we use replication to copy universal data into each shard. Also how to set up replication to provide an additional read slave proxy as well as how to use these replicants for high availability.

We’ll also cover the limitations of Spockproxy and the issues of Sharded databases that we’ve run into as well as how we’re got around them.

Our problems and the solutions to them should be interesting to anyone who is looking to introduce Sharding to their environment even if they choose not to use Spockproxy at all.

Photo of Frank Flynn

Frank Flynn

Spock Networks

Frank Flynn is a Database Architect and Operations Manager at Spock Networks. He specializes in MySQL programming and working miracles wherever possible.

Comments on this page are now closed.

Comments

04/22/2009 5:06pm PDT

Beautifuly presented. Very good description of the Spock proxy rational, implementation and limitations.

Co-presented By:

O'Reilly Media MySQL/Sun Microsystems
  • Kickfire
  • Virident
  • Infobright, Inc
  • JasperSoft
  • Intel
  • Advanced Micro Devices
  • BIRT Exchange by Actuate
  • Calpont
  • Canonical
  • Continuent
  • Dolphin Interconnect Solutions
  • Facebook
  • HiT Software, Inc.
  • IBM
  • iDashboards
  • Oracle
  • Pentaho
  • R1Soft
  • Schooner Information Technology
  • SQLstream
  • Ticketmaster
  • Zmanda, Inc.
  • Linux Journal

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Sharon Cordesse at scordesse@oreilly.com

Download the MySQL Sponsor/Exhibitor Prospectus

Media Partner Opportunities

Download the Media & Promotional Partner Brochure (PDF) for information on trade opportunities with O'Reilly conferences or contact mediapartners@ oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

MySQL Conference Newsletter

To stay abreast of conference news and to receive email notification when registration opens, please sign up for the MySQL Conference newsletter.

Contact Us

View a complete list of MySQL contacts.