Using MySQL Cluster in a High Volume Email Environment

UC Berkeley recently migrated to an open source/free software email system based on Exim, Cyrus, and SquirrelMail. Because users are especially intolerant of downtime when it comes to email, all of our systems are built with redundancy and fault tolerance in mind. Many email systems use LDAP to store account data (often piggybacking on enterprise directory data that are already present), and ours did at first as well. But we soon found that our enterprise LDAP system wasn’t able to meet the reliability and performance requirements demanded by an email system that processes 4 million messages per day.

We built a MySQL cluster on hardware that we were already using to power the mail system itself. NDB data nodes share space with Cyrus on our backend mail storage machines, and MySQL servers acting as API nodes run on each of our redundant frontend MTA servers. The result is an extremely stable database for our account data that is easily used by all of the components of the email system.

In addition to an overview of how we designed and built the cluster, including schema challenges, we will also show how we integrate it into our application components with code and configuration samples:

• We show how Exim can be configured to use MySQL to create a completely dynamic mail routing system.

• We implement the Postgrey greylisting algorithm using only MySQL stored procedures that are called from Exim.

• We use MySQL cluster to make user preference data available to our four webmail servers.

• Model-view-controller based web frameworks are a great way to write web applications that interact with complex databases. We use TurboGears, and will show how we use the SQLAlchemy ORM and Python to make short work of managing our account data.