Taming Your Data: Practical Data Integration Solutions with Kettle

A few years ago, business Intelligence software in general, and data integration software in particular was very expensive. Therefore, it was only accessible to large enterprises.

Recently, this market has been severely disrupted by a number of open source products and companies. Combined with the success of a number of open source database products, data warehousing and business intelligence are more and more becoming a commodity that can successfully be used even by smaller businesses.

In this 45 minute session I will demonstrate a number of practical data integration solutions with Kettle. Kettle (a.k.a. Pentaho Data Integration) is an open source (LGPL), cross-platform data integration tool written in java. It is part of the open source Pentaho business intelligence suite (GPL), and can be run standalone or in conjunction with the Pentaho stack.

Topics include:

• extracting data from compressed text files (using internet movie database data)
• unfolding convoluted data formats into more sensible tabular forms (using internet movie database data)
• reading XML data into a database
• non-intrusively capturing data changes by reading database logs (using MySQL binary logs)
• Scaling-out: running the data integration solution in parallel on a Kettle cluster