John Soer

Archive for June, 2010

Over the last few weeks have been working a lot with computer automation. The challenge with automation is to keep it running even when exceptions are happening.

The project I have been working on moves data on a regular interval between MySQL database and a SQL Server database. The MySQL database lives on a linux machine and is accessed via a SSH tunnel.

The following three rules will ensure that the automation runs quite reliable.

1. Be able to restart the process at any point.
2. Be able to process the same data without causing issues E.g. duplication
3. Be able to chunk the data. E.g. if the process has not run in a long time only process a limited number of records at a time.