Part 1 – The Objective:

We recently set ourselves the goal of streamlining the database for ease of management & development. For those who’s production databases are in the multi-gigabyte size category, you will know what a drama it is to get the dev environment looking as similar as possible to live.

Anyway, our sites get a lot of visits which we like to log and make pretty graphs from. The downside of this from a system maintenance & performance perspective is that a lot of data gets generated. When I say a lot I’m talking gigabytes of space housed in the database, just for the site visit stats.

So, with this in mind we focussed on the stats related tables in our streamlining efforts. We decided to undertake a flavour of Functional Partitioning (FP). I say “flavour” as this strategy can be applied in a variety of ways and to various degrees.

In most cases you perform FP if there is a performance bottleneck in the IO. In our case the objective was to create a more portable (data size) database. I realise it is possible to selectively export/import tables but really, we just wanted a small-as-possible snapshot of the whole database that could easily be imported back into the dev environment.

As I mentioned, Functional Partitioning works on a sliding scale in terms of its implementation. In the fullest degree of FP you dedicate an entire db server to the functionally partitioned data. For example, the system would have a separate server for content, images and whatever other different functional areas you needed. In our situation we didn’t go as far as a new server for the partitioned data, only creating a new database on the same server for the purpose.

The next step in this process was deciding how to best get the data there, then adjusting the application to start recording the visits in the new location (without data loss or downtime).

In order to tackle this problem we identified 3 phases:

• Migrate the data into the new db (preparation).

• Change the live code to look & log in the new location (time sensitive).

• Tidy up the system (post release).

I’ll go into more detail on each of these phases in the next instalment of this post…