techblog 

Databases are very important to us at Nominet. Our main database of domain names has been running Oracle RAC since March 2003, we like the idea of having no single point of failure.

We have been trying to use Dataguard technology and in particular logical standby to set up a replicated instance. Our big idea with this technology and the BIG selling point of logical standby is that you can have your replicated database open and available to query whilst you are updating it with the updates your primary database recieves.

We thought this would be a great mechanism for creating a reporting database, physically separated from the main database that could be used for high volumes of queries without impacting our primary database. There were a few hurdles to overcome as originally our primary RAC cluster was running 9i which did not have support for as many datatypes as we needed. There was also a snazzy new feature in 10g, called immediate apply. This was the killer sell, it means any update done to the primary can immediately be applied to your logical standby as soon as it arrives at the standby destination (prior to 10g, you would define a lag and the primary would switch logfiles on this duration, only at this point could you then apply redo that had been generated).

We were hooked - here was a mechanism of having a real time but offline (as in not pointing at primary db) reporting system. So we took plunge and upgraded to 10g, we are running veritas on our cluster so we had to wait for a patch to enable 10g to happily co-exist with Storage Foundation for Oracle RAC. After weeks of badgering it did appear.

Oh boy does clustering change from Oracle 9i -> 10g. Suffice to say it’s clear Oracle are trying to take business away from cluster vendors, in 9i you had to have vendor clusterware, but from 10g you can run your RAC cluster with only Oracle software, which means if you already have vendor clusterware you will have 2 pieces of software checking for cluster membership.

With 10g you get a rather flaky piece of software to do the cluster checking called Cluster Ready Services. Woe betide you if you try and run the first release of 10g, 10.1.0.2 because you can find yourself in endless rebooting loops. You have to devote aclustered partition to CRS that contains voting disks (of course we already had voting disks thanks to veritas)

So back to logical standby, you can now create this without any downtime on the primary. You first off create a physical standby, and then run a series of commands to go from physical standby (database not available for reporting while applying redo) to a logical standby, we even created standby redo logs just to have the recommended configuration. The final command starts the logical apply engine working:

SQL> ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;

The first time we went through this things started running very, very slowly - and I mean crawling.

Tip #1 Do NOT run the standby database with FULL transaction consistency

Changing to READ_ONLY (Though this is now depracated) makes things much much quicker!

This was only the beginning of the problems on our logical standby.