techblog 

Some time ago I posted on a problem we were having on one of our ASM instances that is used in one of our RAC clusters. Finally after having the Service Request open with Oracle for 1 month, and having got its severity raised to p1 for a couple of weeks we got it fixed. Well I say fixed but it’s really more of workaround. So if you ever see a RAC database instance crash and have the following in your ASM alert log:

ORA-00600: internal error code, arguments: [kfgFinalize_2] 

here is how to get back up and running:

• shutdown asm on the node that has thrown the ORA-600

• shutdown your database instance on another node

• kill the asm instance on this other, working node. I did this with a kill -9 on the pmon process

• the killed asm instance will restart automatically (thanks to CRS) and will perform recovery

• try starting the asm instance that was giving the ORA-600

• check whether all diskgroups are mounted, you may need to manually mount some

It strikes me as a really odd way of fixing a problem, having to kill a running oracle process and crash your ASM instance. As we run only 2 node clusters it also means for some period of time your database is completely down which kind of goes against the RAC high availability goal. It did however work. Oracle swear it is not the same issue as bug 5393792, which is meant to be fixed in 10.2.0.3. The workaround is however identical as is the stack trace thrown by the issue.