techblog 

Some time ago I posted on a problem we were having on one of our ASM instances that is used in one of our RAC clusters. Finally after having the Service Request open with Oracle for 1 month, and having got its severity raised to p1 for a couple of weeks we got it fixed. Well I say fixed but it’s really more of workaround. So if you ever see a RAC database instance crash and have the following in your ASM alert log:

ORA-00600: internal error code, arguments: [kfgFinalize_2] 

here is how to get back up and running:

• shutdown asm on the node that has thrown the ORA-600

• shutdown your database instance on another node

• kill the asm instance on this other, working node. I did this with a kill -9 on the pmon process

• the killed asm instance will restart automatically (thanks to CRS) and will perform recovery

• try starting the asm instance that was giving the ORA-600

• check whether all diskgroups are mounted, you may need to manually mount some

It strikes me as a really odd way of fixing a problem, having to kill a running oracle process and crash your ASM instance. As we run only 2 node clusters it also means for some period of time your database is completely down which kind of goes against the RAC high availability goal. It did however work. Oracle swear it is not the same issue as bug 5393792, which is meant to be fixed in 10.2.0.3. The workaround is however identical as is the stack trace thrown by the issue.

I’ve just got a Nokia E65 to use the built-in VoIP functionality and like most people I can’t really start to use it until I’ve got all my contacts transferred over. I don’t fancy typing in over 100 names and numbers again.

On OSX there is a lovely application called iSync that I use to synchronise data between computers (using .mac) and various devices. So it seemed obvious to synch my new E65 with iSync and away I go.

The first thing I did was to download the Nokia plugin for iSync and install it. This was painless and iSync immediately recognised the phone and configured itself.

However when I first tried to sync (using USB) I got this horrible error message:

[Nokia E65] Device is not available or synchronization was cancelled. Close dialogs and applications on the phone and try again.
Device “Nokia E65″ synchronization failed

I tried all sorts of things to fix it, including attempting to reformat the E65. It turns out that unlike other S60 devices, you can’t actually reformat the E65 all you can do is reset the data.

Then I noticed in the online synchronisation manual that there should be a default synchronisation profile for PC Suite but my list of synchronisation profiles was empty. I had tried adding one already but been daunted by the questions. So I borrowed an E65 from a colleague and found their phone did have one.

The settings for this default PC Suite profile are:

and with those settings using iSync worked fine over USB, even though the profile is for Bluetooth. If you also want to do Calendar or Bookmarks then replace Contacts with Calendar or Bookmarks at the appropriate place.

We have found that dbSwing components (JdbTable, JdbTextField, JdbLabel etc) do not display dates when the application is run using JRE 1.6. The dates display correctly under JRE 1.5 and earlier.

The problem is caused by the method

protected String getDefaultPattern(int variantType)

in com.borland.dx.text.VariantFormatStr. In JRE 1.5 and earlier if no Formatter or Displaymask is set by the program, the function above loads a resource by

SystemResourceBundle.getLocaleElementsBundle(locale);

which gets the following Resourcebundle

sun.text.resources.LocaleElements

This bundle does not exist in 1.6 and an exception is thrown which is handled by setting the default Pattern to “”. These is the reason why nothing is displayed for date, time or timstamp columns.

To work round this we therefore need to set the displayMask for such columns e.g.

yourDateColumn.setDisplayMask("dd/MM/yyyy");

and dates are displayed correctly.

I’ve been reading and thoroughly enjoying Scalable Internet Architectures by Theo Schlossnagle. A few thoughts struck me quite forcefully. First off a lot of the techniques mentioned were quite familiar to us here at Nominet. In particular we are using ipvs in combination with keepalived to perform loadbalancing of a number of our critical public facing applications. Mostly this load balancing technique has performed exceedingly well, though there has been a problem when one of the servers (we run them in pairs) running keepalived died.

I liked the notion that High Availability is a different concept from load balancing, I suspect a lot of people may confuse these terms. The discussion I really liked though, was on peer-based high availability using Wackamole and Spread. Having done a quick straw poll it seems this is not a terribly well known piece of software. This technique for high availability never occurred to me and to my mind it seems a great way of reducing the number of servers you are dependent on in your architecture.

The other thing that stuck in my mind was how often Theo mentions cost of the infrastructure, in some ways I find this surprising in that companies that truly need 24×7 should surely have the resources to obtain expensive hardware, and do those that can’t afford it, actually need it? In the Oracle world there is a lot of talk regarding companies being sold RAC for high availability when they don’t actually need what RAC gives them. In truth a lot of companies can survive being down for a day or so quite easily, they may just not be able to admit this to themselves. There is also an argument that RAC done badly can reduce your uptime.

We have a significant investment in Oracle database technology running several Oracle RAC clusters. In addition to the cost of an Oracle license, there is an ongoing yearly payment for support that entitles us to upgrades, patches and perhaps most importantly, the ability to log so called service requests stating what problem you have and an Oracle support analyst tries to fix your issue. Unfortunately Oracle support went to a global quite a few years and instead of having an analyst in the UK look at the problem, we seem to have analysts from a variety on countries in EMEA. I’ve found this really leads to a huge variation in the quality of analysts assigned to our cases.

Currently we have 6 outstanding cases, three of which are now 2 months old and still awaiting a fix. Mostly these issues have arisen due to us having applied the latest Oracle patchset, 10.2.0.3. Here is a list of our current issues:

• Flashback database fails with ORA-38761, bug:5914784 and has been with Oracle for 5 weeks.

• ORA-600: [kcrfr_resize2], bug:3306010 Oracle have worked on this for 2 months and counting.

• ORA-00355: change numbers out of order, bug:5935278 another 2 month old case.

• ORA-600: [kcrrupirfs.20], bug: 4767278, there is a patch available but I’m still waiting an Oracle response.

• ORA-600: [kfgFinalize_2], bug: 5393792 which is meant to be fixed, but does not appear to be.

• redo transport failing with ORA-16011 fixed by us but still waiting for Oracle to come up with anything.

It was this final case, the ORA-16011 problem that prompted this posting, as we must have ended up with a truly inept support analyst for this case. As stated this problem was regarding sending redo from our primary to our standby. Connectivity between primary and standby was fine and I could use sqlplus to connect to primary -> standby and standby -> primary. The analyst (after I told them I fixed it) asked for the tnsnames entry for our physical standby that the primary was using to connect to it:

STANDBY =
(DESCRIPTION =
(SDU=32767)
(SEND_BUF_SIZE=375000)
(RECV_BUF_SIZE=375000)
(ADDRESS_LIST =
(ADDRESS = (PROTO = TCP)(HOST = localhost)(PORT = 9024))
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 9025))
)
(CONNECT_DATA =
(SERVICE_NAME = STANDBY)
)
)

I also posted the following:

oracle 25366 1 0 Apr25 ? 00:00:00 ssh -N -L 9024:standby-node1:1521
oracle 20759 1 0 Mar04 ? 00:02:18 ssh -N -L 9025:standby-node2:1521

Basically we are sending the redo data to our standby over an ssh tunnel and the tnsnames entry says to connect to the localhost on the ports that the ssh tunnels are forwarding on. This works extremely well at securing the transport of the redo and is completely transparent to the database.

After seeing this information, the support analyst on this case asked “what does localhost mean”. I was utterly speechless to think we are paying money to Oracle to explain to their analysts basic computing terminology.

I’m working on a project involving RSS feeds, and I am interested to know what people are using to read them. Looking at the statistics for Feedburner, it is clear that the majority of RSS Feeds are being read in My Yahoo, Google Reader or Bloglines, but I would like to find out what else is being used, particularly in terms of offline readers. So if you are a user of RSS feeds, please let me know in the comments what tools you are using.