Fun, games and mayhem

Well, the first disaster thus far has been the backup server 'not playing nice'. One tape drive jamming, and another totally dead.

No wonder there'd been 'a few problems' with it. Following on from yesterdays idea for doughnuts to the backup server, can you imagine the look on the support engineer's face when he comes to clean a "jammed" tape drive with a screwdriver, only to realise that we actually mean it has been jammed in the more literal sense :)

Tuesday night on EVE marked my first player kill. OK, so it was one of the events team, on a mission to recover some stolen star charts, but never the less, it was good fun, and a major adrenaline rush. More there I think. (Got the kill on a cruisers, and an assist on the battleship.)

Maelstrom fast approaches, being on friday. I am now able to drive, and I don't anticipate having problems doing so to Maelstrom. My concern is that I'm going to be of marginal use with setting up and striking camp. So I'm rather afraid my contribution towards group activity will be along the lines of supplying beer (or other supplies). I'm fairly certain, I'm afraid, that I'll need assistance even putting up my tent. (Although it'll be that usefully large '2 person + standing up room' so can offer a place to sleep).

This may change in the next week, and if it does, I'll be more than delighted, but assuming it doesn't, is there someone who wouldn't mind lending me a hand to set up?

Oh, and whilst I remember, go vote today. The choices may not be appealing, but writing poetry or "none, they're all wankers" across the ballot slip remains an option.

Why I like Solaris

We have had a problem on Solaris. The machine was 'running badly'.
Load average of around 20-30. ('normal' is 1 per processor) but nothing showing up 'hogging' the processors. Highest process was running at 30% cpu, with the occasional jump to 70% when it re-ran it's indexing. This was normal, although just in case, we disabled it, and tested again.

Same problem.

mpstat, indicated that whilst cpu 1 was running fine, 2 was 100% busy processing. Unusually though, it was 100% in 'system' time, which meant it was the kernel working hard, rather than any of the processes.

Looking closer (and with reference to the Sunmanagers mailing list), we picked out that there were an unusually high number of interrupts and context switches on processor 2. The latter was caused by a dodgy process executing illegal instructions. The former didn't appear to have any immediate cause.

However, a nice chap pointed me at 'lockstat'. Specifically at 'lockstat -kgIW -D 20 -s 40 sleep 5'. Which showed me that a bunch of kernel objects, prefixed 'se_' were generating interrupts regularly.

Googling lead to /kernel/drv/se, which is the serial port device driver.

Having got someone to unplug the console server plugged into the serial port, this machine has returned to normal.

This is the way machine diagnostics should be. All of the above, was accomplished without needing a reboot. I suspect we'll need to schedule an outage to check the exact source of the problem, but that can be done at a more convenient time. (Although as a point of order, it was rebooted by the 'owner' of the machine, before they spoke to me about it, to see if it was a transient error.)

The road cleaner

Once upon a time, there was a good hearted chap, who noticed that the road outside his house was getting a little dirty. So he took out his broom, and started to sweep. He found it was actually rather fun, to work hard, but to have a nice clean bit of road to be proud of. So he started to broaden his efforts, to clean the road all the way along past his neighbours houses too. And for many a month, the road was nice and clean.

His neighbours would not often thank him, but every now and then, one of them would pop out and offer him coffee.

As time went by, his broom started to wear out in a big way. So by mutual agreement, he and his neigbours gathered some funds to buy a new broom, to be used for communual benefit. The good hearted chap agreed to look after the broom, and continued to sweep the road.

But after some months, a new neigbour moved in, with a big dirty car, that liked to strew oil all the way across the road. Cleaning the oil too, was harder, but the good hearted chap liked seeing his road clean, so just put that bit more effort in. Alas, too much effort was required, because rather than sweeping the road once a week, this oil monster of a car would make the road dirty daily. He persevered, struggling onwards, but gradually, the road sweeping stopped being a satisfing experience for him. And he stopped doing it so often.

But now, when the road was dirty, his neigbours looked at him, and wanted to know why their nice shiny road was so filthy. Why hadn't he kept it clean? After all, they'd even given him money for the broom that he used, to sweep the road.

The good hearted man was anguished. He had been sweeping the roads, because he liked having clean roads. He had a new broom, true, but it was getting worn in, and had replaced his previous broom that had been entirely used up in keeping roads clean. So he pushed on, and redoubled his efforts. A determination that nothing would keep his road dirty drove him. He spent many hours each day trying to keep up with keeping the road clean, but each time he looked at it, it had gotten dirty again.

He spoke to his neighbours about the problem, who demanded of him, "how could he let a dirty oilcar come and ruin their neighbourhood", and "why hadn't he dealt with it before". They were disastified with his efforts at keeping the road clean, and told him so. Many of his neigbours were openly hostile to him. Many were rude.

Trying to point out that he kept the road clean as a hobby, for the pleasure of keeping the road clean didn't work. After all, did they not have a right to a clean road, every day? Pointing out that he was just trying to sweep a road, not act as a 'oilcar pollution control officer' was unsuccessful. After all, he had taken responsibility for the clean road, and that clearly included stopping dirty cars from using it.

So one day, he just stopped.

And the road got dirty.