Happyness and joy
Jun. 10th, 2005 09:13 amWell, I was working late last night on a server, that was being 'a bit flakey'.
We got a response that it needed filesystem consistency checking. About a 6 hour job.
Duely, we re-arranged fileystems, made alternative arrangements for the night shift, and got the process started.
This morning, the server in question is not 'a bit flakey' any more. No, it's utterly fucked. It starts, kernel panics, and crashes again.
And again.
And again.
This means, about 800Gb of our 'production' user storage is unavailable. Normally, this would be a cause for minor celebration amongst our users in the appropriate departments. A half day of slacking, free and clear, and time booked to me. Oh yeah, I have my own 'project' code. Well, strictly there's an 'IT fuckup' project code, but as it's usually me in the middle of the storm it's mine. All mine.
However, today is a special case. You see, today is payroll day. And Human Remains are 'on' that fileserver. The word has not yet stated circulating, beyond a bit of twitchyness from those in the know.
For some reason, the prospect of around 3000 people not getting paid on time is one that causes ... nervousness.
So we have a long chain of people frantically fixing the problem.
We got a response that it needed filesystem consistency checking. About a 6 hour job.
Duely, we re-arranged fileystems, made alternative arrangements for the night shift, and got the process started.
This morning, the server in question is not 'a bit flakey' any more. No, it's utterly fucked. It starts, kernel panics, and crashes again.
And again.
And again.
This means, about 800Gb of our 'production' user storage is unavailable. Normally, this would be a cause for minor celebration amongst our users in the appropriate departments. A half day of slacking, free and clear, and time booked to me. Oh yeah, I have my own 'project' code. Well, strictly there's an 'IT fuckup' project code, but as it's usually me in the middle of the storm it's mine. All mine.
However, today is a special case. You see, today is payroll day. And Human Remains are 'on' that fileserver. The word has not yet stated circulating, beyond a bit of twitchyness from those in the know.
For some reason, the prospect of around 3000 people not getting paid on time is one that causes ... nervousness.
So we have a long chain of people frantically fixing the problem.
no subject
Date: 2005-06-10 09:30 am (UTC)no subject
Date: 2005-06-10 03:20 pm (UTC)