sobrique | A day in the life

This morning, I arrived at work at around 08:50, having stopped for a bacon roll on the way in.
The first order of the day was catching up on emails, and checking backup failures. One of the copies of one of our project webserver farms hadn't run properly due to an error in the script, to I amended it, and have scheduled it to run again tonight.

Sorted out a couple of meetings, for next week and then started on the real day's work.

This morning has mostly been messing around in the server rooms (We have 2 datacentre rooms, next to each other). We have a new SAN (storage area network). This includes a disk array, and some fiber switches made by a company called brocade. We've a set of 4 x 4100s, of which we have 2 in each room. In DC1, the switches hadn't been commissioned yet (ITSpeak: Turned on, configured, cabled up, and generally made ready for use).

So I fired them up, plugged them into the nearby Cisco 6500 network switch, and configured some IP addresses. Then installed some licenses on it in order to be able to use all of the 4Gb fiber ports (it has 32 per brocade).

Next step was cabling, which might sound trivial, however, when you're talking about 300 servers, each of which having at least 2 power and network cables, terminal console cables, SAN fiber cables and the odd modem or two, if you're not really careful you end up with a MASSIVE pile of spaghetti.

Each of these two brocades needed to be connected to their partners in DC2. Thankfully we have a patch panel between the rooms (finally) so this was relatively easy. The ISLs (inter-switch links) connect switch 1 in room 1 to switch 1 in room 2. They 'share' the same fabric (fiber* channel term, for a 'configuration domain').

I've been hacking together a cable management spreadsheet, so I can keep track of lables and destinations.
Each cable has two lables on it (at least, in the SAN implementation):
A cable definition: e.g.
00197 F LC-LC 3m (sequence number, media, end connectors, length)
And a cable 'purpose'.
E.G. MYHOST A-DC2BR1 R2Yel05pA(H) - R2Yel06p3(P)
From MYHOST, port A in rack room 2 yellow 05, port A
to DC2BR1 (brocade switch) in Room 2 rack yellow 06 port 3
the '(H)' denotes a host connection, the '(P)' denotes a patch panel.

We have 3 rows of racks, Yellow, Blue and Red. They're correllated to the mains phases that the racks are on. Our mains supply is 3 phase, but it's also going through a flywheel and generator, to allow us to 'clean' the supply before reaching our servers.

I also had to add more cable management clips and some power cable to each of the brocades.

Then it was back upstairs for more coffee, and 'at desk' configuration of the brocade. This afternoon, we'll be putting a 'new' server onto the new SAN.
The process for doing that, losely, is:
Install HBAs (Host bus adaptors. Think fibre scsi controllers)
Install drivers.
Connect HBA A to the Brocade switch, on fabric 1
Configure brocade to allow HBA A to communicate with the Clarion CX 700 storage array.
Configure the disk volumes to be 'visible' to HBA A.
Reboot, find new disks, and format/mount them.
Install multipathing software (called powerpath).
Connect HBA B to fabric 2. Do the same thing with the disks.
Reboot again. Multipathing software should 'discover' that it's got 2 paths to the storage, and make them appear as a single disk to the server.
Carry on as normal.

Fairly straight forward, except for one thing. I've not done it on this new SAN yet, so I'm absolutely certain there'll be some things that don't work the way I expect.

That's this afternoons task, once I've finished a little more coffee and some administriva. I'm actually having to schedule when we're doing about 30 servers, and of course, these are mostly evenings and weekends. So basically, July's cancelled. I've still managed to keep the maelstrom weekend clear, but the rest has me working basically every weekend, and late two evenings a week.

Oh, and because it's a thursday, we in centralised computing go down to the pub for lunch. This week, I'm volunteer to actually phone the order in.

*yes, that's fiber deliberately, as that's the name of the protocol (fiber channel), to distinguish from fibre which is the actual 'optical fibre'.