My Dream Job
Mar. 18th, 2007 10:14 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sobriqueYa'know, my dream job would definitey have to be technology evaluation and development.
One of the things I think I'm good at, and also really enjoy doing is assimilating and appreciating 'new tech'. And then figuring out how it's useful. I wonder if Getronics has something like that. *shrug* anyway.
One of the things I've been thinking that I'd really like to try out, is something involving VMWare ESX, Blade Centres and a SAN.
First of all, you take a Blade Centre. Lots of vendors are starting to do them these days. A blade centre, rather than being some horrific kitchen appliance, is actually a 'multiple computer'. You take a chassis (http://h18004.www1.hp.com/products/blades/components/enclosures/c-class/c7000/index.html), build into it a couple of power supplies, an network switch, and a few other things - in this case, we also need a fibre channel SAN switch (http://h18004.www1.hp.com/products/blades/components/c-class-interconnects.html).
Into this chassis, you place several smaller form-factor 'blades' which are basically standalone systems. ( http://h18004.www1.hp.com/products/blades/components/c-class-bladeservers.html )
And what you get is something that's _relatively_ small in terms of server room footprint, and homogenous hardware.
Then you hook this into a SAN. Doesn't matter especially what, but EMC do Symmetrix and Clariions. These are disk storage arrays, and you basically 'network' the disks with the bladecentre. Create a volume on the SAN per blade, and make it active - booting from the SAN isn't _always_ a good idea, for performance reasons, however it's viable, provided you design for it.
And you install VMWare ESX on each of these blades. VMWare is a virtualisaton product. It lets you run multiple instance of a single server on a single bit of hardware. You install it on all your blades, and use VMWare Virtual Centre to control them. You probably want to designate one of your blades as 'stand alone' VCC.
VMWare ESX can share disks between multiple vmware servers. So you put some storage from your SAN, on each of the ESX servers (the blades). And then you create virtual machines on them.
VMWare Virtual centre implements a cool feature called 'vmotion'. http://www.vmware.com/products/vi/vc/vmotion.html VMotion lets you 'move' a running virtual machine. From testing, it's got about a half second 'outage' which is pretty much un-noticable on any network environment. It starts up a copy of the virtual machine you want to migrate, on the other vmware server. It takes a few minutes to synchronise the memory state between the two, and then it just 'switches' to the other one.
Your VM has moved 'physically' but logically it's still the same system, doing the same thing.
What you then have, is an environment which isn't exactly _resilient_ but because you've got 10 blades, each running almost identical operating system environments, and virtual machines that are 'hardware abstracted' you can move them around the different blades, according to ... well whatever. But performance would be an obvious one, as would if you need to do maintenance on one of them.
And just for bonus points, you set up each of your VMWare servers as 'bootable' scripted installation servers. So if you need to rebuild any single one, for any reason, it's as simple as 'boot from network'.
You see, once you do that, you end up with a trivially scalable environment. You can add additional blades, or SAN storage on the fly. You have something that can be easily managed - VMWare ESX has _nothing_ that needs to be done from the console, aside from whem you're first building it. Especially if you're using scripted installs.
VMWare ESX also has a 'append mode' filesystem. It means you have your 'base' disk image, and after that, it writes changes to a redo log. You can commit this redo log to the disk image, or you can remove it, and leave it at a point in time. Now, that's brilliant, if you want a 'clean' system - just set it up so that when you reboot, it 'updates' anything it needs to - e.g. checks the time or whatever. And trash the redo long, on the OS disk. It guarantees that the flakiness you can sometimes get by a dodgy patch, or someone installing 'something' that they probably shouldn't just 'goes away' after a reboot. Including things like the latest and greatest virus, or whatever.
You also have something that's 'fairly' resilient. If you have a system crash, it'll still go offline, but each VM reboots virtually instantly, and
Better yet, you have an evironment that's trivial scalable - you can literally add new disks or vmware servers as you need. With VMotion, you can move around your VMs (virtual machines) through the day, in response to load changes.
And at a worst case, being a 'hardware fault' - modern servers are getting quite good, between mirrored disks and stuff, but sometimes it happens - you lose an ESX server. But all the VMs on it, can instantly be started on one of the other vmware servers. Functionally, on a 'critical' hardware failure, 'some' of your servers will reboot. OK, so it's not ideal for 'buisiness critical' stuff, but ... well lots of applications and products will actually do some quite clever stuff. A web server for example, if you have several, and one reboots, the end users won't notice. Domain controllers/DHCP servers, if you have multiple, no one will notice.
File servers, are generally pretty tolerant of brief outages these days - you'll get some errors, but then you'll just have to wait 5 minutes to 'save'.
And if you're getting _really_ clever with your VMWare trickery, you take something like SRDF - Symmetrix Remote Data Facility. What that does, is replicate a volume, to another storage array, across a fibre. Which means you can get that 'layer' of resiliency, across physical sites, even geographically distributed. I'm not entirely sure if you could run on the same storage, but you could certainly have 'groups' of disks on one farm, available to another in an emergency.
Your blades are also pretty much entirely replaceable - with a SAN on the back end, it's literally the case of pulling one out, putting a new one in, and doing a _little_ bit of reconfiguration, taking about 5 minutes. And then your 'new' bit of hardware, takes over from the 'old'.
Your entire environment would get 'fairly cheap, fairly resilient'. You'd have something trivially managable, as whilst each VM can be a 'custom build' the underlying thing would be pretty consistent.
And it's actually not all that expensive. Blade centres are a little over the odds. As are SANs. But assuming you'er buying servers and SAN anyway, the 'overhead' is small - you need less physical boxes for one thing. OK, so the VMWare ESX license does add up, but ... well, I think you could probably arrange things such that this overhead was actually relatively minimal, and offset against the 'saving' from buying and maintaining less physical servers.
I believe too, you've got various options out there, for automated migration of VMs - see the load on one spike, move it to a 'high performance' ESX server, and if it drops off, move it back again. That kind of thing.
For a small scale implementation, I have done it, where you take two physical servers, put a 'shared' storage array on each of them, and run VMs on them. This has the same advantages, although admittedly you're not so scalable. But it's perfect for a 'small site' implementation - your 'site' has a 'easy failover' environment, and you get something that's easy to remote manage. And upgrade, and stuff.
Now, wouldn't it be cool to be involved in a project that was trying that out, putting it together, and seeing how viable it was? It might not work, but at the moment, I can't see _why_ it wouldn't work. Now all I have to do is find someone to pay me to do that and find out for them :)
One of the things I think I'm good at, and also really enjoy doing is assimilating and appreciating 'new tech'. And then figuring out how it's useful. I wonder if Getronics has something like that. *shrug* anyway.
One of the things I've been thinking that I'd really like to try out, is something involving VMWare ESX, Blade Centres and a SAN.
First of all, you take a Blade Centre. Lots of vendors are starting to do them these days. A blade centre, rather than being some horrific kitchen appliance, is actually a 'multiple computer'. You take a chassis (http://h18004.www1.hp.com/products/blades/components/enclosures/c-class/c7000/index.html), build into it a couple of power supplies, an network switch, and a few other things - in this case, we also need a fibre channel SAN switch (http://h18004.www1.hp.com/products/blades/components/c-class-interconnects.html).
Into this chassis, you place several smaller form-factor 'blades' which are basically standalone systems. ( http://h18004.www1.hp.com/products/blades/components/c-class-bladeservers.html )
And what you get is something that's _relatively_ small in terms of server room footprint, and homogenous hardware.
Then you hook this into a SAN. Doesn't matter especially what, but EMC do Symmetrix and Clariions. These are disk storage arrays, and you basically 'network' the disks with the bladecentre. Create a volume on the SAN per blade, and make it active - booting from the SAN isn't _always_ a good idea, for performance reasons, however it's viable, provided you design for it.
And you install VMWare ESX on each of these blades. VMWare is a virtualisaton product. It lets you run multiple instance of a single server on a single bit of hardware. You install it on all your blades, and use VMWare Virtual Centre to control them. You probably want to designate one of your blades as 'stand alone' VCC.
VMWare ESX can share disks between multiple vmware servers. So you put some storage from your SAN, on each of the ESX servers (the blades). And then you create virtual machines on them.
VMWare Virtual centre implements a cool feature called 'vmotion'. http://www.vmware.com/products/vi/vc/vmotion.html VMotion lets you 'move' a running virtual machine. From testing, it's got about a half second 'outage' which is pretty much un-noticable on any network environment. It starts up a copy of the virtual machine you want to migrate, on the other vmware server. It takes a few minutes to synchronise the memory state between the two, and then it just 'switches' to the other one.
Your VM has moved 'physically' but logically it's still the same system, doing the same thing.
What you then have, is an environment which isn't exactly _resilient_ but because you've got 10 blades, each running almost identical operating system environments, and virtual machines that are 'hardware abstracted' you can move them around the different blades, according to ... well whatever. But performance would be an obvious one, as would if you need to do maintenance on one of them.
And just for bonus points, you set up each of your VMWare servers as 'bootable' scripted installation servers. So if you need to rebuild any single one, for any reason, it's as simple as 'boot from network'.
You see, once you do that, you end up with a trivially scalable environment. You can add additional blades, or SAN storage on the fly. You have something that can be easily managed - VMWare ESX has _nothing_ that needs to be done from the console, aside from whem you're first building it. Especially if you're using scripted installs.
VMWare ESX also has a 'append mode' filesystem. It means you have your 'base' disk image, and after that, it writes changes to a redo log. You can commit this redo log to the disk image, or you can remove it, and leave it at a point in time. Now, that's brilliant, if you want a 'clean' system - just set it up so that when you reboot, it 'updates' anything it needs to - e.g. checks the time or whatever. And trash the redo long, on the OS disk. It guarantees that the flakiness you can sometimes get by a dodgy patch, or someone installing 'something' that they probably shouldn't just 'goes away' after a reboot. Including things like the latest and greatest virus, or whatever.
You also have something that's 'fairly' resilient. If you have a system crash, it'll still go offline, but each VM reboots virtually instantly, and
Better yet, you have an evironment that's trivial scalable - you can literally add new disks or vmware servers as you need. With VMotion, you can move around your VMs (virtual machines) through the day, in response to load changes.
And at a worst case, being a 'hardware fault' - modern servers are getting quite good, between mirrored disks and stuff, but sometimes it happens - you lose an ESX server. But all the VMs on it, can instantly be started on one of the other vmware servers. Functionally, on a 'critical' hardware failure, 'some' of your servers will reboot. OK, so it's not ideal for 'buisiness critical' stuff, but ... well lots of applications and products will actually do some quite clever stuff. A web server for example, if you have several, and one reboots, the end users won't notice. Domain controllers/DHCP servers, if you have multiple, no one will notice.
File servers, are generally pretty tolerant of brief outages these days - you'll get some errors, but then you'll just have to wait 5 minutes to 'save'.
And if you're getting _really_ clever with your VMWare trickery, you take something like SRDF - Symmetrix Remote Data Facility. What that does, is replicate a volume, to another storage array, across a fibre. Which means you can get that 'layer' of resiliency, across physical sites, even geographically distributed. I'm not entirely sure if you could run on the same storage, but you could certainly have 'groups' of disks on one farm, available to another in an emergency.
Your blades are also pretty much entirely replaceable - with a SAN on the back end, it's literally the case of pulling one out, putting a new one in, and doing a _little_ bit of reconfiguration, taking about 5 minutes. And then your 'new' bit of hardware, takes over from the 'old'.
Your entire environment would get 'fairly cheap, fairly resilient'. You'd have something trivially managable, as whilst each VM can be a 'custom build' the underlying thing would be pretty consistent.
And it's actually not all that expensive. Blade centres are a little over the odds. As are SANs. But assuming you'er buying servers and SAN anyway, the 'overhead' is small - you need less physical boxes for one thing. OK, so the VMWare ESX license does add up, but ... well, I think you could probably arrange things such that this overhead was actually relatively minimal, and offset against the 'saving' from buying and maintaining less physical servers.
I believe too, you've got various options out there, for automated migration of VMs - see the load on one spike, move it to a 'high performance' ESX server, and if it drops off, move it back again. That kind of thing.
For a small scale implementation, I have done it, where you take two physical servers, put a 'shared' storage array on each of them, and run VMs on them. This has the same advantages, although admittedly you're not so scalable. But it's perfect for a 'small site' implementation - your 'site' has a 'easy failover' environment, and you get something that's easy to remote manage. And upgrade, and stuff.
Now, wouldn't it be cool to be involved in a project that was trying that out, putting it together, and seeing how viable it was? It might not work, but at the moment, I can't see _why_ it wouldn't work. Now all I have to do is find someone to pay me to do that and find out for them :)
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2007-03-19 01:51 am (UTC)I'd actually guess that virtualisation on blade servers with centralised storage is one of the main appeals for blade servers. Such a system is impractical on traditional servers, but ideal for blade systems.
It's all relating to the "on demand" buzzphrase, really. Such a setup lets you have server and server power entirely on demand.
no subject
Date: 2007-03-19 06:31 am (UTC)no subject
Date: 2007-03-20 07:55 pm (UTC)especially if used as a plug in to plesk.