Ed's journal

This weekend I have been looking at bikes.
Work offers a Cycle to work scheme, which means - basically - buying a bike without paying tax/NI on your salary.
I used to cycle a fair bit, but ... frankly, hardly at all in the last 10 years (mostly due to storage space issues).
But I was thinking I'd like to again for a variety of reasons. Saving £80/month on diesel is a fairly good start, but it's also a good way of dodging gym fees (I've not made it to the gym in the last year), whilst making time in my daily schedule to 'work out'.

So, the problem is - Bikes have changed a bit in the last 10 years. Price, and technology has shifted somewhat. Back when I was riding last, LEDs weren't 'bike suitable', indexed gears were a new thing, and disc brakes/suspension was something you saw on only the very top end.

Speaking to the bike shops, I've been handed a list of suggestions:
Cube CLS
Scott Sportster
Giant Rapid
Specialized Sirrus
Ridgeback Meteor

This week, I've been trying to work out the relative merits of RAID5 vs. RAID6 as a method of disk protection.
I won't bore you with details of implementation, but the essence of this - RAID 5 is a set of disk, for which one is set aside as parity and error correction.
Losing any single disk within a RAID 5 set means you're fine, but a second means you lose the lot.

RAID6 is - more or less - the same thing, but with dual parity. E.g. in a given sized set of disks, you use two for parity - such that you can lose any two from your set, safely, and a third will take your set of disks out

Into this mix, you have hot spares - a hot spare is _another_ disk, that's set aside, on it's own, to take the place of a failed drive.

So what I'm trying to figure out - given a mean time between failure of the drives (1 million hours), how much better - or worse - are the different RAID types?

When you 'lose' a drive, you have a window of exposure for the rebuild to occur, or your drive to be replaced. I know the chance of failure in that window is (very) low. However, I'm talking in terms of large arrays of drives - 1000 disks or so, and the data that means, which means 'pretty remote' odds, actually start to rack up, and even 'fairly remote' of a critical data loss is bad.

So I'm working on 3 'choices' here.
RAID 5, 3+1
RAID 6, 6+2
As both these 'types' waste 25% of capacity, and therefore cost the same.
For comparison, I'm considering RAID 5, 7+1.

Now, the number crunching goes thus:
MTBF of 1million hours.
Assume a maximum window of 96 hours before a failed drive is replaced and back in service. (Typically it'll be less).

Given 240 drives to put my data on, in which _any_ RAID loss results in total data loss. (So if one group of drives goes pop, I have to recover the whole lot). (IN case you're interested, they're probably 300Gb drives, so we're talking 54 TB of data - this is a lot of data to recover, so we'd rather not have to).

And over a 3 year time period, how likely that circumstance is to show up.

So I make it:
MTBF 1 million hours.
Chance of failure in a given 96 hour block - 0.000096

Taking a 4 disk set - chance of any single failure is:
1 - ( 1 - .000096 ) ^4 = 0.00038
From an 8 disk set, same logic:
1 - ( 1 - 0.000096 ) ^ 8 = 0.00076

Twice as many drives, nearly twice the chance of a failure occurring. (It's not -quite-).

So with the R5 set first drive fail is ok. Second is a total loss.
So chance of losing a second drive out of your 4 disk set is:
For R5, 3+1 we've got:
99.961% chance that of 4 drives none fail.
99.971% chance that of 3 drives, none fail.

So -
3.839x10 ^ -4 x 2.8 x 10 ^ -4.
= 1.1 x 10 ^ -7.
11 in 100,000,000 chance of occurring.

For the RAID 5, 7+1:
Chance of any one out of the 8, is 'chance of not failing' ^ 8.
So 99.923%.

Chance of remain drive from the set of 7 failing, in the same 96 hour
window, is:
'chance of not failing' ^ 7.
So 99.932%

7,677 x 10^-4 x 6.71 x 10^-4 = 5.144x 10 ^ -7.
A 51 in 100,000,000 chance of occuring.

And for the RAID 6, 6+2:
First drive: 99.92322580
Second Drive: 99.93281935
Third Drive: 99.94241382

Which means RAID 6, 6+2 has 2.97 E-10 chance of that scenario.

Now, that's where I get stuck - on the face of it, R6 seems 1000x more reliable than either RAID5, 3+1 or RAID5, 7+1.

If you multiply out across 240 drives, you've 60 4 drive sets, and 30 8 drive sets.
I think you can apply the same rational to that:
Probabity of failure is 1 - ( 1 - one set ) ^ number of sets.

So 240 drives:
R5, 3+1 = 6.63E-006
R5 7+1 = 1.55E-005
R6 6+2 = 8.91E-009

Now, the bit where I get a bit stuck - rolling the time window over 3 years. We're talking about a poisson distribution, (I think?). Can I just take my '96 hour' chance of failure, and do compound probability?
Making the R6, 6+2 scenario - over 3 years = 26280 hours.
Our number is over 96 hours - of which there's 273 chunks.
So ... 1 - ( 1 - 8.91 E-009 ) ^ 273
= 2.43E-006

So, 2 in a million chance of having a really really bad week.
Does my number crunching work out correctly though?

R5, 3+1 = 1.41E-3
R5, 7+1 = 4.22E-3

So ... looking at it, R6 - in terms of pure reliability - is a thousand times safer than R5 in either configuration.
The tradeoff would be performance - RAID 6 carries a write penalty - it must perform reads and writes to calculate parity for each write - which is higher than it would be with RAID 5 (approximately doubled - so halving your write performance).

So, you may have spotted word that there's consideration of a Dr Who film going on.
I'll say - for my part - that given most of the recent stories have been - at most - 2x 45m episodes, I don't mind overly.
http://blogs.indiewire.com/theplaylist/harry-potter-director-david-yates-to-direct-big-screen-reboot-of-doctor-who#

I do however, worry that declaring a reboot of the franchise means that the central point has been missed entirely - specifically, that in a setting where the central character regenerates, and there's time travel involved, then there really is no need to 'reboot' things.
Part of the appeal of of it, is that the Doctor is a mystery with a lot of history, and we see parts of that story.
To 'reboot' is neither here nor there - it's already been done at least once. Still, why would you want to throw out the baby with the bathwater, when it's entirely unnecessary. Have a new incarnation and be done with it.

But I digress. What I was going to ask - if you had the keys to the TARDIS, and a Hollywood budget, what would you do with it?

I'm thinking that Sir Ian McKellen would make an excellent Doctor. So would Christopher Lee, Patrick Steward, or Alan Rickman. And Hollywood might mean I can afford them.
Two companions I think. Alan Tudyk/Nathan Fillion (both would be good, but I think they wouldn't both fit, so either or I'm afraid).
And ... this might seem an odd suggestion, but perhaps someone like Dita Von Teese. Maybe Anna Paquin.

The story would have to be one of the 'more horror' ones. Because lets face it - Doctor Who is a horror story, in a Sci-fi genre. The very best stories are the small ones. We _know_ the universe isn't going to end, and lets face it - it wouldn't matter to us if it did, because we've no real concept of what that _actually means_. But with a small-ish cast of characters, they're expendable - they can die, in order to progress the storyline.

I really liked the Horror of Fang Rock, of the older stuff. I'd be reluctant to 'redo it', simply because... well, why rewrite, when you've a universe of possibilities out there. But it was in 77, so maybe I'd get away with it :).

Otherwise, I'm not sure - I think it would have to be 'gatecrashing' a historical event, because explaining why 'such and such' happened, is always pretty cool.
I'm not sure which one yet. I think I might have to go for one in the US - for the mass market appeal, and lets face it, British history has been poked at a lot.
So lets go with a Wild West, Ghost town story.
There's a gold rush on, and business is booming, but despite all that... there's still more people going missing, and getting lost in the night than would expect. There's rumours of a skin walker...

So, off to Maelstrom later.
One of the things I still have, is a thing steeped in memories. It's my 'Thorn T-Shirt'. It's not much to look at - a tech fabric T-shirt, in olive drab.
But I wore it most of the time as a base layer, when playing Thorn. So it still brings back memories.
It's odd really - Thorn was never real. Was merely a character brought together over the years of playing Maelstrom.
And yet still he persists - I've been used to 'thinking as Thorn' at Maelstrom, and I'd imagine that never quite goes away. After all, it's in playing this character, that's drawn upon parts of me, and gone from there.

It's perhaps odd to wonder who I would be, were it not for that. Would my personality have developed in quite the same way, over the numerous years since Maelstrom started I wonder?

But despite being gone, it's still a memory I treasure. Futility perhaps, but I do still stop and wonder occasionally - What Would Thorn Do? (The answer is usually not something I'd consider, but it still makes me smirk).

So just remember. Next time you see me wearing a plain-ish olive drab t-shirt. I'm probably thinking of mischief.

It's probably caught your attention, that there's been a terrorist incident in Norway. http://www.telegraph.co.uk/news/worldnews/europe/norway/8659028/Norway-shooting-July-24-as-it-happened.html

The thing that's intriguing me, is the response to it. Specifically:
Prime Minister Jens Stoltenberg has stated clearly that the terror will be met with more democracy and more openness.

It's actually a very refreshing approach. The simple problem with the 'war on terror' is - well, as Yoda put it so well.
"Ohhh! Great warrior! [laughs and shakes his head]
Wars not make one great!"

You can't ever change someone's mind by force. You can simply oppress them - and in doing so, you _can_ stop them 'doing it again' but only as long as you always stay vigilant and in control. Once you bully someone into doing something your way, you have to stand over them to make sure they keep on doing it your way.
And in doing so, you steadily increase the size of the backlash if they ever do 'get free.

The same's largely true of extremism. You don't stop 'radical' views, by cracking down on them, and you don't stop... well, any form of hatred actually, by legislating against it. You can make expressing a sentiment illegal, but that's not the same as changing someone's opinion.

So I'm quite gratified to see this approach - fundamentally, the only way to stop terrorism completely, is to stop the perpetrators seeing you as a target.
Something that you can only accomplish by turning the other cheek.

So, one of the things I've been running into more and more lately, is people who are unaware of the distinction between 'moot' and 'mute'.

Mute means to silence or quieten.

Moot on the other hand, means something quite different.
A 'moot point' is something which may be effectively irrelevant, or perhaps remains open to debate.
Wikipedia expands on this more than I could:
http://en.wikipedia.org/wiki/Moot_point

But essentially, Moot is an old word, meaning meeting, or discussion. A moot point originally meant one that was 'for discussion'. More recently, it has evolved, thanks mostly to law students holding a 'moot court' - in which hypothetical cases are debated, which in turn has lead the word to shift from 'for debate' to 'practically irrelevant'. Indeed, that's now become - legally speaking: "Mootness, a legal concept that a case cannot be decided because a decision would no longer have real consequences."

A 'mute point' or to - as I saw in an email just recently - "... let me know if you will be willing to do any of the days muted."
Doesn't really mean the same thing at all.
The problem is, that you _can_ apply 'mute' to a discussion. A muted discussion will be one that's done quietly.
If you 'mute' someone, then you silence them.

There is, however, quite a large divergence of meaning.

Here's a bit more on the subject:
http://www.worldwidewords.org/qa/qa-moo1.htm

One of the things that preoccupies me - well, sometimes - is the nature of faith. And not just when I'm on a wind up mission.
I know amongst my friends, there's a range of viewpoints on the subject. So I often ramble, and in part that's trying to straighten out what I think, and in part, that's to encourage others to do so.

I remain, I think, in my opinion that there may be 'something more'. However, I'm afraid I still don't hold that it follows that that 'something more' might be conscious in ... well, any meaningful sense of the word.

So what I was mulling over - regardless of religious inclination - there's a sort of temptation to believe in a Higher authority. A sort of abdication of responsibility, that 'someone else' knows best. Be that a religious belief, or a secular one, or just an apathetic one.
It's quite clear that this belief is widely held, because - if nothing else - the voter turnout is pretty abysmal.

There's always a temptation to believe in a Higher Authority. Someone who - at the end of the day - holds judgement of what is right.
There's a danger in that though, of abdication of our own responsibility to make the world a better place. To use our own judgment. To accept that we're the arbiters of our own fate. Because lets face it - even if there is that Higher Authority, the feed back timescale is long enough that ... it'll always be too late, if you wait for the judgement.
Even our legal and political systems are - essentially - feedback mechanisms, and deliver a verdict after the fact.
That will always be too late to fix the problem. And no one else is going to do it, either. There's been a few stories lately, in the news, that ... have been really rather shocking, and all because that 'Higher Authority' has failed.
That people have done things, that have ended up... at best ill considered, and at worst outright negligent. And yet they continued, because they were relying on 'someone else' to notice and do something about it.

I think that really is the proof of the adage - that all it requires for evil to triumph is that good men do nothing. But to take it down to a basic level - there's no real thing as absolutes of good and evil. Merely people who are prepared to stand up, and not just demand something better, but commit to making it happen themselves.

It's not complicated, but for all that, neither is it easy.

One of the doctrines I've operated under, for my career, is that after 2 years, you should be asking yourself 'what next?'.

2 years is - in my opinion - about the timescale you should be thinking about a new job. That cuts both ways - assumption is that I expect to be at least 2 years at a prospective employer, but at the same time, past the 2 year point I'm going to be wanting more from them - from a career perspective. (And that goes hand in hand with improving my pay).

So, has it been two years (or more) since you last took stock? I don't mean you _have_ to move on every 2 years, simply that you should be assessing - do I have more challenges? What's next in my career development. It's perfectly fine to decide that you're fine where you are. Just don't make the mistake of doing it because it's easier not to think about.
Go look. See what's available. See if you're worth more (or less!) than you think you are. And then use that, to either think about a new job, or as ammunition for negotiating with your existing employer. There's no defense against being taken advantage of quite like being ready to walk away when it gets too much.

So to that end I'd like to offer people who I know, who are willing to take up the challenge - of assisting in redrafting your CV, and thinking about what might be an option for your skills, mindset and whatnot. I'm at least fairly familiar with the IT industry, and can do - I think - a reasonably good 'private sector' CV.

CVs are a bit of a dark art, and 'advice' on them seems to change ever few years. But my most recent got me several interviews for the kind of jobs I was after, so it can't be _that_ bad, right? (And even if it is, I promise not to take offense if my suggestions don't work for you).

So, my most recent mission at work, has been backups. If you've ever really had to think about backups, then... well, it's one of those little floating chunks of ice on the surface, that hide an iceberg underneath.
What we're doing is database backups. For around 100 Oracle databases. And we're doing it on them all whilst they're 'hot' - all running, with no interference to the running system.
This is usefully large volume of data - of the order of 50Tb.
The objective is to be able to recover any of them within 4 hours with no data lost. Given they're live and active systems... well, it's non trivial to pull that off.

When talking about backups, you talk about Recovery Time Objective, and Recovery Point Objective.
RTO is the time you have to bring a system online after a failure.
RPO is how much data you're allowed to 'lose'. Whilst 'losing data' sounds scary, bear in mind that chances are most of the things you back up have some data loss - because everything you do _after_ the backup isn't backed up any more.

So an RTO of 4 hours, and an RPO of zero. (well, near zero) is pretty aggressive, given that you may need to call someone out, get people out of bed, etc.
What we're doing to achieve this, is use some storage array tricks. We've got two Symmetrix VMAX storage arrays, in two separate datacentres. On the 'primary' side, we're taking snapshot copies of the databases, at 4 hourly intervals through the day.
On the 'remote' side, we're taking a clone copy, and backing that up.

More storage terminology: A snapshot of a disk is a point in time copy of a disk. It's achieved my keeping track of changes. So initially, a snap is zero size, but every time you change something after that snap is taken, the change gets recorded. So you can quickly 'flip it back' to how it was.
A clone is a full copy of a disk, at a disk level - disk signatures, deleted files, the lot.
The advantage of these - as opposed to copying files - is they're actually fairly quick to finalize. Which means we can 'pause' our databases for a matter of a couple of minutes (or less) whilst we take our snap or clone, without anyone really noticing.

So what I've had to do, is write some scripts to make this happen - there are products that can do this, but because of the environment (And timescales) we're working with, they're not an option.
I've been writing a set of perl scripts that are run via a product called Tivoli Workload Scheduler (TWS). They run on Solaris, and 'do the business' of setting hot backup on a (remote) database, create a clone copy, mount the clone copy, and stream it to tape.

I've also been working on a scripted solution that does the snapshots - I'm quite proud of this, as it's not really a trivial matter to manage rotating snapshots - automatically, and on a large number of 'source' devices, which means you can't easily/feasibly do a 'brute force' approach (of defining all the individual relationships).

But ... well, we're coming up to 'go live' on the project, and are just dotting Is and crossing Ts. Last night, was the first time we'd done a recovery 'in anger' as it were (involving callouts, support etc.) and I was very pleased to find that we'd managed it, with 4 minutes to spare. (I'm expecting that to only get better, as we get things a bit more streamlined).

So it's all good, really. I'm kind of watching and prodding it as we go, and writing oodles of documentation, trying to clarify how it all fits together.
On the plus side - because I've known all along that this will be so - I've done my best to ensure that the scripts require minimal amounts of 'hardcore knowledge' to make them work. I think I'm mostly achieving that. I expect to find out that I'm wrong as I start doing knowledge transfer sessions, and doing 'early life support'.

But still, I've been quite enjoying this so far - it's a challenge that uses skills I've built up, with storage arrays, Solaris, Backups, Perl scripting (and XML parsing), and had to bootstrap myself into learning how Veritas Filesystem, Netbackup and TWS work.
IT's been a satisfying challenge.
(So far, at least. Expect a rage post in a week, when it all falls apart again).

For reasons not entirely related to 'just curiosity'. Here's how to do a report on your web history in Chrome.
Chrome is really nice, in that it uses a 'database format' for the various file. SQLite can be used to query it.

So here's how to do it (mostly for my reference):

1: Download SQLite shell from here: http://www.sqlite.org/download.html

2: Run a command prompt

3: Change to your 'chrome' history directory:
On XP – C:\Documents and Settings\name\Local Settings\Application Data\Google\Chrome\User Data
On Vista - C:\Users\name\AppData\Local\Google\Chrome\User Data


4: (Assuming you put the SQLite shell in D:\Tools - amend accordingly)
D:\Tools\sqlite3.exe History "select datetime(last_visit_time/1000000-11644473600,'unixepoch'),url from urls order by last_visit_time desc" > History.txt

This will extract the complete history - at least, assuming you haven't cleared it ever.

5: Open Excel, and import 'History.txt'. Use 'text to columns' (location varies depending on Excel version) and use delimiters of 'space' or '|' (other).

6: Add a couple of columns to your spreadsheet (I used C and D).
Title C as 'Find/' and D as 'Site'.
Populate C2 with: =FIND("/",E2,9)
Populate D2 with: =LEFT(E2,C2-1)

Paste the formula down the full list.

7: Add another column, at the end (Column F?) and put in it '=A1' and paste down. (Yes, that does mean Column F is a copy of Column A. There is method in my madness. )

8: Make a pivot table of your results. Set 'row' to 'Site', and 'values' to 'Count of URLs'.

9: Add a vlookup to your pivot table - if you've done the above, and happen to be on exactly the same versions of everything as me, Column C should be empty. Fill it in with: =VLOOKUP(A5,Sheet1!D:G,3,0) (The 'third column in the range D:G will be 'Column F' so this _should_ populate Column C with the last accessed date of that site)

10: Scroll down the list, and be shocked and awed at how many times you've visited Facebook.Com.