DavsDisorder

This article probably somes up my disposition towards the issues raised by this election process (note I say issues raised by the process, not issues rasied in the election contention itself), better than anything I have read recently:

Worth subscribing to;

http://www.crikey.com.au/2010/08/02/this-is-all-your-fault/

A friend (Carl, thank you), just sent me this...a little reminder of how far data storage technology has come in 50 years.



In September 1956 IBM launched the 305 RAMAC, the  first 'SUPER' computer with a hard disk drive (HDD). The HDD weighed over a ton and stored a 'whopping' 5 MB of data. Not even big to hold a digitised MP3 of John Coltrane's 'Blue Train' recorded the following year.

I thought I would post the text of a comment I made on Scott Lowe's (now of EMC) blog, regarding NetApp's future (like most posts of mine, its half baked and badly worded...thus Dav's Disorder)...

Guys, just a quick thought on these observations…if Scott’s take on NetApp is correct then surely the same applies to HDS? They are apparently releasing a server line themselves (great another fly for the ointment), but in essence they are the storage person’s storage company right? They will even rebuild microcode for customer’s if you have a particularly pressing requirement? I think enterprise storage requirements are breaking out into those demanded by true commercial enterprises (with governance and high uptime needs) and enterprise, as-in-as masses of storage and huge data rates, enterprises (scientific, research, university, aerospace, engineering, Tv and other media). Storage vendor’s cost models vs feature sets will determine their value in either market.

Whilst doing some research around options for moving our internal mail and file serving/collaboration requirements into a 'cloud' provider, I came across the details of the Telstra T-Suite offering, part of which is the Microsoft BPOS suite. As we are thinking of consolidating telecoms with Telstra also, I thought I'd test how their "implementation" (assuming they are actually hosting it in Australia somewhere) of the suite responded (browser refreshes etc). As a Microsoft partner we can use their BPOS but I am guessing it is hosted in an US or other remote DC so I thought I would try Telstra; unfortunately, whilst trying to secure a trial I got the follow screen:

 
Not an encouraging way to greet prospective SaaS buyers!!!

Similar to a post from a few weeks back, I found this little summary amongst some old client correspondence which you may find useful.

8 Steps to Effective DR Planning and Budget Requisition

1. Use the term Disruption Recovery rather than Disaster Recovery.
2. Ensure you currently have some kind of DR/BCP management framework no matter how rudimentary. Show business management that there are current documented processes, metrics and testing that can be optimised. Technology supports the business insurance requirements, it doesn’t necessitate it. You need to show that DR planning is an ongoing process not a point in time flight of fancy.
3. Engage the right people in your organisation. Applications support and owners, facilities management, and of course, when ready, financial and executive management.
4. Conduct a joint Business Impact Analysis or Risk Assessment. What are we insuring/protecting against...delineate the actual dangers. Threats = Impacts = $$.
5. Then proceed to a ‘costs of downtime’ calculation. Dependency mapping of all business applications is the starting point for this. This is never a clean cut equation but, the revenue that a particular application or ecosystem of applications support is the dividend, downtime impact is the divisor and roughly speaking then the cost of downtime is the quotient (which should align roughly with a budget!)
6. Position a DR investment as having some competitive market value – ROI. Present ‘best-in-class’, industry peer data. – Reputational loss, client retention may be affected.
7. Develop a DR services catalogue...align costs to system criticality (RPO/RTO). Relative costs vs. Absolute costs wherever possible. Suggest chargeback to Bus etc.
8. Align DR investment with other IT budgets. Try and link the technologies used in providing DR services to other areas of IT ‘spend’. EG Data centre consolidation, server consolidation (virtualisation), and utilising DR infrastructure for development or test purposes.

I refer here again to topics I mentioned in an earlier post on "Designing for Failure"

I read another blog entry recently by Chuck Hollis of EMC on the parallels between the notorious Gulf of Mexico oil spill and catastrophic system failures in IT. Shortly after reading it I was sent an email with a link to a very interesting video of a US TV show (The Rachel Maddow Show) looking back at a scarily similar disaster in 1979. I include it here for your bemusement!

Just saw earlier today, amidst a rain downpour in Sydney, a courier from a well known tape media offsite storage firm drop his case of tapes on the foot path, hurriedly shove them back in the case and pop them in the truck!!

Hope that organisation has a reliable 2nd tier backup.

www.interpolnyc.com

It's called "Lights", a 5minute power ballad with the signature Paul Banks sinister, misogynistic lyrics.

I recently read a news article about Intermedia's service level agreement 'miss' that was linked to a performance issue on an EMC CLARiiON array.

http://searchstorage.techtarget.com/news/article/0,289142,sid5_gci1510721,00.html

There have also been a couple of subsequent posts and email responses linked to this story;

http://itknowledgeexchange.techtarget.com/storage-soup/one-storage-pros-response-to-intermedias-hosted-email-outage/

http://chucksblog.emc.com/chucks_blog/2010/04/helping-to-avoid-a-really-bad-day.html

I wanted to make a few commetns myself in regards to the story and the responses shown above.

Firstly, I agree with everything Chuck Hollis at EMC says in his post, and I wanted to emphasis and elaborate on his points.

Products Fail?
Damn right they do, all the time...sometimes without causing much of a fuss, but trust me failures don't seem that common because you only hear about the big ones (like Intermedia's). It is a testament to IT hardware vendor's engineering that alot of these "failures" go unnoticed because fo the rigorous redundancy build into their systems...not to mention field support services which, in the case of EMC, are some of the best around.

A short anecdote that relates to this story; an insurance client of ours suffered a similar failure on their IBM N-Series (NetApp) devices a few years back. A controller panicked due to a power supply issue and tried to hand over its load to the other controller but due to incorrect configuration of multi-pathing, dropped all the workloads that it was serving. Result; reboots, reboots, reboots. Missed SLA.

Design for Failure
It will happen...not if, but when. You will have a component failure somewhere in your data path at some point in the future. Design for it (or insure for it!).

CLARiiON arrays (like N-Series, HDS and many other array vendors) have controllers that operate in active/active configuration, which is great when both controllers are working, and 99.99% of the time it works fine when one fails (the beauty of PowerPath). But the disadvantage of running and active/active architecture in a disk array is that, unless you religiously monitor your workloads, you can never be sure if you can meet performance demands in a degredated state (this principle applies all down the data path, even to RAID Group design and LUN layout). My favourite disk array of the last 10 years is EqualLogic's PS Series, now owned by Dell. These fellas only operate in active/passive mode to ensure customers don't accidentally find themselves in Intermedia's situation where peak load cannot be accommodated in degradated mode.

The Alarm is Ringing but Everyone's Asleep
This is an interesting point...vendors and integrators like ourselves put effort into engineering and deploying monitoring and alerting for systems in client sites. That's great but if the client doesn't put in place procedural steps that are triggered into action by these tools, all is for nought. There is no point in having a tight RPO and the ability to deliver a quick RTO unless you have the procedural surety  to act when issues are identified. EMC's DialHome feature is a good example of removing this dependency but its simply not possible (nor do you want it to be possible) for all system or component failures. In short your recovery time is only as good the weakest trigger point and usually that trigger point is simply deciding to act on a error/mis-configuration event.

Practice Failure
Great tip here...I hear clients and prospects talk about their highly redundant environments and their sub-minute failover setups and ask have they tested it...usually the answer is no. Reminds me of people who love to talk about how much their house has gone up in value...inevitably when they actually want to sell they are a little disappointed. Proof is in the pudding. You must test your failure recovery procedures. VMware's SRM product is an excellent tool for doing this non-disruptively. Clients should regularly test failover of their Tier 1/2 applications to ensure that the 'best laid plans' are also the 'tried and true' method.

Advice, especially field tested advice is not risky Happy! But I guess there's sometimes no option but to adopt a bureaucratic approach when positioned as such.

http://www.arnnet.com.au/article/344444/smb_it_supplier_advocate_named/?eid=-4152

"Easter said his first job would be to consult industry bodies and large tech companies to get a sense of the industry. SMB IT companies would need to understand how risky they appeared to Government CIOs"

I would love to get Data Engines to do more government business; it is often challenging work and it feels a little bit like we're using IT as our way to 'give back' in public service!. But...but, but but... $40mil coverage for professional services indemnity is also a regular requirement for tender respondees and that cost is a little too much alongside side a 50 page tender response document for a specification that is often written with a particular solution and/or vendor already in mind.

//TD.