{"id":699,"date":"2014-05-16T16:49:15","date_gmt":"2014-05-16T23:49:15","guid":{"rendered":"http:\/\/itiltopia.com\/?p=699"},"modified":"2017-12-11T08:45:14","modified_gmt":"2017-12-11T16:45:14","slug":"when-disaster-strikes","status":"publish","type":"post","link":"http:\/\/itiltopia.com\/?p=699","title":{"rendered":"When Disaster Strikes"},"content":{"rendered":"

\"1420-3\"<\/a>The true test of an IT organization\u2019s maturity is how it reacts when a significant issue occurs. By significant issue, I mean that a large portion (or even the entire) organization\u2019s ability to function is severely impaired due to an IT-related failure.<\/span><\/span><\/span><\/p>\n

What happens isn\u2019t a test of any single group or process – it is a test of all IT groups and\u00a0three tightly integrated processes: Incident, Problem, and Change.<\/span><\/span><\/span><\/p>\n

Let me give you an example of how the perfect storm can break a business. Once upon a time, back in the 90\u2019s, I got up very early every morning to be the first person in our office so I could take calls at the Service Desk. On this particular morning, it was evident that something was wrong. Some people couldn\u2019t get to fileshares or email. Some users couldn\u2019t get to the internet. Some people were reporting that client\/server applications weren\u2019t working correctly. Between calls, I launched my browser and pulled up CNN. It wasn\u2019t just our organization. Others were reporting similar issues.<\/span><\/span><\/span><\/p>\n

I immediately called my wife and told her that something big was going on and to cut the internet lines to her office. At the time, she was in charge of her company\u2019s IT functions, and when I explained the situation, she went to the network closet and unplugged the cable to the external service provider. She then put a sign on the entry doors to her office telling everyone not to open emails from any external source.<\/span><\/span><\/span><\/p>\n

It was Melissa,\u00a0a worm-based virus that created massive amounts of network traffic by\u00a0sending email\u00a0out from everyone’s Outlook client. <\/span><\/span><\/span><\/p>\n

Of course, we didn\u2019t know that at the time.<\/span><\/span><\/span><\/p>\n

All we knew was that large parts of the network were unreachable. So what did my organization do? The server team obviously thought there was a faulty card on a server sending out massive amounts of packets. The network team assumed that there was a defect in the network routing. The desktop team thought that a software update was being pushed out and that all the traffic was killing the routers. So everyone made countless, uncontrolled changes in their own little world for about\u00a0six hours.<\/span><\/span><\/span><\/p>\n

This was pure Incident Management at its worst. “Get it up and running as quickly as possible” is the first part of Incident Management\u2019s goal. Unfortunately, the second part of the goal was completely ignored. The full goal of Incident Management is \u201cGet it up and running as quickly as possible, while doing the least amount of harm as possible.<\/i>\u201d<\/span><\/span><\/span><\/p>\n

To say that recovery was challenging is an understatement. Our IT groups did more damage to our infrastructure in six hours than the worm could have ever done. Once the news got out that many organizations were being affected by this worm, our Desktop team spent about\u00a0eight hours canvassing the organization and physically touching every PC to remove the virus.<\/span><\/span><\/span><\/p>\n

Our network and servers weren\u2019t completely back to normal for three days.<\/span><\/span><\/span><\/p>\n

What Change Management was done? None. What Problem Management was done? None.<\/span><\/span><\/span><\/p>\n

Let\u2019s just say that it was an excellent learning experience.<\/span><\/span><\/span><\/p>\n

From that point forward, whenever there was a severe issue, the Service Desk Manager (me) started a conference line and\u00a0then notified all the other IT managers (by email, phone or fax) to dial into the conference line immediately.<\/span><\/span><\/span><\/p>\n

There was a formal structure to the conference call:<\/span><\/span><\/span><\/p>\n