January 2019

Greetings 7×24 RMC’ers.

We’re looking forward to you joining us on February 21, 2019 from 11-2 as we ‘launch’ our 2019 Educational Program series.  We’re featuring a speaker from NASA, Matt Melis, who will share his fascinating story on the Root Cause determination of the Space Shuttle Columbia Tragedy.

You can register for the event here… https://7x24exchangerm.org/meetinginfo.php?id=33&ts=1545163186

My LinkedIn post on January 2, 2019 https://www.linkedin.com/feed/update/urn:li:activity:6486322739576651776 I mentioned several similarities between the NASA missions, and our mission critical industry.  One of the parallels we see in our industry failures has to do with schedule pressure. 

You might recall this was a significant factor in the loss of Challenger.  On launch day Cape Canaveral was seeing temperatures around 30°F.  Concerns had been voiced about the effectiveness of the Solid Rocket Booster O-rings at these temperatures.  Despite this, the decision was made to “go for launch” anyway, as there had been some previous delays.  Sadly, all of us know how that ended.

From a Wikipedia Article on the Rogers Commission: “The unrealistically optimistic launch schedule pursued by NASA had been criticized by the Rogers Commission as a possible contributing cause to the accident. After the accident, NASA attempted to aim at a more realistic shuttle flight rate: it added another orbiter, Endeavour, to the space shuttle fleet to replace Challenger, and it worked with the Department of Defense to put more satellites in orbit using expendable launch vehicles rather than the shuttle”

In the beginning of the Shuttle program, the concept was a reusable space vehicle that could be launched monthly or even more often.  Compromises were made in the original design which made this unrealistic.  This led to unmanaged expectations, and ultimately, contributed to schedule pressures within NASA.

I’d now like to share a schedule pressure story in our data center world.

There was a major PM being done at a Tier IV Data Center on a 2N Uninterruptible Power Supply.  The onsite team had asked management for a 48-hour Change Window (from 00:01 Saturday to 23:59 Sunday) to accomplish the work on UPS-A, with the Critical Load being supported by UPS-B.

Management wanted the work completed in a 24-hour window instead, citing the elevated risk of being in a vulnerable condition with only UPS-B supporting the Critical Load.  Management also recommended there be a larger team of electricians assigned to do perform the PM to shorten the Change Window.

What resulted was too many electricians, some without the necessary skill sets, working on the main current-carrying systems as well as the sensitive electronics of a hugely critical component of the Tier IV Data Center. 

A problem later showed up on the output of this UPS System, causing the loss of a portion of the Critical Load on both UPS-A and UPS-B.

The onsite team was easily able to identify the electrician who caused this event.  However, there

were several other causes:

• The decision to try and shorten the Change window by putting too many electricians without the correct skill sets on a critical system
• The onsite team failing to insist on the 48-hour Change Window as originally planned
• Lack of labeling
• Onsite management’s lack of knowledge about some critical components within the UPS

Matt Melis will drive this further with his anecdotal stories of organizational challenges within NASA and beyond.  We’ll then ‘close the gap’ between NASA and our Data Center Industry challenges with a Panel Discussion.

It’s gonna be great!!  Sign up early, this one will sell out quickly.

We have an elaborate Vision, Mission and Strategy that can be found on our website, but it all boils down to a few simple things…

1. Programming…we are committed to bringing you relevant topics that are not sales pitches, topics that are attractive to our target audience: our Data Center End Users
2. Participant Experience…we strive to create a friendly atmosphere, good food and good locations, where our End Users can hopefully glean a few takeaways to apply to their own operations, as well as get connected with our local Vendors and Contractors

Once again, THANK YOU to our Sponsors for TAKING US INTO LOW-EARTH ORBIT!!  Without you, none of this happens.  The venues, the food, the Golf Tournament, the Scholarships.  THANK YOU for your support. 

Warmest Regards,