Waivers, Deviations, and Exceptions

Growing up, my hero – everybody’s hero – was John Glenn.  He was the archetype of heroes:  decorated veteran of combat in two wars, test pilot, speed record holder, astronaut, first American to orbit the earth, later US Senator and Presidential candidate, and all around clean-cut American boy-next-door.  All of the Original 7 were heroes but John topped the list. When he returned to NASA for his second space flight on STS-95 in 1998, I did not have the opportunity to work with him but was thrilled to see him in the hallways and meeting rooms in Houston.  Still my hero.

So it was an . . . interesting . . . experience when, after Columbia, John Glenn requested that I have a telephone conference with him.  The purpose?  To chew me out personally about the number of waivers in the Space Shuttle Program.  I can report that the event was memorable.  Not in a good way.  He had his point:  people had died and somewhat to blame was the acceptance of waivers to requirements in our program.

A number of years ago, I wrote about the use of documented requirements in aerospace systems development (see https://waynehale.wordpress.com/2012/10/07/after-ten-years-the-tyranny-of-requirements/). A good, well run, effective organization or project will define what it intends to do, clearly state exactly what the requirements for the product are, and meet every one of those requirements without exception.  If the product somehow exceeds the requirements, that is OK, but it probably means that you spent more time or money than you should have.

In a complex system like the space shuttle, defining – and enforcing – requirements can be difficult.  All complex programs suffer from ‘requirements creep’ where nice to have features are added.  This can drive up costs and delay delivery.  Not following the requirements can cause all kinds of consequences from having to redesign and rebuild things that don’t work all the way up to death and destruction.  The shuttle had over 40,000 requirements and, as you might expect, not all of them were clear; some of them changed over time, and many were very difficult to do

At NASA, it is recognized that cutting edge projects sometimes don’t quite meet all the requirements and therefore there is a waiver process.  A good engineering review that shows how close the gizmo came to actually meeting the requirement, how safety is still protected, why it is impractical to exactly meet the requirement, etc.  Generally this is a very thorough engineering task to prove that a waiver is acceptable.  However, over thirty years, the shuttle program had granted thousands of waivers to requirements.  No matter how thoroughly this was done, the optics from the outside were atrocious.  So after Columbia – with the gentle encouragement from John Glenn – Bill Parsons and I decided that all waivers must be reviewed and, as far as possible, eliminated.

I got the assignment to lead this effort.  Depending on which section of the requirements documentation was being addressed, these shortfalls were called waivers, or deviations, or exceptions.  But it was all the same.

We adopted a three-fold review process.  First, to examine the requirement to see if it was still good.  After thirty years there had been upgrades, new understanding of how the system works, and changes to what we were trying to do so occasionally some requirement was out of date and no longer applied.  In those rare cases, we changed the requirement and that allowed us to retire some waivers.  In other cases, the fix would be just too hard to do and the risk was small and acceptable.  Those waivers we would keep with better documentation and rationale.  That was a small number.  Most waivers were resolved by redesigning or changing a part – spending money and taking the time to make it right.  Over the years, the shuttle program had worked under very tight budgets and sometimes choices were made that were . . . inappropriate.

The most egregious example of that last case lingers in my memory.  The Shuttle program was responsible for the space suits – EVA suits – used for space walks.  The backpacks were stuffed with all the things necessary to life support, communications, etc., for the space walker.  One well-known and documents standard for wiring in confined places is that you don’t bend the wires too tightly lest the insulation and perhaps the conducting wire inside breaks.  One waiver allowed a section of wiring inside the EVA backpack to exceed industry standards for ‘bend radius’ – it was folded over too tightly.  We reviewed the document and I asked ‘where is the engineering rationale that shows this to be acceptable?’  /The experts replied ‘There is none’ It turns out that there was a proposed way to fix the problem, but it would cost about $100,000.00 to fix all the EVA backpacks.  During the penny-pinching days pre-Columbia, the program did not chose to spend the money on that fix.  It was a short meeting to decide to fix it now, while we were not flying.  So we eliminated the waiver by changing the hardware. For weeks I had nightmares about being at the Flight Director console during an EVA and some astronauts’ backpack shorted out.  Eliminating that hazard was an easy choice.

By far, the silliest waiver concerned the color of the rail car covers for the solid rocket booster segments.  The requirement was that any ‘ground support equipment’ that was attached to shuttle parts during transportation or processing had to be painted bright yellow.  This was intended to draw attention to these parts to ensure that they were removed before flight.  This did not always work, probably a story for another day.  In the case of the rail car covers, recorded instrumentation showed that sometimes during transportation the SRB segments got close to an upper temperature limit which might degrade seals and insulation.  Typically this occurred on sunny warm days.  A smart engineer calculated that if the covers were painted white rather than the heat absorbing yellow, maximum temperatures would be lower.  But this required a waiver to the GSE color requirement.  It seemed pretty apparent to me that we were in no danger of leaving one of these covers attached and accidentally launching them.  I giggle to think about it.  So in this case, we changed the requirement to say that SRB rail car covers could be white.  Waiver eliminated.

Finally, there were hundreds of waivers regarding EMI in the shuttle crew compartment.  It turns out that during the shuttle development there was never an EMI susceptibility test run on the orbiter.  This is a standard test that involves some time and, in the 1970’s – a fairly large facility, of which there were not many.  So the shuttle program decided not to run the test. So we did not have a good idea about how the electronics that ran the orbiter were shielded from stray radiation. Almost every electronic gizmo emits some radio frequency energy:  cameras, laptop computers, even some digital watches.  Since we did not have a good engineering limit to set on equipment brought into the shuttle cockpit, everything must be tested and analyzed.  The results of these engineering evaluations were called waivers – because the basic requirement was nothing could come aboard that radiated any electromagnetic energy.

Resolving all these EMI waivers was going to be herculean. So we adopted a radical strategy.  I turned over the Endeavour to Dr. Bob Scully and his band of mad scientists.  That orbiter had completed its depot maintenance and was literally just sitting until we returned to flight.  They towed OV-105 over to the RLV hanger that once housed the remains of Columbia and outfitted it with sensors.  Tall poles with EMI emitters were erected and the tests began.  I had visions of some Tesla-like lightening discharges playing over the orbiter but they assured me that would not be the case.  I begged them not to break the orbiter, and they didn’t.  And the result?  We found what the real capability of the vehicle was and where the small areas of susceptibility resided.  Thousands of EMI waivers disappeared overnight.

After all these actions, and many more were complete, we drive the number of waivers, deviations, and exceptions to requirements down to just a couple of dozen.  Those were well understood, well documented, and the risks were properly briefed, understood, and accepted.  And that is the only kind of waiver that should ever be allowed.