Waivers, Deviations, and Exceptions

Growing up, my hero – everybody’s hero – was John Glenn.  He was the archetype of heroes:  decorated veteran of combat in two wars, test pilot, speed record holder, astronaut, first American to orbit the earth, later US Senator and Presidential candidate, and all around clean-cut American boy-next-door.  All of the Original 7 were heroes but John topped the list. When he returned to NASA for his second space flight on STS-95 in 1998, I did not have the opportunity to work with him but was thrilled to see him in the hallways and meeting rooms in Houston.  Still my hero.

So it was an . . . interesting . . . experience when, after Columbia, John Glenn requested that I have a telephone conference with him.  The purpose?  To chew me out personally about the number of waivers in the Space Shuttle Program.  I can report that the event was memorable.  Not in a good way.  He had his point:  people had died and somewhat to blame was the acceptance of waivers to requirements in our program.

A number of years ago, I wrote about the use of documented requirements in aerospace systems development (see https://waynehale.wordpress.com/2012/10/07/after-ten-years-the-tyranny-of-requirements/). A good, well run, effective organization or project will define what it intends to do, clearly state exactly what the requirements for the product are, and meet every one of those requirements without exception.  If the product somehow exceeds the requirements, that is OK, but it probably means that you spent more time or money than you should have.

In a complex system like the space shuttle, defining – and enforcing – requirements can be difficult.  All complex programs suffer from ‘requirements creep’ where nice to have features are added.  This can drive up costs and delay delivery.  Not following the requirements can cause all kinds of consequences from having to redesign and rebuild things that don’t work all the way up to death and destruction.  The shuttle had over 40,000 requirements and, as you might expect, not all of them were clear; some of them changed over time, and many were very difficult to do

At NASA, it is recognized that cutting edge projects sometimes don’t quite meet all the requirements and therefore there is a waiver process.  A good engineering review that shows how close the gizmo came to actually meeting the requirement, how safety is still protected, why it is impractical to exactly meet the requirement, etc.  Generally this is a very thorough engineering task to prove that a waiver is acceptable.  However, over thirty years, the shuttle program had granted thousands of waivers to requirements.  No matter how thoroughly this was done, the optics from the outside were atrocious.  So after Columbia – with the gentle encouragement from John Glenn – Bill Parsons and I decided that all waivers must be reviewed and, as far as possible, eliminated.

I got the assignment to lead this effort.  Depending on which section of the requirements documentation was being addressed, these shortfalls were called waivers, or deviations, or exceptions.  But it was all the same.

We adopted a three-fold review process.  First, to examine the requirement to see if it was still good.  After thirty years there had been upgrades, new understanding of how the system works, and changes to what we were trying to do so occasionally some requirement was out of date and no longer applied.  In those rare cases, we changed the requirement and that allowed us to retire some waivers.  In other cases, the fix would be just too hard to do and the risk was small and acceptable.  Those waivers we would keep with better documentation and rationale.  That was a small number.  Most waivers were resolved by redesigning or changing a part – spending money and taking the time to make it right.  Over the years, the shuttle program had worked under very tight budgets and sometimes choices were made that were . . . inappropriate.

The most egregious example of that last case lingers in my memory.  The Shuttle program was responsible for the space suits – EVA suits – used for space walks.  The backpacks were stuffed with all the things necessary to life support, communications, etc., for the space walker.  One well-known and documents standard for wiring in confined places is that you don’t bend the wires too tightly lest the insulation and perhaps the conducting wire inside breaks.  One waiver allowed a section of wiring inside the EVA backpack to exceed industry standards for ‘bend radius’ – it was folded over too tightly.  We reviewed the document and I asked ‘where is the engineering rationale that shows this to be acceptable?’  /The experts replied ‘There is none’ It turns out that there was a proposed way to fix the problem, but it would cost about $100,000.00 to fix all the EVA backpacks.  During the penny-pinching days pre-Columbia, the program did not chose to spend the money on that fix.  It was a short meeting to decide to fix it now, while we were not flying.  So we eliminated the waiver by changing the hardware. For weeks I had nightmares about being at the Flight Director console during an EVA and some astronauts’ backpack shorted out.  Eliminating that hazard was an easy choice.

By far, the silliest waiver concerned the color of the rail car covers for the solid rocket booster segments.  The requirement was that any ‘ground support equipment’ that was attached to shuttle parts during transportation or processing had to be painted bright yellow.  This was intended to draw attention to these parts to ensure that they were removed before flight.  This did not always work, probably a story for another day.  In the case of the rail car covers, recorded instrumentation showed that sometimes during transportation the SRB segments got close to an upper temperature limit which might degrade seals and insulation.  Typically this occurred on sunny warm days.  A smart engineer calculated that if the covers were painted white rather than the heat absorbing yellow, maximum temperatures would be lower.  But this required a waiver to the GSE color requirement.  It seemed pretty apparent to me that we were in no danger of leaving one of these covers attached and accidentally launching them.  I giggle to think about it.  So in this case, we changed the requirement to say that SRB rail car covers could be white.  Waiver eliminated.

16784709a

Finally, there were hundreds of waivers regarding EMI in the shuttle crew compartment.  It turns out that during the shuttle development there was never an EMI susceptibility test run on the orbiter.  This is a standard test that involves some time and, in the 1970’s – a fairly large facility, of which there were not many.  So the shuttle program decided not to run the test. So we did not have a good idea about how the electronics that ran the orbiter were shielded from stray radiation. Almost every electronic gizmo emits some radio frequency energy:  cameras, laptop computers, even some digital watches.  Since we did not have a good engineering limit to set on equipment brought into the shuttle cockpit, everything must be tested and analyzed.  The results of these engineering evaluations were called waivers – because the basic requirement was nothing could come aboard that radiated any electromagnetic energy.

Resolving all these EMI waivers was going to be herculean. So we adopted a radical strategy.  I turned over the Endeavour to Dr. Bob Scully and his band of mad scientists.  That orbiter had completed its depot maintenance and was literally just sitting until we returned to flight.  They towed OV-105 over to the RLV hanger that once housed the remains of Columbia and outfitted it with sensors.  Tall poles with EMI emitters were erected and the tests began.  I had visions of some Tesla-like lightening discharges playing over the orbiter but they assured me that would not be the case.  I begged them not to break the orbiter, and they didn’t.  And the result?  We found what the real capability of the vehicle was and where the small areas of susceptibility resided.  Thousands of EMI waivers disappeared overnight.

After all these actions, and many more were complete, we drive the number of waivers, deviations, and exceptions to requirements down to just a couple of dozen.  Those were well understood, well documented, and the risks were properly briefed, understood, and accepted.  And that is the only kind of waiver that should ever be allowed.

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in Uncategorized. Bookmark the permalink.

15 Responses to Waivers, Deviations, and Exceptions

  1. Bob Scully says:

    Thanks for giving us the opportunitiy to make a difference, Wayne! … and thanks vrey much for writing us up in your blog! I for one greatly appreciate the nod and the tip o’ the hat!

  2. Charley S says:

    Thank you. Not only for the work on the waiver system but to give a clear, understandable depiction of how you get to stacks of waivers for each mission to chopping those stacks down.

    Any insight on the TPS? I remember clearly at both the Challenger and the Columbia hearing the question of “How, when the original requirement was for no launch damage to the TPS, did a hundred or more strikes become acceptable?” Inference would suggest the answer to be a better understanding of the risk coupled with a reduction of strikes. Memory says they were also using a tougher tile for problem areas.

    • waynehale says:

      We should have some of the earlier program experts comment on why such a poor design was picked and allowed to be flown in the first place

      • Dave H. says:

        Wayne,

        One must remember the timeframe in which the STS was designed and what materials were available to construct it with. Add the considerable amount of time and testing required for certification in and, like the nuclear power industry, you end up with devices obsolete before they see service. The TPS was simply the “best available technology” in its time. Compromises are part and parcel of engineering.

      • waynehale says:

        There were other choices that would have worked. Robustness and impact tolerance were traded for weight savings. Was it a good choice?

  3. Bob Scully says:

    Thanks for the acknowledgement of the work we did for return to flight, Wayne. It was an honor and a privilege to have been able to contribute.

  4. Jay Estes says:

    Good read. It is astonishing that 40,000 requirements are required for a human rated spacecraft, it’s so much that I can’t help but ask myself, it is really important to insist that all GSE be yellow, or other similar issues that “seem” innocuous. Did the SR-71 have that many requirements? It flew near the edge of space, and carried humans with comm and life support systems – I know not the same, but honestly, it’s not a outlandish comparison. I am assuming it had significantly less than shuttle, but it was also from a different era.

    In any case, I have had to push a waiver or 2 thru in my time. Both since the shuttle waiver review season you describe above. I still use what I call “Wayne Hales’ 3 rules” for waivers (from a letter you wrote) to defend the need for one. It has been an effective tool for me. I do greatly appreciate your leadership in this area. It’s a very important process!

  5. Peter Keats says:

    Thanks for the Twitter bump on this, very interesting reading and though the waivers seemed like a bad idea, hindsight is a wonderful thing. I wish I was born with it. The fog of competing pressures will always tend to make things lose focus, and will continue to do so, despite all good intentions otherwise.

  6. Pam Melroy says:

    Wayne, this is really interesting. Every developmental program has a lot of issues that there isn’t time and money to address, with the assumption that in the sustainment part of the program they will eventually get worked through. There were constant improvements to the shuttle that I am sure did deal with many of them. But that isn’t true for every issue, particularly if it’s hard, and everyone is under the impression it’s all good enough. It makes me wonder how to ensure this type of review in new developmental programs like Orion and Commercial Crew after they have been operating a couple of years.

  7. Harold Doiron says:

    It would be interesting to compare the number of requirements and waivers granted to those requiements in the Space Shuttle Program vs previous programs such as Mercury, Gemini, Apollo, Apollo Soyuz, Skylab, Shuttle/Mir and ISS.

    Two critical Shuttle Level II requirements that were known to be violated by the 2nd Shuttle test flight were:
    (1) A requirement for fully redundant SRB Field Joint O-Ring seals that the design being flown was clearly violating with observed hot gas blow by during liftoff dynamics.
    (2) A requirement that limited the size (mass) of debris shedding from any Shuttle System element that could strike another System Element

    Neither of these requirements violations were shown by a rigorous engineering test and analysis process to be acceptable risk, yet waivers must have been granted to allow the Shuttle System to continue to fly with these Level II Systems Requirements violation. In my opinion, this decision by Level II Shuttle Program Managers to grant waivers to these requirements violations without the necessary test and analysis process used to certify a design is the primary root cause of both the Challenger and Columbia fatal flight failures.

    There is a NASA LESSONS LEARNED here that has not been thoroughly assessed to determine what to do differently next time.

    Wayne,

    You, I and Robert F Thompson, Shuttle Level Ii Program Manager throughout design and development through the 1st flight, have been discussing this issue for several years. I am surprised and disappointed it was not emphasized in your essay on Waivers and Deviations. John Glenn was right to be concerned about flying with too many waivers in the Space Shuttle Program.

    • waynehale says:

      By the time it was my job to review the waivers, that one had been retired with a new design.

    • waynehale says:

      Hal,
      I have disapproved several of your comments as being off topic to my blog post. I generally also disapprove comments that are longer than the original post. I suspect you and your friends need to get a blog of your own if you want to pursue you particular interests on social media. I am writing from my own personal experience and do not care to speculate on what might have happened at some points in the past on topics that I have no direct knowledge about.

  8. Paul451 says:

    “we drive the number of waivers, deviations, and exceptions to requirements down to just a couple of dozen. Those were well understood, well documented, and the risks were properly briefed, understood, and accepted. And that is the only kind of waiver that should ever be allowed.”

    Can you explain why these waivers were still required? Or rather, why the requirements were left intact if there was no way to meet it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s