Ten Years After Columbia: Pennywise and Pound Foolish

“Relentless budget reduction pressures necessitate more dramatic program actions” – Brewster Shaw, Space Shuttle Program Manager, December 1994

Polls show that the general populace rates the Internal Revenue Service as the most disliked agency in the US Federal government.  Among members of the US Civil Service, the agency is most disliked is the Office of Management and Budget.   Some years ago I pondered what it must be like to be a member of the OMB.  Charged with getting the most value from every federal dollar (what taxpayer could be against that?), the OMB challenges each and every federal agency and program to justify their existence and their budget.  Every year they ask every bureaucrat ‘Can’t you do that for less money?’  Every year they look for new ways to decrease the deficit by decreasing spending.  Every day of the year they nag, scold, harass, and browbeat each and every government program to find ways to do business more efficiently.  From the outside that makes great sense.  From the inside it makes you obnoxious and disliked. 

Most program managers, when challenged to save some reasonably small amount, say 5% in a year’s budget, have the ego to think that they probably can do that.  Of course, the OMB will be back next year to ask for another 5% reduction.  Or 10%.  Hey, if it was that easy to save 5%, surely they can do more! 

It is understandable that most people blame the OMB for the plight of the space program.  I don’t.  Dealing with the OMB is just part of normal business for federal agencies and their management.  Remember not to promise things that you can’t deliver; that should be rule #1.

So I have sympathy for the OMB. 

Well, not really.  Sorry guys, it’s hard to be unloved.

Back to basics: the shuttle cost too much to operate.  No argument.  The intention, back in 1972 or so was to lower the cost of putting payload into low earth orbit significantly.  Compared with the Saturn V or the other expendable launch vehicles of the day, the shuttle did decrease cost per pound to orbit; but not nearly as significantly as what was wanted.  So from day 1, the national leadership (and their minions, the OMB), beat up the space shuttle program management to lower costs.  In the early 1980’s it still seemed possible that increasing flight rate and decreasing launch costs might be possible – not to the levels envisioned a decade earlier, but by a significant amount.  Then came the loss of Challenger and it was clear to everybody that this was an experimental, fragile system that would cost more to fly. 

There is a lot of discussion about lack of vision in the space program.  I would submit that the space program has had the same vision for over 50 years; the one promulgated by Willy Ley and Werner von Braun and Walt Disney in the 1950s.  The one made famous by Colliers magazine and the artistic work of Charles Bonestall.  To wit:  an infrastructure to explore and colonize the solar system; a winged shuttle to low earth orbit, a space station as waypoint to the planets, and a fleet of planetary vehicles to carry great expeditions to Mars and other destinations in the solar system.  Scratch every blue ribbon panel recommendation since then and you find the same vision, just variations on the theme, schedule, and  . . . ahem . . . how to pay for it.  Flexible path is just the latest blue ribbon panel name for this vision.

NASA was devastated when President Nixon, just after the last moon landing, killed the proposed Mars program and the space station, leaving only the shuttle.  OK, it was the first step, but all true space cadets knew it was only the first step.  So maybe, just maybe, we all believed that if we did this shuttle thing right and make no mistakes, the next step will be approved.  The ‘next logical step’ being a space station.  I can remember waiting, on July 4, 1982, in Mission Control after we landed STS-4 in California to meet President Reagan.  We had great hopes that he would announce the space station program.  We were disappointed again that day.  It would be almost four more years before Freedom was announced. 

When President Clinton entered the White House, his policy advisors made it know that they intended to cancel Space Station Freedom.  So NASA Administrator Dan Goldin had to make a Faustian bargain.  First, the Russians had to be included – and the myth established that they were not on the critical path.  Everybody below the rank of Deputy Administrator knew to be an outright falsehood.  Second, the cost of the space station development would not be allowed to increase the agency budget.  Third, the agency budget had to be reduced by almost one quarter.  All of that or no space station, no next logical step; and we would be left with a space shuttle without a destination.  There is a lot of criticism in the ranks that Goldin made a poor deal.  I think he probably got the best deal he could.  Anyway, we kept the space station and gained in international partner – which paid off in the longer run in ways that we could not have imagined in 1992.

This resulted in the incredible and relentless pressure that was put on the entire NASA budget including the space shuttle.  Shuttle program manager after program manager were squeezed to make the program more efficient, cheaper.  And the continuing, grinding pressure to decrease costs paid off.  Between 1992 and 2002 the annual operating budget of the shuttle program per year (adjusted for inflation) decreased by 45% – almost half.  Where did that money come from?  It came from hard work to squeeze out inefficiencies and duplication.  It continued with harder work to eliminate activities that only slightly added to the program.  Then it continued with even harder work to re examine every operation with an eye to its “requirement”. (See my last post “The Tyranny of Requirements”). 

In the decade of squeezing, a lot was cut out.  Some of that was good and necessary.  Some of it was, well, counterproductive.  See the graph that John Muratore made after the Columbia accident.  SE&I Decreases Systems Engineering and Integration plays a vital role in the safety of any highly complex high risk operation.  But vitally important functions were no longer performed when the personnel levels fell too low.   Lambert Austin who headed SE&I in those days fought tooth and nail against these cuts; he lost.  Then after Columbia, NASA management skewered Lambert as if it was his fault for losing to their earlier ideas. After all the reasonable reductions were made, the requests became just plain silly.  Mel Peterson from NASA HQ browbeat Shuttle Program Manager Ron Dittemore to privatize the shuttle; one of the dumbest ideas ever floated.  After repeated beatings, Ron responded by giving them a plan to do what they wanted.  It never got implemented.  I can remember a discussion in mid-2002 to eliminate maintenance on one of our crosswind lakebed runways so the program could reduce its budget by less than $10,000.  Now is that a reasonable requirements reduction?  I just remember all those shuttle deorbit decisions when we were worried about the winds shifting around and the forecast was uncertain.  More than one day I was glad to have options to land our billion dollar spaceship safely on its one and only gliding landing.  Who was right on that one?  We never used that lakebed runway.  Like insurance, we paid for it and never used it.  Was a good money savings idea to eliminate that $10,000 bill?

This is probably the point to trot out that hoary old story about how to cook a frog.  If you put a frog in a pot of boiling water, he will immediately jump out.  If you put a frog in a pot of cool water and start heating it slowly, he will never jump out.  We all live in a world where there are slowly trending situations; we can’t deal with them all at once, you have to make choices.  Where do you draw that line? 

In the end, I am convinced that the “relentless budget reduction pressures” were a major cause of the Columbia accident that cost us a crew and an orbiter.  Not the only cause, but a major cause. 

So where do you draw that line, between prudent and acceptable expenses and extravagance?  What do you do when you depend on a vehicle that just flat costs more to fly than you can afford?  Across several administrations, across multiple program managers, across the terms of more than one NASA administrator, these reductions slowly ground out.  Good people, trying very hard to do what they thought was right, missing something critical, resulting in a tragedy.  Like some classical Greek tragedy, in hindsight it seems inevitable. 

When my children were small I read them bedtime stories; it was always easy to pick out who was the hero and who was the villain.  But when you read adult stories – real life stories – the heroes always have flaws, the villains can exhibit admirable traits.  Real life is not black and white, but gray.  We all need to learn how to discern the shades.

And when to draw the line.

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in After Ten Years and tagged , , . Bookmark the permalink.

20 Responses to Ten Years After Columbia: Pennywise and Pound Foolish

  1. Lonnie Moffitt says:

    Wayne….great article. I appreciate your candidness.
    Thanks
    Lonnie

  2. Beth Webber says:

    Thank you for the insight, and the economic history, of pre-Columbia. It dismays me to realize how close we came to not having an International Space Station, and the lack of importance our Government places on space exploration. I’m just a citizen that wants to see our country explore space; with Apollo, the Shuttle, Orion, Curiosity, with the Russians, just do it. It should be as important (not more, not less) than Medicare, Social Security, and National security. It is one of the things, if not the only thing,our country does that is an investment in our future.

    • waynehale says:

      At one point a vote in the US House of Representatives came within one vote of canceling the space station. Please make sure your elected representatives know how you feel. And don’t forget to vote!

    • Rob says:

      You have been following the economic situation I assume. Our space budget was built into the general budget that was built on a false economy the last 30 years. We never could afford it but didn’t know. Let’s wait and see where the economy bottoms out before we try to say that the space program is as important as Medicare, Social Security, and National security.

      • waynehale says:

        Interesting comment that I am trying to understand. The USA is a rich country, biggest economy and largest GDP in the world. We spend more money on pet food than space, we spend more money on cosmetics than space. It is not a question of having enough money to spend on space, it is a question of priorities, I think.

  3. Dave Huntsman says:

    Wayne, with respect, I disagree with you.

    “In the end, I am convinced that the “relentless budget reduction pressures” were a major cause of the Columbia accident that cost us a crew and an orbiter.  Not the only cause, but a major cause. ”

    They may have been; but they were not the major cause. There was one major cause which had its genesis before the ‘relentless budget pressure’, and it was squarely on shuttle management, not the budget.

    We knew in 1989, after the severe tile damage of STS-27, that the issue of crap from the SRBs and ET coming back and hitting the orbiter was just not going to go away. The panel that investigated the incident, besides the technical recommendations made to fix the immediate problems with the SRBs, made one management recommendation. Realizing that this was a serious issue that was not going to just disappear, and that business as usual was inadequate to handle it, the failure team also recommended that a full-time Level II technical manager be assigned to follow this problem in all its aspects for the duration of the program. Not just tracking debris hits and their possible entry impacts during a mission; but working and integrating the systems engineering, integration, testing, etc. across the program to reduce the incidents over time. The Shuttle program manager at the time, Dick Kohrs, agreed, and appointed an existing Level II technical integration manager to take on that job full-time, and gave him full authority across the program. That person was myself.

    Over the next year I established new or used existing teams to do everything from re-evaluate TPS designs, institute new testing and aging programs, integrate the pre-launch, mission ops, and post-mission photo analysis and assessment teams. During missions I reported on the results of the various photo analysis teams’ efforts directly to each days’ MMT (Mission Management Team). Orbiters did not leave the runways unless I signed off on documents stating that we had gotten all the analysis our teams needed to do post-flight damage and TPS assessment, for all flight elements. I also had several flights where I put aircraft in the air during launch at different altitudes to get data on when debris impacts were occurring, though the latter proved of little use in the end and were cancelled. We had a focused, long-term effort whose job was to protect the orbiter before it ever flew, monitor it intensively during flight phases, report constantly to management in both real-time and between flights. We were determined never to lose a vehicle due to debris damage.

    However, not all agreed that debris damage was a serious threat; and when after one year I was promoted to NASA HQ to serve on the international space station program, I was never replaced. The lesson of STS-27 was ignored – and continued to be, right up until Columbia. The Shuttle Program Office SE&I manager who made that decision was Larry Williams; but Gene Kranz of the Mission Operations Directorate concurred with the decision to no longer have someone follow, integrate, and advocate for the entire debris damage assessment issue. I went to Larry and Gene to lobby both of them about the importance of having someone full-time, over a period of years, provide focus, advocacy, and integration of all related debris damage efforts; neither would be swayed.

    So you see, I maintain the single biggest root cause was a shuttle program management issue; essentially, the same type of root cause which caused the Challenger accident as well. Did the shuttle program have schedule pressures on them pre-Challenger that made things worse? Of course it did. But managers at various stages of the program then, in both Huntsville and also Houston, were operating in a bubble where ‘normalization of deviance’ became, well, the norm; and the bureaucracy stopped questioning things and didn’t communicate well.

    Pre-Columbia, it was shuttle management’s decision to stop following the STS-27 incident team’s findings and to not have a single person continue the full-time, focused, engineering, integration, and operations efforts to minimize debris damage that was, in my view, the single biggest cause of the Columbia accident. It had nothing directly to do with money, but was mainly a ‘well, we never need to do this before, so…….’ type of mindset that I kept running into among senior managers.

    • waynehale says:

      I certainly appreciate your comment. Poor management decisions, apart from the budget pressures, will be the focus of an upcoming post that I am drafting.

    • Karen says:

      Thanks for writing this comment, Dave. I remember working on an analysis after STS-27 regarding foam impact to the Orbiter windows. I remember discussing with TPS experts the possibility of RCC damage from similar kinds of debris. I was too new in my career to understand how all of the work flowed into the big picture and when the Columbia accident happened I couldn’t understand how work we’d already done had been forgotten. I don’t remember you running an SE&I effort like you describe in your comment but it makes sense now that it didn’t exist long enough for me to take notice of it, so many layers down in the org chart as I was at that time. I’m sorry that we weren’t able to do better on managing the debris risks to the Orbiter.

    • no one of consequence says:

      First, thank you and Wayne for having this conversation. Glad its being heard.

      I believe that what each of you are saying is true and not in conflict. But that the nature of the problem is slightly more abstract in the context of the political institutions involved.

      Political institutions have inherent biases that after a while, everyone adjusts to, sometimes even almost at the subconcious level. Wheither the issue is oversight or budgetary effectiveness, the short term impetus gets damped by the long term fallacy of status quo.

      Yes, you’re exactly right about shuttle management’s flawed decision to stop following the STS-27 incident team’s findings ongoing. In general, every weakness of Shuttle needed long term attention as well – from the original compromises made (including in the SRB addition with its issues some of which you raised – experiences predating RSRB). I’d go further and suggest that the nature of running an X- vehicle long term as if it is a institution leads to a kind of reductionist thinking that tends to blind after a while. So its not “just more money”. May I suggest possibly where operations/management issues … might emerge from is such “blindness”?

      Though there is a component of this due to budgetary pressures, whereby after a while people fail to notice what is lost when a program is “slimmed down” – could it be that compromises are occuring but not being noticed, because the fallacy that we “know it works” is too attractive? My concern in ther early days of the Shuttle’s design was that a great deal of assumptions were being made … and … we got away with … but was anybody going to put in the effort to see if end to end that was always true … all along? I doubted then if the budget could be found … to afford even a complete understanding of the entire vehicle, processes, … and flaws … such that the experience of having done the Shuttle would have lead to refinement of materials, structures, and practices at a fine grained level. The scope was too large. As apart from, for example, what I directly saw with the ALT, cutting short the testing to mitigate cost / risk to maintain schedule/budget. And never returning to those items missed. “Through a glass darkly” became after a while unnoticed.

      As to budget reduction – its hard to get this into government, but the biggest reduction in costs require taking additional budget/risk like doing significant IR&D – for my two cents, the follow-on way underestimated and done way, way to little. The easier “penny pinching” stuff only gets two-bit results or worse (increased liability).

      Bottomline – I think the Shuttle was “too cheap” a program. Reductionism was a poor replacement for well-justified minimalism – this going back also to your comments on Larry and Gene. My criticism pre-TAOS all through to program conclusion unchanged. Doesn’t let anyone off the hook, just putting it in context. Why I said both you and Wayne are right as parts of a larger incomplete puzzle.

      Back to budget reduction in general … Worse yet is that it points out original flaws of congressional deals that shouldn’t have been done all along, and also puts at risk “sacred cows”. Which is why the real things that need to be done that are obvious … don’t happen. Of course any major weapons programs face these same issues, often further compounded. To restructure this to work requires a degree of cooperation at the political/organizational/industrial level currently out of reach.

      NASA has pioneered outrageous successes already for all of history. Perhaps there is room for it in this area too. Or so I shall continue to hope.

    • Dave H. says:

      Reading this tale is truly fascinating. It reminds me of the 1960s TV show “Wild Wild West”. Every 15 minutes what we today call a “screen capture” became part of a tapestry.

      I appreciate those posters who were part of that tapestry and their recollections and insights. Learning that Gene “Failure Is Not An Option” Kranz was willing to risk loss of crew and vehicle has given me pause as yet another childhood hero is shown to have feet of clay.

      Even so, I’ve built my career on his Apollo 13 creed, and for that I thank him.

      • waynehale says:

        Life is full of tradeoffs. I don’t know the details of how the decision that Dave H referred to was made. I will tell you that everybody who has been in a position of authority has made decisions that were not right, some that they came to regret. That is called being human. Gene Kranz has repeatedly stated on many occasions his regret about not speaking out soon enough to (potentially) prevent the Apollo 1 fire. I’ve had my own regrets. Heros don’t have feet of clay because they made mistakes or sometimes failed; real heros are heros because they tried to do something difficult that most people didn’t even attempt.

      • Dave H. says:

        Quite true, Wayne.
        In my case, it is simply the difference in how the world looks when you’re age 14, as I was during Apollo 13, versus how the world looks at age 57. Neither perspective is “wrong”, but they are quite different.
        The body of Gene’s work stands on its own merits.

  4. dblumentr says:

    Wayne, really enjoyed this post. Comment for you, in the words of Rod Serling, submitted for your approval regarding squeezing efficiencies out of the Shuttle. When I got out of school, after some help from my relatives and friends, my plan was to be a contractor, which in my mind gave me flexibility to pursue new development projects, one of those was the multi mission upgrades dating from the early eighties. After I got to JSC, I worked in was Shuttle Mission Simulator (SMS) reconfiguration upgrades and some early work on the Control Center upgrades. We squeezed some square pegs into round holes, had to do some arm twisting and arm wrestling, but eventually got it working, tested and into operations. Now when you work development, you don’t really get to see the fruits of your labor. Much later, while working Shuttle upgrades, I got a chance to fly the SMS and I was amazed at how few people there were to reconfigure the SMS for our made up mission and how hands off the operations were. We aren’t making routine flights to the moon, exploring mars and don’t really have routine access to space, like I thought we might back when, but we have an ISS and a partnership that gives us access. Got a new exploration vehicle in development and a rocket to ride on coming. Nature of the beast with budgets as they are, we just have to chalk up successes and feel a sense of accomplishment, even if they fall short of what the original ambitious goals might have been.

  5. Wayne, I just want to say, you are a wonderful writer. I’m interested in space travel (who isn’t?) but the history and politics of the space program are not normally subjects I’d follow with any particular interest–particularly because I am a working mother of two teenagers and just at this moment stressed beyond belief. But I stop to read your blog whenever it arrives in my box, because it is such a joy of fluid writing–it draws me right in. (And you should know I’m a professional writer, with a blog about writing, so I can be pretty critical.) Wonderful job!

  6. outstanding article. I apprise your candidness. Thanks

  7. Frank G says:

    I was going to write a comment, but I think Robin stated it perfectly!

  8. I like how you are pointing out that everything in reality is shades of gray. Certainly having an international station with the Russians, Europeans and Japanese joined with the US is a good outcome of saving money, it has created a great symbol of multiple parts of the world, even former adversaries, working together for a common cause and vision. And certainly there were some good outcomes in reducing the operating costs of the shuttle program.
    And of course, a number of the reductions that were made for budget reasons were completely the wrong things to do. While I continue to agree with Neil deGrasse Tyson that space spending needs to be significantly increased, you also opened my eyes for some positive effects that spending cuts can have, next to the detrimental effects playing an important role in your overall story.
    Thanks for your insights – even if I’m just some space (and fellow Star Trek) fan casually following your stories.

  9. Rod Wallace says:

    I wonder what would have happened after STS-113 if the Orbiter Projects managers of the past had invested in impact testing which provided data equivalent to the post Columbia testing. I wonder what would have happened if every ET Project manager had invested in finding the root cause of foam loss. I wonder what would have happened if every Program manager, SE&I manager and element manager had listened to what the hardware was saying instead of assuming that mere processing schedule issues could not be safety of flight issues. Budget pressures and needing to fly pressures are not excuses for not having good data to make better decisions. We are all humans, but without data we can’t be good engineers. Keep telling the story Wayne because all future human flights depend on not making our mistakes.

  10. Jim Banke says:

    Wayne, reading this particular installment comes days after I just finished the (disappointing) John Young “Forever Young” book. Toward the end he rattled off a great number of suggestions he made to improve shuttle safety, lamenting that most if not all of them were refused because of cost. So yeah, where do you draw the line with competing requirements of safety and budget and politics and engineering and many others? No clear answers then, or now. You do your best and hope for the best. And the saga continues….

    • waynehale says:

      I had the great honor (or something) to be the official “responder” to John Young memos for several years. The Shuttle Program office delegated that duty to the Flight Directors and somehow I failed to duck in time and wound up with the assignment. John had many interesting observations and suggestions. Not all of them were practical and some of them had consequences that Capt. Young had not considered.

      A truly great man, I am proud to have known him, and his continued active dedication to safety kept all of us on our toes.

      I am sorry that at some level some people started ignoring him. I know I frequently differed with his suggestions, but I always respected him. A true hero in an age when we don’t have enough of those.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s