After Ten Years: Working on the Wrong Problem

One of the toughest problems the Ascent Flight Director faced was how to get the crew back home safely if the shuttle engines quit during the launch phase. We studied and worked out procedures and techniques for over thirty years. Single engine failures were automatically handled; it was the multiple engine out cases that were tough. The orbiter is a glider with a terrible life over drag ratio. Many of the situations we just didn’t have enough range to get to a runway. Plopping the crew down in the middle of the Atlantic Ocean was not a satisfactory answer; stretching the glide to make a runway – like Shannon in Ireland – that was a much better solution. The orbiter does a belly flop during re entry to dissipate the energy; this keeps the temperature down on the heat shield; particularly the hottest part of the wing leading edge which is made up of 02 inch thick composite material: reinforced carbon carbon. At 40 degrees nose high, the temperatures there stayed right around 3,000 degrees F for about half an hour. If the nose were lowered, the temperature climbed, but if the wing stayed intact, the lift over drag ratio was much better. During the early 1990’s we worked very hard with the RCC experts to determine exactly how much we could lower the nose, increase the glide, make the runway, and not destroy the wing. For transatlantic aborts, that number was 31 degrees nose high. The recognized expert on the RCC was Dr. Don Curry of Johnson Space Center. He knew everything there was to know about the wing leading edge materials, structure, testing, and capabilities.

During the last week of Columbia’s flight, I was in Houston and attended the MMT on Monday morning in person. Calvin Schomberg of JSC’s Engineering organization gave the discussion of preliminary results on possible damage to the shuttle tiles from the ascent debris strike. Much has been made of this analysis in the CAIB report. There were flaws in the analysis, but post accident testing showed that the bottom line was correct: a glancing foam strike on the underside of the left wing would have damaged the soft thermal tiles but probably not to the point at which fatal heat would reach the interior of the wing. Calvin was a recognized expert on the shuttle tile system. After discussion of other minor issues on the mission and the status of the ongoing experiments, the MMT was adjourned.

In the hall outside the meeting, I encountered Don Curry. I asked him if there was any concern with the RCC. His reply ‘Oh, the RCC is tough stuff. You know during qualification testing we even shot ice at it. The RCC is OK.’ That was good enough for me. The expert had spoken. It never occurred to me to ask anyone else; nor did the question come up formally during the MMT review.

And of course, the accident investigation – all those pieces picked up in East Texas – showed that the tiles were intact; the RCC had taken the strike – and had broken.

So all the discussion in the accident report about the flaws in the tile analysis are simply not applicable. We were working the wrong problem. The hard RCC panels in the very front of the wing, not the soft silica thermal tiles on the bottom of the wing were at issue.

I spent a lot of time the early part of the week in the Mission Evaluation Room where the engineering analysis teams were headquartered; I sat through more than one MMT; and I visited with my fellow Flight Directors in the Flight Control Room. All was quiet, nobody talked about any serious concerns about anything; just the usual logistical administrivia of getting on with a routine shuttle mission.

Jon Harpold was the Director of Mission Operations, my supreme boss as a Flight Director. He had spent his early career in shuttle entry analysis. He knew more about shuttle entry than anybody; the guidance, the navigation, the flight control, the thermal environments and how to control them. After one of the MMTs when possible damage to the orbiter was discussed, he gave me his opinion: “You know, there is nothing we can do about damage to the TPS. If it has been damaged it’s probably better not to know. I think the crew would rather not know. Don’t you think it would be better for them to have a happy successful flight and die unexpectedly during entry than to stay on orbit, knowing that there was nothing to be done, until the air ran out?”

I was hard pressed to disagree. That mindset was widespread. Astronauts agreed. So don’t blame an individual; looks for the organizational factors that lead to that kind of a mindset. Don’t let them in your organization.
After the accident, when we were reconstituting the Mission Management Team, my words to them were “We are never ever going to say that there is nothing we can do.” That is hindsight.

That is the lesson.

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and has retired from consulting and is currently a full time grandpa. He might be available for speaking engagements for the right incentives (coffee and donuts work!)
This entry was posted in After Ten Years and tagged , , . Bookmark the permalink.

41 Responses to After Ten Years: Working on the Wrong Problem

  1. rt steinhoff says:

    Life over drag?

  2. Dave Huntsman says:

    Concur.

  3. charlie barber says:

    Never say never……..

    Thanks for the continuing effort of yours to tell a difficult story… Take care.

  4. Fredric Mushel says:

    The odds of the Columbia accident occurring with that shuttle being on a science only mission, without the remote manipulator system (robot arm), in an orbit out of reach to the ISS, and the foam striking the leading edge of wing (RCC panel), is probably higher than the (odds) chance of having a winning ticket in all the state, and multi-state lotteries combined. But unfortunately it did happen.

    • Phil says:

      Add to that the odds that it was one of the only two missions (the other was STS-109) in many many years where the entry path took place over land and made it possible to recover the debris.

      Or that if the mission had delayed one more time (either the BSTRAs or the damage to Endeavour’s robot arm) would have resulted in Wayne in the Mission Management Team chair instead of Linda Ham.

      Or if a single quality inspector hadn’t noticed the flowliner cracks in the summer of 2002 then STS-107 would have flown before STS-112 (where bipod foam was lost and not addressed with enough concern).

      or a dozen other more subtle coincidences.

      No deep thoughts – but it’s something I’ve thought about many times over the past decade.

  5. Dave H. (not Huntsman) says:

    I’m glad today, after reading this, that I’m “unqualified” to be an astronaut! It would seem that the trust of those who fly was sorely misplaced.

    Wayne, you can embargo this post, but you can’t change what happened nor can you take anything away from me. The following story is the truth, like it or not.

    You see, sometimes real life IS like Hollywood, not as you believe. I sent my journals to our mutual friend Reads, perhaps someday you might want to read them. My children and grandchildren will have an amazing story to read.

    In the past ten years, I’ve attended a CAIB hearing, walked beneath Endeavour, been invited to Sean O’Keefe’s farewell gathering, and been “inside” to watch as Atlantis closed out the STS program. But here’s where I, along with Dr. Douglas Osheroff, made a difference for STS-133.

    Do you remember the issue with ET stringers exposed by intertank region foam liberation being cracked on STS-133? They planned to launch on December 17th, but after Thanksgiving we acted.
    If you found termites in one floor joist you’d have all of them inspected, right? That’s a precious commodity called “common sense”. If you found one stringer cracked it made sense to have them all inspected but no one was seeing this! Launch fever had taken hold. If you believe in serendipity it makes sense why I carry the card of a KSC engineer who was Chief-ET/SRB & Launch Accessories Systems Branch in my wallet.
    When I e-mailed him and explained why inspecting all of the stringers made sense, he failed to reply. Two days later, Dr. Osheroff lent his voice with a missive reminding the engineer that acceptance of normalization of deviance was a major factor in the loss of Columbia.

    Two days later, the December 17th launch of Discovery and STS-133 was postponed and a root cause investigation begun. The investigation would find that the stringers had been improperly heat-treated and that they were also used on the tank slated for STS-135.
    Failure of any four adjacent stringers would have caused catastrophic failure of the ET during launch. They ended up stripping all of the foam and reinforcing 94 of 108 stringers on Discovery’s tank.

    There were many other “coincidences” along the way, but you’ll need to talk to Reads because you don’t want to hear it from me, a nobody. All that mattered was that no one else got hurt or killed while the program was alive.

    No, Wayne, it’s not like Hollywood at all. It was better. In the beginning, it was a lot like “Field of Dreams”, then it morphed into “Phenomenon”, then a little bit of “The Mothman Prophecies”, and at the end I had a “Big Fish” story that my family and everyone at Process Instruments lived.

    Thanks to you and everyone else who let me in and encouraged me. I can only hope that you, and everyone else who was part of Columbia, can find a certain sense of redemption in the fact that we kept Discovery from going into history alongside Challenger and Columbia.
    Not me…WE did it!

    I couldn’t do it alone.

    • waynehale says:

      I’m glad to hear, that after I retired from NASA, they kept high standards and did what was necessary to keep the shuttles flying safely until the end.

      BTW, Readdy was also retired during the STS-133 period so I don’t know how that would have helped

      Anyway, the only comments of yours that I haven’t posted are those where the Hollywood movie references were just too much. Try to move away from those, please.

      • Dave H. says:

        Wayne,

        Thanks for writing back. If you don’t like the movie references, you have my permission to edit them out. It still changes nothing for me.
        The blog posting that got Sean O’Keefe’s attention and opened the door for me to contribute further made reference to “For Love Of The Game”.

        If Homer Hickam can make a movie about a kid with a dream going from the coal mines to the launch pad then I can compare my “great adventure” to any number of movies. People from West Mifflin just don’t go to NASA.
        But I did.

        My “adventure” inspired a great number of people in my world. Starting with my son, now three years at Penn State studying Architectural Engineering, to three of the “kids” from Process Instruments accepting positions with Siemens Medical, to two who took engineering positions with Range Resources and Duquesne Light, to the staff at Elrama Power Station who was there with me at the conclusion.
        It made a difference. It’s a long way from the steel mills to LC39A, but I made the trip.

        But if I could, I’d trade all of it if it would bring Columbia and her crew home safely.

        We can’t change the past, and in order to change the future we have to have a certain vision and a drive to stick to what you know is right.

        Doug summed it up best:

        Dave:

        Let’s hope that we never have the opportunity to be heroes again.

        Doug

        Like I said before, you really should read my journals. I had to learn things I never knew I had to learn, and to take leaps of faith.
        The goal was to keep any more names from going up on that gray wall in Florida, and it didn’t matter what we had to do to accomplish that goal.

      • Dave H. says:

        After all of this time, I finally found the answer.
        Reads, Sean, Doug, and others provided a key ingredient: encouragement. Somehow they understood what I was trying to accomplish and guided me to that end. I was the “apprentice” and they were the “journeymen”.
        When I was working every so often I’d encounter a young kid with skills and initiative. Skills are easy to find; initiative, not so much. When I found a kid with both I shared my knowledge with him and gave encouragement and guidance. After having been a tradesman since 1978, I found myself to be an apprentice in the spaceflight arena in early 2003.
        Twenty years later, I’m retired, with my body unscathed by the last 13 years of my career spent keeping Elrama and Cheswick power stations running. I keep in touch with my former apprentices, and while they occasionally ask a question or two they thank me for sharing what I learned with them. And I suppose that learning and passing that knowledge and experience on to the next generation is really what it’s all about.
        Encouragement and guidance.

  6. Beth Webber says:

    ‘After the accident, when we were reconstituting the Mission Management Team, my words to them were “We are never ever going to say that there is nothing we can do.” ‘

    This is what I expect of NASA.

  7. Frank Ch. Eigler says:

    “We are never ever going to say that there is nothing we can do.”

    Excellent. It clearly doesn’t guarantee success, but at least makes its probability non-zero.

  8. Frank says:

    Thanks for writing these Wayne. I know some of this must be difficult to put down, but your watermark on this moment of history is very valuable to the future.

  9. J. Osborn says:

    “‘Oh, the RCC is tough stuff. You know during qualification testing we even shot ice at it. The RCC is OK.’”

    Sheila Widnall, member of the CAIB, said during a lecture at MIT that the only evidence of RCC impact testing that the CAIB could find were some tests involving firing some small metallic spheres, like BBs, at some RCC during the 1970s. That the RCC is “tough” seems to be one of those things that existed almost as folklore. Everybody seemed to know it, or thought they knew it, but didn’t have any evidence to back it up. One lesson I took from all this was to be the annoying guy who keeps asking “How do you know this? What’s your source?” You’d be amazed at how often the source is something like “I think so-and-so might have said something about it…”

    • Karen says:

      This is what makes me saddest. We did study bird and ET foam impact on the Shuttle forward windows, widely recognized as brittle and sensitive components of the thermal protection system. We did that work in the late 80s. We should have done the same for the RCC. It was a sad and tragic mistake to overlook the brittle nature of the RCC.

  10. Frank, your blog entries are the most difficult and most important essays I read each week. Thank you for doing them.

  11. Kathy Kaminski says:

    Wayne,
    Thank you for telling your story. As heart-wrenching as it is for us to revisit, it must be told. If we don’t continue to reiterate the lessons learned, the tragedy will be far greater.

  12. cthulhu says:

    A technical questions: Don Curry of JSC knew more about the RCC than the people who designed it and made it, the former LTV Aerospace (which was part of Lockheed Martin by the Columbia accident in 2002)? LTV’s Grand Prairie TX operation was the acknowledged world leader in RCC in the’70s, ’80s and ’90s, which is why they were picked to to the Shuttle RCC; they were doing stuff like building gas turbine hot section rotors out of RCC to try to improve efficiency by burning hotter. Were all of those expert folk gone in 2002?

    • waynehale says:

      Don probably knew as much about the actual material and its properties, but he certainly knew more about the environments in which it operated. The LTV operation in the early 200o’s was mainly a production and repair facility, the engineering support was long assigned to other projects and not available to us. That Grand Prairie operation has been completely liquidated with the retirement of the shuttle

      • cthulhu says:

        Too bad; there were some seriously sharp people working on that stuff when I was there. But I left in the mid ’90s for greener pastures, just after LockMart bought us.

  13. Andrew W says:

    Don Curry didn’t run the numbers.

    Something that’s been bugging me, though a moot point, is:
    If a Orbiter were trapped in orbit, how long could a crewed Orbiter be kept habitable through resupply, what systems would be the first to fail because of the impracticality of replacing parts, or topping up consumables?

    Hope you don’t mind me going a bit off topic Wayne, but it’s the people who read your blog who are most likely able to answer.

    Thanks.

    • waynehale says:

      They had less than about three weeks of air. How do you propose to resupply them?

      • Andrew W says:

        At Rand’s place I pointed out that there was a Pegasus launched 9 days after the loss of Columbia, and a Delta launched 5 days after that, both from CC,

        Nemo replied:
        “Neither of those two were considered by the CAIB. The CAIB did consider an Ariane 4 that launched from Kourou on February 15 but offered no conclusions (CAIB D.13 section 5.2).

        Keep in mind that none of the vehicles were capable of active rendezvous; Columbia would have needed to conserve enough consumables to power-up and perform the rendezvous. That would appear to rule out both the Delta and the Ariane, and the Pegasus would have been marginal.

        STS-107 was not a rendezvous flight and did not carry rendezvous checklists, but checklists could have been uplinked. Husband and Chawla had rendezvous experience and McCool was rendezvous qualified.

        In order to stretch the crew consumables that far, the crew would have discontinued exercise and would likely have been too deconditioned for a repair EVA. Therefore this option was less likely to succeed than the Atlantis rescue option.”

        http://www.transterrestrial.com/?p=46830#comments

        If we assume that a rendezvous with an unmanned vehicle was at least a possibility, were there systems on the orbiter that simply couldn’t have been maintained for very long, as an example: Topping up the fuel cells with LH2, or could that issue have been bypassed by plugging another power source into the system (as I imagine is done when the Shuttle is waiting on the launch pad).

      • waynehale says:

        A very interesting option but one that has probably more challenges than scrambling the next shuttle. In any event, we missed any opportunity to do anything. If the organization had been more prone to take action then some type of option might have worked; all were long shots, all would have taken immediate radical action by the entire spaceflight team. That didn’t happen.

        Avoiding that organizational mindset for future human spaceflight – or other hazardous activities – is one of the main reasons I am writing this account.

      • Andrew W says:

        It’s hardly beyond the realms of possibility that in the future we will see an isolated manned spacecraft in LEO unable to deorbit, so maybe it’s worth some effort to think about the design of manned spacecraft and contingencies for last minute changes of mission for unmanned launch vehicles with the practicalities of emergency unmanned resupply in mind.

      • Dave H. says:

        Wayne, were no-win scenarios ever simulated and/or practiced at Mission Control?

      • waynehale says:

        All the time . . . just not on purpose. Hoot’s law: “There is no situation so bad that you can’t make it worse”

      • Michael Grabois says:

        Andrew W asked:

        “… Were there systems on the orbiter that simply couldn’t have been maintained for very long, as an example: Topping up the fuel cells with LH2, or could that issue have been bypassed by plugging another power source into the system (as I imagine is done when the Shuttle is waiting on the launch pad).”

        None of the orbiter’s consumables could be refilled on orbit, there was no way for the crew to access any of them via EVA. The fuel cells used cryogenic O2 and H2, while the crew used the same O2 (warmed up) and N2 for breathing, all under high pressure. The O2 and H2 were around 800 psi and the N2 was at 200 psi in the tanks. And there’s nothing portable that could be used to power the orbiter via the T-0 umbilicals, that’s all connected to ground power sources when the shuttle is on the launch pad or in the OPF. Similarly, the propellant tanks are not accessible in space.

      • Phil says:

        If you assume – as an axiom – that it’s worth having the capability to resupply a space mission on an emergency basis in case anything unexpected happens, then it would have made sense to have a Pegasus or Athena-class launch vehicle on “hot standby” with a miniaturized MPLM available and always ready to go.

        Once an emergency is declared the module could be loaded with whatever emergency supplies (LiOH cans, batteries, jumper cables, etc.) and launched into an appropriate rendezvous orbit with the stranded spacecraft.

        I’m kind of surprised that something like this wasn’t developed after Columbia, certainly it would have added an additional safety margin, in particular for the STS-125 mission.

        The concept of hijacking a Pegasus, Delta, Ariane, or even MX ICBM during the two weeks Columbia was in space, developing a new flight profile, making some kind of carrier for emergency supplies, etc. etc. is really grabbing at straws. While it doesn’t violate any laws of physics it seems to me to be the same as those who use 20/20 hindsight to describe how the passengers on the Titanic could have been saved, Pearl Harbor and 9-11 could have been anticipated and avoided, or any other Monday Morning quarterbacking. It’s always easier with hindsight.

  14. Chelsea Bak says:

    I was only 10 when I heard the news of Columbia’s horrific end, and I cried for many many hours that day (while no one in my household seemed to blink an eye). I’ve always had a deep love for space and the work that goes into learning about it, and I’ve finally allowed myself to find out the true details of 2-3-03. I was shocked by the thought of so many suppressing their own concerns that something should be done to save those seven beautiful people, however it is a lot easier to say in hindsight that they could have been rescued. While I don’t agree that nothing should have been done, I think I agree that I would not want to know if I was going to die after an incredible spaceflight. The only reason I’d want to know would be to say goodbye to my family. All of the processes that allowed for the repression of the call to action to save the crew are normal and occur everyday in all settings. It is apart of a psychological phenomenon known as groupthink. I can see why this happened even though others may view it as common sense in hindsight. The only thing we can do from here is continue to learn from the mistakes, as is demonstrated amazingly in another’s comment, and remember those we’ve lost not only on Columbia, but aboard Challenger, Apollo, and all along the incredible journey. My 10 year old self is finally able to vocalize all the thoughts and feelings from that horrific day.

    • Chelsea Bak says:

      I am very sorry. I obviously meant 2-1-03.

    • Dave H. says:

      Chelsea,

      In the end, the best way to remember those who lost their lives in the cause of science is to not repeat the errors that cost them their lives.

      Had we, as a nation, turned away from this cause because the risks were “too great”, well, we would have done them a great disservice.

      In early July 1971, I was 15 as we toured KSC. I never imagined that exactly 40 years later I would return, not as a tourist, but as a Launch Day guest.
      I give you this advice…learn as much as you can, remember as much as you can, don’t be afraid to take a stand for the right reasons, and don’t ever let anyone move you off your spot!

      At age 57, there’s a lot more I could tell you, but why should I spoil your learning experience?

      Now, go out there and make a difference!

      • Chelsea Bak says:

        Dave,

        Thank you for your comment. I did want to mention, as I saw in another comment of yours, I also attend Penn State and am a third year student as well. Maybe I have crossed paths with your son. Anyway, thanks for the advice. I was touched by your story about Discovery, and applaud your leadership. Thank you for, as Mr. Hale said, “[keeping] high standards” so that the integrity and intelligence of NASA can live on!

  15. Jason L. says:

    Mr. Hale,

    In this article you discussed an analysis that determined 31 degrees nose high was the minimum angle that provided maximum reentry glide range without burning up the vehicle. Could a higher angle been used to reduce the amount of time Columbia was exposed to heat during reentry? In other words, could a higher intensity reentry of shorter duration been better than the nominal reentry?

    Thank you for this compelling and sobering series. Also, thank you for your willingness to take our questions and comments.

    • waynehale says:

      Lower the nose and you increase the thermal environment on the wing leading edge RCC panels; raise the nose and you increase the thermal environment on the silica tile covered aero surfaces at the rear of the vehicle. Threading the needle is very close work. 40 deg (+/-2) optimized the heat loads.

  16. Richard Merrill says:

    Even if there was nothing that could be done to save the orbiter and crew, there were still range safety considerations. Had we known there was a significant chance of the vehicle breaking up on entry, would we still have chosen a deorbit that took the vehicle over major populated areas? I would hope not.

  17. Kelly M. says:

    I just read the article that ABC has that refers to your blog. The title of the article caught my attention…”If Space Shuttle is Doomed, Do you Tell the Crew?” What a dilemma that would have been if you all had known what was to happen that fateful day. I was intrigued that all had agreed earlier in the week, when the hypothetical question about issues was asked, that it would be best if the crew didn’t know of problems. Looking back, all agree that if the issue had been known, everyone would have done everything in their power to bring the crew back safely and the crew would have had the comfort in knowing that. Ethical decisions face all of us everyday and according to philosopher Charles Taylor, ” Ethical judgements are simply matters of individual personal opinion anyway, so there are no final answers; and it is presumptuous, perhaps even unethical, to judge the ethics of others.” With that being said, I can see that not telling the crew would have been ethical because the intent would be to protect their peace of mind. On the other hand, I can see that telling them would have been the right thing to do as well, because the intent would be to show that everyone was doing everything in their power to help them and they could take comfort in that knowledge. Thank you so much for you blog. It is very obvious that this tragic event had a huge impact on your life.

    Johnnesen, Richard l.,Valde, Kathleen S., Whedbee, Karen E. 2008. Ethics in Human Communication.Long Grove, IL. Waveland Press, Inc.

    Kelly M.
    Drury University Student

    • waynehale says:

      No ethical dilemma at all; we would have told the crew immediately. The subtle point that some have missed is not a discussion of what to tell the crew, but how hard to work to find the truth. We told them exactly what we thought . . . but we didn’t think about it hard enough. Do you see the difference?

  18. Kelly M. says:

    I do see the difference so thank-you for clarifying. It seems that the article I read might have been a bit miss-leading. Again…I appreciate your response. I’m in an Ethical Communication class and much of what we have been discussing concerns the media and ethics.

  19. John Dominic says:

    My grandpa worked for NAVY during WWII. He always told me one thing, “Never ever say, there is nothing I could do. Instead its worth dying doing something, allowing others what it takes to be a fighting spirit.” He was right. Now, dear Wayne, I appreciate your courage. I know what it takes to be different. God bless you.

  20. RIP kalpana chawla ,really sad about the incident happened ……

Leave a comment