Ten Years After Columbia: Balancing Life and Work

When you are in a High Reliability Organization, you have to pay attention.  The more extreme the risk, the more difficult the environment, the more complex the technology, the more attention you must pay.  But life intrudes.  As the song says, there are planes to catch and bills to pay, and too there are little children to play with.  You cannot work all the time, and if the balance gets too far either way, a price will be exacted.  Once upon a time I wrote:  “pay attention, do good work, have no regrets”.  True, but far too simplistic. 

In 2002 I thought we were paying the right level of attention to the shuttle.  I thought I was paying the right level of attention to the shuttle.  I was a Flight Director.  I was also a husband and a father and active in my community.  I thought I could do it all. 

I was wrong.

Later on, I will write about the MLK three day weekend that cost us a crew just because we took three days off.  But how can you know in advance where the proper balance is between work and life when you work at extreme risk?

This is my personal recollection of the events leading up to the Columbia accident nearly ten years ago, so let’s get personal. I will start with the work side; later I will visit the ‘life’ side.

In 2002 I was a Space Shuttle Flight Director at the Johnson Space Center.  Not only that, I was the Deputy Chief of the Flight Director’s Office for Space Shuttle Operations. That meant that I had managerial responsibilities; personnel responsibilities, budgetary responsibilities.  Oh, that last one, about budget – we will visit that again in a later post.  How I hated budget responsibilities in an era of cost cutting.

Life was a continuous round of meetings, teleconferences, and management decisions.  I regularly attended the Shuttle Program management meetings and had input into many of the decisions that were made in those days.  We evaluated in flight anomalies, provided recommendations on the budget, questioned technical issues, and tried to pay attention to everything.  Even though the organization chart showed I was in the Mission Operations organization, we were all part of “the Program”.

I had just come off a successful Lead Flight Director assignment in December 2001 for STS-108/ISS UF-1.  Being a lead flight director is a year long assignment; the Lead Flight Director is responsible for all aspects of mission planning, training, and execution for a shuttle flight.  It was a tremendous amount of pressure and a huge satisfaction when it was over.  ISS Utilization Flight-1, commanded by Dom Gorie, was a logistics flight with a Multi-Purpose Logistics Module full of supplies and a large quantity of scientific experiments for the ISS.  The most emotional part of the flight was the 9/11 remembrance which got punted to me.  The list of mission objectives was long.  We completed them all, just another 100% successful shuttle mission.  Except for the inflight anomalies on the shuttle system, nothing untoward occurred, and even those were nothing to worry about.  Or so we thought.

It was my third, and as it turned out, last assignment as a Lead Flight Director.

Every shuttle flight has three to five Flight Directors assigned.  One FD was the Ascent/Entry guy – my usual job.  The A/E FD worked shifts prelaunch, launch, went to in standby/monitor mode for the middle of the flight, and then worked one pre-deorbit shift (Flight Control System Checkout day) and the Entry shift.  Or Entry shifts, as weather caused frequently delays.  During most of the flight, while the on-orbit flight directors were in charge, the A/E FD could be found most often in the JSC Weather office kibitzing with the meteorologists, eating their food (they always had the best snack layout), hanging out in the back row of the MMT, and generally being a nuisance.  During the ‘on-orbit’ period, there were three shifts of roughly eight hours each, cleverly named the Orbit 1, Orbit 2, and Orbit 3 shift.  Sometimes we called the Orbit 3 shift (when the crew was asleep), the Planning shift because every “night” the plan would be reviewed, modified, and changes sent up to the astronauts.  If the flight was more than about 10 days long, an additional Orbit team would be assigned to allow the three primary shifts to have a day off.  Shift schedule planning could be complex.  Typically the Lead Flight Director worked the Orbit 1 shift.  But on every flight, because the FD had to stay in the Mission Control Room working the minute by minute details, a manager from the Mission Operations organization was assigned to be the go between from the flight control team and the on-console flight directors to the Mission Management Team, aka NASA senior management.  The Mission Operations Director had several formal duties, most important of which was as a member of the MMT.  The MOD did not work a specified shift but was always in the MCC during the most critical and dynamic times.  So when the on-console FD was under the most stress, that was the precise moment when the MOD, the FD “boss” would show up: to sit on the console directly behind the FD, and offer “advice” and “encouragement” and write up performance appraisals.  The MOD, along with the Lead Flight Director, also flew to KSC for the Flight Readiness Review about two weeks before launch where about a hundred of the NASA senior managers met face to face to ensure the shuttle was ready for launch. 

I was assigned to be the MOD for the 4th Hubble Space Telescope servicing Mission, STS-109, which flew in March of 2002.  It was a hectic mission with lots of little issues, but no big ones that I recall.  I hope I wasn’t too much of a nuisance to the flight control team doing the real work!

Finally, for 2002, I was assigned as the STS-113 A/E Flight Director.   STS-113 or ISS Assembly Flight 11A carried a huge solar array – Port Array #1 – up to the ISS.  Jim Wetherbee was the commander.  We were all informed of the in flight anomalies from the previous flight, STS-112.  Among those anomalies was the loss of a large segment of insulating foam from the External Tank which fell and struck the left hand Solid Rocket Booster.  This strike left a large smudge on the steel case.  We Flight Directors speculated that if that piece of foam had struck a critical electronic box on the SRB not far from where the smudge was located, that BAD THINGS might have happened. But then, we Flight Directors were always worry warts.  At the FRR the foam loss was deemed “not a safety of flight issue”.  The A/E FD does not go to FRRs.  In retrospect, of course, I wish I had gone.  Maybe I could have gotten the captain to turn the Titanic away from the iceberg.  But that didn’t happen.  No significant foam loss occurred on STS-113, so we all shrugged and agreed it had been a one time thing. 

We were wrong.  STS-113 was the last successful entry of a space shuttle flight for three years. 

We all knew that Jim Wetherbee was slated to take over JSC’s office of Safety and Mission Assurance right after his flight; so for a humorous moment, I got on the Air to Ground radio after the landing and spoke with him.  It is not typical, but not all that unusual, for a Flight Director to pre-empt the CAPCOM and talk directly with the crew.  After the successful landing I was in a jocular mood so just before Wetherbee was to leave the cockpit, I called him up (you can probably find the recording somewhere) and congratulated him on a successful flight and told him “your desk in Building 1 [JSC’s management building] is waiting for you”.  Fate is funny;  it turned out that prophecy applied to me too, but I didn’t know it at the time.

It was my last time to sit at the Flight Director’s chair in Mission Control.  Best job of my life; I miss it every day; but it was over and I didn’t know it.

So work was a 60 hour a week job; it was always on my mind.  There were a thousand details to plan and decide and brief to management.  Even then I recognized that there were a million details in the shuttle business that I couldn’t participate in; there just wasn’t enough time or energy for any one person to do it all.  The folks in the shuttle program office, Ron Dittemore, Linda Ham, and the rest, they had a tough job.  Tougher than you know.  Tougher than I knew at the time.  Everybody played their part, everybody was highly motivated, everybody wanted the shuttle to succeed, and yet, within months, we were all to fail. 

Why?  Listen and learn.  Maybe you will avoid failure.  But don’t count on it.

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in After Ten Years and tagged , , . Bookmark the permalink.

33 Responses to Ten Years After Columbia: Balancing Life and Work

  1. Beth Webber says:

    Another insightful post. Some of this is surely hard for you to write, but thank you for letting us see your view inside this tragic event.

  2. Mike says:

    Thank you for sharing this, Wayne.

  3. Marie says:

    Thank you for writing about this difficult time. It’s the best way for us to learn from the past, and prepare for the future. You honor your comrades with this writing. Peace.

  4. Bill Killingsworth says:

    Wayne, Is there any credible evidence that once that vehicle got to orbit the crew could have been saved, even if the extent of the damage was precisely known? Certainly heroic efforts would have been made, but I just don’t see a solution.

    • waynehale says:

      There is an appendix in the CAIB report devoted to a possible crew rescue. John Shannon lead the effort. If we had known early and decided to launch 114 ASAP there was just the barest long-shot of a chance to pull that off.

      • Dan Adamo says:

        I’m sure you share my frustration, Wayne, but the Shuttle Program’s decision to even look for port wing damage during STS-107 ranks as the biggest missed opportunity of my 60-flight career in Mission Control. Without an RMS aboard, the crew would have had a sporty EVA to perform an in-situ inspection, but remote sensing “resources” at the Program’s disposal might have justified this EVA at relatively negligible expense. A successful STS-114 rescue mission would certainly have rivaled Apollo 13 as Mission Operations’ finest hour. I look forward to your discussion of this missed opportunity and whether or not the Program’s culture led to it, as the CAIB report seems to suggest.

      • waynehale says:

        Careful Dan; you and I were both part of that “culture”. I feel very keenly the need not to point my finger at others. Better we should look to what we could have individually done to keep failure at bay. What could you have done? Hindsight is 20/20 and courses of action that seem obvious in retrospect are not at all clear at the time the decision has to be made.

      • Dan Adamo says:

        Copy all and concur, Wayne. It wasn’t my intent to pass judgment on historic events but rather to suggest how close we were to obtaining that critical additional situational awareness leading to a rescue attempt. All we needed to do was look, and we’d “find ourselves in a role where our performance has ultimate consequences.” I honestly can’t think of a greater missed opportunity in my flight control career and will carry the regret of not scrutinizing Columbia on orbit with my USSTRATCOM colleagues to my grave.

      • waynehale says:

        As do we all. I wish we had taken the opportunities to save the crew when we had them.

      • Dave H. says:

        Dan,

        History shows that prior to STS-107 foam strikes were commonplace. Until then foam had never caused a “serious” problem, so if you base your organizational safety upon “In God we trust, all others bring data” the then-available data didn’t merit additional attention.
        Don’t forget that up until the second Scott Hubbard’s “Mythbusters” test blew a hole in the RCC the conventional wisdom was that the foam impact had the same effect as your car impacting a foam beer cooler at 55 miles per hour.

        Replace “In God we trust, all others bring data” with “I don’t have data to prove my point but what harm is there in checking it out?”

        “I don’t think there is much we can do” gets “but have the photos taken and let me know what you find” added to it.
        Wise engineers know to include input from anywhere and everywhere. They know how to winnow the chaff from the wheat.

    • Bill, like Wayne said above, the CAIB report says that in theory, there are things that MIGHT have been done that MIGHT have helped save the crew. But the analyses that came up with these took literally months of work by dedicated teams, and there would have been only about 3 weeks to rescue the crew with an all-out effort (the limiting factor would have been lithium hydroxide canisters to scrub the CO2 out of the air, just like on Apollo 13). Not all of that analysis would have been available at the time so NASA would have been forced to go with a “this is the best we could do on short notice” plan.

  5. Charley S McCue says:

    I’m listening.

  6. no one of consequence says:

    Was there any inspection of the impact on the RSRB casing post recovery? E.g. did it dent the steel or simply scrape it?

    In the FRR’s assessment of the impact, what technical basis did they use to dismiss the STS-112 foam anomaly?

    Appreciate the candor as always.

    • waynehale says:

      Didn’t even scrape it, just left a smudge.

      Since previous foam losses hadn’t caused serious damage, it was a ‘failure of imagination’ to jump to the conclusion that they never would. Things are so much clearer in hindsight; its predicting problems in advance that is hard.

  7. Steve Pemberton says:

    Although you didn’t say so, I have always assumed based on the photo at the top of your blog that Flight Director is also the job that you are the most proud of.

  8. Rod Wallace says:

    Great start. I am not sure anyone knew how hard the Program Office jobs were and trying to influence dicisions proved just as difficult.

  9. Pinpointing those moments where something went wrong…. It seems like you’re still on the job while writing this. Your insight gave me a glimpse into a place I’d have never really known about had you not written so clearly but also so emotionally. And, then, there’s that word failure that’s so familiar to us all. Thank you.

  10. Dave H. says:

    Wayne,

    I never would have surmised that you are a Harry Chapin fan! He was a good story teller. “Mr. Tanner” is a good story.

    “In retrospect, of course, I wish I had gone.  Maybe I could have gotten the captain to turn the Titanic away from the iceberg.”

    You’ve done well in how you’ve constructed this opening segment of your story. Set the stage, introduce the players, sprinkle with a few well-seasoned clues. I look forward to more.

    One question for you to wrestle with: in an environment defined by “In God we trust, all others bring data” how would you have earned the Captain’s trust?

  11. Gijs says:

    Surely another FD was present at the FRR, so why didn’t he bring up the FD worries about the foam potentially hitting the electronic box or some other critical part (e.g. a leading edge of a wing)? Or wasn’t it given the proper attention because it came from the worry warts?

    Anyway, great buildup…. I am very eager to find out whether giving up the FD position was your own choice, so please don’t make us wait another month for the next chapter…

  12. scribblechic says:

    You have created a new opportunity for those of us who witnessed history to feel more intimately engaged. Thank you.

  13. BenP says:

    I attended the STS-113 FRR and had no dog in that fight (I was ISS at the time), but the foam loss issue was just one of many potentially catastrophic failures that was evaluated. It did not even appear to necessarily represent the most imminent risk to the shuttle, but was just one among many risks including concerns over the SRB hold-down bolt pyro failures, flowliner cracks, turbopump seal damages, ET stringer manufacturing problems, and other standard items (such as MMOD risk). I remember saying to someone at the time that I was glad I was not the Program Manager, as I thought it would be impossible to arrive at a “go” decision with the volume of open issues and concerns that had been expressed (my “error chain” violation sense was high), and no quick fixes available. The meeting broke late in the day for key participants to consult outside the FRR room. When we reconvened, many of us expected that the discussions would continue, but instead the Chair only stated that open issues would be worked and readdressed at the L-2 MMT, and then just ended the meeting. It was a very unsatisfying conclusion to the FRR, but I quickly forgot about it after the successful 113 flight. When the sea is full of icebergs, which way would you turn?

    • waynehale says:

      Ben, as you are well aware, assessing risk can be very subjective. Every launch there were a dozen “issues” any one of which could have been fatal. And the program paid a lot of attention to all the issues. But once, maybe twice, we didn’t pay enough attention; we didn’t mitigate the risk, we didn’t slow down. How many times did we overwork other problems? How many times did we delay launches that could have gone on time? I don’t know, but more than a few I suspect. So you and I, we together need to write down our experiences so that others, faced with similar difficult decisions in the future, can have the benefit of our experience.

  14. michaeloder says:

    Wayne,
    I stumbled, quite by accident, onto this post yesterday. Funny how the universe works because I was thinking about Columbia and Challenger earlier in the day. I was too young to remember anything about Challenger, but Columbia, I can remember vividly. I am very much looking forward to your insight and perspective on a very difficult time in our Nation’s space history. Thank you for your service.

  15. Yusef Johnson says:

    No…The Flight Dynamics MPSR had the best spread…lol

    • waynehale says:

      Food was plentiful in the MCC. In an earlier life I blogged about the ice cream social we had during STS-3. Look at the NASA.gov web page under archived blogs and you will find my old stuff and there is the story of how the planning team didn’t get the plan done one night . . .

    • Kerri Hagee says:

      Agree Yusef. I think we always had the best spread, but then could go down to SMG for cheesecake, and toward the end, carrot cake that rivaled my grandmothers:) Still waiting for your “Cookbook” :)

  16. Michael says:

    Thanks so much for these stories. As I’m just starting my career, I hope to learn from the wisdom of others.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s