When you are in a High Reliability Organization, you have to pay attention. The more extreme the risk, the more difficult the environment, the more complex the technology, the more attention you must pay. But life intrudes. As the song says, there are planes to catch and bills to pay, and too there are little children to play with. You cannot work all the time, and if the balance gets too far either way, a price will be exacted. Once upon a time I wrote: “pay attention, do good work, have no regrets”. True, but far too simplistic.
In 2002 I thought we were paying the right level of attention to the shuttle. I thought I was paying the right level of attention to the shuttle. I was a Flight Director. I was also a husband and a father and active in my community. I thought I could do it all.
I was wrong.
Later on, I will write about the MLK three day weekend that cost us a crew just because we took three days off. But how can you know in advance where the proper balance is between work and life when you work at extreme risk?
This is my personal recollection of the events leading up to the Columbia accident nearly ten years ago, so let’s get personal. I will start with the work side; later I will visit the ‘life’ side.
In 2002 I was a Space Shuttle Flight Director at the Johnson Space Center. Not only that, I was the Deputy Chief of the Flight Director’s Office for Space Shuttle Operations. That meant that I had managerial responsibilities; personnel responsibilities, budgetary responsibilities. Oh, that last one, about budget – we will visit that again in a later post. How I hated budget responsibilities in an era of cost cutting.
Life was a continuous round of meetings, teleconferences, and management decisions. I regularly attended the Shuttle Program management meetings and had input into many of the decisions that were made in those days. We evaluated in flight anomalies, provided recommendations on the budget, questioned technical issues, and tried to pay attention to everything. Even though the organization chart showed I was in the Mission Operations organization, we were all part of “the Program”.
I had just come off a successful Lead Flight Director assignment in December 2001 for STS-108/ISS UF-1. Being a lead flight director is a year long assignment; the Lead Flight Director is responsible for all aspects of mission planning, training, and execution for a shuttle flight. It was a tremendous amount of pressure and a huge satisfaction when it was over. ISS Utilization Flight-1, commanded by Dom Gorie, was a logistics flight with a Multi-Purpose Logistics Module full of supplies and a large quantity of scientific experiments for the ISS. The most emotional part of the flight was the 9/11 remembrance which got punted to me. The list of mission objectives was long. We completed them all, just another 100% successful shuttle mission. Except for the inflight anomalies on the shuttle system, nothing untoward occurred, and even those were nothing to worry about. Or so we thought.
It was my third, and as it turned out, last assignment as a Lead Flight Director.
Every shuttle flight has three to five Flight Directors assigned. One FD was the Ascent/Entry guy – my usual job. The A/E FD worked shifts prelaunch, launch, went to in standby/monitor mode for the middle of the flight, and then worked one pre-deorbit shift (Flight Control System Checkout day) and the Entry shift. Or Entry shifts, as weather caused frequently delays. During most of the flight, while the on-orbit flight directors were in charge, the A/E FD could be found most often in the JSC Weather office kibitzing with the meteorologists, eating their food (they always had the best snack layout), hanging out in the back row of the MMT, and generally being a nuisance. During the ‘on-orbit’ period, there were three shifts of roughly eight hours each, cleverly named the Orbit 1, Orbit 2, and Orbit 3 shift. Sometimes we called the Orbit 3 shift (when the crew was asleep), the Planning shift because every “night” the plan would be reviewed, modified, and changes sent up to the astronauts. If the flight was more than about 10 days long, an additional Orbit team would be assigned to allow the three primary shifts to have a day off. Shift schedule planning could be complex. Typically the Lead Flight Director worked the Orbit 1 shift. But on every flight, because the FD had to stay in the Mission Control Room working the minute by minute details, a manager from the Mission Operations organization was assigned to be the go between from the flight control team and the on-console flight directors to the Mission Management Team, aka NASA senior management. The Mission Operations Director had several formal duties, most important of which was as a member of the MMT. The MOD did not work a specified shift but was always in the MCC during the most critical and dynamic times. So when the on-console FD was under the most stress, that was the precise moment when the MOD, the FD “boss” would show up: to sit on the console directly behind the FD, and offer “advice” and “encouragement” and write up performance appraisals. The MOD, along with the Lead Flight Director, also flew to KSC for the Flight Readiness Review about two weeks before launch where about a hundred of the NASA senior managers met face to face to ensure the shuttle was ready for launch.
I was assigned to be the MOD for the 4th Hubble Space Telescope servicing Mission, STS-109, which flew in March of 2002. It was a hectic mission with lots of little issues, but no big ones that I recall. I hope I wasn’t too much of a nuisance to the flight control team doing the real work!
Finally, for 2002, I was assigned as the STS-113 A/E Flight Director. STS-113 or ISS Assembly Flight 11A carried a huge solar array – Port Array #1 – up to the ISS. Jim Wetherbee was the commander. We were all informed of the in flight anomalies from the previous flight, STS-112. Among those anomalies was the loss of a large segment of insulating foam from the External Tank which fell and struck the left hand Solid Rocket Booster. This strike left a large smudge on the steel case. We Flight Directors speculated that if that piece of foam had struck a critical electronic box on the SRB not far from where the smudge was located, that BAD THINGS might have happened. But then, we Flight Directors were always worry warts. At the FRR the foam loss was deemed “not a safety of flight issue”. The A/E FD does not go to FRRs. In retrospect, of course, I wish I had gone. Maybe I could have gotten the captain to turn the Titanic away from the iceberg. But that didn’t happen. No significant foam loss occurred on STS-113, so we all shrugged and agreed it had been a one time thing.
We were wrong. STS-113 was the last successful entry of a space shuttle flight for three years.
We all knew that Jim Wetherbee was slated to take over JSC’s office of Safety and Mission Assurance right after his flight; so for a humorous moment, I got on the Air to Ground radio after the landing and spoke with him. It is not typical, but not all that unusual, for a Flight Director to pre-empt the CAPCOM and talk directly with the crew. After the successful landing I was in a jocular mood so just before Wetherbee was to leave the cockpit, I called him up (you can probably find the recording somewhere) and congratulated him on a successful flight and told him “your desk in Building 1 [JSC’s management building] is waiting for you”. Fate is funny; it turned out that prophecy applied to me too, but I didn’t know it at the time.
It was my last time to sit at the Flight Director’s chair in Mission Control. Best job of my life; I miss it every day; but it was over and I didn’t know it.
So work was a 60 hour a week job; it was always on my mind. There were a thousand details to plan and decide and brief to management. Even then I recognized that there were a million details in the shuttle business that I couldn’t participate in; there just wasn’t enough time or energy for any one person to do it all. The folks in the shuttle program office, Ron Dittemore, Linda Ham, and the rest, they had a tough job. Tougher than you know. Tougher than I knew at the time. Everybody played their part, everybody was highly motivated, everybody wanted the shuttle to succeed, and yet, within months, we were all to fail.
Why? Listen and learn. Maybe you will avoid failure. But don’t count on it.
Wayne Hale's Blog 
Another insightful post. Some of this is surely hard for you to write, but thank you for letting us see your view inside this tragic event.
Thank you for sharing this, Wayne.
Thank you for writing about this difficult time. It’s the best way for us to learn from the past, and prepare for the future. You honor your comrades with this writing. Peace.
Wayne, Is there any credible evidence that once that vehicle got to orbit the crew could have been saved, even if the extent of the damage was precisely known? Certainly heroic efforts would have been made, but I just don’t see a solution.
There is an appendix in the CAIB report devoted to a possible crew rescue. John Shannon lead the effort. If we had known early and decided to launch 114 ASAP there was just the barest long-shot of a chance to pull that off.
I’m sure you share my frustration, Wayne, but the Shuttle Program’s decision to even look for port wing damage during STS-107 ranks as the biggest missed opportunity of my 60-flight career in Mission Control. Without an RMS aboard, the crew would have had a sporty EVA to perform an in-situ inspection, but remote sensing “resources” at the Program’s disposal might have justified this EVA at relatively negligible expense. A successful STS-114 rescue mission would certainly have rivaled Apollo 13 as Mission Operations’ finest hour. I look forward to your discussion of this missed opportunity and whether or not the Program’s culture led to it, as the CAIB report seems to suggest.
Careful Dan; you and I were both part of that “culture”. I feel very keenly the need not to point my finger at others. Better we should look to what we could have individually done to keep failure at bay. What could you have done? Hindsight is 20/20 and courses of action that seem obvious in retrospect are not at all clear at the time the decision has to be made.
Copy all and concur, Wayne. It wasn’t my intent to pass judgment on historic events but rather to suggest how close we were to obtaining that critical additional situational awareness leading to a rescue attempt. All we needed to do was look, and we’d “find ourselves in a role where our performance has ultimate consequences.” I honestly can’t think of a greater missed opportunity in my flight control career and will carry the regret of not scrutinizing Columbia on orbit with my USSTRATCOM colleagues to my grave.
As do we all. I wish we had taken the opportunities to save the crew when we had them.
Dan,
History shows that prior to STS-107 foam strikes were commonplace. Until then foam had never caused a “serious” problem, so if you base your organizational safety upon “In God we trust, all others bring data” the then-available data didn’t merit additional attention.
Don’t forget that up until the second Scott Hubbard’s “Mythbusters” test blew a hole in the RCC the conventional wisdom was that the foam impact had the same effect as your car impacting a foam beer cooler at 55 miles per hour.
Replace “In God we trust, all others bring data” with “I don’t have data to prove my point but what harm is there in checking it out?”
“I don’t think there is much we can do” gets “but have the photos taken and let me know what you find” added to it.
Wise engineers know to include input from anywhere and everywhere. They know how to winnow the chaff from the wheat.
Bill, like Wayne said above, the CAIB report says that in theory, there are things that MIGHT have been done that MIGHT have helped save the crew. But the analyses that came up with these took literally months of work by dedicated teams, and there would have been only about 3 weeks to rescue the crew with an all-out effort (the limiting factor would have been lithium hydroxide canisters to scrub the CO2 out of the air, just like on Apollo 13). Not all of that analysis would have been available at the time so NASA would have been forced to go with a “this is the best we could do on short notice” plan.
Wish we had the chance, now we’ll never know if we could have pulled it off.
I’m listening.
Was there any inspection of the impact on the RSRB casing post recovery? E.g. did it dent the steel or simply scrape it?
In the FRR’s assessment of the impact, what technical basis did they use to dismiss the STS-112 foam anomaly?
Appreciate the candor as always.
Didn’t even scrape it, just left a smudge.
Since previous foam losses hadn’t caused serious damage, it was a ‘failure of imagination’ to jump to the conclusion that they never would. Things are so much clearer in hindsight; its predicting problems in advance that is hard.
Although you didn’t say so, I have always assumed based on the photo at the top of your blog that Flight Director is also the job that you are the most proud of.
yep
Great start. I am not sure anyone knew how hard the Program Office jobs were and trying to influence dicisions proved just as difficult.
Pinpointing those moments where something went wrong…. It seems like you’re still on the job while writing this. Your insight gave me a glimpse into a place I’d have never really known about had you not written so clearly but also so emotionally. And, then, there’s that word failure that’s so familiar to us all. Thank you.
Wayne,
I never would have surmised that you are a Harry Chapin fan! He was a good story teller. “Mr. Tanner” is a good story.
“In retrospect, of course, I wish I had gone. Maybe I could have gotten the captain to turn the Titanic away from the iceberg.”
You’ve done well in how you’ve constructed this opening segment of your story. Set the stage, introduce the players, sprinkle with a few well-seasoned clues. I look forward to more.
One question for you to wrestle with: in an environment defined by “In God we trust, all others bring data” how would you have earned the Captain’s trust?
Don’t tell us here…it’s a plot point we’ll learn as the story unfolds.
Surely another FD was present at the FRR, so why didn’t he bring up the FD worries about the foam potentially hitting the electronic box or some other critical part (e.g. a leading edge of a wing)? Or wasn’t it given the proper attention because it came from the worry warts?
Anyway, great buildup…. I am very eager to find out whether giving up the FD position was your own choice, so please don’t make us wait another month for the next chapter…
That story comes later
which story are you referring to, to come later; “the disregarding of the worry warts at the FRR” or “the giving up of the FD position”?
Oh, both i guess.
You have created a new opportunity for those of us who witnessed history to feel more intimately engaged. Thank you.
I attended the STS-113 FRR and had no dog in that fight (I was ISS at the time), but the foam loss issue was just one of many potentially catastrophic failures that was evaluated. It did not even appear to necessarily represent the most imminent risk to the shuttle, but was just one among many risks including concerns over the SRB hold-down bolt pyro failures, flowliner cracks, turbopump seal damages, ET stringer manufacturing problems, and other standard items (such as MMOD risk). I remember saying to someone at the time that I was glad I was not the Program Manager, as I thought it would be impossible to arrive at a “go” decision with the volume of open issues and concerns that had been expressed (my “error chain” violation sense was high), and no quick fixes available. The meeting broke late in the day for key participants to consult outside the FRR room. When we reconvened, many of us expected that the discussions would continue, but instead the Chair only stated that open issues would be worked and readdressed at the L-2 MMT, and then just ended the meeting. It was a very unsatisfying conclusion to the FRR, but I quickly forgot about it after the successful 113 flight. When the sea is full of icebergs, which way would you turn?
Ben, as you are well aware, assessing risk can be very subjective. Every launch there were a dozen “issues” any one of which could have been fatal. And the program paid a lot of attention to all the issues. But once, maybe twice, we didn’t pay enough attention; we didn’t mitigate the risk, we didn’t slow down. How many times did we overwork other problems? How many times did we delay launches that could have gone on time? I don’t know, but more than a few I suspect. So you and I, we together need to write down our experiences so that others, faced with similar difficult decisions in the future, can have the benefit of our experience.
Wayne,
I stumbled, quite by accident, onto this post yesterday. Funny how the universe works because I was thinking about Columbia and Challenger earlier in the day. I was too young to remember anything about Challenger, but Columbia, I can remember vividly. I am very much looking forward to your insight and perspective on a very difficult time in our Nation’s space history. Thank you for your service.
No…The Flight Dynamics MPSR had the best spread…lol
Food was plentiful in the MCC. In an earlier life I blogged about the ice cream social we had during STS-3. Look at the NASA.gov web page under archived blogs and you will find my old stuff and there is the story of how the planning team didn’t get the plan done one night . . .
Agree Yusef. I think we always had the best spread, but then could go down to SMG for cheesecake, and toward the end, carrot cake that rivaled my grandmothers:) Still waiting for your “Cookbook” 🙂
Thanks so much for these stories. As I’m just starting my career, I hope to learn from the wisdom of others.
I sure dislike this time of year when we are reminded of our failures and the incredible lives we lost. We all wanted to be successful. However, we often make mistakes because we don’t have sufficient data, ask the right questions, or allow our biases to cloud our reasoning.
Twenty years ago, a conscious decision was made NOT to request imagery of the leading edge. That was the ‘culture’ at that time. But I would argue, at T-0, the Columbia crew’s fate was sealed. Sadly, there was no going back. Foam damage from the previous flight was not fully appreciated.
So, given the decision to forgo imagery–not that it would have secured the crew–to suggest today ‘NASA’ would have agreed to accelerate the launch of another shuttle and execute a rescue mission without any real strategies or procedures, doesn’t seem a reasonable scenario. The next shuttle was not ready to launch. Shortcutting it’s ground preparation would have required deviating from nominal processing, posing extreme risk to crew and vehicle. And given the damage to the SRB from the previous flight, and wing leading edge damage to Columbia, the odds of another critical foam strike could not have been justified–certainly not in the timeframe of the remaining consumables. How many links in the ‘error chain’ would have had to have been deliberately broken to execute a rescue mission? How many hours were regularly spent debating issues during routine readiness reviews, where operations were all preplanned? This rescue would be no routine mission.
In response to arguments about saving the crew–who were among the most brilliant and talented, national treasures, including an esteemed Israeli fighter pilot–recall that in 2014, two brothers were among four workers who lost their lives in a hazardous chemical leak at a Texas industrial plant. One employee entered an area and was quickly overcome by Methyl Mercaptan fumes. Three others were subsequently lost in separate rescue attempts. In each case, all protocols, procedures and required equipment were disregarded. These rescue attempts only made the incident more costly.
I know there is a place where the buck stops, and in catastrophes, those who might have had a role in changing the outcome will always second guess what might have been done. I haven’t heard your story, but I know enough about Columbia to know the circumstances leading to the decisions that were made were no one single person’s fault or one condition, like a long weekend, where some singular event tilted the scales towards imminent failure. To be honest, one of the other element’s rationale for going ahead with launch despite a specious rationale was almost enough to delay the flight. But the decision was made “not to call the other Element’s baby ugly”. The winds aloft played a role. A delay could have tipped the scales the other direction. Players in the pre-FRR and post flight assessment look back and wish they had spoken up about misgivings. They all carry a burden of personal accountbility, real or imagined, many to this day. While I know you have fantastic insight and your approach to changing the culture and educating the actors in the great dance of Shuttle missions, I think the real story is one of ego and personal risk tolerance – not about the vehicle but about one’s career and standing. I saw the humble ones focus on the problem. I saw the proud leverage fear in a show of control. The vehicle had become 4 Elements flying in close formation, and that can never be good. The swirling environment of details, risks, and interactions on such a system inevitably overwhelm the best and the brightest. While I am for life/work balance, I don’t think spending more time at work would have prevented the Columbia accident. It wasn’t just the launch decision that drove the fatal outcome. One can only hope the same hubris will not relegate the next human space flight systems to the same type of fate.