After Ten Years: Too Little, Too Late

First the official disclaimer: I can neither confirm nor deny that other national agencies might or might not have had capabilities that could have helped NASA during the last flight of Columbia.

The fact of the matter is that in January 2003 I had never had a security clearance of the type to let me know anything about such matters. We had all had low level clearances during the DoD flights of the late 80’s, which were allowed to lapse. Some Flight Directors had low level security clearances but these were almost never used. There were rumors, innuendos, lies, and tall tales told during the post flight parties or other occasions about what might be out there, but I never actually saw any evidence of anything. That is where things stood in January 2003.

After the MLK holiday weekend, I traveled back to Florida, to continue my preparation to be the Space Shuttle Program Launch Integration Manager fully trained and in place for February 1. I was to spend that short work week in Florida, and the next week closing out any lingering assignments in Houston before driving back to Florida to start my permanent assignment Jan 30 & 31.

As I recall we had two MMTs that week; Tuesday morning and Friday morning. I participated by teleconference from KSC.

It was midmorning Wednesday January 22 when Lambert Austin called me. Lambert was the Manager of the Space Shuttle Systems Integration office, a senior position in the program based at JSC. The conversation was pretty short; Lambert said that several of his people wanted more information about possible damage to the left wing of Columbia. He thought that the DoD might have some ways to get that information. Would I please contact the DoD and see what they could do? I told him that I would do what I could.

It was much much later before I questioned why Lambert – who certainly had the authority and contacts – did not make the request himself. His position in the organization was the same level as the position that I was in training to take over – he was official and I wasn’t yet there.

First I called Dave Phillips down at Patrick AFB. He was the NASA/DoD liaison for all things related to launch, range safety, search and rescue, etc. I described the situation to him and asked him to pass along the request to do whatever they could to get us more information.

About an hour later, I realized that the right way to make such a request was through the Mission Control Center in Houston. So I called the MOD console and talked with my old colleague Phil Engelauf. Some weeks later we talked about that call; my intention was to tell Phil to make the request; Phil heard that I was asking his opinion about making such a request. As it turns out Phil did not pass the request along. I thought he did.

Later in the afternoon, Linda Ham called me. I don’t know how the news traveled to her, but she heard that I had initiated a request for help and information from the DoD. She told me that she had checked and ‘nobody’ had a ‘requirement’ for more information (see my previous post on ‘The Tyranny of Requirements’). She asked me to terminate the request for information.

Much later on, I found out that Linda had contacted three fairly senior managers and none of those three had any interest in more information. Where and how the desire for more information died in the various management chains is probably a long story. During the investigation many engineers said they had requested more information but those requests never made it to the top.

After I hung up the phone with Linda, I found that I was mad. I don’t like to be overruled (who does) but this was unusual. I was very tempted to ignore her direction and let the requests stand. I went out of the office into the third floor hall at KSC HQ and paced up and down for several minutes. Finally I reluctantly decided to follow her instructions. I went back into my office, called Dave Phillips first, then Phil Engelauf. I told them to stand down on any request for more information from the DoD.

As I explained in my last post, it wouldn’t have made any difference. If there were some magical way to find out Columbia’s status, a week after launch it was too late. The best case scenario – which had virtually no chance of succeeding – would only have worked if action had been taken on the second or third day of the flight; by the sixth day it was too late.

For the rest of the week, whenever I was in my office, I listened to the air-to-ground and flight director squawk boxes. Lead Flight Director Kelly Beck was leading a very successful and complex science mission. Rick Husband and the entire crew seemed to handling everything in stride.

Kelly Beck had been the best Ascent Guidance and Procedures officer I ever had on any of my 27 ascent teams. She knew all the emergency and contingency procedures to save the crew during an ascent abort, no matter how many problems the sim team put on us. When it was time to select new flight directors I encouraged her to apply and then lobbied for her selection. After she was certified as an orbit Flight Director, I asked if she wanted to enter the Ascent/Entry Flight Director training. To my surprise she turned me down; the responsibility was too great she said; too much chance of losing a crew during those phases. STS-107 was her first assignment as lead orbit flight director and she got as close to the crew as any flight director in the office.

At the end of the week I flew back to Houston. No MMTs were held that next weekend either. I would go to my first MMT meeting in person at JSC on Monday January 27. Even though we did not know it, all options were behind us at that point.

Years later, a senior NASA official told me that after the accident, I was picked to be the new Deputy Program Manager because I asked for help during the mission. If true, that is the saddest comment of all. A handful of phone calls one Wednesday in January, started, then stopped. Too little, too late.

After Ten Years: Death Never Takes a Holiday

During the accident investigation there were several efforts to determine what might have been done to save Columbia and her crew. None of the concepts to plug the hole in the wing would have worked; most would have caused even earlier failure of the structure as the incredible heat of reentry vaporized the plug. The only possible scenario – and it was a slim one – was to immediately discover the hole by a never previously attempted complex spacewalk, immediately power down Columbia to survival levels to stretch the life support consumables, and scramble Atlantis, already stacked for launch in the VAB, with an incredibly complex formation flying and crew vacuum transfer. It would have made Apollo 13 look like a cakewalk. And the key was early detection and immediate radical action.

Whether or not we might have been able to pull that off is problematic; but psychologically we were not ready to take such radical action and as it turns out, we never had the chance to find out.

The launch had appeared to be perfectly normal. After the usual ceremonies, all the senior visiting managers left, while I stayed behind with the KSC folks. I had plane reservations to go home to Houston for the long MLK holiday weekend and I was anxious to get both some training done at KSC before I left, and to take care of things at home since I would be spending increasingly longer periods of time at KSC.

Friday morning was the first Mission Management Team teleconference. All the senior managers and engineers in the shuttle program at the various centers and NASA HQ tied in to a voice conference to hear the status of the mission and possibly to make decisions about its conduct. Nobody expected anything significant; after all this was not a complex ISS assembly mission with a lot of spacewalks; this was a straightforward science mission with many zero gravity experiments in the SpaceHab module in the payload bay. The biggest management decisions expected were to decide experiment priorities between quarrelling principle investigators whose experiments invariably ran into problems.

The MMT started promptly at 8 AM Houston time, and we had a good crowd in the Launch Integration conference room on the third floor of the KSC Headquarters meeting where it was a more civilized 9 AM Eastern time. All the element managers reported in on the quick look evaluations of the engineering data from the launch, all was ‘nominal’. The Mission Operations team (my old buddies) gave a quick status of mission events past and forecast; and again everything was normal. The meeting wasn’t very long, much less than an hour. It was announced that due to the three day federal holiday weekend, the MMT would not meet until next Tuesday, January 21.

Bob Page worked in the Shuttle Launch Integration office at KSC as the Head of the InterCenter Photo/TV Working Group. Among his most important duties was to collect all the findings from the launch videos and distribute them to the engineers. At that time, most of the data came from the film cameras. All those films had been collected within a couple of hours after launch, rushed down to a lab in Miami, developed and sent back overnight with copies going to the three engineering review teams at KSC, MSFC, and JSC. They had started looking at those films as soon as they arrived early Friday morning.

Bob’s office was just a few doors down the hall from mine. Not long after the MMT had ended, Bob burst into my office with the news that there had been a debris strike on Columbia’s left wing during launch. He was understandably upset that one of the key cameras had been so out of focus that its imagery was useless. But all three review teams had agreed that a significant piece of debris had been lifted off the External Tank as the shuttle stack accelerated through Mach 2.5 and that debris had hit the left wing near the front and exploded into a cloud of dust. Most probably insulating foam, but the picture could review few other details.

Immediately we called Linda Ham in Houston. As chairman of the MMT, she was the senior person to be notified in an event like this. Linda picked up the phone in her JSC office and shortly put us on the speaker so that Ron Dittemore, whose office was next to hers, could join in. Bob described what was seen and asked if they knew of any way to get more data. Since we did not have a robot arm on Columbia for that flight, there was no way to look at the front or underside of the wing. No one mentioned EVA and if one of us had thought of it, the likelihood is that we would not have agreed to take that risk – spacewalks always involve risk – on such slim grounds.
We agreed that Bob would extract a video clip of the strike and email it that day to all parties who might be concerned. After we hung up the phone, I felt I had done my duty by informing program management and ensuring the data would be distributed to the engineers who could perform the analysis. Bob pressed me to discuss options about how to get more data about possible damage to the wing; those options were severely limited.

Sometime after Bob left my office, Linda and I had another short phone conversation in which she told me that Bob was an excitable guy. I had to agree; he was pretty excited. But it seemed to be justified, rather than a reason to downplay the concern. Then she delivered the sentence that would define the rest of the tragedy; a sentence that was repeated as common wisdom by almost every senior manager that I talked to over the next two weeks: ‘You know, if there was any real damage done to the wing, there is nothing we can do about it.’ As unsettling as that was, I had to agree; going back to the first shuttle flight it had been well known that there was no way to repair the heat shield in flight. Nobody, not even me, thought about a rescue mission. Why would we?

So I caught my plane back to Houston that afternoon, the MMT did not meet all weekend and I had no reason to go over to mission control.

Saturday evening was a scout leader awards dinner meeting. I had worked with the Boy Scouts for almost a decade at that point, my son having long ago graduated and gone off to college, I had served in various leadership positions ending up as District Chairman. The council was reorganizing and our district was being renamed and geographically changed. Most of the leaders I had worked with for years had sons that likewise had grown out of the scouting program. It seemed like a good time for many of us to hang up our uniforms for the last time. So we had a nice dinner and awards assembly and said our goodbyes. I told people that I wouldn’t have missed that for the world.
It was the last meeting of the leadership of the Challenger District. Yes, named after the shuttle lost 17 years earlier. The name Challenger was not being reused by the scout council. It made me sad to think that the memory of Dick Scobee and his crew was fading away.

This would be the first shuttle flight since we lost Challenger to fly over the anniversary date of that loss.
That Saturday evening on the late TV news, I saw the very video clip that Bob Page had emailed out to NASA management and engineers. It was broadcast with the anchorman commenting “NASA says there is nothing to worry about.”

After Ten Years: Counting Down to Disaster

Early on I decided that riding the NASA ‘corporate’ jet was not a real advantage.  NASA had acquired a number of used Gulfstream II corporate jets to be converted to Shuttle Training Aircraft.  Supposedly the Gulfstream people had upgraded to G III and then G IV aircraft and the engineering required to modify the airplanes to simulate shuttle landings would have to be redone if newer models were to be used.  Since the STAs had a defined lifetime (putting the planes into screaming power dives dozens of times a day put a lot of stress on the airframe), it was good to have some low time used planes in the inventory to be held for later modification.  Put that together with the well-known aviation principle that planes sitting unused deteriorate faster than planes being flown, and voila’ you get a number of executive class airplanes to ferry people around.  We put them to a lot of use.  Unfortunately, it was like being locked in another meeting with your co-workers while you flew.  Productive?  Somewhat.  Enjoyable?  Let’s just say that I preferred to fly commercial.

For the FRR I was on the plane from JSC going to Florida.  They returned without me as I spent a few more days getting acquainted with my new duties as Shuttle Launch Integration Manager, the office staff, and looking for an apartment to rent.  Between the FRR and the launch, I spent another week in Florida.  Lots of work to do – getting all the safety classes so I could go the all the shuttle facilities was a non-trivial objective.  Ron Dittemore had instructed me to be “in place and operating on February 1.”  I was doing my best to get “operational.”  The date seemed to be less meaningful than it turned out to be.

One of the more interesting assignments for the Shuttle Launch Integration Manager was to administer the shuttle budget for “infrastructure revitalization.”  This means maintenance on all the facilities – not just at KSC, but around the country – that had been neglected for many years.  My predecessor had gotten work started on putting a new roof on the VAB.  Since being built in the middle 1960’s, no serious maintenance had been done to the roof of that gigantic building.  It was a multimillion dollar to replace the roof and that had not been in the budget for decades.  The fact that the roof leaked so badly that material was falling out the underside and a wooden lower deck had to be installed to protect the shuttles shows how shortsighted that policy had been.  It was a good thing we had that roof installed when three hurricanes blew through the KSC area in 2004.  Similarly, the huge factory at Michoud, Louisiana needed its 43 acre roof replaced; we got that done just in time for Katrina.  If the roof had been in poor shape when that hurricane struck the shuttle would have been out of business for good.  So I got acquainted with deteriorating facilities all around the country and started parsing out money to the direst situations.

I watched Linda Ham a lot in those days.  She was extremely active, hardworking, and intelligent.  Normally pretty brusque but in the pantheon of NASA leadership she treated subordinates better than the old line managers.  I had worked with her for a long time and appreciated her commitment and abilities.  She was teaching me the ropes as the new person in the Shuttle Program Management team.

During those days, a crew of photo/TV people went down to a little concrete building about a block from Ron John’s Surf Shop on Cocoa Beach.  They loaded film in the long range tracking cameras in preparation for the STS-107 launch.  Maintenance had been cut way back on the photo/TV equipment and checking the focus and operations of the cameras was not a “requirement.”  The short staffed team loaded the film, locked the building up, and hurried to the next camera location.  We were going to sorely miss the pictures that long range tracking camera could have made for us.

Launch day was Thursday January 16.  Thursdays were generally picked as launch days since the countdown would pick up Monday and the post launch cleanup could be finished on Friday, thereby avoiding paying weekend rates to all the workers.  Yes, that is how tightly the program managed its money.

I flew down early in the week, commercially; I didn’t want to get stuck in the flying management meeting.  All the last minute meetings went well; the L-2 day meeting closed all the final actions although there was some issue being worked about structural components in the SRB attach ring that would stay active until launch morning.  We all came in early for the Tanking Weather Briefing and most of us stayed through the countdown, even though we had the option to get a little more shuteye before the formal MMT call to stations at L-5 hours.

Linda did what was to be my future job as Launch Integration manager; she chaired the MMT for all the prelaunch activities and then would continue that role as was her normal job as the Flight Operations Integration Manager.  On January 16, the weather cooperated, the structural analysis for the SRB attach rings came back positive, and the launch was on time and beautiful.  After all the speeches and the ritual beans and cornbread meal in the Launch Control Center, the MMT rushed to the SLF to board their various management aircraft to fly back home to JSC, MSFC, and Stennis Space Center.  I planned to stay another day and continue learning the new job.

The next morning, Friday January 17, was when it all started.

After Ten Years: the Fateful FRR

George Abbey never allowed a shuttle flight to be scheduled over the last days of January.  He was too steeped in the events of the Apollo 1 fire and the STS-51-L loss of Challenger and her crew to put another space flight operation over those days.  But Mr. Abbey was long gone by the time of the STS-107 Flight Readiness Review, sent into an unwilling retirement.  But in his inimitable way, Mr. Abbey wanted all of us to stop and remember on those days how exacting the business we are in can be.

Not that delaying the flight a few weeks until February would have made any difference.  All the testing and evaluation after the accident indicated that there was nothing particular that the weather or winds on January 16 that caused a large section of insulating foam to once again detach from the External Tank during launch.  It is perhaps too existential a question to wonder if an additional two weeks, with its reminders of past vulnerabilities, would have caused someone to ask more questions, do more research, and just maybe . . . but such speculation is useless.  George Abbey was gone and schedule pressure as discussed in earlier posts did not allow for sentimentality or superstitious delay.  The clock was ticking toward the all-important US Segment completion date.  Standing down to contemplate safety was not a ‘requirement’.

Since the loss of Challenger, all higher level managers of the shuttle program were required to attend the Flight Readiness Review in person at the Kennedy Space Center, about two weeks prior to launch.  As a Lead Flight Director, I had attended a few FRRs before; as Launch Integration Manager trainee, I attended the STS-107 FRR the first week of January, 2003.  Ron Dittemore, Linda Ham, and I had a long history of working together:  we were all members of the Propulsion Systems Section of Flight Controllers early in our careers; we had all been Flight Directors together.  Ron was senior to me by about a year; Linda was junior by about two years.  But Ron had early on left the Flight Directors office to move up in shuttle program management and Linda, who had been somewhat his protégé for over a decade, had followed the same path.  I dawdled as a Flight Director, the job I really loved.

So in a role reversal, Linda was assigned to act as Launch Integration Manager – she had been through the drill many times – while I was to watch, listen, and learn from her.  The irony was not lost on me, but in truth there really was a lot to learn.  The chairperson for the FRR was the Associate Administrator for Human Space Flight, Bill Readdy.  It was not lead by the Shuttle Program manager, or the Launch Integration Manager.  The Launch Integration Manger and his office staff were responsible for orchestrating the event.  In theory, the FRR was a presentation by the shuttle program management to the most senior NASA leadership to gain permission to launch.  Bill Readdy, of course, was an experienced shuttle astronaut himself who had moved on to Washington to help run the agency and learn the machinations required to navigate the political minefields of the nation’s capital.

The FRR was always held in the cavernous Mission Briefing Room in the Operations and Checkout building.  The O&C had been built in the 1960’s and the MBR always had the atmosphere of some soviet function.  Not only was the room huge, with a vaulted ceiling, but it was cold, devoid of any decorations.  The layout included a u-shaped table where the principals sat, each before a microphone to amplify their questions or comments.  Two large projection screens in the front would show the seemingly endless powerpoint slides, and an oversized lectern was provided for the presenter.  In the back and along the sides were rows and rows of seats, mostly with assigned name placards.  The front row of seats was nicer chairs and was reserved for the NASA Administrator, Deputy Administrator, and other high dignitaries.  Generally these seats were empty and no one was brave enough to commandeer one of them.  In the very back, with the least comfortable chairs, there were a handful of unassigned seats.  Most of the time, the back wall was lined with folks standing.  Somewhere between three and four hundred people would crowd the room for the duration, generally about a day and a half.

Most oppressive was the atmosphere.  Presenter after presenter would lay out all the work their part of the program had completed, project pages of signatures showing that all the proper checks had been made.  Rarely were questions asked, and almost always by those at the head table.  In fact, the general impression was that people were not to question topics outside their area of concern or expertise.  The long table in the hallway outside always had coffee, tea, water, and sometimes pastries or cookies.  In the hallway, disgruntled lower level managers would gather and complain about the opacity of particular presentations.  Questions would be lobbed about concerning flight rationale or engineering test results and their interpretation.  But these discussions almost always took place in the lobby, not in the MBR.  Nobody wanted to start a riffle in the FRR.  After all, if you asked questions about their topics, they might ask questions about yours.  Everybody wanted to get on the stage and off without questions.  It was an oppressive atmosphere. And I had a front row seat to the proceedings; observe, wander the lobby, ask questions (outside), and generally think about what was going on.

Anything that might have been a topic of discussion at the FRR had been flagged days earlier.  Side bar discussions, small group meetings, and multiple phone calls between senior managers had taken place.  The solutions to any concerns were agreed to before the FRR started.  As my future boss Bill Parsons – who had served with the US Marines in Okinawa – put it the FRR was a kabuki dance:  pre-scripted, with every player knowing his or her part.

About the only person not intimidated by the proceedings, was John Young, bless his heart.  He would stand up, ask questions in a clear and loud voice, and expect and answer.  Even if his point was well taken, the folks at the table never seemed sure of how to handle it.  Sometimes an action was assigned “due at the L-2 day briefing” or some such.  But John had come to be considered a gadfly with a few topics that were his personal interests.  He had no standing and no organization, and worse he sometimes asked goofy questions.  Other than the comic relief of watching the head table squirm when John came to the microphone, little came of it.

Discussion of foam losses from the External Tank never came up as far as I can recall.  That issue had been ‘dispositioned’ ages ago at the STS-113 ET/SRB Mate Review.  There was no evident reason to reopen that topic.

So at the end of the day, everybody at the table said “go for launch.”

I had heard no reason to disagree.

January 16 was set for the launch of Columbia and her crew for her 28^th flight.

After Ten Years: STS-113, the Calm before the Storm

STS-113 Ascent/Entry Mission Control Team

Of course the title is wrong; there was nothing calm about STS-113. I was the Ascent/Entry Flight Director as you can see in the picture of the mission control team. This ISS assembly and crew rotation flight was jam packed with activities Mike LA added to his spacewalk tally by working with John Harrington to install the P1 truss and other gear – three EVAs. We carried up the ISS Expedition 6 crew who got a much longer stay on ISS than they expected.

Jim Wetherbee was the commander and his right hand man was Gus Loria as pilot. But Gus had an unfortunate accident late in the training and was replaced by Paul Lockhart. Paco was very happy – just as Gus was not. Of course the flight was delayed while the collective space shuttle program worked on analyzing BSTRA ball cracks (see my previous post on STS-112).

Training all done and issues all dispositioned, we got ready to launch on November 9. With a two week flight, we should be done before Thanksgiving. A Sunday launch was unusual, but not unheard of; everybody knew the schedule was tight and we were behind because of all the technical issues earlier in the year. The ascent team came in Saturday to do the usual prelaunch crew briefings and get the weather forecasts. Everything seemed to be in order. We went home to get a good night’s sleep for the launch. Sunday, Murphy had a little surprise in store for us: leaking oxygen in the payload bay. Seems that some of the metal bellows flex hoses that carried crew breathing oxygen from the tanks in the mid-fuselage had been stepped on by some of the workers preparing Endeavour for flight. The hoses blew out and the gas detectors showed higher than normal concentrations of oxygen – not enough to be a flammability hazard, but demonstrating that a leak was occurring. So the ascent team never even made into mission control to man the consoles.

The launch was scrubbed and plans were made to replace the faulty hoses. This required opening the payload bay doors and inserting work platforms so that the techs could get access to the hoses. Unfortunately, putting a work platform into the front end of the payload bay did not go so well; it bumped the shuttle’s arm which was clamped in its launch restraints. The shuttle arm is a composite material and impact damage can greatly reduce the strength of the material. So in addition to fixing the oxygen hoses, we now had to fix the arm. Time passes. Word from NASA HQ is that people in high places are not pleased with the delay.

New launch date: November 22. Rats, Thanksgiving on console again.

On Friday, November 22, the shuttle was flawless, everything working perfectly. On November 22, the weather at KSC was beautiful, even the weather at Edwards and White Sands was beautiful in case we needed to land early. But in Spain we had a real problem. A basic safety requirement for the space shuttle was that there had to be a safe landing place all throughout the launch phase. If one of those three high pressure main engines were to shut down early on, the shuttle could ‘return to launch site’ for an early shutdown; and later on it could ‘abort to orbit’ or ‘abort once around’ to White Sands or Edwards. But for an engine failure in the mid section of the launch, a ‘trans Atlantic abort’ was the only option. The shuttle program had several sites just across the ocean which were prepared to receive the shuttle. For flights near the equator – like the Hubble space telescope missions – we could use a West African airport, Yundum, at the capital city of Banjul, The Gambia. Before I worked on the shuttle program I’d never heard of it. For flights to the space station, farther away from the equator, runways in Europe could be used, but the weather was worse. Ireland, England, Germany, even Sweden had airports that could be used in an emergency, but we didn’t staff them because the weather was generally poor. Later on, after Columbia, we added the French Air Force flight test center, Istres, at Marseilles, but in November 2002 that was not available. We had a great runway at a very old Strategic Air Command base from the 1950’s in the desert of Morocco not far from Casablanca. But after Sept. 11, 2001, sending NASA personnel to a largely Muslim nation was not considered to be a good safety risk. So we had two landing sites in Spain, south near Seville, and north near Zaragoza. And today, November 20, 2002, there were thunderstorms at both places. The shuttle could not survive a lightning strike, so that was a firm, solid, no questions asked NO GO. Abort landing weather was the responsibility of the Ascent/Entry Flight Director and I made the call the Mike Leinbach the Launch Director. We were all disappointed but understood that tomorrow would be a better day.

It nearly got me fired. One of the senior managers at NASA HQ, a former military flier, was incensed that we did not fly. Fortunately, I was well insulated from that kerfluffle and didn’t even know about it until days later.

On Saturday, November 23, we had better luck with the weather and launched uneventfully. Of course, I held my breath through first stage, and kept my fingers crossed all the way until FDO reported over the loop “Nominal MECO, Flight, no OMS-1 required.” Probably the best call that an Ascent Flight Director could ever hope to hear.

The Ascent/Entry Flight director checks in with Mission Control every day, goes to the Mission Management Team meetings as they are scheduled, and generally prepares for the end of the mission. As long as you don’t bother the Orbit Flight Directors, you can watch EVA’s, monitor arm ops, and in general have the best seat in the house. Of course, if it’s slow, the PAO officer may trap you into an on-console mini-interview on NASA TV, but that is a small price to pay.

Finally landing day rolled around. We had done so well on this flight that we had an unprecedented 4 days of consumables to try to land at the Kennedy Space Center. And since Endeavour was going into a depot maintenance period, there was no hurry to land. Program management told me to use all those days if necessary to get back to KSC. Weather in the late fall being a little unsettled, of course it took us three days to find a good day to land. But we did; and again all was well. Of course, I always held my breath during re-entry. I always did. But nothing untoward happened, and the commander and I had a nice chat on the radio before he climbed out of Endeavour on the KSC runway.

It would be the last safe landing of a space shuttle for a very long time.

Weather was something we just had to deal with; no issues there. Metal bellows flex hoses became the bane of my existence in my next act in life as we went through and replaced almost all of them on the three remaining orbiters after Columbia. BSTRA balls also continued to be an issue. Human errors like driving a work platform into the arm will plague high risk endeavors until the end of time. But all in all, STS-113 was calm, no biggies. No foam lost. Not yet.

Personally, I had spent nearly 15 years in the Flight Director’s office. That is more than twice as long as average. I enjoyed the job, thought I was good at it, but there were younger people that I was holding back. And my eyes were going: presbyopia meant that I couldn’t wear contact lenses and see all the screens in the control center – there are only so many ‘granny glasses’ that you can use. Years of having a headset plugged into my ears were deteriorating my hearing. And everyone seemed to be advising me that I should look for a promotion, a job where I could do more important things for the agency. That advice turned out to be poor.

Parker Counts had been the External Tank Project manager at MSFC. I really respected him. After a long career, he took a position at NASA HQ working for the Assistant Administrator for Space Operations, Bill Readdy. Now, Parker was retiring and his job was offered as a one year rotational assignment at HQ. I decided to put my paperwork in for a tour.

Ron Dittemore, my old collegue and Space Shuttle Program Manager caught wind of it, and gave me a call; Jim Halsell, heading up the shuttle program at the Kennedy Space Center, had just been assigned to command STS-120, about a year away. Ron needed a replacement for Jim. His argument that it would be more fun than being at headquarters sounded good. My old collegue Linda Ham would train me. What Ron really wanted was a permanent replacement but we agreed for a one year trial as a rotational assignment. If I liked it, I could have the job permanently, if it didn’t work out, in a year I’d be back in the Flight Director’s office. Even my wife agreed.

Robert Lightfoot got Parker Counts’ old job at NASA HQ. Ron Dittemore didn’t tell me that he was quietly negotiating retirement and would soon be working for industry.

None of us expected what was going to happen in early 2003.

Over the Christmas holidays I stopped at my office preparing for the move. On the way out the door, I ran into Kalpana Chawla. She was so ready to fly and so excited – she told me all about it with a big smile. It was the last time I saw her.

Ten Years After Columbia: STS-112, the Harbinger

“You will never remember the many times the launch slipped, but the on-time failures are with you always” – Walt W. Williams, NASA Program Manager for X-15 and Mercury

In the summer of 2002, the word got out about the NASA HQ screen saver counting down to the launch of Node 2 (US segment of the International Space Station completion).  That was well over a year away and the screen saver was counting down in fractions of a second toward that scheduled event.  All of us in Shuttle operations were offended.  We had been schooled over and over again about the dangers of “launch fever” where people lose their judgment just to make a launch happen on schedule.  To a person, we all were committed to not let that happen.

And the program management agreed.  Prime example:  early that year, the schedule took a real series of delays over BSTRA balls. The Ball-Strut-Tie-Rod-Assembly was a complex part of the main propulsion system plumbing on the shuttle.  Flexible pipes over a foot in diameter carried a huge flow of cryogenic hydrogen or cryogenic oxygen into the main engine inlets.  Due to the large temperature changes the pipes must have flexibility to respond as the metal contracts or expands.  A complicated mechanism ensured that the light weight pipes would move properly; a spiderweb of struts in the curving pipe met at a ball made of incredibly hard material.  Cracks were found in one of these balls.  This was not good.  If chunks of the super-hard BSTRA ball came off, they would go right into the turbopumps where catastrophic damage would occur.  But the assembly was too far back down the pipe to inspect easily, even with long optical fiber camera contraptions.  Studies, experiments, analysis, and as much inspection as we could do gave us some confidence that the worst would not happen, but all of that work took time out of a compressed schedule.  And the HQ countdown clock kept running.  We ignored that.

NASA is not a military outfit.  Sometimes there are new leaders appointed that believe they can, by fiat, make changes in policy, direction, or actions.  NASA is more like a benevolent anarchy where leadership pronouncements generally are considered the starting point for discussion.  So I don’t buy schedule pressure as a major cause of the Columbia accident.  Earlier blogs have given my point of view on those causes.

A sad side note is that months later, in the Columbia debris that filtered into the reconstruction hanger at KSC, I got my best and close up look at a BSTRA ball, still packaged inside its spiderweb of struts.  No cracks.  Not the way I wanted to see that item.

But on STS-112 we had a full blown harbinger of what was to come, and we missed it.  Here is my part of that story.

Early in the year I had been assigned to be a lead flight director, and a Mission ops director rep, and for STS-113 I was supposed to requalify as an Ascent/Entry Flight director.  In addition to all the simulations and training, I needed to work closely with the A/E FD on the flight before mine.  John Shannon was the A/E FD for STS-112; Bob Castle was the MOD, and I was “Weather Flight”.  That meant I sat next to John (and in front of Bob) during pre-launch and pre-entry operations and kept track of all the weather information.  As if John couldn’t do that himself.  But it got me in the control center during a real flight and helped to get down the cadence and tempo.  Weather, in a real flight, was more often than not, the major problem.  Simulations and training attempting to emulate weather problems always came off lame and easy.  Real life weather observations and forecasting was always dynamic, complex, and hard to follow.

After the usual difficulties, the countdown clock for STS-112 ticked down the final few seconds of the count.  During that last part, the shuttle’s onboard computers – known as the “redundant set” – were in control.  The last signal from the ground was “go for main engine start” at T-10 seconds.  Unless the ground system – either automatically or manually – sensed a problem and issued an “RSLS Abort” the onboard system would launch itself.  The Redundant Set Launch Sequence was software program that did all the onboard checking and commanding in the last 31 seconds.  The four redundant set general purpose computers executed the same software in lockstep to the millisecond.  The RSLS software commanded the main engines to start in 120 millisecond staggered sequence at 6.6 seconds to go, listened for any failures detected by the engine controllers, checked to see that all three engines were at full thrust, checked a dozen other items, and in the milliseconds right at T=0 sent the pyrotechnic commands which separated the shuttle from the launch pad and ignited the solid rocket boosters.  Once the solids lit, you were going someplace in a big hurry.  Last step in the RSLS, terminate itself and start the onboard software programs required to actually fly.  Of course, when the RSLS terminated, there was no program looking for a ground commanded “RSLS Abort.”

A real fear – alleviated by a million software verification checks – was that somehow the solid rocket boosters would be ignited and something – an RSLS Abort command for example – would stop the launch sequencer in the last milliseconds.  That would be a disaster.  If the hold down posts and T=0 umbilical panels and the GUCP arm did not separate, or the liquid engines were commanded to shut down, the consequences would be immediate and devastating.  So that software was tested over and over again with all the variations of inputs that could be devised.

On STS-112, sitting with the Ascent Flight director, I got to do the one thing that was not allowed for any of the critical flight controllers – watch the television.  By some quirk of communications routing, the TV picture would beat the data to the screens of mission control.  So I knew the shuttle had lifted off before the DPS officer sung out the magic words:  “Liftoff Confirmed”.  The Data Processing systems officer was looking at the termination of the RSLS and the start of the software to fly the vehicle, not engines or bolts or anything physical.  Out of the corner of my eye, I caught a red light on the screen in front of John:  “RSLS Abort”.  My heart stopped.  But the TV showed that everything was OK, no fireball of explosion, so we road it out.

Much later we found out the cause:  bad pin connectors.  The pyrotechnics to blow all those holddown bolts are actually part of the launch pad.  The signal from the shuttle onboard computers has to go through a set of pull away electrical connectors in what is called the T=0 umbilical panel (because the panel is to separate at T=0) and then the signals are routed to the explosive charges.  One set of signals did not make it across the interface due to corroded or misaligned electrical pins.  The ground system caught that only half of the pyrotechnics would fire and requested a hold – but it was too late.  Of course, the system was redundant, so half the explosives did the job – just barely.  A failure of one more pyrotechnic initiator and it would have been a very bad day.

For the next flight, new cables and pins were installed.  But that was not enough; during the long return to flight process, a special working team examined every possible aspect of what causes pin connectors to fail.  New processes were built into the last two dozen shuttle launches starting with a prohibition against re-using connector pins in those critical areas.

Meanwhile, back to the launch:  there was a new camera was attached that looked down from the side of the external tank at the earth falling away below.  It was powerfully mesmerizing and I watched the TV until the solid rocket booster separation motors fogged the lens and nothing more could be seen.  It should be noted, that the camera was on the wrong side of the tank to see the really interesting development that happened during early ascent:  the loss of a big chunk of foam from the outside of the external tank.

A few days after launch the report came in that on the left hand solid rocket booster, one of the foam areas that are sprayed on the booster case to alleviate splash down water loads, had a big dent in it.  No real issue there; and some sleuthing turned up pictures of the big foam loss off of the external tank that must have caused it.

After the flight, as Ascent/Entry flight director for STS-113, I followed all the “anomalies” with great interest.  We simply had to fix those connector pins for protection against disaster at launch.  The program did that.  The BSTRA ball issue was still not well understood and more work had to be done to check the Endeavour’s plumbing and make sure no fragments clogged a turbopump.  Oh, that foam thing?  It was not categorized as an “anomaly.”  The program reviewed this ‘event’ at the STS-113 ET/SRB Mate Review.  That review was chaired by Jim Halsell, the Shuttle Launch Integration Manager.  At that review, the ET Project Manager, Jerry Schmeltzer, categorized the loss as “not a safety of flight issue” and the potential for future losses was “accepted”.  In the NASA shuttle system, once a program review had “dispositioned” an item, it was not reviewed further.  If there was any discussion at the STS-113 Flight Readiness review, I was not there to hear it; Ascent/Entry Flight Directors did not get to go to Florida for that review.  The Lead Flight Director, Paul Dye, and the Mission Ops Director, Bob Castle, attended the FRR but they were focused on the in flight operations, the assembly tasks.  It was never a subject for any later flight – STS-107 – because it had already been “dispositioned.”  There were simply too many issues to rehash every thing, every launch.  Later on, when I talked to Linda Ham, she was worried that the foam issue was not properly addressed; the rationale was lousy.

But by that point we were way behind schedule for the all-important Node 2 launch.  Jim Halsell was assigned as commander for that flight, STS-120, and they needed a replacement for him.  It turned out to be me.  But that is a story for another day.

Turns out that we were more vulnerable to schedule pressure than I thought.

After Ten Years: A Few Words from Admiral Gehman

I’d like to interrupt my personal recollections of the Columbia accident and its aftermath to give a few words from Admiral Gehman.  You might as well know that there are still people out there who will tell you the CAIB got it wrong.  I am not one of them.  Of course there are minor details that are not exactly right, no report is perfect.  But in the main points the CAIB did get it right, in my opinion.  Admiral Gehman lead the CAIB and in December 2005 he made a remarkable speech to a group of Navy officers.  His talk included a discussion of the terrorist attack on the USS Cole and the USS Iowa main gun battery explosion which killed a number of sailors.  He also mentions various aspects of other naval programs and investigations.  In the interest of brevity, I have deleted all of that other material and just retained his comments on the Columbia accident.  You should read this carefully.

———————————————————————-
In the course of working closely with NASA engineers and NASA scientists as we tried to solve what had happened to the Columbia, we became aware of some organizational traits that caused our eyebrows to rise up on our heads. After not very long, we began to realize that some of these organizational traits were serious impediments to good engineering practices and to safe
and reliable operations. They were doing things that took our breath away.

We concluded and put in our report that the organizational traits, the organizational faults, management faults that we found in the space shuttle program were just as much to blame for the loss of the Columbia as was the famous piece of foam that fell off and broke a hole in the wing. Now, that’s pretty strong language, and in our report, we grounded the shuttle until they fixed these organizational faults.

I need to give you the issue from the NASA point of view so you can understand the pressures that they were under. In a developmental program, any developmental program
The program manager essentially has four areas to trade. The first one is money. Obviously, he can go get more money if he falls behind schedule. If he runs into technical difficulties or something goes wrong, he can go ask for more money. The second one is quantity.  The third one is performance margin. If you are in trouble with your program, and it isn’t working, you shave the performance. You shave the safety margin. You shave the margins.  The fourth one is time. If you are out of money, and you’re running into technical problems, or you need more time to solve a margin problem, you spread the program out, take more time. These are the four things that a program manager has. If you are a program manager for the shuttle, the option of quantity is eliminated. There are only four shuttles. You’re not going to buy any more. What you got is what you got. If money is being held constant, which it is—they’re on a fixed budget, and I’ll get into that later—then if you run into some kind of problem with your program, you can only trade time and margin. If somebody is making you stick to a rigid time schedule, then you’ve only got one thing left, and that’s margin. By margin, I mean either redundancy—making something 1.5 times stronger than it needs to be instead of 1.7 times stronger than it needs to be—or testing it twice instead of five times. That’s what I mean by margin.

It has always been amazing to me how many members of Congress, officials in the Department of Defense, and program managers in our services forget this little rubric. Any one of them will enforce for one reason or another rigid standard against one or two of those parameters. They’ll either give somebody a fixed budget, or they’ll give somebody a fixed time, and they forget that when they do that, it’s like pushing on a balloon. You push in one place, and it pushes out the other place, and it’s amazing how many smart people forget that.

The space shuttle Columbia was damaged at launch by a fault that had repeated itself in previous launches over and over and over again. Seeing this fault happen repeatedly with no harmful effects convinced NASA that something which was happening in violation of its design specifications must have been okay. Why was it okay? Because we got away with it. It didn’t cause a catastrophic failure in the past. You may think that this is ridiculous. This is hardly good
engineering. If something is violating the design specifications of your program and threatening your program, how could you possibly believe that sooner or later it isn’t going to catch up
with you? For you and me, we would translate this in our world into, “We do it this way, because this is the way we’ve always done it.”

The facts don’t make any difference to these people. Well, where were the voices of the engineers? Where were the voices that demanded facts and faced reality? What we found
was that the organization had other priorities. Remember the four things that a program manager can trade? This program manager had other priorities, and he was trading all right, and let me tell you how it worked. In the case of the space shuttle, the driving factor was the International Space Station. 

In January of 2001, a new administration takes office, and the new administration learns in the spring of 2001 that the International Space Station, after two years of effort, is three years behind schedule and 100 percent over budget. They set about to get this program back under control. An independent study suggested that NASA and the International Space Station program ought to be required to pass through some gates. Now, gates are definite times, definite places, and definite performance factors that you have to meet before you can go on. The White House and the Office of Management and Budget agreed to this procedure, and the first gate that NASA had to meet was called U.S. Core Complete. The name doesn’t make any difference, but essentially it was an intermediate stage in the building of the International Space Station, where if we never did anything more, we could quit then.  And the date set for Core Complete was February 2004. Okay, now this is the spring of 2001.

In the summer of 2001, NASA gets a new administrator. The new administrator is the Deputy Director of OMB, the same guy who just agreed to this gate theory. So now if you’re a worker at
NASA, and somebody is leveling these very strict schedule requirements on you that you are a little concerned about, and now the new administrator of NASA becomes essentially the author of this schedule, to you this schedule looks fairly inviolate. If you don’t meet the gate, the
program is shut down; they took it as a threat.  If a program manager is faced with problems and shortfalls and challenges, if the schedule cannot be extended, he either needs money, or he needs to cut into margin. There were no other options, so guess what the people at NASA did? They started to cut into margins. No one directed them to do this. No one told them to do this. The organization did it, because the individuals in the organization thought they were defending the
organization. They thought they were doing what the organization wanted them to do. There weren’t secret meetings in which people found ways to make the shuttle unsafe, but the
organization responded the way organizations respond. They get defensive. We actually found the PowerPoint viewgraphs that were briefed to NASA leadership when the program for good, solid engineering reasons began to slip, and I’ll quote some of them. These were the measures that the managers proposed to take to get back on schedule. These are quotes. One, work over the Christmas holidays. Two, add a third shift at Kennedy Shuttle turnaround facility. Three, do safety checks in parallel rather than sequentially. Four, reduce structural inspection requirements. Five, defer requirements and apply the reserve, and six, reduce testing scope. They’re going to cut corners. That’s what they’re going to do. Nevertheless, for very good reasons, good engineering reasons, and to their credit, they stopped operations several times, because they found problems in the shuttle, and they got farther and farther behind schedule.

Well, two launches before the Columbia’s ill-fated flight—it was in October—a large piece of foam came off at launch and hit the solid rocket booster. The solid rocket boosters are recovered from the ocean and brought back and refurbished. They could look at the damage, and it was significant. So here we have a major piece of debris coming off, striking a part of the shuttle assembly. The rules and regulations say that, when that happens, it has to be classified as the highest level of anomaly, requiring serious engineering work to explain it away. It’s only happened six or seven times out of 111 launches. But the people at NASA understand that if they classify this event as a serious violation of their flight rules, they’re going to have to stop and fix it. So they classify it as essentially a mechanical problem, and they do not classify it as what they call an in-flight anomaly, which is their highest level of deficiency.

Okay, the next flight flies fine. No problem. Then we launch Columbia, and Columbia has a great big piece of foam come off. It hits the shuttle. This has happened two out of three times.
Now, we go to these meetings. Columbia is in orbit, hasn’t crashed, and we’re going to these meetings about what to do about this. The meetings are tape-recorded, so we have been listening to the tape recordings of these meetings, and we listen to these employees as they talk themselves into classifying the fact that foam came off two out of three times as a minor material
maintenance problem, not a threat to safety. Why did they talk themselves into this? Because they knew that, if they classified this as a serious safety violation, they would have to do all these
engineering studies. It would slow down the launch schedule.  They could not possibly complete the International Space Station on time, and they would fail to meet the gate. No one told them
to do that. The organization came to that conclusion all by itself.  They trivialized the work. They
demanded studies, analyses, reviews, meetings, conferences, working groups, and more data. They keep everybody working hard, and they avoided the central issue: Were the crew and the shuttle in danger? [This was] a classic case where individuals, well-meaning individuals, were swept along by the institution’s overpowering desire to protect itself. The system effectively blocked honest efforts to raise legitimate concerns. The individuals who raised concerns and did complain and tried to get some attention faced personal, emotional reactions from the people who were trying to defend the institution. The organization essentially went into a full defensive crouch, and the individuals who were concerned about safety were not able to overcome.
________________________________________________
So back to me.  I would tell you nobody intentionally did anything to compromise safety.  Everyone that I worked with always held the safety of the crew to be their highest priority.  So the real question that you have to answer is how did so many hard working, intelligent, safety minded people come to make such a fundamentally unsafe set of decisions?  And do you think that you or your organization is immune?
Think again.