After Ten Years: the Fateful FRR

George Abbey never allowed a shuttle flight to be scheduled over the last days of January.  He was too steeped in the events of the Apollo 1 fire and the STS-51-L loss of Challenger and her crew to put another space flight operation over those days.  But Mr. Abbey was long gone by the time of the STS-107 Flight Readiness Review, sent into an unwilling retirement.  But in his inimitable way, Mr. Abbey wanted all of us to stop and remember on those days how exacting the business we are in can be.

Not that delaying the flight a few weeks until February would have made any difference.  All the testing and evaluation after the accident indicated that there was nothing particular that the weather or winds on January 16 that caused a large section of insulating foam to once again detach from the External Tank during launch.  It is perhaps too existential a question to wonder if an additional two weeks, with its reminders of past vulnerabilities, would have caused someone to ask more questions, do more research, and just maybe . . . but such speculation is useless.  George Abbey was gone and schedule pressure as discussed in earlier posts did not allow for sentimentality or superstitious delay.  The clock was ticking toward the all-important US Segment completion date.  Standing down to contemplate safety was not a ‘requirement’.

Since the loss of Challenger, all higher level managers of the shuttle program were required to attend the Flight Readiness Review in person at the Kennedy Space Center, about two weeks prior to launch.  As a Lead Flight Director, I had attended a few FRRs before; as Launch Integration Manager trainee, I attended the STS-107 FRR the first week of January, 2003.  Ron Dittemore, Linda Ham, and I had a long history of working together:  we were all members of the Propulsion Systems Section of Flight Controllers early in our careers; we had all been Flight Directors together.  Ron was senior to me by about a year; Linda was junior by about two years.  But Ron had early on left the Flight Directors office to move up in shuttle program management and Linda, who had been somewhat his protégé for over a decade, had followed the same path.  I dawdled as a Flight Director, the job I really loved.

So in a role reversal, Linda was assigned to act as Launch Integration Manager – she had been through the drill many times – while I was to watch, listen, and learn from her.  The irony was not lost on me, but in truth there really was a lot to learn.  The chairperson for the FRR was the Associate Administrator for Human Space Flight, Bill Readdy.  It was not lead by the Shuttle Program manager, or the Launch Integration Manager.  The Launch Integration Manger and his office staff were responsible for orchestrating the event.  In theory, the FRR was a presentation by the shuttle program management to the most senior NASA leadership to gain permission to launch.  Bill Readdy, of course, was an experienced shuttle astronaut himself who had moved on to Washington to help run the agency and learn the machinations required to navigate the political minefields of the nation’s capital.

The FRR was always held in the cavernous Mission Briefing Room in the Operations and Checkout building.  The O&C had been built in the 1960’s and the MBR always had the atmosphere of some soviet function.  Not only was the room huge, with a vaulted ceiling, but it was cold, devoid of any decorations.  The layout included a u-shaped table where the principals sat, each before a microphone to amplify their questions or comments.  Two large projection screens in the front would show the seemingly endless powerpoint slides, and an oversized lectern was provided for the presenter.  In the back and along the sides were rows and rows of seats, mostly with assigned name placards.  The front row of seats was nicer chairs and was reserved for the NASA Administrator, Deputy Administrator, and other high dignitaries.  Generally these seats were empty and no one was brave enough to commandeer one of them.  In the very back, with the least comfortable chairs, there were a handful of unassigned seats.  Most of the time, the back wall was lined with folks standing.  Somewhere between three and four hundred people would crowd the room for the duration, generally about a day and a half.

Most oppressive was the atmosphere.  Presenter after presenter would lay out all the work their part of the program had completed, project pages of signatures showing that all the proper checks had been made.  Rarely were questions asked, and almost always by those at the head table.  In fact, the general impression was that people were not to question topics outside their area of concern or expertise.  The long table in the hallway outside always had coffee, tea, water, and sometimes pastries or cookies.  In the hallway, disgruntled lower level managers would gather and complain about the opacity of particular presentations.  Questions would be lobbed about concerning flight rationale or engineering test results and their interpretation.  But these discussions almost always took place in the lobby, not in the MBR.  Nobody wanted to start a riffle in the FRR.  After all, if you asked questions about their topics, they might ask questions about yours.  Everybody wanted to get on the stage and off without questions.  It was an oppressive atmosphere. And I had a front row seat to the proceedings; observe, wander the lobby, ask questions (outside), and generally think about what was going on.

Anything that might have been a topic of discussion at the FRR had been flagged days earlier.  Side bar discussions, small group meetings, and multiple phone calls between senior managers had taken place.  The solutions to any concerns were agreed to before the FRR started.  As my future boss Bill Parsons – who had served with the US Marines in Okinawa – put it the FRR was a kabuki dance:  pre-scripted, with every player knowing his or her part.

About the only person not intimidated by the proceedings, was John Young, bless his heart.  He would stand up, ask questions in a clear and loud voice, and expect and answer.  Even if his point was well taken, the folks at the table never seemed sure of how to handle it.  Sometimes an action was assigned “due at the L-2 day briefing” or some such.  But John had come to be considered a gadfly with a few topics that were his personal interests.  He had no standing and no organization, and worse he sometimes asked goofy questions.  Other than the comic relief of watching the head table squirm when John came to the microphone, little came of it.

Discussion of foam losses from the External Tank never came up as far as I can recall.  That issue had been ‘dispositioned’ ages ago at the STS-113 ET/SRB Mate Review.  There was no evident reason to reopen that topic.

So at the end of the day, everybody at the table said “go for launch.”

I had heard no reason to disagree.

January 16 was set for the launch of Columbia and her crew for her 28th flight.

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and has retired from consulting and is currently a full time grandpa. He might be available for speaking engagements for the right incentives (coffee and donuts work!)
This entry was posted in After Ten Years and tagged , , . Bookmark the permalink.

21 Responses to After Ten Years: the Fateful FRR

  1. charlie barber says:

    It would be interesting to know what Capt. Young had to say that day long ago….

    Interestingly enough today I happen to be reading through the STS-27R Orbiter TPS Damage Review Team Summary Report Vol-1. 7.0 Findings and Recommendations (page 60), Recommendation 1.a. “In the immediate future, equipment, systems, procedures, and resources be put in place to gather sufficient data to understand the causes for and to propose changes to eliminate the damage to Orbiter TPS. Specifically, this effort should include the following: (1). A detailed assessment of the systems design criteria influencing potential debris for all elements.”
    So on and so forth through Recommendation 10.
    I guess I must ask then, how come waiver after waiver occurred regarding ET foam losses Wayne?

    • waynehale says:

      Technically, as I understand it, there were no waivers issued. Foam losses were ‘dispositioned’ as maintenance items and despite the recommendation you cited, there was never any baselined program requirement to eliminate all foam losses. No requirements violation, no waiver necessary. A recommendation in a report does not to be a requirement without the program taking action. I’ll leave it to Dave Huntsman (who has commented extensively in this blog on previous posts) to help explain the nuances.

  2. Beth Webber says:

    The Flight Readiness Review sounds like the common template for any X-RR (for X put software, engine control, you name it). You already know the questions, and go prepared with the answers, and hope to hell you get out of there without someone coming up with a question you or your organization hadn’t anticipated. In hindsite the X-RR only serves the purpose of making Management feel they are doing their job.

  3. cthulhu says:

    In my (military airplane) experience, you present a Flight Readiness Review to a Flight Readiness Review Board: the FRRB is typically made up of company and customer graybeards, chosen for specific areas of expertise but not limited to those areas (anybody on the board can ask questions at any time). Key to the whole enterprise is that the FRRB is independent: the members don’t work on the project in question, they are instead advisors to project and eventually company and even customer management, responsible for an independent assessment of the project’s readiness to fly.

    In your description here, I’m not seeing where the independent review comes in; am I missing it? Without a truly independent review, I’m inclined to agree with Beth Webber above – it’s just a feel-good party.

    • waynehale says:

      Independence is supposed to come from the “independent” safety and mission assurance organization, the “independent” engineering organization, and the “independent” health and medical organization. Sometimes that works. They don’t work ‘for’ the shuttle program but they all work ‘for’ the agency head and thus are exposed to the direction and desire of the leadership of the agency.

      • cthulhu says:

        OK, I get it now; relating back to my experiences, ultimately the independence of my style of FRRB is whether top level company management back up the independent FRRB if the FRRB disagrees with project management about the level of risk.

        So, to ask one more question along these lines, as a way to informally assess the de facto independence of the FRR: were there times when the FRR recommended NOT to launch, to the chagrin of overall NASA management? (in a similar vein, did FRR decisions / recommendations ever cause flights to slip?)

      • waynehale says:

        Never that I recall

  4. Dave H. says:

    Best of the New Year to you and yours, Wayne, and every one of your readers.

    Once again, there were no red flags, obvious or hidden, that would have caused alarm and forced a stop to the countdown. While there were no weather issues ala Challenger, the CAIB would note that Columbia encountered wind shear shortly before the foam liberation event.
    Hindsight, yet again.

    “Anything that might have been a topic of discussion at the FRR had been flagged days earlier.”
    This tells me that if nothing else, good communication was taking place between all of the groups represented at the FRR. Whether issues were being swept under the rug due to budget/schedule/other “pressures” is something only those who were there can answer…and will have to live with the consequences of. But to my eyes, viewing this from the MCC observation room behind the glass, it looks like everything that needed to be done was being done.
    If the FRR appeared to be a scripted, well-rehearsed “dance”, it was because issues had been handled in advance…no surprises. Which brings me to John Young…

    Every project or organization needs someone who can see the forest for the trees. From your description of him, John Young had willingly taken up the mantle of that role and gladly embraced its consequences. Being labeled a “gadfly” is a small price to pay if the issues you bring to the table have meaning. By “meaning” I mean that they have enough merit to cause someone to think twice. John had the “chops” so that what he said had that meaning, and his intent was probably simple: to make sure those who went into space came safely home again.
    Tell me, Wayne, did he have a sense of humor which occasionally manifested itself as James Doohan saying “You canna change the laws of physics, you know”?

    George Abbey’s heart was in the right place, but ahead of his time. Perhaps if Remembrance Day had begun to be observed in 1968 the loss of Challenger might have been avoided? I’ll leave that answer to the temporal mechanics people.
    In the aftermath of the loss of Columbia, I posted a blog message in a humorous vein in which I suggested that NASA not launch anything in the month of January. There are simply too many distractions…the holidays, football playoffs, figuring out your taxes. Things like that. Spaceflight demands your full and complete attention, and it’s tough to do that when Brett Farve is on your fantasy team.

    This year, February 1 falls on a Friday. I expect that Remembrance Day will be observed on that day as well. It’s only fitting. I think I’ll take a personal day and watch the observance on TV.

    • Rod Wallace says:

      The wind shear had nothing to do with the foam loss. Foam internal pressure increased because of leak paths for the liquid nitrogen state change which caused a significant pressure increase. While John had a great sense of humor, his main emphasis was on abort landing capability. I don’t remember him bringing up foam issues prior to STS-107. He always ended whatever he was bringing up with the phrase “I’m just asking”. I think many of us listened to John while looking for anything that we had not addressed or thought about. Many of us spent a lot answering John Young letters. He did not limit his comments to just FRRs.

      • Dave H. says:

        Rod, the following article is from Spaceflight Now, I believe. Strangely enough, during The Weather Channel’s show “When Weather Changed History: Challenger” they also touch upon Columbia and the role wind shear may have played.

        Thanks for the insight into John Young’s MO. When modern bloggers want to say something that might offend others it’s commonplace to say “I’m just sayin'” after it. John’s intent was to get someone’s attention and thought process moving in his direction. It falls into the same category as Columbo’s famous “One more thing…”

        Board studies wing edge, wind shear, foam repair
        BY WILLIAM HARWOOD
        STORY WRITTEN FOR CBS NEWS “SPACE PLACE” & USED WITH PERMISSION
        Posted: March 11, 2003
        The Columbia Accident Investigation Board today showed video of Columbia’s launching that indicates foam debris falling away from the ship’s external fuel tank slammed into the lower leading edge of the orbiter’s left wing within a few feet of where it merged with the fuselage.

        The white flash is the moment of impact as the external tank material strikes Columbia’s wing. Photo: NASA video
         
        The board also revealed that 20 seconds before the foam fell away from the so-called “bipod ramp” area of the tank just under the shuttle’s nose, Columbia rocketed through unusually high wind shear. The steering system in the shuttle’s left-side solid fuel booster swiveled the rocket’s nozzle slightly to counteract the effects of the shear, putting some additional stress on the left side of the vehicle.

        This event occurred three seconds or so after Columbia endured “max Q,” or maximum dynamic pressure. This number varies from flight to flight, but for Columbia, it was around 741 pounds per square foot. Whether that had anything to do with any subsequent damage to the wing is not yet known, but investigators are looking into the matter.

        “At 62 seconds on launch, we saw one of the larger transients we’ve seen on the solid rocket motor,” said Maj. Gen. John Barry. “It was well within parameters, but interestingly enough, the two largest ones we’ve seen on ascent both happen to be Columbia, both happen to be going on 39-degree inclinations (trajectories), both have lightweight tanks. So we’re trying to identify if there’s any commonality there as an additional stress load on the left-hand side of the orbiter, because it was with the left solid rocket motor that had this input.”

      • waynehale says:

        I would add to Rod’s comment about wind shear. The DOLILU team would bring the wind plots out several times during the countdown and there was almost always a wind shear at some point in the ascent trajectory. Compared to the aero forces generated by the launch itself, the shear forces were not large. Did it contribute to foam loss? Possibly. Would having launched a different day or even a different time of the same day have prevented the accident; probably not.

      • Yusef Johnson says:

        You forgot the “This is a highly complex technical matter, which I will be glad to discuss with anyone at anytime….”

      • waynehale says:

        That was John’s standard tag line at the end of his memos. Unfortunately, when I tried to talk to him – which I did on several occasions – it rarely went well.

  5. It’s always seemed to me, as someone who has studied, attended and reported on fifty or so shuttle launches for various media, including the final flight of Columbia, that the final Flight Readiness Review should be an explanation of NASA’s work getting the shuttle ready for flight given in great detail not to management but to the astronauts about to fly in it. As a pilot, I require that the mechanic go over in great detail all maintenance/repair work done to my plane and explain the the ramifications of same before I ever start the engine. Yet, it appears that crews asked to put their lives on the line in on top of a rocket are not afforded the courtesy of doing the same. Perhaps if those “disgruntled lower level managers” had to look directly into the eyes of the people who would die if they screwed up, the whole process would be taken a lot more seriously.

    • Sorry for the typos, I’m down to only two hands.

    • waynehale says:

      The chief of the astronaut office represented the crew. Typically they had other things on their schedule and did not come to the FRR. I’m not sure that it would have made a lot of difference; most of the presenters were removed from the real work and multiplicity of small decisions over months that made up the hidden risks. Those who were disgruntled probably still would have waited until they were out in the hall to express their opinions.

  6. Yusef Johnson says:

    CAPT Young, God Bless him!! That man made my career with all the CA issues…I can remember the panic that ensued after he got a hold of the STS-107 Ascent Checklist and the BDA numbers weren’t the ones I delivered. I can remember the fit he raised about CA aero data before STS-105 and the long night Bill Jacobs and I put in running sims to prove it wasn’t a problem.

    I hear he’s been pretty out of sight these days….

    • waynehale says:

      John was focused on crew safety and the knowledge that he was out there typing up his famous memos made us all work harder to do a good job

    • John Hallstrom says:

      @Yu, Good times pre-105, me getting STAMPS Aero upgraded with the CA increments and feeding it on to you to fly out. I believe somebody sketched out something on a napkin about high Mach high Alpha with their stability concerns and slid it to Young which kicked it all off. I recall RCS was still active in the regime in question and could handle the Aero even with the increments.

Leave a reply to Rod Wallace Cancel reply