“Do good work” – Virgil Ivan “Gus” Grissom
This January 30 has been designated ‘Remembrance Day’ in honor of the astronauts who lost their lives in the Apollo 1, Challenger, and Columbia accidents.
It is altogether fitting that we remember their dedication, bravery, and sacrifice.
But it is not enough to partake of fuzzy emotional sentiments about loss; nor is should our remembrance day devolve into a contest of ‘I remember exactly where I was when I heard’ stories.
For those of us engaged in pushing back the space frontier it means we must work to learn something from those accidents; lessons which will keep us from adding a future catastrophe to the list for some remembrance days to come.
So I ask you: what did you learn?
How about this: Pure oxygen atmosphere in a space craft may be necessary but the entire process should be approached cautiously; ignition sources must be eliminated by design and workmanship and flammable materials must be minimized if not completely eliminated.
So there is that. What about other lessons?
Attempting to fly space systems outside their design and tested environment (e.g. cold temperatures) is not allowable. Systems should be qualified by test as well as analysis for temperature extremes and never operated outside those bounds.
Do you have another lesson?
Re-entry heat shield integrity should be established in orbit by direct inspection. Repair material and techniques should be available if damage is detection. In design, all efforts should be made to protect re-entry heat shields from debris impacts which could cause damage.
Does that cover the entire spectrum of lessons?
Not hardly. I can think of at least one more.
In each of the accidents there were people who believed that the programs were proceeding into unsafe territory. These people tried with varying degrees of success to alert the management of their concerns. In some cases, they fell silent quickly. In other cases, they were overruled and gave up. Later, in all three of the accidents, the top leaders unanimously said ‘we didn’t know anybody was concerned’.
The lesson to take away here is not to give up. If it is unsafe say so. If overruled, appeal. If denied appeal, make your case to the highest level manager you can find. Do not give up until you have been heard at the very top.
Because you might be the only one that sees what no one else can.
Don’t live with regret.
Wayne Hale's Blog 
Amen…
Reblogged this on Bud Branch and commented:
Well said, Wayne.
Not entirely sure of that “Re-entry heat shield integrity should be established in orbit by direct inspection” lesson. That was never done for any vehicle except the shuttle, and (as far as I know) not likely to be done on any of the new crew capsules under development. Is that a real lesson learned, or just a reflexive refight of the last battle? “Don’t expose your heat shield to debris” seems like the more worthwhile lesson, we just learned it too late to apply to shuttle.
Good start to a dialog about lessons leaned
Wayne, Thank you for your constant reminders of our obligations for being both vigilant and persistent. As one who lived through one of these disasters and came into NASA directly as a result of another one, I understand the context of your article and agree fully with its purpose. I retired 5 years ago and my memory fades a little every day with the details of all that was discovered as a result of the accident analyses and the systemic analyses from the inevitable “Blue Ribbon Panels.” These are great documents to understand the “what,” the “how” and the “why.” I hope that everyone, from line worker to Administrator, involved in Human Spaceflight (and even those in NASA that are not) are encouraged at the end of January to immerse themselves in reading the accounts of these accidents and the resultant conclusions and recommendations. Millions of dollars were spent to determine ways of preventing recurrence and we must take advantage of the insight gained. It is a very good way to recall and learn the lessons and if there is even one disaster averted because of a reminder of someone else’s failure to notice and act it will be worth the time spent. Space exploration is exhilarating by its nature but intolerant of exceeding the known design and operating margins. The real key is to know the margins and to have the fortitude and foresight to heed them.
Very good advise, applicable for much more than spaceflight. Any endeavor that has the potential of causing injury or death can apply this lesson; be vigilant, communicate and be persistent.
‘Don’t live with regret’ may be as important as any lesson; otherwise we will stand paralyzed.
We should all be glad if we could flatter ourselves that we come as near to the central idea of these tragedies in our all posts as you have here (ref Edward Everett re: Lincoln’s Gettysburg Address). Thanks for posting this.
Lesson: If your management does not listen to you, in the case imminent loss of human life and billion of dollars of equipment (a space shuttle in this case), my advice is to take your case to the public media.
In the case of Roger B. who worked at Thiokol, he was ostracized, demonized and lost his job in his field anyway (after his testimony before the Challenger Commission), so if I were in his shoes I would have gone to the media with all the paper work that proved the O-ring problem needed to be fixed before any more shuttle flights.
The loss of human life, and in Challenger’s case, very intelligent, top of their field people, I would do everything possible to prevent their deaths.
That’s just my opinion.
I took a presentations class at JSC in 1987, and the instructor stressed the importance of being able to make a good case, and pitch it well, because the Challenger mishap might not have happened if someone had done that better.
After the return to flight in 1988, there was a Crit 1 problem that became known with cracking in the temperature probe wells in the cryo piping upstream from the SSME’s that was discovered by a contract metallurgist who was in NH, I believe. I wonder how much trouble he had convincing the program management there was an issue?
Had not heard that one
Sort of short and sweet and a good lesson for certain. The problem is that nobody wants to be “that guy” that is always trying to throw a wrench into things.
Have a good ‘un, Bobbo!
On Wed, Jan 27, 2016 at 6:07 AM, Wayne Hales Blog wrote:
> waynehale posted: ““Do good work” – Virgil Ivan “Gus” Grissom This January > 30 has been designated ‘Remembrance Day’ in honor of the astronauts who > lost their lives in the Apollo 1, Challenger, and Columbia accidents. It > is altogether fitting that we remember their ” >
““Those who fail to learn from history are doomed to repeat it”
~ Winston Spencer Churchill
After every such accident we hear “Never Again”, then it happens again. Sally Ride, the only member of both the Challenger and Columbia investigations, noted a disturbing “echo” during the STS107 investigation 17 years after 51-L. We can only hope this time the lessons are not only learned, but remembered.
Thank you Wayne for your continued and candid discussions of the events which lead to the loss of 17 great Americans.
There does seem to be a pattern (and not just in the space program) that no matter how many people try to raise the alarm, nothing actually happens until somebody dies. In my latest short story I speculate a little about whether a sufficient close call could be sufficient:
“The speed with which events had unfolded made it difficult to examine the timelines and determine the effects of slight changes. Even a minute too early and the flames didn’t have enough time to spread to the point that would convince the program managers that problem lay in the spacecraft, and not the attitudes of the astronauts. If they started before it actually ignited, there was a sweet spot of a few seconds when they could finishing opening the hatch just as everything was catching alight and the three of them would emerge with flames licking across their suits, a visual so dramatic no one could miss the message that they’d escaped a deathtrap. They’d bear the scars for the rest of their lives, but they’d get to have those lives, even go on to fly later missions.”
Wayne: I enjoyed your Blog. It brought up some painful memories. The backstory is that before Challenger’s tragic events, I decideded to review the Filament Wound Case Solid Rocket Motor RFP. In the RFP, Thiokol and Hercules were given an SOW that called for them to build the FWC for 19 reuses and to the 28 degree low temp. When I asked how MSFC expected them to demonstrate that they had met the specifications and thereby become certified, I was informed that they were just restating the SRM specs.
Since Thiokol didn’t have a test chamber which would allow them to cool the stack to 28 degrees, I knew they couldn’t have demonstated they were in compliance with the thermal spec. And, as for 19 reuses, MSFC agreed that Thiokol could demonstrate compliance by continuing to fly the cases until it got to 20 uses.
And, during the fateful reviews that led to a go for launch, the new replacement {whose name escapes me] for Bob Lindstrom as the program manager for Marshall Systems actually questioned Thiokol as to why they were having reservations about launching when the experienced stack temperatures were well above the spec. As I recall, he had been the IUS program manager before he was promoted to the post, and God help us, he had read the SRM specs.
And, there are many other instances (e.g.,Orbiter RCC resistance to impacts) where NASA managers had never tested the hardware’s capabilities. In most cases, the hardware was “qualified by analysis.” So, the lesson to be learned is not only to read the specs but ask how they had certified compliance.
In all cases that I researched during my tenure as the Shuttle Program’s Independent Assessment analyst for Code B, it was clear that doing the tests were deemed to be unaffordable by the program management. “Unaffordable” translated into “if we asked for the funds to do the tests, they would come out of our hide or would impact some other Center priority.”
But, the real shortcoming was not being upfront and transparent about their decisions and documenting them for the benefit of the new engineers coming on the programs and the program control/budget managers.
As to the “we don’t have the money”, when I became NASA Comptroller under Dan Goldin, I brought this excuse up to him as a cultural issue in the agency. He directed me to repeat and repeat to all NASA managers that either I or my staff encountering using that excuse that it was totally inappropriate and unacceptable to Dan Goldin, because by God, we would find the money to do the tests and ensure a safer Space Shuttle.
On Wed, Jan 27, 2016 at 7:06 AM, Wayne Hales Blog wrote:
> waynehale posted: ““Do good work” – Virgil Ivan “Gus” Grissom This January > 30 has been designated ‘Remembrance Day’ in honor of the astronauts who > lost their lives in the Apollo 1, Challenger, and Columbia accidents. It > is altogether fitting that we remember their ” >
“So I ask you: what did you learn?”
What I learned goes far beyond mere physics and human performance. Someday, either here or beyond the veil, I’ll explain it to you, and you’ll laugh at how simple – and how complex – it was.
“Because you might be the only one that sees what no one else can.”
Thanks for the tip-o’-the-hat, Wayne.
“The lesson to take away here is not to give up. If it is unsafe say so. If overruled, appeal. If denied appeal, make your case to the highest level manager you can find. Do not give up until you have been heard at the very top.”
And, if you have what you need to prove your case, either common sense or hard data, don’t be afraid of finding a bigger voice to present your case.
Take that leap of faith! It’s far better to have tried and failed than never to have tried at all. A hero dies one death, a coward a thousand.
“The highest level manager you can find”, let me remind everyone, is the Administrator.
If he won’t talk to you; if he blows you off…
If you’re willing to stake your career on it…
Go to the public.
That’s what I stated. Roger B. and Arnold T., both Thiokol Engineers argued for hours that based on previous o-ring erosion, that temperature was a factor in the sealing capabilities of the o-rings, yet NASA Engineering at Marshall Spaceflight Center “pressured” Thiokol management (which had a renewable contract upcoming with NASA to continue to provide the SRB’s) in to making a “management” decision to approve the launch of Challenger. They were afraid if they did not approve the launch that the contract for SRB’s might go to a competitor.
And as we saw, NASA management and culture did not change after Challenger for any extended period of time, because the Columbia disaster could have been prevented also if engineering and management at NASA fixed the foam loss off the ET. Two flights before Columbia, from almost the exact place on the ET, foam fell of and impacted the lower portion of an SRB casing, which is 1/2″ thick steel, and caused some damage to that case.
Yet, NASA ignored this foam loss, which happened many times before, until the ill-fated Columbia mission which just happened not to be going to the ISS and also not having the remote manipulator arm onboard. Those odds (not going to ISS and no RMS along with continued foam loss from the approximate same place on ET impacting the leading edge of Columbia’s left wing) are probably as high as winning the Powerball lottery.
The STS-112 foam loss impacted the SRB case but did not cause damage. A streak or smudge was the ‘witness mark’. But even without damage it was still a warning
STS-112 foam loss didn’t cause damage because as you know, the SRB casing is made of 1/2 inch thick steel. I know that the leading edge of the wing is RCC, but how much of the leading edge is RCC? I seem to recall reading in the book “Wings In Orbit” that the RCC was just 0.2 inches thick. If that is true, what material composes the “thick” physical structure of the wing leading edge? John Young, in the program “When We Left The Earth: The NASA Missions” stated that he was told that hitting the leading edge of the wing with a (baseball) bat would cause no damage by engineers, and said “they weren’t exactly telling the truth.”
Lastly, I suppose as “HAL” from the film “2001:A Space Odyssey” would say, that it was “human error” that the ET foam shedding problem was not addressed by the Rogers Commission investigating the Challenger disaster. If it was, then perhaps the Columbia disaster would have been prevented. I believe it was STS-27, with Hoot Gibson as Commander, on a DOD mission, not long after return to flight (Challenger) that the side of the right wing chine’s tiles were severely damaged and that they were lucky then that they did not suffer the fate of Columbia 17 years later.
Spaceflight question – is there a way to extend the re-entry to reduce the overall heat load on the spacecraft? Is it simply that you’re going to be creating plasma, and slowing-down-and-falling long before you can use the air that you’re hitting is thick enough for you to climb back up? Also, if you stretched the re-entry out from ~15 minutes to ~45, would that make a difference in how you managed the heat?
No,
Conservation of energy means you get all that heat back. If adjusted for a slightly lower max temp, it goes on longer.
We did that sad calculation many times after Colmbia
Mr. Hale,
I’m curious as to your thoughts on the O-ring issue as a whole. Was the design safe enough to use as long as the temperature was above a certain point? I seem to remember reading that it took some effort in tests after 51L to get the joints to fail.
Seems like an academic question. All I can really say now is that the joint design was deficient against the environments in which it was required to perform. But that is 20-20 hindsight
“Seems like an academic question. All I can really say now is that the joint design was deficient against the environments in which it was required to perform.”
Early on in my son’s engineering classes he took a required course about material properties. The primary lesson was that everything in the universe has physical properties that are temperature-dependent.
Had the “stack” been certified for flight in a -10C- +40C environment then the environmental conditions on January 28th, 1986 wouldn’t have been an issue.
My son knows that buildings are designed for high percentage ambient seasonal conditions. Anything beyond that falls into statistics and probability…i.e.; what are the chances of an event exceeding the design conditions and more importantly, what will the most likely consequences be?
The SRB joint design was not “deficient”, it wasn’t certified for the expected flight conditions, yet like the captain of the El Faro (who for reasons still unknown chose to sail directly into a Cat 4 hurricane) the decision to launch was made despite engineering concerns.
I commissioned a gas turbine power generator at El Sauz, in Mexico. Located at roughly 6000 feet above sea level, the turbine was rated at 100MW *at that altitude at 20 degrees C* versus the 150 MW rating at sea level. Gas turbines are basically jet engines and follow the same physical laws. Less air means less power. Our customer wouldn’t expect to be able to make 150MW under those conditions. On our first run we made 128MW because the ambient temp was 5C with 100% humidity – thick fog. Once the sun came up, the air warmed, the fog lifted, and the power output dropped off to 100MW.
Not knowing is one thing. Knowing and ignoring is another thing entirely.
Regarding Challenger & Columbia, let’s go further still. If the Space Shuttle had been built as originally designed (i.e., with the Orbiter atop the stack, instead of adjacent to the External Tank and the Solid Rocket Boosters), the crew of the Challenger would likely have survived the accident (since the Orbiter atop the stack would have been well away from the explosion). Moreover, the Columbia accident would not have happened, since ice falling and hitting the Orbiter wing and compromising its thermal protection would not have been possible with an Orbiter atop the stack. Bottom Line: Design decisions have consequences. Lest we forget, the Challenger & Columbia accidents represent terrible, terrible – and unfortunately avoidable – tragedies.
Correction: As you know, it wasn’t ice that fell and struck the Columbia Orbiter – it was foam. Sorry for the brainburp.
“Design decisions have consequences.”
Certainly. Some decisions are made as compromises, some as cost considerations, but they all have consequences.
The O-ring issue that doomed 51L was somewhat speculative because no shuttle had ever launched in the conditions they launched in. If it hadn’t failed, “see, whatcha so worried about” would have carried the day.
By 2003 there was a considerable case history of foam strikes and the results but the data didn’t point anyone towards trying to engineer out a resolution. Why devote money and time to resolving a non-issue?
Read about the cracked stringers on STS-133’s ET and see how they caught it in time. How they resolved that issue is how they should have resolved 51L and 107. The reason you never heard about it is because near-hits aren’t quite as visible as disasters.
Hindsight is *almost* 20-20!
I’m not sure my point was fully understood. The Space Shuttle design change (i.e., moving the Orbiter from the top of the stack to adjacent to the ET & SRBs) had loss of life implications. I remember going to NASA-JSC when the Galileo mission first was manifested on the Shuttle in the late 1970s, and pointing out that to fly a nuclear-powered spacecraft on the Shuttle, NASA, DOE, and an Interagency Nuclear Safety Review Panel (INSRP) would require a Shuttle Databook that identified and characterized the probability and environments associated with all pre-launch, launch, ascent, and orbital injection accidents that could credibly occur. I was told that the probability would be 1E10-5 per launch. My response was that was not at all likely since no launch vehicle had ever achieved 1E10-3 per launch, and a two order of magnitude improvement would be extremely difficult to achieve, and virtually impossible to validate. The response was that the Shuttle would be man-rated; thus, would have a failure probability of 1E10-5 period. I think NASA-JSC at that time was trying to send a signal about planned high reliability of the Shuttle, and was trying to avoid or delay production of such a Databook. As we know, the Shuttle failure probability proved to be ~1E-2 per launch. History (i.e., reality) is a source of data; here, it is serves to provide insight for launch vehicle failure probability. But that isn’t the whole picture. Failure of the launch vehicle does not necessarily result in loss of crew. For the original Shuttle design with the Orbiter atop the stack, crew escape (and even Orbiter separation) was relatively straightforward. But that was not so, and acknowledged as such, for the final Shuttle design (with the Orbiter adjacent to the ET & SRBs). My point is that human safety goes beyond launch vehicle reliability, and the probability of a failure during launch. Human safety requires robustness, forgiveness, and protection – given failures.
Wayne, can I share some of your blogs on my FB page. I feel that you may want to put them all in a book some day and would prefer me not to……I can hear you in Sunday school class when I read your posts. I wonder if you are still in Friendswood ….. I am taking a road trip this summer and would love to say hello.
Yes
I’m trying to understand how much of this is hindsight. Were the three accidents unusual in having people who believed that the programs were proceeding into unsafe territory, or was this par for the course?
Par
Industrial safety consultants will tell you that a large percentage of disasters and accidents are caused by human performance issues. They will also tell you that most disasters are the result of a number of small issues followed by an unexpected event that brings the disaster about.
Wayne was there for both shuttle losses and thus has a unique perspective on them.
In that respect, Challenger and Columbia share many human performance issues but the results were the same…plenty of post-disaster retrospective salted with a liberal amount of “I wish I would have said/done something.”
Mr. Hale, I am so glad I found your site. Your humanity and humility honors your organization and your Country. And you are a great writer and storyteller. You sir are a steely eyed missile man! It takes a special man to do what you did. Like I heard Gene Kranz say on a TV doc one time, ‘We teach all our controllers that between your faith in your God and your Country and your own abilities–you can solve any problem.” Words that have inspired me ever since.
While this could flood the upper management hierarchy with concerns, maybe it is still better than having concerns be “dispositioned as not a safety-of-flight issue” by lower level groups and thereafter be a closed subject.
Lesson: Listen to warnings regardless of the rank or status of the person. No retaliation, no poking fun. Demonstrate to your team, especially the interns, that it’s safe to speak up.