The School of Hard Knocks

Learning hard rocket lessons in the 1930's

Robert Goddard Launches a Rocket

It has been a tough sledding for rocket boys recently.  Just look at what has not worked right:  a Chinese Long March 2C rocket failed on August 18.  Russia lost two last month. a workhorse Proton upper stage failed to put a communications satellite into the proper orbit, and the most launched rocket in history, the Soyuz launch vehicle, suffered a third stage engine failure which resulted in the cargo intended for the ISS scattered all across northeastern Russia. Here at home, the Blue Origin commercial spacecraft development organization had a test go awry when their vehicle lost control during supersonic flight.   And last but not least, SpaceX reported an incident at engine shutdown during a launch several months ago.  Not that the list wouldn’t be longer if we look a little further back in time; but this is lengthy enough to make the point.

Does this mean that these organizations are incompetent or about to fail?  Or does it mean that we should give up trying to travel into space?  Not at all.  In fact, just the opposite.  Rocketry is a difficult subject; and failures – although we try our best to prevent them – have the salubrious effect of teaching more lessons in a short time than can be learned in a long time from a string of many successes.  In fact, success has a funny way of playing on human nature and breed unwarranted confidence, maybe even arrogance. ‘Pride goeth before destruction’ to quote a proverb, and a long string of successes generally over-stimulates pride. 

One of the hallmarks of a successful organization is being a learning organization.  Everyone has new lessons to learn.  In the high risk, high energy, low margin endeavor that is spaceflight, no matter how hard you try, little things are going to go wrong.  What you hope is to learn the lesson before the little things grow into big things. 

So in all the incidents last month, the affected organization will learn the technical and organizational lessons which come from the failure, and will thus insure future success.  Or if they don’t learn, natural selection will take them out of the gene pool in short order.

When I was a young flight controller at NASA, my bosses taught me to immediately and publicly confess my shortcomings and errors.  This was seen as the only way to improve.  If anyone hid a mistake, they were quickly shown the door.  Mistakes can happen, especially with young and inexperienced personnel.  Recognizing and admitting a mistake, then learning from that mistake – which includes taking advice from those who have been down the path before you – is the key.  I have a strong memory from one of my early bosses who advised that after making a mistake, you should find the most senior manager in the area and immediately and in great detail admit your mistake in all its gory detail and quickly explain how you are never ever again going to let that mistake happen.  I had more than one occasion to practice this advice. 

There is truly something to be said for transparency.  Putting an accident investigation report on public display allows others to learn from your mistakes and makes the entire industry better and safer.  Involving others in the dissection of a failure leads to a richer, deeper, and more fruitful set of corrective actions.  All of this goes to preventing accidents in the future.

When someone keeps a mistake hidden, nobody learns from it; nobody recognizes that the activity is hazardous; and more accidents will occur.  Secrecy, whether it is in the name of national security or corporate stability, has a price.  Nervous investors may not bear up well listening to a rocket failure report, but this enterprise is not for the faint of heart.

Besides, these things always get out.  You cannot hide them forever, and eventually the story gets publicized.  Then you have lost credibility and your reputation for integrity is in tatters, all that in addition to the real technical problem that has most likely gone unsolved because it was hidden under a bushel basket.

Nope, honest is the best policy in the rocket business; transparency will bring help and advice that you would not have gotten by holding a secret close. 

Good luck to all those who put their hearts and sweat into building machines to leave the planet.  I know it is a hard business.  But the rewards for ultimate success make it all worthwhile.

 

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

10 Responses to The School of Hard Knocks

  1. Beth Webber says:

    Excellent post, Wayne. Thank you for putting into perspective the utility of failure.

    Beth

  2. Jim Hillhouse says:

    Wise words as usual from Wayne Hale.

    In aviation, we have the 4 C’s: Confess, Communicate, Climb, Comply. All four are relevant in rocketry. Confess a fai…er, event. Communicate it to the community. Start working on the incident review. Make changes based on the incident review.

    There’s nothing in aerospace that cannot be solved if openers rules.

  3. Jim Hillhouse says:

    Ummm…if openess rules.

  4. David F says:

    I remember vividly hearing Ed Fendell say to someone over the loop, after hearing how they had done something or other, “Fix it, and don’t do it again!” (I’m sure this has meaning only to people who were at JSC back in the day…)

    I have come to dislike the term “mistake” and “error.” Because they are arbitrary labels, placed on an action/decision after the fact by investigators, and then managers, who usually make no effort to understand the circumstances that led to that decision. I’d rather think of it as decisions that have undesirable outcomes.

    The problem with using the label “mistake” or “error” is that we tend to stop there. We start thinking in terms of “re-training,” “motivate,” and worst of all, “discipline.” It’s a problem because that way of fixing the problem keeps most people from admitting they made a “mistake” unless they have to.

    By thinking in terms of decisions, we first of all shed some of the biased baggage that is associated with “errors.” Then we can start thinking in terms of “why did they make THAT decision instead of another?” That can lead us to recognizing and understanding how manager and peer pressures, training, fatigue, distraction, and cognitive traps can lead an individual down a path.

    Best of all, we can avoid the typical reaction to hearing/reading someone made a mistake, which is usually, “I’d never do that, because I’m (trained/smarter/better looking/ – pick one)!” and dismiss it as not applying to us.

    How many times have you sat through a presentation or read endless lists of Lessons Learned and two days later couldn’t remember any of it? That’s because we fail to make an emotional connection with persons who were involved. We fail to see that they are just like us, that they too thought, “I’d never do that…!” and then some set of events happened in just the right sequence and context to lead them to a decision that had an undesirable outcome.

    We fail to think in terms of “I could have made that same decision if I had been presented with the same evidence, and missed my normal cup of coffee that morning.”

    A true learning organization presents lessons learned in a way that evokes an emotional response within the audience. Without that emotional response, we won’t remember it, because it has no meaning.

  5. reader says:

    Just to add to the list. Armadillo Aerospace has planted two vehicles recently.

  6. Louis Cioletti says:

    Bravo! It’s against our human nature to publicize our failings. Heck, I feel I hedge on my confessions to my pastor! Who am I fooling???
    The irony of your message is that is doesn’t strip you of your dignity as one might imagine. Instead, it reinforces the courage of your convictions that you can be trusted to “own up”. One might believe it’s a very lonely position when admitting one’s mistakes. But Andrew Jackson said, “One man with courage makes a majority.” Sounds like you and Jackson have something in common.

  7. nooneofconsequence says:

    We have a “gotcha” culture of separate teams, that do this “enemy of my enemy” stuff. Makes it hard to be transparent when someone else uses your “open-ness” as collateral to gear up the noise machine against you … imperfect as a form of impure in a faux purity age.

    With so much ill will, perhaps the point of manned spaceflight might get lost. The hard fight is always against the fates. And Murphy.

    Nice to see you posting more.

  8. Cheryl desLandes says:

    Dear Wayne
    TYour thoughts were sent to me at a very low time as a manager – thanks. Phew.
    I habe a daughter wanting to be an astronaut – please take her and good luck on her keeping her room tidy. My other daughter wants to develop the programs that drive the s[ace ship that takes her sister away. You on the other hand – please help me manage my personnel problems and if you want a NZ Holiday – you are welcome (car provided). YOu will be busy

  9. Ed Minchau says:

    While admitting mistakes or malfunctions is good, how much detail can be made available outside the company before running afoul of ITAR restrictions?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s