How to Avoid Train Wrecks

There has been a recent renewal of interest in a post I made five years ago
“The Coming Train Wreck for Commercial Human Spaceflight”
If you find that interesting perhaps you should read my post on Standards from just four years ago
Some people have called those posts prophetic. Not really. I don’t have all the answers. Just sad, tough experience from the inside out.
So how can we avoid a train wreck, not just in commercial human spaceflight but with the NASA exploration systems as well.
Spaceflight is exacting and very hard; the environments are tough, the energy levels are extreme, and the margin between success and failure is very slim. All of this is said so often that is sounds like a trite cliché. But no matter how trite it sounds, it is true. In the 21st century, still true.
We know a lot about launching satellites and people into low earth orbit – it has been done for decades. If you grew up with the Space Age, as I did, you got very accustomed to pictures and videos of rockets blowing up. Nowadays, rockets seldom blow up. I think the recent Atlas launch was the 100th success in a row. How did that occur?
Digging through the debris from failure after failure, engineers learned a little at a time about the complex and exacting process necessary to improve the chance of success in rocketry. No secrets anymore, nowadays those processes are well understood, well documented. And mostly followed.
To cite a couple of recent incidents; it is well know that a new heavy payload on the top of a new and spindly rocket (they are all spindly) it is imperative that the springy interaction be analyzed. If you don’t do that, well, you run the risk of failure like 59P last April.
Or if you try to lower the cost of your rocket by purchasing non-aerospace-standard parts, and don’t test each and every one of those parts, you run the risk of something breaking like it did on Spx-7 last June.
Or if you use really old rocket engines, you really should . . . well, enough recent history.
The point is, success in spaceflight is not easy and comes at a high price because the work is exacting and unforgiving. But we know how to do it well. Just follow the process.
But there is a rub, because the process, the standards, are not static. They are always being improved. Added to. Never reduced, unfortunately.
After Columbia, part of what we did was over the top. In order to satisfy all the critics, we did everything anybody asked for. Much of it was necessary, most of it was good, some of it was no value added bureaucracy to scratch some independent reviewer’s itch. All of it got codified as ‘lessons learned’ and added to the process.
The James Webb Space Telescope is a ‘must work’ project for the agency. There have been serious challenges, both technical and management. NASA has learned a lot about how NOT to manage a complex development project in the JWST experience. Much of that has been codified into new and ‘improved’ processes. Added processes. Always more, never less.
I have a cheap seat view of the Orion/SLS development. My basic observation: those efforts are drowning in ‘process’. The biggest threat to their success is not technical; it is schedule and cost. If the design and development processes drag the projects out too far, Congress or a new Administration will throw up their hands and call a halt to the whole thing. They did once before; my intuition is that they will again unless something significant happens.
The secret of a good program – as a very senior spacecraft designer once told me – is knowing how much is enough and then not doing anything more.
Right now, inside NASA, we have trained our workforce to do it perfectly. And perfection is very costly and takes a long time. Over in the Commercial Crew Program, the senior leadership is making some progress in toning down the drive for perfection. It is a slow effort and uphill at all times. Over in the Exploration systems area, it all seems to be going the other way. Whatever anybody calls necessary for safety or improvement – without evaluating the real cost or schedule or other impact – seems to be adopted.
So I am guardedly optimistic about the commercial teams actually succeeding in flying humans in space in the next couple of years.
Not so much optimism for the exploration systems, drowning in ‘process’.
Most engineering problems have an optimum solution, a point where doing more actually results in less performance. This situation is not an exception.
Somebody needs to be able to just say no. In order to succeed.
It’s not really rocket science, it’s just good engineering.
Everybody in the policy world and the blogosphere wants to debate destinations – Moon, Mars, Asteroids – or the shape and size of the rocket. All interesting but not really relevant. Pick a place. Make a design. If it’s going to actually fly, do what is necessary – and not one thing more.
That’s what real rocket builders need to focus on.

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in Uncategorized. Bookmark the permalink.

25 Responses to How to Avoid Train Wrecks

  1. Robert Barron says:

    Hi Wayne,
    The links in your blog post send me to login to WordPress, not to other places in your blog.

    Thanks for your insights, as usual.

  2. Vince says:

    Wayne, as always thought filled and thoughtful. Yes, NASA needs to commit to an objective then get there. I am skeptical that the agency can or wants to do that absent White House push. We can agree it does not have to a JFK-esque commitment “…by the end of this decade”, but it needs to be something can span the yawning chasm/political divide in Congress. Ugh.

    I thought your mention of recent industry mishaps was apropos. You have written previously–paraphrasing, that safe exploration/operations entail and turn on certain kinds of organizational thinking. Thanks.

  3. Steve says:

    “A good solution applied with vigor now is better than a perfect solution applied ten minutes later.” – Gen. George S. Patton, Jr.

    Words to live by. Or, as I’ve been constantly reminded in my enginerding career: once the car has been sold, stop selling it!

    Excellent advice here, Mr. Hale. It should be part of any engineering curriculum. But alas, the exorbitant price placed on lives by the public–who have no understanding that those who’ve signed up to ride the rockets have already considered (and accepted) the potential costs–will prevent this sort of practical thinking. I hate to be the pessimist!

    FYI..might want to check those links to your previous blog posts; they require sign-in to view.

  4. “do what is necessary – and not one thing more.” requires actual flight tests, not computational analysis. Most of the commercial players out there live by “Fly early. Fly often.” because that’s really the only reliable way of learning what is “good enough”.

    • patb2009 says:

      Not to be directly obstreperous, but the Success rate of the Atlas 5 and Delta 4 show that the analysis heavy, process heavy approach of ULA is “A reliable way of learning”.
      The ULA approach is not cheap but neither is blowing up 10% of the fleet.

      There is a fine balance, and finding out that something may only fail 1:10,000 times
      may be an expensive lesson when you have 500 of those in your bird.

  5. spacebrat1 says:

    the voice of reason. why is that such a tough position to espouse?

  6. Charley S says:

    Strangely enough, recently sat across an engineer and who said that was his biggest problem in training new engineers was teaching them ‘when you get it to work, stop there’.

  7. “Do what’s necessary, and not one thing more” seems to be what the commercial space ventures are doing now. The hard part of that is knowing what’s necessary and what is not. There will be times when it’s something seems to be not be necessary, but actually is, and it’s exclusion causes a failure.

    In some ways, NASA hasn’t contributed to identifying the line between necessary and not. NASA, unlike the commercial space ventures, cannot tolerate failure. For NASA, if there is even a very small chance that something is necessary, it gets included. For the commercial space ventures, if something has a low probability of being necessary, they have the option of excluding it. When these things turn out to be necessary causing rockets to fail, it expands the bounds of knowledge of what’s necessary.

    Instead of building new, perfect rockets, NASA should work towards building the knowledge of what’s necessary. NASA should be building rockets that fail. Let commercial space build rockets that are safe and work. The roles of NASA and commercial space are completely backwards.

  8. rktsci says:

    I had a mid-priced contractor seat on Orion for a while. Among the problems that we had:
    – NASA keeps changing things. Ares interface keeps changing. Launch on Ares, now on SLS. US built Service Module, now ESA built one (but only for a couple of missions!). Missions deleted.
    – Late delivery of specs from NASA. God help us if we were a day late on a document, but NASA documents were often months late.
    – Rehashing. There were multiple studies done of a composite structure for the Crew Module. Every one said it wasn’t worth doing, mainly because of all the penetrations and stuff hanging off it. Yet the higher ups in NASA kept insisting on looking at it over and over. Never mind that there were composites being used in many other areas. Then the big rehash took place. As the Shuttle and ISS ramped down, NASA had to do something with all the civil service staff being offloaded from those projects. So they added many to Orion. And the questions started, rehashing all the items previously decided on. Requirements that had been nailed down for years were reopened.
    – Schedule stretch. As you stretch out the schedule, costs rise as you have to keep staff around longer.

  9. Andrew_W says:

    The real advantage of commercial space is survival of the fittest.

  10. Jacob Krall says:

    The links in the first paragraph are unreachable to non-Wayne people 🙂

  11. Ed Fendell says:

    Wayne: I find your blog very interesting. It is very true. A good deal of this problem I believe is caused by the loss of your experience base. When NASA stopped exploring and went to what I call daily operations their experience base started to leave and that has continued on to this date, where it is all gone. Like it or not you and I are one of those. The continual process of making items better and spending more continues on. I like to call it the “More Better Syndrome”

  12. astronist says:

    Many thanks for fixing the links. Interesting reading (I’m a British Interplanetary Society member and blogger, but not in the space industry myself).

  13. Gordo says:

    Not sure you are sending consistent messages. From your last blog:

    “In space, the enemy is physics and chemistry . . . and finances. It may be that flexibility and leaving options open provides a better path for our long term ambitions in space. Who knows what may be invented in the next five years that could change the entire game plan?”

    From your latest:
    “Everybody in the policy world and the blogosphere wants to debate destinations – Moon, Mars, Asteroids – or the shape and size of the rocket. All interesting but not really relevant. Pick a place. Make a design.”

    Care to give some examples on how Constellation is “drowning”?

  14. Gordo says:

    Whoops, meant NASA Exploration….

    • waynehale says:

      Do you know 7120.5E ? That is exhibit 1

      • Charley S says:

        Bottom of page 3 of 7120.5e (all the other documents that pertain) certainly reinforces that we are experiencing the ‘death of a thousand paper cuts’! If each of those documents are also a 150 pages, wow.

  15. jbensted says:

    Wayne, did NASA ever consider this:
    Since the major problem with the Space Transportation System (STS) was its side mounted configuration, I wonder why NASA did not consider placing the Shuttle on top of the STS stack. The Space Shuttle Main Engines (SSME) could have been moved to the bottom of the External Tank (ET), possibly in a recoverable configuration. The Space Shuttle could have been modified to account for the removal of the SSMEs and placed on top of the ET. An adaptor and aerodynamic fairings could have been constructed to mount the Shuttle safely on top of the ET. Was not the ET able to have the Shuttle mounted to its side? Why not on top of it? Perhaps minor changes to the ET would be required, but we would still have the Shuttle flying to this day.

  16. These rare, long form blog posts are much more interesting than your twitter posts.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s