How to Avoid Train Wrecks

There has been a recent renewal of interest in a post I made five years ago
“The Coming Train Wreck for Commercial Human Spaceflight”
If you find that interesting perhaps you should read my post on Standards from just four years ago
Some people have called those posts prophetic. Not really. I don’t have all the answers. Just sad, tough experience from the inside out.
So how can we avoid a train wreck, not just in commercial human spaceflight but with the NASA exploration systems as well.
Spaceflight is exacting and very hard; the environments are tough, the energy levels are extreme, and the margin between success and failure is very slim. All of this is said so often that is sounds like a trite cliché. But no matter how trite it sounds, it is true. In the 21st century, still true.
We know a lot about launching satellites and people into low earth orbit – it has been done for decades. If you grew up with the Space Age, as I did, you got very accustomed to pictures and videos of rockets blowing up. Nowadays, rockets seldom blow up. I think the recent Atlas launch was the 100th success in a row. How did that occur?
Digging through the debris from failure after failure, engineers learned a little at a time about the complex and exacting process necessary to improve the chance of success in rocketry. No secrets anymore, nowadays those processes are well understood, well documented. And mostly followed.
To cite a couple of recent incidents; it is well know that a new heavy payload on the top of a new and spindly rocket (they are all spindly) it is imperative that the springy interaction be analyzed. If you don’t do that, well, you run the risk of failure like 59P last April.
Or if you try to lower the cost of your rocket by purchasing non-aerospace-standard parts, and don’t test each and every one of those parts, you run the risk of something breaking like it did on Spx-7 last June.
Or if you use really old rocket engines, you really should . . . well, enough recent history.
The point is, success in spaceflight is not easy and comes at a high price because the work is exacting and unforgiving. But we know how to do it well. Just follow the process.
But there is a rub, because the process, the standards, are not static. They are always being improved. Added to. Never reduced, unfortunately.
After Columbia, part of what we did was over the top. In order to satisfy all the critics, we did everything anybody asked for. Much of it was necessary, most of it was good, some of it was no value added bureaucracy to scratch some independent reviewer’s itch. All of it got codified as ‘lessons learned’ and added to the process.
The James Webb Space Telescope is a ‘must work’ project for the agency. There have been serious challenges, both technical and management. NASA has learned a lot about how NOT to manage a complex development project in the JWST experience. Much of that has been codified into new and ‘improved’ processes. Added processes. Always more, never less.
I have a cheap seat view of the Orion/SLS development. My basic observation: those efforts are drowning in ‘process’. The biggest threat to their success is not technical; it is schedule and cost. If the design and development processes drag the projects out too far, Congress or a new Administration will throw up their hands and call a halt to the whole thing. They did once before; my intuition is that they will again unless something significant happens.
The secret of a good program – as a very senior spacecraft designer once told me – is knowing how much is enough and then not doing anything more.
Right now, inside NASA, we have trained our workforce to do it perfectly. And perfection is very costly and takes a long time. Over in the Commercial Crew Program, the senior leadership is making some progress in toning down the drive for perfection. It is a slow effort and uphill at all times. Over in the Exploration systems area, it all seems to be going the other way. Whatever anybody calls necessary for safety or improvement – without evaluating the real cost or schedule or other impact – seems to be adopted.
So I am guardedly optimistic about the commercial teams actually succeeding in flying humans in space in the next couple of years.
Not so much optimism for the exploration systems, drowning in ‘process’.
Most engineering problems have an optimum solution, a point where doing more actually results in less performance. This situation is not an exception.
Somebody needs to be able to just say no. In order to succeed.
It’s not really rocket science, it’s just good engineering.
Everybody in the policy world and the blogosphere wants to debate destinations – Moon, Mars, Asteroids – or the shape and size of the rocket. All interesting but not really relevant. Pick a place. Make a design. If it’s going to actually fly, do what is necessary – and not one thing more.
That’s what real rocket builders need to focus on.

Posted in Uncategorized | 25 Comments

Specific Plans

“A long term strategy and corresponding plans must also be developed . . . a set of notional milestones, launches, and hardware developments that are sufficiently defined so as to allow a cost estimate” – NASA Advisory Council finding April 2015

There is a story, perhaps apocryphal, that in the backseat of the car from NASA Headquarters to the White House to brief President Kennedy on the possibility of a moon landing, the legendary NASA Administrator James Webb decided to double the estimated cost of the program. Whether that part is true or not, the Webb estimate delivered that day in the spring of 1961 was significantly lower than the actual Apollo program.
Norman Augustine’s famous book of “Laws” concerning government acquisition states that all program cost estimates are subject to a correction factor of [1+ 0.52/(1+8t3)] where t is the percent of the procurement period completed. Or as he finishes the chapter with Law XXIV: “The most unsuccessful three years in the education of cost estimators appears to be fifth-grade arithmetic.”
During the so-called Space Exploration Initiative (SEI) days of the late 1980’s, the ’90-day-study’ came up with a very detailed plan to go to Mars . . . and the cost estimate made that plan dead on arrival at Congress. This lesson has not been lost on the NASA leadership.
A historical example may be in order. Look at the Apollo program hardware, specifically the Service Module and its rocket system the SPS (Service Propulsion System). That rocket engine is tremendously more powerful than the subsequent lunar landing flights needed. Why was such a large rocket engine installed on the Apollo SM? In 1961 when the first real plans for lunar landing were baselined, Direct Lunar Ascent was the designated mode. Some sort of huge lander would drop the entire CM/SM stack onto the lunar surface and the SPS had to be big enough to lift the astronauts, the Command Module, and the Service Module off the lunar surface and put them on a trajectory for the Earth.
To put that big stack – the CM/SM and the Landing Stage on a trajectory to the moon, the puny Saturn V was not big enough. Developing a much larger rocket was required – they called it Nova. Nova would have twice the number of F-1 engines as the Saturn V, tanks twice the diameter, much taller, more stages, etc., etc., etc. Exactly how the Nova rocket would be built was never figured out – it would be too big to fit under the ceiling of the factory at Michoud where the Saturn V first stage was made. The notions of how to transport that rocket to the launch pad were . . . notional.

Then along came some bright boys at Langley headed by John Houbolt who advocated an operationally more complex idea called Lunar Orbit Rendezvous – which only needed the Saturn V already under development.
The Nova rocket, the 100 foot tall Lunar Descent Stage, all went in the dust bin of history were never developed. But the contract for the SPS engine had already been let. Any real need to downsize that engine? No, but much less propellant would be carried in the tanks. If the Apollo CM/SM were somehow magically transported to the surface of the moon, the SPS had enough oomph to lift them off . . . but probably not enough gas onboard to get very far.
LOR was a good idea. Lots of folks are proposing ideas for future space travel. Some of them are actually pretty good. Locking a plan down means new, good ideas can’t complete.
History cries out with lessons. Some of them are subtle. Having detailed plans is generally good; believing in them too much is not. In the military they are fond of quoting the maxim: “No battle plan survives its first encounter with the enemy”. In space, the enemy is physics and chemistry . . . and finances. It may be that flexibility and leaving options open provides a better path for our long term ambitions in space. Who knows what may be invented in the next five years that could change the entire game plan?
Would we have made it to the moon if we tried to build the Nova rocket to do it? Maybe, maybe not.
The wrong plan can easily come with a forecast cost – a shock to the system – such that the program is never approved. Having a reasonable plan for the next step while keeping the goal in sight might actually be better. Waiting a little while doing some testing and development might be a good idea. Finding creative ways of controlling costs is mandatory.
Meanwhile, anybody seen Zephram Cockrane out there? Or at least the ghost of John Houbolt?

Posted in Uncategorized | 17 Comments

Pilot Error is Never Root Cause

Most accidents originate in actions committed by reasonable, rational individuals who were acting to achieve an assigned task in what they perceived to be a responsible and professional manner.
— Peter Harle, Director of Accident Prevention,Transportation Safety Board of Canada and former RCAF pilot, ‘Investigation of human factors: The link to accident prevention.’ In Johnston, N., McDonald, N., & Fuller, R. (Eds.), Aviation Psychology in Practice, 1994

Recent news stories have made me think about STS-28 landing. That flight is special to me because it was my first shuttle flight to sit in the big chair in the center of mission control. I was the Flight Director on the planning shift. New flight directors always start on the night shift when the crew is asleep. Get in less trouble that way. But your first time is always special and I won’t forget that flight.

STS-28 was a ‘classified’ flight that carried a national security payload. Someday, perhaps a long time from now, they will declassify it and let me know what exactly it was we were carrying. But for now, all I know is that they told me it was ‘important’. Important enough, in that post-Challenger era, to put a flight crew at risk. Because every shuttle flight is risky.

Brewster Shaw was the commander of STS-28, his first time in that role. Brewster is a remarkable pilot, one of the best, and went on to demonstrate significant skills as Program Manager for the Space Shuttle and later a leader in the Boeing Space and Defense organization. Not all astronauts make good managers but Brewster certainly did. But in those days Brewster was best known for his piloting ability.

Immediately prior to the flight of STS-28 a problem was uncovered with the way the flight software worked in connection with the small sensors on the landing gear. These so called ‘squat switches’ made contact as the landing gear was compressed and the software moded from flying to rolling on the wheels control. I’ve forgotten the particulars but there was a failure mode that if the switches made contact in a certain way that the computers would put the flight control system into the wrong mode – steering with the nose wheels when steering should be controlled by the rudder and elevens or something like that. Could lead to catastrophic loss of control.

It was too late to modify the software, and the switches were inaccessible with the shuttle attached to the external tank. A manual workaround by the pilot was required to ensure safety. So the Commander and Pilot got briefed – multiple times – in the last few days before flight about the need to land very softly – with a low ‘sink rate’ at touchdown – so the switches and software would work properly.

On the last night of the flight, I supervised the team as we prepared the entry messages for the crew. One of those was a reminder to land ‘softly’. The Entry flight control team came on and I went home hoping for a good landing. One of the first calls that the Capcom made – the crew was waking up as I was leaving the MCC – was a reminder to land softly.

So we set Brewster up.

Nominal deorbit burn, nominal entry, TAEM and HAC acquisition all normal, Commander took over flying manually as planned just as the orbiter decelerated to subsonic speeds. A perfect final glideslope. And now for the moment of truth, would the landing be soft enough to prevent the software glitch?

Normally an orbiter lands with a heavyweight payload in the bay at 205 kts – that is really fast for airplanes, but those stubby delta wings on the shuttle don’t create a lot of lift. With the payload bay empty – as it was for STS-28 – the lightweight landing speed is targeted at 195 kts. Under special circumstances, the pilots were allowed to land as slow as 185 kts. Brewster kept working and working to get the landing sink rate low and the speed kept dropping and dropping. At some point, as any fixed wing aircraft slows down, the wings will ‘stall’ and the aircraft will drop like a rock. Also, as the speed goes down, the pilot has to adjust the nose higher and higher – increasing the ‘angle of attack’ – to maintain lift. At some point with a high angle of attack at low altitude, the tail will scrape on the runway – always considered to be a catastrophic event for the shuttle.

The shuttle touched down at 154 kts. It is still the record for the slowest shuttle touchdown speed by a wide margin. It was less than 5 kts above stall speed. The tail avoided scraping by inches.

Oh, and by the way, the squat switches and software worked perfectly. No issues.

The post flight debriefings were all very positive and constructive – except for the entry and landing analysis. You can look back in my posts for the one called ‘Hockstein’s Law’ for a flavor.

I’ve never seen Brewster so embarrassed. In trying to avoid one hazard he nearly created another. In colorful pilot language (which I won’t repeat) he told us all that ‘on any given day the pilot can foul things up’. And it’s true. But I never blamed Brewster. We had set him up.

By concentrating on one issue to the exclusion of all others, and not reminding him of the training – probably years earlier – about very slow landing hazards – we, the flight control team, the program office, the NASA management – we set him up.

When doing an accident (or close call) investigation, I’ve been told to ask ‘why’ seven times before getting to root cause. The root cause, for example, can never be “the bolt broke”; a good accident investigator would ask “why did the bolt break”. Otherwise, the corrective action would not prevent the next problem. Simply putting another bolt in might lead to the same failure again. Finding out the bolt was not strong enough for the application and putting in a stronger bolt, that is the better solution – and so on.

The Russians had a spectacular failure of a Proton rocket a while back – check out the video on YouTube of a huge rocket lifting off and immediately flipping upside down to rush straight into the ground. The ‘root cause’ was announced that some poor technician had installed the guidance gyro upside down. Reportedly the tech was fired. I wonder if they still send people to the gulag over things like that. But that is not the root cause: better ask why did the tech install the gyro upside down? Were the blueprints wrong? Did the gyro box come from the manufacturer with the ‘this side up’ decal in the wrong spot? Then ask – why were the prints wrong, or why was the decal in the wrong place. If you want to fix the problem you have to dig deeper. And a real root cause is always a human, procedural, cultural, issue. Never ever hardware.

So it is with pilot error. Pilot error is never ever a root cause. Better to ask: was the training wrong? Were the controls wrong? Did the pilot get briefed on some other problem that cause distraction and made him/her fly the plane badly?

Corrective actions must go to root causes, not intermediate causes. Really fixing the problem requires more work than simply blaming the pilot.

Posted in Uncategorized | 24 Comments

Peeking Behind the Curtain

When I worked for the government, I never really understood what industry was doing; it was all behind a curtain. They gave only glimpses of what they wanted the government to see. Those of us in the civil services always had theories about what was going on in the corporate boardrooms or in the private research labs. But we really didn’t know and it was always the cause of puzzlement.
Nowadays, I’m retired from the government and work as a consultant, mostly to private industry. I find that the industry guys don’t have a lot of insight into how the government works, surprising to me. I always thought we had been fully open and transparent. Now I know better; the government and its decision making processes are pretty impenetrable from the outside. In fact, a lot of the leaders and workers out in the aerospace industry have established theories about how the government works internally, about what the government leadership wants, etc. I find most of these theories incredibly funny, terribly inaccurate, and I am astounded that otherwise knowledgeable people have some very odd ideas about what goes on behind the walls of government offices.
Thus my consulting work is very busy. Now that I’ve had a foot in both camps, I find I do a lot of theory correcting. Interpretation of what is motivating this party or that. Understanding what they want so needs can be efficiently met. Really keeps me busy.
Oh yeah, there is that technical work, too. Lots of that.
One of the reasons that I don’t get around to updating this blog as often as I used to is that my clients keep me busy. And my old government colleagues are always asking for my time, too.
A lot of what I do – make that almost all of what I do – is covered by ‘Non-Disclosure Agreements’. In other words, I can’t tell anybody about what anybody else is doing. There are a lot of times when I have to bite my tongue, but that is the nature of the job. Reminds me of the old days when I worked on ‘classified’ shuttle flights. In the name of national security I had to keep a lot of things from a lot of people. Kept me busy trying to remember who I could say what to and who I couldn’t. Interesting mental exercise to partition your memory and thoughts like that. Good training for my current work.
So, while I’d like to blog about what my clients are doing, well, you will just have to wait for them to tell you themselves; I’m not authorized
But what I can tell you is that it’s amazing. There are so many organizations working on so many aspects of space flight: new vehicles, new engines, new capabilities. Whew. I don’t know if they are all going to make it but I’m sure at least some of them will.
There is a renaissance coming in space travel. Some of it is from the government, yes, but a lot of it is not. Some of it is coming from garage shop inventors and some of it is coming from the biggest industrial corporations, and a lot of it is coming from folks in between.
Much of the really interesting advances won’t be the big jobs programs that the politicians like. If you are a politician and want to help the space program – you can send money, but better to open doors to private industry, remove barriers, reduce red tape.
Now that made me sound like I read the Wall Street Journal too much. Lest you think I’ve gone over to the ‘anything goes’ camp, I will quickly say that there is a very real place for the government to make sure that adequate safety precautions are followed. Not exactly like what is done for airliners, but something more fitted to this new, higher risk, higher energy field.
Anyway, I’ve got to say it’s been a great ride: all those years working on the forefront of the big government space programs, and now helping all the industry geniuses break through to the future.
Just stand by.
You will be amazed.

Posted in Uncategorized | 13 Comments

February first, again

No matter what is going on with the world, no matter what is happening in my life,  when the calendar turns to February 1 I have to stop, remember, and rededicate.

This year we have another gold star on the wall honoring those who have given the ultimate sacrifice in the conquest of space.  Michael Alsbury died last October trying to reach for the edge of space.  I trust that the NTSB will have several lessons for all of us to pay attention to when their report comes out.


It is a day to honor those brave souls.  I would particularly point out the last phrase on the Apollo 1 plaque at LC 34:  “Remember then not for how they died, but for those ideals for which they lived.”

It is not adequate to get emotional, and think about our losses; what is required is that we actually do something – make spaceflight safer, more reliable, and more common.  It has been too long to get maudlin.  It is time to get busy.

Over the next several blog posts I intend to visit the work we had to do to return the space shuttle to flight after Columbia.  It was much more than just technical.  Oh yes, much more than technical.

In the meantime, think on these words from the American author Jack London:

“I would rather be ashes than dust; I would rather that my spark should burn out in a brilliant blaze than it should be stifled by dry-rot; I would rather be a superb meteor, every atom of me in magnificent glow, than in a sleepy and permanent planet; the proper function of man is to live, not to exist.”

Posted in Uncategorized | 9 Comments

Space Thanksgiving

Today is the day in America that we set aside to count our blessings and give thanks for all the good that is in our lives.  So I will set aside my curmudgeonly ways and ignore the future and all the imperfections to concentrate on what is good and right.

Like most of you, but mindful that not everyone had these, I give thanks for faith, family, friends, food, freedom, and finances.  I wish that everyone could be as fortunate as I have been.  Good health, too, is a blessing that not everyone can share.

But today, as is the theme of this location, I would like to think about space.

I am thankful for the opportunity to live in such a time; with all its challenges and imperfections this is a unique time in history.  My generation, born before Sputnik, living yet, is the only generation that has gone from wondering to knowing.  Before, we all thought there were canals on Mars and it was full of intelligent life; now we know that it is a marvelous place but not like that.  Before, we thought Venus was a swampy, wet, and warm world; not we know it is not like that.  Before, we had no idea what the far side of the moon even looked like, now we know.  The mysteries of Saturn’s rings, the icy moons of Jupiter, and much much more, we could only guess about; now we know, at least in part.  This is the great age of solar system exploration.  We are fortunate to live during the excitement of these days. And there is more to come.

I am thankful to personally know and interact with the giants of this age of space exploration.  To work alongside many of the heroes of Apollo, to talk with Nobel laureates in cosmology and astronomy; to meet with the leaders of the organizations which carried out all these works, is truly amazing.  Better than meeting the Hollywood stars — and I’ve been fortunate to meet many of those, too.

I’m thankful that I have been able to contribute at least in a small way to the advancement of humanity into the cosmos.  To sit in the big chair in Mission Control, with all the power and responsibility, is an opportunity that few will have.  To be responsible for an organization which regularly launched humans into space to accomplish great tasks – repairing Hubble, building the ISS, and so much more; that has been a blessing, too.

To know those who put their lives on the line to fly on the fiery rockets and plunge into the unknown, that has been an awesome blessing and lesson in courage.

To have good work to do even in these days, helping new organizations build the next generation of rockets and spacecraft – safer and more efficient than before – that is a blessing.  To contribute to the nation’s policy discussions and shape a more hopeful future, that is a blessing.

Today I pray for blessings on all our spacefarers at work on the ISS, bring them home safely after long and productive work high above us.  I pray for blessings for those building new spacecraft and rockets, help them be diligent and creative so that we may all be successful.  I pray for the leaders of our nation and other nations who make space exploration a priority; grant them wisdom and vision to use resources wisely to make our lives better here on earth and in the future in the cosmos.

Finally, I am thankful for the dreamers who inspire us.  Ideas can be outlandish or practical but they challenge all of us to do more to bring the future into focus.

So I really have quite a lot to be thankful for today.  I hope you d,o too.




Posted in Uncategorized | 7 Comments

Unfamiliar Terms

During the summer of 2009, we were working on the history book about the Space Shuttle, ‘Wings in Orbit’. We had hired three summer interns, college students, to help with the book. Their primary assignments were to build the appendices, check references, make tables, and the like. Unusual for NASA interns, these three were not technical majors, but history, English, and social science majors. They did a great job for us.

One of the ‘other duties as assigned’ that we gave the interns was to read a chapter a week and report on it to the editorial board. All the writing was done by the engineers and scientist who worked on the shuttle program, and, sadly, engineers are not always known for excellence in written communications. We asked the interns to provide a critical look at the readability of each chapter and to be especially on the lookout for unfamiliar terms or acronyms that would make the text hard to follow for the general public.

One memorable week, one of the interns drew the assignment of reading the draft chapter on the historical setting of the space shuttle. I thought this might be unnecessary since Dennis Webb is a great writer and there was almost nothing technical in the chapter. At the end of the week, we convened the editorial board for a number of topics and had the intern’s reports at the end of the agenda. First up was a review of the chapter on the APUs and hydraulics. As you might expect, the intern pointed out several instances of very technical jargon and a number of undefined acronyms, all of which would have to be cleared up by rewrite. Then we covered a chapter on another technical subject with similar results. Finally it was time for the report on the history chapter. The young lady, probably a sophomore level college student, said that the chapter was very well written, easy to understand, and she had no recommendations for rewrite except for one term that she was unfamiliar with. ‘What was that?’ we inquired. She replied that the term she did not understand was:

‘Cold War’


I was thunderstruck. For someone of my generation, the idea that anyone would not know about the cold war is unthinkable. A short discussion ensued to make sure we had communicated correctly, but the bottom line was that she really hadn’t heard the term before and was unfamiliar with the concept.

More recently I had a conversation with a friend whose children are early high schoolers. He and his son were home alone one evening and decided to watch a movie together. On the schedule that evening was “The Hunt for Red October” based on the book by Tom Clancy. My friend reports that his teenage son didn’t quite get it. He kept asking why there was a big deal; after all the Russians are our friends, right?

Both of these young people were born after the fall of the Berlin Wall, after the dissolution of the Soviet Union. Those were ‘current events’ shortly before they were born. Given the lag in grade school history texts, those events were too recent to be covered.

When I was their age, nuclear annihilation stared us in the face. I can remember, probably when I was in 4th or 5th grade, when my parents came home from a Civil Defense meeting with plans of how to build a fallout shelter (we didn’t build one). In middle school the civics teacher showed us the AEC (Atomic Energy Commission – forerunner to the Department of Energy) films with the bomb tests in Nevada. You know the ones where they set up houses, mannequins with clothes, cars, household goods, etc., to see how a nearby bomb blast would affect those things. The message from those documentaries was that it was unsurvivable. I got to practice the ‘duck and cover’ method of hiding under our classroom desks if there was a bright flash in the sky. Again, the message came down that we were on the edge of annihilation and not likely to survive.

In high school history we studied the Cuban Missile Crisis and how close we came to the end in October of 1961. There were B-52 bombers on armed standby at air force bases near my home. I can remember as a high school student, plotting the likely fallout path from ‘targets’ near my home based on prevailing winds so that we would know which way to travel to survive following a nuclear exchange. That wouldn’t have helped either.


How that affected the psychology of two generations has been studied by sociologists.  It motivated people in so very many unusual ways.
And today’s kids don’t know what the term “cold war” means. I think that is a good thing. Not that they don’t need to understand history, but that those days are behind us. Hopefully for good, notwithstanding some burbles in the geopolitics these days.
There are a lot fewer nuclear weapons in the world these days – still too many – but the finger on the trigger seems to be a lot more relaxed. For our grandchildren’s sake I hope it stays that way.


We worry about a lot of things these days; there are serious problems all around us. But I think the level of worry is at a much lower intensity than it was 30 or 40 years ago.



And that is a good thing.

Posted in Uncategorized | Tagged , , , | 21 Comments