Remembrance is Not Enough

“Do good work” – Virgil Ivan “Gus” Grissom

This January 30 has been designated ‘Remembrance Day’ in honor of the astronauts who lost their lives in the Apollo 1, Challenger, and Columbia accidents.

It is altogether fitting that we remember their dedication, bravery, and sacrifice.
But it is not enough to partake of fuzzy emotional sentiments about loss; nor is should our remembrance day devolve into a contest of ‘I remember exactly where I was when I heard’ stories.

For those of us engaged in pushing back the space frontier it means we must work to learn something from those accidents; lessons which will keep us from adding a future catastrophe to the list for some remembrance days to come.

So I ask you: what did you learn?

How about this: Pure oxygen atmosphere in a space craft may be necessary but the entire process should be approached cautiously; ignition sources must be eliminated by design and workmanship and flammable materials must be minimized if not completely eliminated.

So there is that. What about other lessons?

Attempting to fly space systems outside their design and tested environment (e.g. cold temperatures) is not allowable. Systems should be qualified by test as well as analysis for temperature extremes and never operated outside those bounds.

Do you have another lesson?

Re-entry heat shield integrity should be established in orbit by direct inspection. Repair material and techniques should be available if damage is detection. In design, all efforts should be made to protect re-entry heat shields from debris impacts which could cause damage.

Does that cover the entire spectrum of lessons?

Not hardly. I can think of at least one more.

In each of the accidents there were people who believed that the programs were proceeding into unsafe territory. These people tried with varying degrees of success to alert the management of their concerns. In some cases, they fell silent quickly. In other cases, they were overruled and gave up. Later, in all three of the accidents, the top leaders unanimously said ‘we didn’t know anybody was concerned’.

The lesson to take away here is not to give up. If it is unsafe say so. If overruled, appeal. If denied appeal, make your case to the highest level manager you can find. Do not give up until you have been heard at the very top.

Because you might be the only one that sees what no one else can.

Don’t live with regret.

Posted in Uncategorized | 36 Comments

Messy Accidents

Messy Failure

“…the messy interior of engineering practice, which after the accident investigation looks like ‘an accident waiting to happen’ is nothing more than ‘normal technology.’ Normal technology…is unruly.”
– Dr. Diane Vaughn “Challenger Launch Decision” (Chapter 6: Engineering Culture)

I just read a very interesting article that you should also read:
“Launch failures: the Predictables” by Wayne Eleazer Monday, December 14, 2015

However, I think we need to examine the subject with a little more rigor.
It seems to me to be a little too glib to say that people make mistakes because they get in a hurry (launch pressure) and ignore information.


That seems awfully shallow in a complex real world. And misleading – because you can begin to think that you are smarter than those folks who got in a hurry and made that mistake. I would never make such a mistake.

That was certainly the mode that many of us at NASA were in after the Challenger accident. After all, an incredibly stupid middle manager let launch pressure force him into making a dumb decision that cost the lives of seven astronauts. Right?

Well, er, no. You need to read Dr. Vaughn’s book. It’s a little more complex than that.

If you settle for a simple, easy cause for an accident, you might just miss an important lesson that – if learned early enough – just might help you keep from making the same mistake later in life.

“Most accidents originate in actions committed by reasonable, rational individuals who were acting to achieve an assigned task in what they perceived to be a responsible and professional manner.
— Peter Harle, Director of Accident Prevention, Transportation Safety Board of Canada and former RCAF pilot, ‘Investigation of human factors: The link to accident prevention.’ In Johnston, N., McDonald, N., & Fuller, R. (Eds.), Aviation Psychology in Practice, 1994



After Challenger, almost all of us at NASA failed to learn the complex, rich, and ultimately effective lessons there because we accepted the glib, easy answer that all we had to do was avoid ‘launch fever’ and keep those middle managers from doing the same.


A necessary step for flight safety but hardly a sufficient method to avoid future mistakes.

Doing an accident investigation, I have been told, involves asking “why?” seven times. Such and such happened. Why? Because somebody did so and so. Why? Because they were trained wrong. Why? Because nobody thought that could happen. Why?

You get the picture. If you want to find a root cause for an accident you have to go deep. If you want to keep from having an accident, it follows, you must go equally deep.

The problem with most engineering projects – particularly complex, highly coupled, high performance, extreme environment engineering projects –is that there are too many issues to deal with. A great leader will organize the team to look at all the possible problems, issues and triage them into what needs the most attention. There is never enough resource (time, people, money) to get to the depths on all the issues that are out there. By their very nature, complex problems require priority setting and resource allocation.

“Absolute certainty can never be attained for many reasons, one of them being that even without limits on time and other resources, engineers can never be sure they have foreseen all possible contingencies, asked and answered every question, played out every scenario.” – Dr. Diane Vaughn

In the months leading up to the Columbia accident, the Space Shuttle program staff – engineers, safety specialists, managers, operators – worked through more than two dozen potentially fatal issues; some arising from inflight anomalies, some arising from new technical analysis or ground test. All major issues. Worked hard. Delayed flights to solve them. Put in place new equipment, new procedures, and new safety checks to ensure that the probability of success was maximized.

Nothing was ignored. Nothing. Nothing. Not even the foam.

But issues were evaluated, ranked, and resources applied as it was felt appropriate.

Nothing was ignored. But some things were mis-evaluated.

“Judgments are always made under conditions of imperfect knowledge” – Dr. Diane Vaughn



In High Reliability Organization Theory, one of the postulates is “a reluctance to simplify interpretations”. This makes sense. The deeper an understanding of a subject, the more likely it is that a proper judgement can be made.

One of the leading causes of the Columbia accident is clearly the simplification of one issue that led to a mis-categorization.

In a larger sense, having a simplistic understanding of how accidents occur in complex engineering systems will prevent learning and lead to a continuation of the accidents.

A really stupid organization is one that ignores critical issues. Those organizations are not in business very long. A smarter, but still accident prone organization, addresses critical issues but improperly. A truly smart organization addresses all issues with the best possible judgement applied. A successful organization is very smart and always worried that something has been missed – or improperly evaluated. “Preoccupied with Failure” is the term.


Or you can just remember to think “I’m not as smart as I think I am.”  Properly applied, that can work too.

Posted in Uncategorized | 9 Comments

The Ancient and Honorables

Friday was a busy day for me and I was off comm most of the day with meetings and whatnot. At the end of the day, I hustled over to the Saturn V barn at JSC to help with a tour of JSC co-ops. It was a good thing to do.

Then, at 5:30, the new class of Flight Directors sponsored a gathering of the Ancient and Honorable Society of Flight Directors at Space Center Houston. It was a wonderful event. Most of the society made it. Chris Kraft, Glynn Lunney, Chuck Lewis were the elder statesmen. Many true and almost true stories from the MCC were exchanged as the new Flight Directors were initiated into the team. As the evening wore on, we settled into a circle around the most venerable members. We heard many stories of the early days at the remote site locations around the world. Most of the stories are unrepeatable. Some of them were even about space flight.

At the end of the evening, when the SCH cleaning crew through us out about 2 hours past when we were scheduled to leave, the last story was from Dr. Kraft. How they found out that the LM would not be ready for checkout on Apollo 8. How the team came up with bold idea to orbit the moon instead. How Gilruth and Kraft sold the idea to a skeptical assembly of General Phillips, George Mueller, Wehner Von Braun. How the mission transformed the entire program, the entire race to the moon. A bold gamble that paid off.

Only after I left the gathering, and turned on the car radio, did I learn the awful news from Paris. Somehow that contrast with the final story brought to my mind the essay that the Poet Laureate of the United States wrote back in December of 1968. About Apollo 8. And how this story is profoundly appropriate:

Riders on Earth Together, Brothers in Eternal Cold
December 25, 1968
Men’s conception of themselves and of each other has always depended on their notion of the earth. When the earth was the World — all the world there was — and the stars were lights in Dante’s heaven, and the ground beneath men’s feet roofed Hell, they saw themselves as creatures at the center of the universe, the sole, particular concern of God — and from that high place they ruled and killed and conquered as they pleased.
And when, centuries later, the earth was no longer the World but a small, wet spinning planet in the solar system of a minor star off at the edge of an inconsiderable galaxy in the immeasurable distances of space — when Dante’s heaven had disappeared and there was no Hell (at least no Hell beneath the feet) — men began to see themselves not as God-directed actors at the center of a noble drama, but as helpless victims of a senseless farce where all the rest were helpless victims also and millions could be killed in world-wide wars or in blasted cities or in concentration camps without a thought or reason but the reason — if we call it one — of force.
Now, in the last few hours, the notion may have changed again. For the first time in all of time men have seen it not as continents or oceans from the little distance of a hundred miles or two or three, but seen it from the depth of space; seen it whole and round and beautiful and small as even Dante — that “first imagination of Christendom” — had never dreamed of seeing it; as the Twentieth Century philosophers of absurdity and despair were incapable of guessing that it might be seen. And seeing it so, one question came to the minds of those who looked at it. “Is it inhabited?” they said to each other and laughed — and then they did not laugh. What came to their minds a hundred thousand miles and more into space — “half way to the moon” they put it — what came to their minds was the life on that little, lonely, floating planet; that tiny raft in the enormous, empty night. “Is it inhabited?”
The medieval notion of the earth put man at the center of everything. The nuclear notion of the earth put him nowhere — beyond the range of reason even — lost in absurdity and war. This latest notion may have other consequences. Formed as it was in the minds of heroic voyagers who were also men, it may remake our image of mankind. No longer that preposterous figure at the center, no longer that degraded and degrading victim off at the margins of reality and blind with blood, man may at last become himself.
To see the earth as it truly is, small and blue and beautiful in that eternal silence where it floats, is to see ourselves as riders on the earth together, brothers on that bright loveliness in the eternal cold — brothers who know now they are truly brothers. apollo-8-earth-rise-8

Posted in Uncategorized | 13 Comments

Von Braun Symposium speech Oct. 29, 2015

Wayne Hale Speech at
Von Braun Symposium October 29, 2015
First Slide – Pluto Limb from New Horizons
In his 1950 book ‘Interplanetary Flight: An Introduction to Astronautics’, Arthur C. Clarke wrote:
“the choice, as H. G. Wells once said, is the Universe – or nothing. Though men and civilizations may yearn for rest, for the Elysian dream of the Lotus Eaters, that is a desire that merges imperceptibly into death. The challenge of the great spaces between the worlds is a stupendous one, but if we fail to meet it, the story of our race will be drawing to its close. Humanity will have turned its back on the still untrodden heights and will be descending again the long slope that stretches, across a thousand million years of time, down to the primeval sea.”
What a challenge – and progress is being made as seen in this picture from the robotic New Horizons spacecraft of the planet Pluto
second slide: Saturn
The Universe or nothing. That is what all of us here believe, I think. Space exploration is important for a number of reasons that are all familiar to the crowd here. We don’t need to sell anybody here on the idea do we? In fact this is one of those aerospacy kind of meetings where the true believers are who are in attendance. This is the epitome of us talking to ourselves – something we do too much of.
So let’s take advantage of this opportunity to talk among ourselves. We can hope the media will turn off their cameras and stop their recordings because what I have to say here today is amongst the family – a message for us, to us, about us.
third slide: O’Neill
Personally I am an O’Neillian – persuaded decades ago by Gerard K O’Neill about the place of humanity in the universe – which is everywhere.
So – inside the family now – we must stop the internecine, take no prisoners debate, and not allow it to start again. Moon, Mars, Asteroids – which should we do? My answer is yes, all of the above, and all the other places in the solar system too. We will do them all, it is just a matter of how fast and in what order. But if we keep squabbling amongst ourselves so as to confuse the rest of the world about goals and objectives, then we will not get anywhere.
Slide: Von Braun
Similarly the argument about New Big Rocket vs Smaller Existing Rockets. Develop what you can afford. Use what you’ve got when you’ve got it. Von Braun himself once said about developing the Saturn V that going to the moon without it would be like flying the Berlin airlift with piper cubs – no one can say it can’t be done but the logistics are overwhelming. (Those of you too young to know what the Berlin airlift was will have a remedial course later).
slide: piper cub
If we can build an affordable big rocket, then by all means do so; if not, then we need to start dividing the mission up into piper cub sized packages and get on with it.
That is really my message for this family, it is not a happy message: get on with it.
slide: Orion & SLS
Two years ago I spoke to this conference and told you that we need to fly soon and fly often. We need to figure out how that can be done. Today we are not two years closer to the first human flight of our exploration systems. If anything we are farther away. No wonder I’m frustrated. If we are honest with each other, then we will acknowledge we are all frustrated.
In our frustration it is natural to try to place blame somewhere. John Adams once famously remarked that ‘One disreputable man is a disgrace; Two disreputable men are a law-firm, and Three or more disreputable men are called a Congress.” Ho ho ho. We all like that.
slide: Congress
But listen to me: It is not the Congress’s fault we are where we are. If anything, they are accurately representing their constituent’s views. Do you want to blame the President? It is not the President’s fault. Do you want to blame OMB, OSTP, the Big Aerospace Corporations, the little New Space disruptors? It is easy to point the finger and blame somebody else. But I am here to tell you – all in the family now – that they are not to blame. If you want to know who to blame, look in the mirror.
slide: rockwell mirror picture
Young Padawan – Learn new skills you must. Master yourself you must, if successful you are to be.
Or as a general officer I once knew often said: I’m not interested in your little problems, I’m only interested in results.
Yes, the blame falls on all of us the true believers. Why is that? Because we expect too much from others.
One of my mother’s favorite stories was about an immigrant family; the oldest son approached his father one day – a man who came to America, worked hard all his life in menial jobs to make a better life for his family – and the son said to his father ‘I would like to go to College’ The father thought for a long time and finally said ‘You have my permission’
slide: Heart of the City cartoon about money

I guess some of you got the point of that story.
You see, we have permission to go to Mars and the Asteroids and the other places, even the Moon.
What is holding us up is that we are asking for the taxpayers to foot the bill – we are asking for public money. That is the problem. Once you ask for public money you have to play the political game — and if you want to play that game, you can’t complain about the rules.
slide: transcontinental railroad
Two years ago, I told you a historical tale about the building of the transcontinental railroad – and how that was a great technical and engineering feat – but the really impressive part was the creative financing and the organization that made it successful.
Two years ago, I told you that we need to learn from that example, how to be creative, innovative, flexible, and hardworking – not just technically but financially and organizationally as well. It appears that my message was not understood, so I am here to present it again in blunter terms.
slide: Mercury capsule/Intelsat IV rescue/Watney
Remember Apollo 13: you need to make a CO2 removal device out of ‘this stuff’. And they did it. Like our predecessors in the 1960’s or like cinematic hero Mark Watney we need to be innovative with what we have, finding opportunities to thrive and succeed in the face of overwhelming circumstances
If those of us in this family that believes in space exploration are to make the dream come true – to save humanity in the long run – we must work harder, must be more creative, must do whatever it takes to be successful. And fighting internally or blaming the world as it exists are not productive ways to proceed.
I have two recurring nightmares that wake me in the wee hours of the morning: first, that a piece of MMOD takes out the ISS and that ends human space flight for a generation. In the cold light of day we know that the probability of that happening is very low.
The second nightmare is more probable: that on January 20, 2017 a new administration will roll into office; they won’t know what to do about space because it is never a big election issue;
slide: funny old people
They wheel Norm Augustine out from the retirement home to lead another blue-ribbon commission study on what the nation should do in space. The commission says the exact same thing that every other study and blue-ribbon commission has said for the past 40 years – NASA should go back to the moon, to Mars, to the other places, but that unless there is a significant increase in the budget NASA will not be able to anything very interesting.
And so on February 1, 2018, the new administration, with lots of higher priorities, cuts the big rocket and the deep space capsule and we are left to try to figure out how to get to Mars with Piper cubs . . .
slide: Multiple Piper Cubs
What do you call it when you do the same thing over and over again and expect different results?
If we are to conquer the universe then we have got to change our tactics. We must steal the playbook from those disruptive forces who can get things done IN THE EXISTING ENVIRONMENT. This is not an easy lesson for a bunch of techno-geeks who would rather debate the mixture ratio selection for the main stage rocket engines and gets uncomfortable whenever the conversation turns to finances or organization.
In his fictional tale of the rich industrialist who bankrolled the first moon mission, Robert Heinlein created Delos D. Harriman. Intent on getting to the moon he put his complete business empire behind the effort. And when money ran short he never gave up finding new ways to earn or raise more. His business motto: “We also walk dogs”. That is the attitude we have to adopt. Be creative. Do what it takes. Never give up.
slide: dog walking
We have got to be able to figure out how to be safe and successful enough and go fly. Remember this: in exploration it is very important to keep safety always in our plans. But in exploration, safety is not the most important thing. In exploration, the most important thing is to actually go.
slide: safety workers
We need to quit doing non-value added paperwork. You can figure out what is worthwhile and what is a bureaucratic knee jerk to something that happened in the past.
So enough of a rant. Let’s get our act together, buckle down, and get this done.
The organizers said I can’t leave you without a historical lesson. . So now the media can turn back on he cameras again. (what? they hadn’t really turned them off?) Some of you may have heard this before, but here it is:
Then I proceeded to tell the symposium the history lesson of the Ming Navy vs. Portugal. I won’t repeat it here. If you want to read it try my old, archived NASA blog post:
But the point was in the first part of the speech. The rest is just . . . history.

Update:  the AAS has posted the video from the conference, you can watch it here:

My speech starts at about the 6 hr 55 min point

Posted in Uncategorized | 29 Comments

How to Avoid Train Wrecks

There has been a recent renewal of interest in a post I made five years ago
“The Coming Train Wreck for Commercial Human Spaceflight”
If you find that interesting perhaps you should read my post on Standards from just four years ago
Some people have called those posts prophetic. Not really. I don’t have all the answers. Just sad, tough experience from the inside out.
So how can we avoid a train wreck, not just in commercial human spaceflight but with the NASA exploration systems as well.
Spaceflight is exacting and very hard; the environments are tough, the energy levels are extreme, and the margin between success and failure is very slim. All of this is said so often that is sounds like a trite cliché. But no matter how trite it sounds, it is true. In the 21st century, still true.
We know a lot about launching satellites and people into low earth orbit – it has been done for decades. If you grew up with the Space Age, as I did, you got very accustomed to pictures and videos of rockets blowing up. Nowadays, rockets seldom blow up. I think the recent Atlas launch was the 100th success in a row. How did that occur?
Digging through the debris from failure after failure, engineers learned a little at a time about the complex and exacting process necessary to improve the chance of success in rocketry. No secrets anymore, nowadays those processes are well understood, well documented. And mostly followed.
To cite a couple of recent incidents; it is well know that a new heavy payload on the top of a new and spindly rocket (they are all spindly) it is imperative that the springy interaction be analyzed. If you don’t do that, well, you run the risk of failure like 59P last April.
Or if you try to lower the cost of your rocket by purchasing non-aerospace-standard parts, and don’t test each and every one of those parts, you run the risk of something breaking like it did on Spx-7 last June.
Or if you use really old rocket engines, you really should . . . well, enough recent history.
The point is, success in spaceflight is not easy and comes at a high price because the work is exacting and unforgiving. But we know how to do it well. Just follow the process.
But there is a rub, because the process, the standards, are not static. They are always being improved. Added to. Never reduced, unfortunately.
After Columbia, part of what we did was over the top. In order to satisfy all the critics, we did everything anybody asked for. Much of it was necessary, most of it was good, some of it was no value added bureaucracy to scratch some independent reviewer’s itch. All of it got codified as ‘lessons learned’ and added to the process.
The James Webb Space Telescope is a ‘must work’ project for the agency. There have been serious challenges, both technical and management. NASA has learned a lot about how NOT to manage a complex development project in the JWST experience. Much of that has been codified into new and ‘improved’ processes. Added processes. Always more, never less.
I have a cheap seat view of the Orion/SLS development. My basic observation: those efforts are drowning in ‘process’. The biggest threat to their success is not technical; it is schedule and cost. If the design and development processes drag the projects out too far, Congress or a new Administration will throw up their hands and call a halt to the whole thing. They did once before; my intuition is that they will again unless something significant happens.
The secret of a good program – as a very senior spacecraft designer once told me – is knowing how much is enough and then not doing anything more.
Right now, inside NASA, we have trained our workforce to do it perfectly. And perfection is very costly and takes a long time. Over in the Commercial Crew Program, the senior leadership is making some progress in toning down the drive for perfection. It is a slow effort and uphill at all times. Over in the Exploration systems area, it all seems to be going the other way. Whatever anybody calls necessary for safety or improvement – without evaluating the real cost or schedule or other impact – seems to be adopted.
So I am guardedly optimistic about the commercial teams actually succeeding in flying humans in space in the next couple of years.
Not so much optimism for the exploration systems, drowning in ‘process’.
Most engineering problems have an optimum solution, a point where doing more actually results in less performance. This situation is not an exception.
Somebody needs to be able to just say no. In order to succeed.
It’s not really rocket science, it’s just good engineering.
Everybody in the policy world and the blogosphere wants to debate destinations – Moon, Mars, Asteroids – or the shape and size of the rocket. All interesting but not really relevant. Pick a place. Make a design. If it’s going to actually fly, do what is necessary – and not one thing more.
That’s what real rocket builders need to focus on.

Posted in Uncategorized | 25 Comments

Specific Plans

“A long term strategy and corresponding plans must also be developed . . . a set of notional milestones, launches, and hardware developments that are sufficiently defined so as to allow a cost estimate” – NASA Advisory Council finding April 2015

There is a story, perhaps apocryphal, that in the backseat of the car from NASA Headquarters to the White House to brief President Kennedy on the possibility of a moon landing, the legendary NASA Administrator James Webb decided to double the estimated cost of the program. Whether that part is true or not, the Webb estimate delivered that day in the spring of 1961 was significantly lower than the actual Apollo program.
Norman Augustine’s famous book of “Laws” concerning government acquisition states that all program cost estimates are subject to a correction factor of [1+ 0.52/(1+8t3)] where t is the percent of the procurement period completed. Or as he finishes the chapter with Law XXIV: “The most unsuccessful three years in the education of cost estimators appears to be fifth-grade arithmetic.”
During the so-called Space Exploration Initiative (SEI) days of the late 1980’s, the ’90-day-study’ came up with a very detailed plan to go to Mars . . . and the cost estimate made that plan dead on arrival at Congress. This lesson has not been lost on the NASA leadership.
A historical example may be in order. Look at the Apollo program hardware, specifically the Service Module and its rocket system the SPS (Service Propulsion System). That rocket engine is tremendously more powerful than the subsequent lunar landing flights needed. Why was such a large rocket engine installed on the Apollo SM? In 1961 when the first real plans for lunar landing were baselined, Direct Lunar Ascent was the designated mode. Some sort of huge lander would drop the entire CM/SM stack onto the lunar surface and the SPS had to be big enough to lift the astronauts, the Command Module, and the Service Module off the lunar surface and put them on a trajectory for the Earth.
To put that big stack – the CM/SM and the Landing Stage on a trajectory to the moon, the puny Saturn V was not big enough. Developing a much larger rocket was required – they called it Nova. Nova would have twice the number of F-1 engines as the Saturn V, tanks twice the diameter, much taller, more stages, etc., etc., etc. Exactly how the Nova rocket would be built was never figured out – it would be too big to fit under the ceiling of the factory at Michoud where the Saturn V first stage was made. The notions of how to transport that rocket to the launch pad were . . . notional.

Then along came some bright boys at Langley headed by John Houbolt who advocated an operationally more complex idea called Lunar Orbit Rendezvous – which only needed the Saturn V already under development.
The Nova rocket, the 100 foot tall Lunar Descent Stage, all went in the dust bin of history were never developed. But the contract for the SPS engine had already been let. Any real need to downsize that engine? No, but much less propellant would be carried in the tanks. If the Apollo CM/SM were somehow magically transported to the surface of the moon, the SPS had enough oomph to lift them off . . . but probably not enough gas onboard to get very far.
LOR was a good idea. Lots of folks are proposing ideas for future space travel. Some of them are actually pretty good. Locking a plan down means new, good ideas can’t complete.
History cries out with lessons. Some of them are subtle. Having detailed plans is generally good; believing in them too much is not. In the military they are fond of quoting the maxim: “No battle plan survives its first encounter with the enemy”. In space, the enemy is physics and chemistry . . . and finances. It may be that flexibility and leaving options open provides a better path for our long term ambitions in space. Who knows what may be invented in the next five years that could change the entire game plan?
Would we have made it to the moon if we tried to build the Nova rocket to do it? Maybe, maybe not.
The wrong plan can easily come with a forecast cost – a shock to the system – such that the program is never approved. Having a reasonable plan for the next step while keeping the goal in sight might actually be better. Waiting a little while doing some testing and development might be a good idea. Finding creative ways of controlling costs is mandatory.
Meanwhile, anybody seen Zephram Cockrane out there? Or at least the ghost of John Houbolt?

Posted in Uncategorized | 17 Comments

Pilot Error is Never Root Cause

Most accidents originate in actions committed by reasonable, rational individuals who were acting to achieve an assigned task in what they perceived to be a responsible and professional manner.
— Peter Harle, Director of Accident Prevention,Transportation Safety Board of Canada and former RCAF pilot, ‘Investigation of human factors: The link to accident prevention.’ In Johnston, N., McDonald, N., & Fuller, R. (Eds.), Aviation Psychology in Practice, 1994

Recent news stories have made me think about STS-28 landing. That flight is special to me because it was my first shuttle flight to sit in the big chair in the center of mission control. I was the Flight Director on the planning shift. New flight directors always start on the night shift when the crew is asleep. Get in less trouble that way. But your first time is always special and I won’t forget that flight.

STS-28 was a ‘classified’ flight that carried a national security payload. Someday, perhaps a long time from now, they will declassify it and let me know what exactly it was we were carrying. But for now, all I know is that they told me it was ‘important’. Important enough, in that post-Challenger era, to put a flight crew at risk. Because every shuttle flight is risky.

Brewster Shaw was the commander of STS-28, his first time in that role. Brewster is a remarkable pilot, one of the best, and went on to demonstrate significant skills as Program Manager for the Space Shuttle and later a leader in the Boeing Space and Defense organization. Not all astronauts make good managers but Brewster certainly did. But in those days Brewster was best known for his piloting ability.

Immediately prior to the flight of STS-28 a problem was uncovered with the way the flight software worked in connection with the small sensors on the landing gear. These so called ‘squat switches’ made contact as the landing gear was compressed and the software moded from flying to rolling on the wheels control. I’ve forgotten the particulars but there was a failure mode that if the switches made contact in a certain way that the computers would put the flight control system into the wrong mode – steering with the nose wheels when steering should be controlled by the rudder and elevens or something like that. Could lead to catastrophic loss of control.

It was too late to modify the software, and the switches were inaccessible with the shuttle attached to the external tank. A manual workaround by the pilot was required to ensure safety. So the Commander and Pilot got briefed – multiple times – in the last few days before flight about the need to land very softly – with a low ‘sink rate’ at touchdown – so the switches and software would work properly.

On the last night of the flight, I supervised the team as we prepared the entry messages for the crew. One of those was a reminder to land ‘softly’. The Entry flight control team came on and I went home hoping for a good landing. One of the first calls that the Capcom made – the crew was waking up as I was leaving the MCC – was a reminder to land softly.

So we set Brewster up.

Nominal deorbit burn, nominal entry, TAEM and HAC acquisition all normal, Commander took over flying manually as planned just as the orbiter decelerated to subsonic speeds. A perfect final glideslope. And now for the moment of truth, would the landing be soft enough to prevent the software glitch?

Normally an orbiter lands with a heavyweight payload in the bay at 205 kts – that is really fast for airplanes, but those stubby delta wings on the shuttle don’t create a lot of lift. With the payload bay empty – as it was for STS-28 – the lightweight landing speed is targeted at 195 kts. Under special circumstances, the pilots were allowed to land as slow as 185 kts. Brewster kept working and working to get the landing sink rate low and the speed kept dropping and dropping. At some point, as any fixed wing aircraft slows down, the wings will ‘stall’ and the aircraft will drop like a rock. Also, as the speed goes down, the pilot has to adjust the nose higher and higher – increasing the ‘angle of attack’ – to maintain lift. At some point with a high angle of attack at low altitude, the tail will scrape on the runway – always considered to be a catastrophic event for the shuttle.

The shuttle touched down at 154 kts. It is still the record for the slowest shuttle touchdown speed by a wide margin. It was less than 5 kts above stall speed. The tail avoided scraping by inches.

Oh, and by the way, the squat switches and software worked perfectly. No issues.

The post flight debriefings were all very positive and constructive – except for the entry and landing analysis. You can look back in my posts for the one called ‘Hockstein’s Law’ for a flavor.

I’ve never seen Brewster so embarrassed. In trying to avoid one hazard he nearly created another. In colorful pilot language (which I won’t repeat) he told us all that ‘on any given day the pilot can foul things up’. And it’s true. But I never blamed Brewster. We had set him up.

By concentrating on one issue to the exclusion of all others, and not reminding him of the training – probably years earlier – about very slow landing hazards – we, the flight control team, the program office, the NASA management – we set him up.

When doing an accident (or close call) investigation, I’ve been told to ask ‘why’ seven times before getting to root cause. The root cause, for example, can never be “the bolt broke”; a good accident investigator would ask “why did the bolt break”. Otherwise, the corrective action would not prevent the next problem. Simply putting another bolt in might lead to the same failure again. Finding out the bolt was not strong enough for the application and putting in a stronger bolt, that is the better solution – and so on.

The Russians had a spectacular failure of a Proton rocket a while back – check out the video on YouTube of a huge rocket lifting off and immediately flipping upside down to rush straight into the ground. The ‘root cause’ was announced that some poor technician had installed the guidance gyro upside down. Reportedly the tech was fired. I wonder if they still send people to the gulag over things like that. But that is not the root cause: better ask why did the tech install the gyro upside down? Were the blueprints wrong? Did the gyro box come from the manufacturer with the ‘this side up’ decal in the wrong spot? Then ask – why were the prints wrong, or why was the decal in the wrong place. If you want to fix the problem you have to dig deeper. And a real root cause is always a human, procedural, cultural, issue. Never ever hardware.

So it is with pilot error. Pilot error is never ever a root cause. Better to ask: was the training wrong? Were the controls wrong? Did the pilot get briefed on some other problem that cause distraction and made him/her fly the plane badly?

Corrective actions must go to root causes, not intermediate causes. Really fixing the problem requires more work than simply blaming the pilot.

Posted in Uncategorized | 24 Comments