Finding Columbia

IMG_0877[1] IMG_1087[1]A couple of days ago, I had the rare opportunity to visit the Columbia Debris Repository on the 16th floor of the VAB at Kennedy Space Center.  It is a solemn experience to walk among the remaining parts of the space shuttle which bear witness to the tremendous forces that broke the vehicle apart and ended the lives of seven brave astronauts. 

This experience is not available to the general public.  But you can find places to see parts of Columbia.  The data recorder, found miraculously intact in a field in east Texas, is on display in public areas at the Johnson Space Center in Houston, and the frame for the side hatch window is in the entrance display case of the Kennedy Space Center headquarters building in Florida.  There is another part reportedly on display at the Marshall Space Flight Center in Huntsville, Alabama, although I have never been able to find it on my many visits there.

These artifacts have great engineering value showing how future spacecraft should be built to withstand the extreme conditions of space flight. 

But there is a backstory that is more important. There is a story that must be told of how poor decisions were made, mistakes, and hubris.  This more important story is not found in the artifacts, although they emphasize the consequences. It is for us who lived through those days to share that story.

It is not enough to remember the sacrifice of our brave friends, although we should do that.  It is not enough to study the wreckage for clues of how to build better aerospace systems although we should do that.  It is mandatory that we remember how those sacrifices and that wreckage came to be.  And to prevent it from happening in the future.

Recently, some have said that loss of human life in spaceflight is to be accepted as a part of the cost of exploring the universe.  That may be, but those of us in the business must always believe that going forward we have to all we can to prevent accidents.

Because forward into the universe we must go.

Posted in Uncategorized | 18 Comments

Keeping Eileen on the Ground: Part II – or – How I Got Launch Fever

The STS-93 launch attempts proved the adage we knew well in the Flight Director office: When the weather is good, the vehicle will break; when the vehicle works perfectly the weather will be bad. On the first launch attempt – scrubbed due to hydrogen sensed in the aft compartment – the weather was good. The Space Flight Meteorology Group post mission report says all sites were GO – launch, RTLS, TAL, AOA, etc. All GO. Hard to believe that there were no issues anywhere. And of course, something broke.

Two days later, we had another great weather forecast, no issues. The SMG post mission report for this day says “Thunderstorms occur at KSC about 3.2% of the time for a 0400 to 0500 UTC launch window.” Guess Murphy doesn’t believe in statistics.

Deploying the Chandra X-ray telescope on its Inertial Upper Stage required a complicated launch window. The plan was to deploy the payload early in the flight. But you can’t just kick it out just anywhere. The IUS had a complicated navigation and guidance system and the telescope needed to get its solar arrays going pretty quickly – a long period of time in the earth’s shadow wouldn’t be good. And there probably were other constraints that I’m not remembering. Given the orbital constraints on the deployment window, a launch window could be calculated. But that is not the whole story; we should plan a backup deploy opportunity just in case we ran into trouble. And an alternate deploy window in case we wound up in a lower than expected orbit. And another backup deploy window in case we had to wait until the second day of the flight to deploy. Etc., Etc., Etc. Each of these deploy windows worked back into a constraint on the launch window. Covering for all the backups and contingencies meant that the launch window would be 47 minutes long.

As the countdown progressed, rain showers which built into thunderstorms popped up just off shore – not over the pad where they would be real trouble, but offshore to the east. And the prevailing winds were pushing the storms to the east, away from the launch pad.

After Apollo 12, we knew that thunderstorms meant lightening and lightening was bad for our space ship. No launching if the edge of a thunderstorm was within 20 miles of the launch pad. And one of the storms was right there. On the edge.

Sitting behind John Shannon, on the MOD console, Bryan Austin and I were not happy. The Launch Director was not happy. The entire team, having scrubbed once, was not happy.

The Ops Manager, who chairs the MMT down in Florida in the firing room bubble, called me on the loop and asked about the constraints on the launch window. Could we extend it? Yes, by golly, we could, I decided. Bryan and I got on the loop with the IUS team out in Sunnyvale and with the Telescope POC. If we gave up the third backup deploy opportunity, that would add ten minutes to the launch window. Everybody agreed that it was unlikely that we would need that backup deploy opportunity. We all wanted to go. We were all frustrated that the launch had scrubbed earlier. This flight had been delayed several times in the planning cycle before we got to the pad; it was time for Chandra to fly!

The SMG forecaster put his electronic cursor on the satellite image of the edge of the storm, projected the storm movement based on wind speed, and opined that we would be good to go at the end of the window. So we waited.
Time passes very slowly.

Then the storm started building up, and expanded, and all of a sudden, and now that ten minutes would not be enough.

We were back on the loops again. Giving up another backup deploy opportunity would gain another 7 minutes or so. Were we all OK with that? Sure, let’s light this candle. GO.

I could tell that the Ascent Flight Director was not really comfortable with this broken field running. When John queried his FDO about the launch window close time, the FDO said he didn’t know when the window would close based on all the back room deals that the MOD was making.

That should have been a red flag. Instead, it made me mad. Hadn’t FDO been listening?

Just about that time, the thunderstorm on the edge of the launch area exploded. The trailing edge of the storm rapidly grew farther and farther to west. Against the wind. Over the launch pad. Time was up, no matter how you calculated it. We were done.

STS 93 weather

The crew rode the elevator back down. Launch was reset for 24 hours later.

We had an interesting debrief out by the coffee pot in the hall. That is the place all the really significant conversations in Mission Control happened. Not on the loop. In the hall at the coffee pot.

It became suddenly clear what a risk I had been running by negotiating real time changes to the launch window. We could have launched into a miscalculated window that left us no options but to return with the telescope in the payload bay. Of if we needed one of those backup deploy windows we would have been out of luck. What had I been thinking?

So when folks point to the damage that has been done on other launches when officials got launch fever and let go of good judgment, I think about the second scrub for STS-93. And how sometimes an errant thunderstorm can save you.

Next time: what happened when we actually launched.

Posted in Uncategorized | 17 Comments

STS-93: Keeping Eileen on the Ground, Part 1

The summer of 1990 is remembered in shuttle circles as the hydrogen leak summer. We tried to launch Columbia and found that there was a leak in some of the plumbing carrying that volatile gas. The launch team scrambled into a series of hydrogen loading tests with various sensors to find the source. During the trouble shooting on OV-102, we tried to launch Discover and found leaks in OV-103 too. All summer we sat on the ground running test after test. There is a really good summary on the NASA lessons learned page: This was the set up for our story about STS-93, the most interesting shuttle launch of them all.

Hydrogen is a powerful fuel for rockets, but it is hard to work with. The molecules are very small and can slip through the tiniest of openings; and exposed to any oxygen (in the air), well, everybody has seen the Hindenburg movie. Not good.

The aft compartment is the engine room of the space shuttle. It is surprisingly large and open on the inside – the volume is bigger than your bedroom but oddly shaped. And it is stuffed full of pipes of various sizes running every which way; large ones and small ones, some carrying gases and some carrying cryogenically cold fluids, hydrogen and oxygen lines, ammonia and Freon lines, hydrazine in two varieties and nitrogen tetroxide – a powerful oxidizer so deadly that it was said if you whiffed its fishy vapor in the air you were already dead. During main propellant loading and right up to engine start, dry nitrogen gas is pumped into the compartment to keep the possibility of fire as low as possible.

And to detect if we had a problem? We went through an evolution in sensor technology, but in 1999, inside the shuttle aft compartment was one (1) inlet to a hose to detect a hazardous gas leak. The detection equipment was actually located well away from the shuttle itself, buried deep in the mobile launch platform. A variety of a mass spectrometer searched for various gases and forwarded concentration information to a console operator in the firing room, three miles away. A sample of gas “collected” in the top of the aft compartment wended its way down through the tubing and plumbing maybe a hundred feet before hitting the analyzer. And the analyzer needed frequent calibration to ensure that it was operating correctly. This system could tell us, within a reasonable expectation, that we had a leak and what gas it was. But the detection system was practically useless to tell us where the source of the leak could be found. This system was slow, unwieldy, and the best we had at the time.

The launch commit criteria was carefully written to tell the operators exactly what concentrations at what times were allowable and which were not – for example, sometimes when External Tank loading was started the seals weren’t thermally stabilized and leaked a little until they chilled down. Later on, that same concentration of hydrogen free in the aft compartment wouldn’t be acceptable.

And at certain times, parts of the shuttle system were “committed” to launch – the firing room personnel were not longer required to monitor nor call a hold. But there was the big exception statement in the front of the LCC that required “senior console operators” to use their “best judgment” to call a hold to stop a hazardous situation, no matter how late in the countdown.

So it was a real surprise that during the first countdown attempt for STS-93, the senior haz gas operator called a cutoff – the technical term for a launch scrub that late – at T-8 seconds for high hydrogen gas concentration. There is a really good youtube video here: – which has an erroneous title and writeup, but the video is good.

At T-10 seconds, the ground computers send the very last command the onboard system needs to fly: Go for Main Engine Start. With no other electronic word from the firing room computers, the shuttle launches itself. Right after that T-10 seconds command, the big sparklers at the base of the MLP towers fire off to ignite any small patches of free hydrogen that might have escaped from the main engines. At T-6.6 seconds, the orbiter computers command the main engines to start, staggered by about 120 milliseconds – center first, then left, and finally right. Shutting down the count at T-8 seconds is dicey. If the mains start there is a lot of free hydrogen left around and plenty of heat to ignite it. The ROFI sparklers were firing – those take 48 hours to replace – and if even the first engine cracks open one valve in the start sequence, it is a three week turnaround to remove and replace any SSME that started. So it was a surprise to get the call, especially after haz gas was “committed” for launch. It was one of those senior console operator things.

It is not technically a pad abort, no fire or smoke occurred – other than the sparklers – but the adrenaline is very high all around the program when it gets that close.

Safing was complete, the crew unstrapped and rode the elevator down to wait for the next attempt; 48 hours later at the earliest.

The KSC management was ready to pin a medal on the guy that called the hold. But the first story we got in Houston was that the sensor had just gone through an automatic calibration and the high concentration reading was a ghost.

I never heard the end of that story. Did Ozzie get a medal? Did we unnecessarily scrub a launch? Somebody needs to fill me in. All I remember is it set me up for a bad case of launch fever two days later.

That story next time.

Posted in Uncategorized | 15 Comments

Von Braun Symposium

Several people have requested the text or a link to the video of my keynote address on Tuesday. Thanks to the AAS for providing the link. Get your popcorn and take your seats and when you are through listening please let me know what you think about it

Or if the first link doesn’t work try getting the video through this

Back to shuttle launch stories in a few days!

Posted in Uncategorized | Tagged | 3 Comments

STS-93 and the Flight Director Office

FD Group 1999, Flight Director Party at Linda's house, 1999

One of the most interesting shuttle flights was STS-93 which put the Chandra Advanced X-Ray Telescope into space. Not only was it the first shuttle flight to be commanded by a woman – Eileen Collins – and the heaviest payload to that time – 59,000 lbs – but we had the most eventful launch scenario of all the shuttle missions – clearly a close call which turned out well.

But before I start, some background is in order. Especially about the Flight Director office as it existed in 1999.

Being a Space Shuttle Flight Director was the best job I ever held; it was the toughest, the scariest, and the most rewarding. I never wanted to leave the office but was propelled out by events beyond my control – something I talked about in my rambles about the Columbia accident earlier this year.

The FD office held the best people in the world and at the same time many of whom caused me to contemplate homicide. Being a Flight Director was not for the faint of heart; we held an awesome responsibility to the crew, to the program, and to the nation. We learned early not only not to suffer fools gladly but not to suffer fools at all. Every Flight Director had a highly tuned BS detector and would swiftly attend to anyone who was not prepared, organized, and thorough. Every Flight Director thought that he (or she) was the best in the office and the other guys and gals needed help.
Monday morning staff meeting in the Flight Director office resembled a school of sharks swirling about looking to devour the weakest member in the room. It could be intense. Most of the office chiefs encouraged this behavior. It made you really think through any position on a controversial subject or approach. As we went around the table, each FD had to describe the status of an upcoming shuttle flight for which he had been assigned responsibility. Nobody got through their discussion unchallenged. It was brutal but many mistakes were corrected early because shortcomings in a plan were identified on Monday morning.

As a flight approached and integrated training schedules started appearing, the team of Flight Directors would be assigned; the Lead Flight Director generally started at least a year in advance but the rest of the team could be lined out as little as six weeks before a flight. The Lead got the primary shift where the biggest activities occurred during the crew work day; this was generally called the Orbit 1 shift and covered the period from just after crew breakfast through the middle of the crew ‘afternoon’ – which could be any time on the clock in Houston. Another experienced flight director would be assigned to the Orbit 2 shift which wound up the crew day and got them in bed. If a rookie flight director was assigned, he would draw the Planning shift when the crew was asleep and plans for the next day were modified as required and uplinked for the crew to read when they awoke. And of course there would be the Ascent/Entry Flight Director (sometimes split for two) whose job was to get the mission off the ground and back down again – and who paid little attention to the ‘cg management device’ found in the payload bay.

Additional flight directors could be assigned depending on the type of flight; for ISS assembly or logistics flights a single flight director was sent to the Russian control center (TsUP or MCC-M) to help our partners understand what the crazy Americans were doing. For a flight with the big Boeing upper stage – the IUS – which was the case for STS-93, a Flight Director was assigned to travel to the USAF/Boeing control center at Sunnyvale, California, to keep those guys in line with what Houston wanted to do. And finally there was the Mission Ops Director. For all the early Shuttle flights there was only one MOD – the legendary Gene Kranz. The function of the MOD was to keep the program leadership, the headquarters guys, and any other management lookie-loos who might be present out of the way of the flight control team. After Gene’s retirement, the position of MOD was circulated between senior managers (always former flight directors) in the Mission Operations organization and some of the senior flight directors. For STS-93 the MOD of record was Randy Stone, former Flight Director, Director of Mission Operations, and an expert on the IUS. But Randy was never an Ascent Flight Director, so as senior A/E FD in the office, I was tapped out to be the MOD just for the launch and landing shifts. John Shannon was the Ascent/Entry Flight Director, Brian Austin was the Lead (Orbit 1) Flight Director.

So midway through the countdown of STS-93, I would take my seat in Mission Control, right behind John Shannon, and right next to Brian Austin – who could not stay away. Everything that happened later, I blame on being sandwiched between two Texas Aggies . . .

Well, maybe not.

To be continued

Posted in Uncategorized | 12 Comments

Monuments and Birthdays

Back Camera

Just by happenstance I share a birthday, separated by century and a half – with the great American Naval hero David Farragut. On visit to Washington a while back, I snapped this picture of the statue of America’s first Admiral in the park bearing his name.

Really unfortunate about the pigeons, isn’t it?

Admiral Farragut took New Orleans from the Confederacy, led his fleet through the mines and torpedoes of Mobile bay, helped Grant take Vicksburg, and transformed the Union Navy in his spare time. No wonder there is a statue and a park honoring him. A century and a half later, few people can remember why we honor him.

Recently, I have been working with the first program manager of the Space Shuttle, Bob Thompson. Mr. Thompson lead the development of the space transportation system from the end of the Phase A studies in 1972 through the first flight in 1981. An incredibly important time in the history of space travel and he was at the center of the whirlwind ensuring that the thing actually worked. Mr. Thompson is concerned that certain aspects of the early history of the space shuttle have not been correctly recorded by the professional historians who have written the books about those times. This is not surprising; professional historians rely on sources, professional historians were not present at the events. There are some significant elements about which there is disagreement. I tend to lean with Mr. Thompson’s account; after all, he was there.

Unfortunately, talking with historians and other folks who were also present, there is a tendency to discount Mr. Thompson’s recollections. “After all” they like to say “he was just the program manager at a field center and he did not have a good understanding of what was happening [in Washington, in the Administrator’s office, at the White House]. The program manager just isn’t that important.”
Ouch. And I thought being Program Manager was pretty important . . . and once upon a time, I held that job. That loud hissing noise was my ego deflating. ‘Just not that important’.

My last birthday wasn’t one of the big ones ending in a zero, but the calendar is pointing that way. Probably a function of my age, I frequently get funeral notices for people that I know. To quote Yogi Berra: “If you don’t go to their funeral, they won’t go to yours.” Think about it.

Hearing eulogies which categorize the importance of someone’s life makes you think about what is important in your own life. What great victories were accomplished, what projects were completed, what you did that improved the lives of those around you. These thoughts tend to a realization of humility and an impetus to do more with the time ahead.

Recently I have been leading a frenetic schedule both professionally and personally. It has simply got to slow down. All of this scurrying around certainly doesn’t seem to have been very productive in terms of making a difference where it matters. Perhaps I’ve been planting seeds which will grow to fruition at a later date, but right now the results of all my busy-ness seem pretty slim.

So I’m making a new start this fall – promising to slow down and spend my time where it is most important; eliminating the busy work as much as I can. I promise to put new emphasis on blogging to share my perspectives and some of my personal history lessons. And I also promise to spend much more family time – in the final analysis, that is where the most important work lies.

That’s much better than a statue with a pigeon on your head.

Posted in Uncategorized | 10 Comments

For the Record

Here is my written testimony for the US Senate Commerce Science subcommittee from a couple of weeks ago. My oral statement was a shorter version of this.

Testimony of N. Wayne Hale, Jr. before the United States Senate Subcommittee on Science and Space of the Committee on Commerce, Science, and Transportation, May 16, 2013

I thank the committee for inviting me to testify concerning the growth of the space industry including the private sector space transportation.
In the interest of full disclosure, I am hardly a disinterested party in this topic. I am and have always been a passionate believer that space exploration and the industries that may derive from it will benefit humanity in ways beyond our imagining. I have spent most of my professional life working in the large government space programs of the Space Shuttle and the International Space Station. During those years I have seen NASA at its very best and at its worst. The hard working dedication of my colleagues at NASA personnel is nothing short of phenomenal, and their talent and creativity is second to none. However, their endeavors have frequently been stymied due to the inherent bureaucratic inefficiencies of government work and the frequent shifts in priorities and funding that whipsaw most space initiatives. This has led me to believe there must be a better way to develop and operate space systems.
In my last assignment before retirement from government service, I worked with Frank Bauer, the Chief Engineer of the Exploration Systems Directorate, to define the management philosophy, protocols, and processes for the then new Commercial Crew Program within NASA. After my retirement, my work has continued as a consultant. My company, Special Aerospace Services, and I are paid advisors to a number of entities involved in the commercial crew and commercial space cargo enterprises. And I have volunteered my time to work with the Commercial Spaceflight Federation to establish safety, management, and engineering standards for all the members of this fledgling industry. So the committee can see that I am hardly a disinterested party and should weigh my testimony as such.
Establishing good, effective safety, engineering, and management standards in a voluntary industry association is the hallmark of any reputable and mature industry. I am pleased to report that the CSF is making good progress in setting up voluntary processes which will ensure public safety and promote general success in this difficult business. Industry group standards can alleviate the need for government regulations by allowing the members of a trade association to tailor best practices specifically for their industry. Evolution of these industry standards inevitably proceeds more rapidly than the development of government regulations and can therefore take rapid advantage of best practices as they emerge.
The most singularly vexing problem with space flight is the high cost of getting to low earth orbit. As the noted science fiction writer Robert Heinlein once observed, ‘when you are in earth orbit you are half way to anywhere in the universe’ which accurately reflects the physics of the situation.
The lack of low cost transportation to that point located just above the earth’s atmosphere and moving at 17,500 mph forward velocity has prevented potential space entrepreneurs more than any other factor. Hundreds of potential business opportunities in the limitless resources of the solar system have floundered on the high cost of transportation to low earth orbit. Asteroid mining, energy production, zero gravity manufacturing are all within our grasp technologically but will not be profitable until reliable and reasonably affordable transportation systems are in place.
New systems for transportation to low earth orbit have enormously high development costs. Private investors, with a few exceptions, are loath to provide the capital needed to develop low earth orbit transportation without clear and immediate business ready to purchase tickets.
So we are in a ‘chicken or the egg’ paradox. Space business needs low cost transportation to become profitable, while potential private transportation services need established business to justify the cost of construction. This is not the first time that America has been in this situation. Both the early railroads and fledgling air transportation industries found themselves becalmed in similar straits. In both these cases, and others, the federal taxpayers stepped in to provide critical resources to help new industries develop. Those investments have been paid back myriad-fold in tax revenues when the new industries caught fire and provided transportation systems that were the envy of the world.
NASA and its predecessor agency the NACA provided needed aeronautical research to make air transportation as inexpensive and safe as we find it today. The federal investment in aeronautics development has paid off handsomely in the development of a multi-billion dollar industry. Indeed, one of the largest sectors of net exports in the American economy is aerospace with billion dollar sales a common occurrence.
The history of space flight – after the first early steps to demonstrate that space flight was even possible – has been marked with the goal of decreasing the cost of transportation to low earth orbit. In my home I have an entire shelf of books populated by volumes of studies and proposals from a multitude of thinkers spread over decades on that subject: how to provide reliable safe space transportation on the cheap.
The space system that consumed much of my professional career, the space shuttle, was established to achieve just such a low cost goal. But the technologies of the 1970s, harnessed to a risk adverse government apparatus resulted in a system that was only slightly less expensive than those which went before.
In the last decade, the United States embarked on a bold new experiment to turn over the creative reins of spacecraft development to entrepreneurial, nimble, flexible, creative private commercial teams. Bolstered with a modicum of taxpayer resources, these businesses have leveraged private investment to create the critical mass to develop new, much cheaper transportation systems. We see the first fruits of success today with cargo carrying craft: the SpaceX Falcon and Dragon, and the Orbital Antares and Cygnus. These cargo carrying privately developed vehicles are starting to supply our government outpost, the International Space Station. In future years others, the Boeing CST-100 and the Sierra Nevada Dream Chaser will be added to the fleet to carry human beings as well as cargo.
Poised on the cusp of these new systems, we run the risk of being penny wise and pound foolish as we make the same mistake that doomed the space shuttle to much higher cost operations: starving a spacecraft development program in the name of saving a few pennies for today’s budget bottom line resulting in the compromised systems that, if they fly at all, will not be cheap enough to enable business in space.
This is not to devalue the development of truly deep space exploration systems by the government. Those high risk, high cost systems payback over such are long term that they would never be funded by private investment. But, like the expenses incurred by Lewis and Clark, Captain Zebulon Pike, and a host of other government expeditions in our history, the payback from exploration will be enormous for both the country and for all of humanity. Just at a more distant point in the future than business spreadsheets normally run. The SLS and the MPCV should be developed in conjunction with the commercial low earth orbit transportation systems. Flying to cis-lunar space to inspect a captured asteroid is an engineering and operations test worthy of a first deep space mission. But that mission can only be a first step. More should follow.
The commercial systems will enable the deep space exploration initiative in substantial ways. First of all because the ISS is our space test laboratory for the technologies and systems that deep space exploration will need. Operation in space, aboard the ISS, is the most effective means to wring out life support, communications, propulsion, and other technologies. Commercial transportation of cargo and crews to the ISS directly support deep space systems development. As deep space exploration proceeds, commercial cargo and crew vehicles will likely be called upon to aid with assembly and fuel delivery to low earth orbit where we will finalize preparations to head into the vasty deep. Cost effective commercial transportation to low earth orbit can make a vital difference in equipping the deep space fleet.
So the two efforts go hand in hand. Funding equity between the two programs is necessary to ensure the timely success of both. Currently, the commercial space effort stands uncomfortably close to the brink of financial starvation. Deep space transportation development is being stretched out by similar restrictions. Business is looking to see if the government is serious about providing the critical support or whether this effort will be wasted as so many earlier government programs which withered away on the very cusp of success: National Launch System, Orbital Space Plane, and others.
I urge the Congress to fully fund these vital activities, both the commercial crew program and the exploration systems. They will allow America and American industry to lead in the exploration and development of human activity in our solar system. When the historians of the future look back on our era, they will recognize the movement of humanity from planet earth into the solar system as the pivotal event of our times. There is no project that is so important for the long term success of humankind. I would hope that those historians record that at this crossroad of history that a creative, enterprising, farsighted nation called America led the way.
The prizes both economic and historic are too great to bypass. If America does not lead in these enterprises, somebody else will. And the leader will reap the greatest rewards both in the near term and in the longer term.
For all our limitations, America is a very rich country. There are many things which America needs to do for the present moment: provide for a strong military to protect us in a dangerous world, educate our children, care for our elderly and infirm, revitalize our transportation infrastructure of roads, bridges, airports, and more. All of these activities are of vital importance today. Space exploration is about the future. Space exploration is possibly the only line item in the federal budget that is all about the future. Currently we spend one half of one percent of our nation’s treasure on the future. Isn’t the future worth that investment? Or more!

Posted in Uncategorized | 20 Comments