Into the Memory Hole

In 2008, as part of my new job at NASA Headquarters, I was told to ‘try out’ some of the new social media applications.  There was no other guidance given (that may have changed since then).  In mid-2008 and continuing until I retired in the summer of 2010, the agency gave me a blog site on the main NASA web page.  After I retired from NASA, that site went into the ‘archives’ – still available, could be found by a simple search.  And of course, I continued writing a blog, now hosted on a non-government site.

Recently, with some housekeeping, my old NASA blogs have disappeared.  Poof.  If you try to access them, the return message will state:

“404 The cosmic object you were looking for has disappeared beyond the event horizon.”

I guess NASA doesn’t need or care about my work anymore.  I certainly take it to mean that they have relinquished any claim on those writings since they have been declared ‘beyond the event horizon,’ e.g., in a black hole and unretrievable.

But have no fear, foreseeing such a possibility, I have printed them all out: all the blog posts, all the comments.  On paper, kept in my far too numerous paper file folders. Some day my heirs may shred them or burn them, but not today.  I have them for my personal reference.

If you noticed that my blog postings are less frequent these days, it is because I am spending my literary energy working on compiling many of those posts into a book.  Or rather, updating, rewriting, and rearranging some of those posts- plus many other stories – into a potential book.  Or maybe two books.  Or maybe three.  Some publishers have expressed an interest in such a book.  Or two.

So, if there are any blog-o-sphere fans of mine still out there – be patient.  Coming soon (I hope) will be lessons from my years in the space industry.  Hope you buy a copy!

Meanwhile, the electronic memory black hole has grown.  But things past will not all be forgotten.  I still have the paper.  And maybe, soon, you will too. 

Reflections on a Pilgrimage

I’ve been on a pilgrimage. 

A pilgrimage of the old-fashioned kind.  It was not something I thought I would ever do. 

The terms pilgrim and pilgrimage are used frequently these days and often without much consideration of their historic meaning.  In the old-fashioned definition, a pilgrimage is a journey undertaken for spiritual or religious reasons. 

Since I am a good protestant and a modern scientific man, I set little store in ancient relics, nor do I have a need to work my way out of purgatory.  I have no argument with Roman Catholicism, but my personal beliefs vary in some ways from their orthodoxy.  But at the core, on the most significant subjects, my beliefs and theirs coincide.   

I’ve been on a pilgrimage.  One of the old-fashioned kind.  All along the way, the townspeople and passers-by wished me ‘buon camino’ – have a good pilgrimage.  Because they knew what was happening. 

So, what was my experience?  And what did I learn? 

My boon companions on this trip were eight other protestants, six of whom were ordained Methodist clergy.  They planned the trip before I joined the group.  They selected the time and route according to their needs and schedules, and happily welcomed me as a latecomer into their troop.  All of these men are much younger than my 70 years, and in better physical condition. 

Because of time commitments, my group needed to be away from home no more than twelve days.  To earn our certificate, we needed to walk at least 100 kilometers, or about 62 miles. And we had to do it the last two weeks in February when the temperatures were chilly and it was officially the rainy season.

I went because I wanted to spend time with my adult son; I like nature; I have always enjoyed a good hike.  In my younger years, with a group of like-minded friends I would undertake strenuous backpacking trips.  But those ended more than a decade ago.  This trip sounded easier – no mountains to climb, a warm shower and a soft bunk ready at a hostel each night, food readily available in any number of restaurants along the way.  No need to carry a tent, cookstove, food, or more than a liter of water.  My pack was about a third the weight of those I carried in my backpacking days.  It would have been even lighter without the winter clothes required for February.  How hard could it be?  The leader sent out a rigorous preparation walking schedule for almost three months before departing.  I mostly followed it.

The Camino de Santiago has history going back to the early middle ages.  Pilgrims came from all over Europe, using different paths, taking much longer time than two weeks.  These days, the most famous route to Santiago de Compostella starts in France, but there are many options:  starting in Lisbon – the Portuguese route – is very popular.  There are at least a half dozen more popular options.  We planned to walk the Camino Ingles – the ‘English’ route.  It was started by pilgrims from the British Isles back in the day when they were all Catholic.  Those pilgrims actually took ship to the north coast of Iberia – Galicia – and finished their pilgrimage by walking the shortest route of all: officially 113 kilometers, just over 70 miles.  I made most of it. 

In a good year, almost half a million pilgrims make their journey to Santiago; thousands can come in each day in the warm summer months.  The day we arrived, the official count for the day was 145 pilgrims; 19 having come on the English way.

Rather than focus on the religious art we saw, or the churches and cathedrals we visited, beautiful as they were, the focus is on the spiritual insights I gained walking along the way.

Any walk in the woods, seeing the beauty of nature, hearing the birds, can cause contemplation and a sense of being closer to God.  Spending special time with my son was a cause for rejoicing.  Making new friends, and even depending on them for help – that was a spiritual experience.  Even visiting new places and seeing new sights makes one thoughtful.  Getting away from the daily schedule, the cares and concerns of home and work for a time always allows spiritual growth.  Not knowing the language, other than a few basic phrases, and staying away from electronic devices – the constant noise of modern life was turned off.  All these things are positive spiritual influences.

Old, everything is so old in Spain.  We crossed one bridge, still in use today, that was built by the Romans.  We have nothing like that in America, although I had nearly the same feeling of time when visiting the ancient pueblo ruins of Mesa Verde.  Time is broad and we are merely at one tick in its everlasting stream.

Everything is so very old in Spain, except for what is new.  Modern high speed train travel is amazing to this Yankee, something we don’t have.  But the Camino is ancient by any standard.  There is spiritual significance remembering that one is in the presence of so many who traveled the same path for more than a thousand years.  Awareness of time, and one’s place in time; that is a profound spiritual experience. Decreasing one’s self importance puts everything into perspective.

Modern travel has obliterated distance.  On the plane crossing the ocean, 113 kilometers would pass in moments.  Walking is to experience distance in its raw essence: every meter, every foot.  Little rises that would be unnoticeable traveling by automobile become challenges to be overcome when walking. Traversing downhill slopes on wet cobblestones increases anxiety and heart rate dramatically.  Distance passes slowly; there is time to contemplate the countryside, the houses, the animals in the fields, the trees.  Traveling on foot is so completely out of the ordinary in these days that the mere walking becomes a spiritual experience too. 

Rousing up and starting to walk before sunrise, without breakfast or coffee, in the cold and often in the rain: these were privations.  Privations are part of a pilgrimage too.  Somehow it wouldn’t count as a pilgrimage if there wasn’t some pain involved. 

There were a couple of days when my old body just wouldn’t go the distance.  It was disappointing, somewhat maddening, but it was a fact of (my) life.  Confronting the limitations of age is a spiritual blessing I was not looking for. 

Fortunately, we were in civilized territory and a taxi ride was just a phone call away.  The old-time pilgrims didn’t have that option.  I didn’t keep strict accounting but I probably missed walking more than 13 of those English Way kilometers, making my certification slightly suspect. 

Finding a hot meal in a warm and dry restaurant after a day walking in the cold and rain was indescribable.  Putting on dry clothes after being soaked was cause for thanksgiving.  Simple material blessings loom much larger in context.

Entering the Cathedral at Santiago was more moving than I had anticipated.  We entered the crypt where the relics of St. James (Saint Iago in Spanish) are kept in a silver casket.  I would never insist that the bones of James, the brother of John, one of the sons of Zebedee, one of the sons of Thunder, were in that box.  But I could not dismiss the vague feeling that they might just be there.  It is just possible that I was in the presence of one who followed the Master for three years.  That feeling was overwhelming.  The Mass was in Spanish, unintelligible to me; but the music was moving and magnificent.  The altar was gaudy, all silver and gold, intricate beyond belief, and offensive to my puritan sensibilities.  But spending time with others of our faith, even when there are variances in the details of our belief, even when they spoke a veritable babble of languages; that was a spiritual experience, too. 

I don’t think I got any time off from Purgatory, if it exists.  There were times on the trail, I confess, that I felt I was experiencing Purgatory!  That is part of the pilgrim experience; part of the spiritual journey too.

Certainly, I did receive a spiritual blessing from my pilgrimage.  I’m wiser, better traveled, and more thoughtful than when I left.  It will take some time to fully process the journey.  That is what a pilgrimage is:  a journey undertaken for spiritual reasons.

I have been on a pilgrimage; one of the old-fashioned kind. 

My son tells me that he my try a different route on the Camino de Santiago in a couple of years. 

I told him I would take the plane and meet him at the end.

All eggs, One basket

I am told that there is a principle that both the President and Vice President never ride on the same aircraft, for a pragmatic if ghoulish reason.  We can’t afford to lose both of them in one accident. 

During a space shuttle flight, the entry flight director – specially trained and certified for the unique demands required of the decision maker during that critical operation – was always near at hand, only a phone call and quick drive away, night or day, all hours.  If something were to happen to that critical individual, it was comforting to know that there was always at least one other trained and certified entry flight director available; maybe not assigned to the flight at hand, but ready willing and able to take over in a pinch.  Usually there were up to about four flight directors so certified but various travel duties, vacations, etc., might take some of them out of the picture at least for a quick call up.

Only one time . . . well, that is this story.  It starts with a very technical debate.

A long and contentious debate over some of the fine points of main engine limits management.  That is as arcane and complex subject as there ever was about space shuttle operations. A subject that many felt passionately about.  Unfortunately, there were passions on both sides of the argument. 

The argument is highly technical so I have postponed that discussion to a post script rather than delay the main story.  But both the main story here and the technical post script that follow are about risk management in very unusual circumstances and how NASA decides what to do in such cases.

Those who persevere to read and perhaps understand the technical details in the post script will be awarded the space cadet nerd award.

Suffice it to say that the debate had been going on for literally years as improvements were made to the engines and various launch abort procedures; the risk was always a balance between the two.  It became essentially an argument between the Flight Directors and the Astronauts.  Strong discussions frequently marked the process of arriving at the best possible resolution of an issue at NASA.  This was more contentious than most but not the worst I ever saw. 

We in the Flight Director’s office were smug in our confidence that we understood both sides of the risk trade; after all the developer of the main engine was our primary source of information.  The JSC engineers responsible for the space shuttle orbiter were extremely nervous about the contingency abort profiles that the other side proposed using.  The JSC engineers were little use in the discussion because they would not commit to allowing flight conditions to go one iota past the limits that were tested and certified (and which had safety factors built in). 

Finally, we in the Flight Director’s office were suspicious of the arguments the Astronauts made.  Most of the pilot astronauts were fighter pilots and test pilots.  They were supremely confident in their ability to fly any vehicle under any circumstances.  Probably too confident, we suspected.  The Astronauts had practiced flying convoluted contingency abort scenarios in the Shuttle Mission Simulator.  The SMS was great for near normal flight characteristics, but its computer models did not cover extreme cases where, for example, the wings could melt off.  We believed the Astronauts were being deceived by a simulator that did not accurately represent the consequences of their flying techniques. 

By 1998 the impasse had reached a boiling point: each side was convinced it was right, the other wrong, and the senior management confused about the arcane details of the arguments.  There were numerous ‘white papers’ written with varying degrees of statistical calculations based on limited data and limited modeling and testing.  It grew very heated.

Finally, the Space Shuttle Program Manager, Tommy Holloway, had enough.  He directed the entire contentious contingent to travel to the Marshall Space Flight Center in Alabama where the engine experts resided.  A face-to-face meeting with no holds barred.  We were instructed to not come home until a consensus was reached.

The problem was getting a time when all the parties could make the trip.  All of our schedules were overbooked.  The astronauts and their management could fly their T-38’s, but the flight directors could only make the trip in one day by using the JSC management airplane, an ancient Gulfstream I turboprop nicknamed ‘NASA 2’.  The only date ‘open’ was during a shuttle mission: STS-90 in April of 1998.  Columbia was on an extended 16-day SpaceLab mission.  All the astronauts not in space could go.  The A/E Flight Directors were free. 

All the certified Ascent/Entry flight directors plus those in training were sent.  That’s right, all of us.  The mission was going well, so the management felt it was OK for us to be gone for a day – up in the morning, back in the evening.  After all, the orbit certified flight directors had some training in how to execute an emergency deorbit if necessary. 

Ha.  An emergency deorbit is not like a regular deorbit.  Most orbit flight directors were overly confident from their one (1) simulation during training.  The A/E flight directors had sweated over the real thing multiple times and knew it was not easy.

No emergency deorbit was required.  The trip was stormy both literally and figuratively, but a consensus was hammered out.

That day of the trip there were thunderstorms forecast along the return flight path of our airplane as there often are that time of year.  We were really in turbulence for much of the return trip; the Gulfstream bounced around significantly.  Added to the old turboprop’s normal waddling tendency to dutch roll, this herky-jerky made all the flight directors wish they were on the ground.  The T-38s could fly high enough to get out of the worst of the turbulence but the old Gulfstream was limited to a lower ceiling where the weather was stronger. 

Later on, we talked about how stupid that trip had been.  Clearly the odds of the plane crashing were low, but they were not zero.  If all of us had been wiped out at once, there would have been a heck of mess to deal with and put the crew of STS-90 at significant risk. 

Of course, that would not have been our problem!

Such an event never happened again in our shuttle history.  Putting all the certified decision makers on the same plane at the same time, and especially not during a mission.  I wonder if the STS-90 crew knew.  I don’t think we told them. 

Oh, the upshot of the meeting?  A very bright young MSFC engineer named Mike Kynard gave a very convincing presentation that swayed us all to the side of the astronauts.  It was a hard pill to swallow because I was the leader of the other side. 

As a matter of fact, it took me a couple weeks of fuming to come around to the new way of thinking.  Pride, I guess.  But before the next shuttle launch, we had put the new procedures, rules, and other documents into effect. 

Risk?  Of course.  Necessary?  Maybe.  Maybe not.  The Program Manager got what he wanted: a decision that (eventually) all of us agreed to. 

Late addition and thought: After all the effort: years of study, analysis, calculations, and most of all the contentious debates which affected interpersonal team relationships, what was the true outcome?

We practiced over and over again the procedures for these cases. Thousands of hours were spent in simulations reacting to multiple engine failures.

And the bottom line: Never used in real life. No crew member ever again touched the limits switch in actual flight.

Was it worth it?

At least we were prepared.

Postscript

So, for those interested in the technical details, here it comes.  Non-technical folks should leave now.  Abandon All Hope Ye Who Enter Here.

Large liquid fueled rocket engines are extremely complex, light weight, low margin, and temperamentally subject to coming apart in spectacular ways.  The Space Shuttle Main Engines (SSME) were definitely a worrisome product and the most worrisome were the turbopumps rotating at 35,000 revolutions per minute.  Every SSME had a computer built right onto the engine that monitored and controlled it.  The main shuttle orbiter computers – the General Purpose Computers (GPC) – would issue the ‘big’ commands to the engines:  start, shutdown, throttle up/down to x%, etc.  The SSME controllers (Main Engine Controllers – MECs) would translate those commands into smoke and fire.  For example, when receiving the ‘start’ command, the MEC would send a choreographed time sequenced set of valve position commands to spool up the engines.  Each step exquisitely timed to build up thrust without overtaxing any part of the engine.  And then the MEC would continuously monitor the ‘redline’ parameters:  some critical temperature measurements, some well placed pressure measurements, and a few other parameters such as turbine shaft speeds.  Each measurement was monitored to ensure the instrumentation was reading within reasonable values, which ‘qualified’ each measurement.  Then, if all the qualified measurements for a monitored measurement exceeded or fell below a safe value (‘the redline’), the MEC would issue a Major Component Fail (MCF) which has an associated Failure Identification code (FID).  And then check on instructions from the GPC before doing anything else. 

One of the most worrisome values which was monitored was the temperature of gas exiting the high-speed turbines that powered the fuel pumps, normally about 1,300 degrees Fahrenheit. If something went wrong in the turbine it would quickly show up in a dramatic temperature rise; hopefully tripping the safety limit before the metal case melted through.   Early versions of the engines had two sensors in the turbine exhaust gas manifold; later models had four sensors. If all were working properly (‘qualified’) and all measurements exceeded the programmed limit, the redline software would issue the MCF and FID.  If the MEC had received a shutdown limit enabled command from the GPCs, the MEC would start shutting down the affected engine.  This is what happened on STS-51F when both sensors on one of the turbines actually failed and indicated erroneously high temperatures.  That was the only in-flight shutdown of an SSME in the entire program.  That flight has a story of its own.

Back to the hypothetical story.  As the engine shuts down, the chamber pressure – a direct correlation to engine thrust – drops below the 30% level, the MEC sends a software notice to the GPCs that the engine is in ‘shutdown mode’.  In a benign shutdown, from the start of the MEC sequence to shut down the engine – the issuance of a MCF – to 30% thrust could take a long as 3 seconds.  Keep that in mind.  The GPC is not going to change commands to the other engines until it gets that ‘shutdown mode’ indication from a failing engine.  In the rocket engine business, 3 seconds is a long time. 

Once the GPCs received the ‘L, C, or R engine in shutdown’ bit, red light lit up on the cockpit dashboard – something we never wanted to see but often practiced in simulated mission.  Simultaneously telemetry went out to the flight controllers on the ground and the red light on several consoles in the Mission Control Center would come on as well.  Flight controller heart rates are not monitored by the flight surgeons, a good thing. 

Next, depending the position of a certain switch in the cockpit, another command could go out from the GPCs – or not.  On the orbiter flight deck center console, between the commander and pilot, were a number of switches, one of which was the SSME Limits switch.  The position and actuation of that switch was the center of the controversy.  It had three positions:  Inhibit, Auto, and Enable.  If the Limits switch was in the Enable position, and an SSME in shutdown mode bit were received, the GPCs would take no action, leaving the enable bit out there for the remaining engines.  The automatic redline shutdown function of the two other engines which were running would continue as if nothing had happened.  If a second or third engine MEC detected a redline violation, their MECs would execute shutdown as well. 

If the Limits switch were in the Auto position, it meant that the automatic redline shutdown function on all three engines, which had been enabled until one of them entered shutdown mode, would be changed.  At that point the GPC software would send the redline limits shutdown inhibit command to the two running engines.  That means that if either or both of the two still running engines had a violation of any of their redline sensors, those engines would continue to run no matter what – or at least try. 

But remember that there was a three second or so delay from the time that an SSME with Limits enabled would be spinning down before it issued the ‘I’m in shutdown’ bit.  A bad problem that affected either or both of the other two engines could lead to them shutting down in that little time window.  Good news/bad news depending on how you looked at it.

If the Main Engine Limits switch was in the Inhibit position, all three engines would have already received the redline shutdown inhibit command back whenever the switch got set.  With redline shutdown inhibit set, any sick or failing engine would continue to try to run no matter what the MCF/FID status was.  The MEC would not start the shutdown sequence no matter how bad it got. 

Read those last few paragraphs again if they went by too fast. 

The only SSME premature (unplanned) shutdown in shuttle flight history occurred in November of 1985 on mission STS-51F.  A pair of faulty temperature sensors erroneously indicated a problem with the engine and the MEC shut that engine down.  This occurred late enough during the boost phase that the mission continued to a completely successful conclusion.  On STS-51F, those temperature measurement devices had a generic flaw, and in addition to the one engine that shut down, there were indications of measurement failures on other engines.  The booster officer, Jenny Stein, saved the day by telling the Flight Director to instruct the crew to put the Main Engine Limits switch to inhibit to preempt the redline shutdown software.  None of the engines actually had a problem, it was just sensor failures.  After that flight, the temperature sensors were upgraded to a more robust, more reliable sensor type.

And remember that the turbine exhaust temperatures were not the only sensors being monitored by the redline detection software; there were other pressures, temperatures, and turbine shaft speeds being monitored.  Any of those could shut an engine down if the software was enabled.

High performance liquid rocket engine turbopumps have a tendency to come apart in a hurry when something goes wrong.  The SSMEs (now RS-25s) have been extensively instrumented and tested.  Their computer control brain – the MEC – has a number of ways to detect an impending failure and turn the engine off before it comes apart. 

The system was not completely foolproof, but should prevent an explosive catastrophe in most cases. 

Mr. Bob Biggs was the Rockedyne SSME Program Manager all through development and into the middle of the shuttle flight program.  He wrote a white paper describing the philosophy behind setting up that redline software.  He knew from history that liquid rocket engines can fail very spectacularly in an incredibly short time period, or they can run in a degraded manner for some significant amount of time under some circumstances.  The result of a degrading engine which had some redline exceedance was described by Mr. Biggs as having three equally probable outcomes.  First, the engine could fail catastrophically in spite of the redline detection; that is to say the mechanism would violently come apart too fast for the redline detection software to react: ‘Inevitable catastrophe’.  Second, the engine which would have failed catastrophically in a slightly longer time would be successfully shut down by the redline software and thus avoid the catastrophic failure.  These were termed ‘intact engine failures’.  The engine might not be reusable but it would not blow the back end of the vehicle off.  Third, the engine was running in a degraded mode that exceeded the redline limit but would continue to operate – produce thrust – for an ‘extended’ period of time if the redline shutdown software were inhibited. 

Mr. Biggs noted that the cutoff values for each redline were chosen so that equal probabilities would result in each case.

So, for an engine flashing a red light, 1/3 of the time it would blow up before the safety mechanisms could prevent it – always resulting in the loss of the vehicle; 1/3 of the time the safety software properly shut the engine down before it came apart and save the day; and 1/3 of the time the engine could have continued to run if the safety redline software were inhibited. 

Clear?

On the other side of the risk discussion are the abort black zones.

The shuttle had a remarkable capability that most other rockets do not.  In virtually all expendable rockets, if any one of the booster engines shut down prematurely — even if that shutdown is benign — the mission is over, the payload is going into the ocean somewhere, and the Flight Control Officer (FCO) at the Range Safety console is going to “send functions” (aka kaboom).  On the other hand, the shuttle is designed — required — to be able to safely return the orbiter, crew, and payload to a runway landing following the premature benign shutdown of any one of the three SSMEs.

So, if any single SSME shuts down (fails) at any point in the launch phase, a safe return of the shuttle and crew is certified (read ‘guaranteed’) to result.  All the various conditions have been examined, analyzed, simulated, and verified by computer analysis, wind tunnel testing, etc. 

From launch to about 4 minutes into flight the shuttle can perform the scariest type of abort – a Return to Launch Site abort (RTLS).  Prior to the first shuttle flight, somebody proposed that we do an RTLS on purpose as a test — they called it the “Sub-Orbital Flight Test (SOFT)”.  Capt. John Young, the chief of the astronaut office and the commander of STS-1 was noted for his colorful memos that he would regularly send on topics of the day.  The SOFT proposal drew a classic response: “RTLS requires continuous miracles interspersed by Acts of God to be successful” John wrote in 1980.  And in fact, on STS-1, a trajectory bug lofted the shuttle trajectory higher than expected and an RTLS probably would not have been successful.  John was right at least for those days.

Since those early days, RTLS was significantly improved and later in the program would most likely have worked if required — but I’m just as glad we never found out for real. 

From about 2 1/2 minutes into flight until almost orbital insertion, the premature shutdown of an SSME could result in a Trans-Atlantic Landing abort (TAL).  The shuttle keeps going forward but aims for Europe or Africa rather than orbit.  The entry is very similar to a normal end-of-mission entry and the landing would occur at a prepared runway in Spain, France, or western Africa.

Later in flight, from about 4 1/2 minutes on, the premature shutdown of an SSME would result in an Abort To Orbit (ATO).  In this case the shuttle presses forward and the system is designed to scavenge all the reserve propellant in the External Tank to get to orbit.  Sometimes a dump of propellant from the Orbital Maneuvering System is required, sometimes other adjustments to the trajectory are required, but ATO missions can range from landing after a few orbits on launch day to having a fully successful mission depending on many variables.  The longer the remaining main engines run, the closer to normal the shuttle can get. STS-51F, the only real-life example of ATO, performed a basically nominal mission.

The Abort Once Around (AOA) mission – which is exactly what it sounds like – was not used except for ‘systems’ problems like a big air leak from the crew cabin. From a trajectory standpoint, ATO and AOA looked very similar until the crew executed a ‘deorbit’ burn well after the main engines were done. 

All of that is fine as long as two of the three SSMEs continue to operate and the shuttle remains under control.  If control is lost, then all is lost.  The shuttle does not fly sideways very well.  A capsule might right itself, but the shuttle will break up.

The speeds and energy required to achieve earth orbit is almost beyond conventional understanding.  To maintain a low earth orbit, a satellite must travel at over 5 miles each second.  At even a fraction of those speeds in the “lower” atmosphere (below say, 80 miles high), air friction converts that vast kinetic energy into tremendous heat.  Meteors or re-entering space junk are vaporized in a flash.

For those who may have forgotten their high school physics class; getting to high speed is critical to establish an orbit.  To compare with commercial air travel may be helpful.  Typical airline travel is around 6 miles high (30,000 feet or higher).  Typical airline speed at cruise is around 500 miles per hour.  To be in a safe orbit, a satellite needs to be at least 20 times higher (120 miles is a safe orbit for a few weeks) but must also be going about 35 times faster (17,500 mph).  Energy, the real measure of the difference, is directly related to height (altitude) and the square of the speed.  To achieve earth orbit requires roughly 1,000 times the energy that an airliner has at cruise. 

This explains why war-surplus V-2 rockets with WAC Corporal second stages could reach orbital altitude in the late 1940’s but it took another decade to develop rockets that could not only get that high but propel a payload to the extreme velocity required for earth orbit.

Satellite launchers seek the most efficient way to get to orbit — they want to use the least energy (total impulse is the correct term) to get the maximum payload to orbit.  Simplistically, one would want to go straight up to altitude first, then pitch over horizontally and accelerate, accelerate, accelerate.  Expendable satellite launch vehicles go high early and then pitch over toward the horizontal for the largest part of the rocket burn. Frequently, the trajectory goes higher than needs to be and the rocket accelerates horizontally why simultaneously falling back toward earth! An expendable rocket sending a satellite on a one-way trip to orbit optimizes its trajectory by lofting high early on.  If an engine fails, the mission would be lost no matter what the trajectory; abort modes and crew rescue are not a consideration. 

Unfortunately, this does not work well if you want to protect a crew from a failure of the rocket engines.  In a planned re-entry, the shuttle flew through the upper atmosphere at a fairly shallow angle so that it encountered thicker atmosphere gradually and the lifting body plus its stubby wings created lift.  As the re-entry proceeded, the speed (or more precisely, kinetic energy) was bled off gradually limiting the maximum heating temperature and using lift generated by the wings to hold structural loads relatively low.  For a suborbital ballistic re-entry, the trajectory is quite steep, encountering the denser parts of the atmosphere while the speed and energy are still quite high leading to a high heat impulse and very high structural loads. 

For the space shuttle, a steep re-entry would result in a structure demise.  This is called a ‘black zone’.   Unsurvivable by the crew. 

The use of ‘expendable’ Atlas rocket booster for the Boeing CST 100 Starliner or the Falcon 9 for the Dragon capsule were only made possible by reshaping their launch trajectory lower than that used by those same boosters in an expendable mode.  This eliminated what would be the black zones even for those capsules for a premature engine failure.  The cost of this ‘abort shaped’ trajectory is performance:  the amount of payload to orbit is decreased by flying a safer, more depressed trajectory.

The trajectories for manned spacecraft try to avoid these steep re-entries even on an emergency case.  The two real life cases using capsules turned out moderately well.  On April 5, 1975 the crew of what would be known later as Soyuz 18A, Vasili Lazarev and Oleg Makarov, were more than half way to orbit – at orbital altitude and traveling at over 10,000 mph – when their second stage refused to be jettisoned.  Separating the crew module from the malfunctioning rocket stage resulted in a very high altitude but ‘slow’ horizontal speed ballistic entry.  During a Soyuz entry, decelerations of 5 g are normal.  Due to the steep angle of the Soyuz 18A abort trajectory, the crew endured up to 21 g.  Fortunately, they survived, the capsule did not break up, and they landed safely.  But the two crew members never flew in space again. On October 11, 2018 the Soyuz MS-10 carrying NASA astronaut Nick Hague and Russian Cosmonaut Alexey Ovchinin suffered a booster failure at an altitude of just over 30 miles and a much lower speed than the 18A case.  The spacecraft successfully executed an abort and the crew landed safely, experiencing only about 6 g, and were able to fly again another day. 

The shuttle flew an ascent trajectory that is more depressed than expendable launch vehicles.  This allows for potentially graceful abort trajectories following one premature engine shutdown.  The program management never intended, never required, and generally never funded the studies or tests to define a capability to safely land with more than one SSME prematurely shut down.  In the Flight Operations Directorate, that did not stop us from trying to figure out what the best way out was for those situations.

If two of the SSMEs quit but one remains running, there are some options.  Early in the shuttle program there were no good options.  However, capabilities were added over the years which slightly improved the possible outcomes in some parts of the trajectory.  For missions headed to the International Space Station, plans for ‘single engine’ contingency cases called for steering toward the east coast of the United States with the potential to land at an emergency airfield somewhere on the Atlantic Coast of North America.  However, many of these trajectories result in entry conditions that exceed the capability of the shuttle orbiter either thermally or structurally:  the dreaded black zones.  The possibility of executing a successful East Coast Abort Landing (ECAL) was far from guaranteed, but in that situation, it was felt to be worth a try.  What is the other choice?  If the shuttle doesn’t break up or burn up on the steep ballistic trajectory for an ECAL there is every reason to believe that a safe landing might occur.  As Capt. John Young put it colorfully: “this gives you something to do while you’re waiting to die.”

If the shuttle could not get to a runway, the shuttle still possessed minimal crew escape capability.  If the shuttle could achieve a straight and level glide (actually not very level since the shuttle glides like a rock), at around 30,000 feet the crew could jettison the side hatch and bail out with parachutes like some WWII bomber crew.  This was considered better than ditching in the ocean or rough terrain.  All studies show touchdown “off-runway” would likely not be survivable since the shuttle touched down at around 250 mph and would roll up in a ball.  The subsonic, aircraft-in-control, bailout was really the only option.  And in most cases the crew would have probably wound up sitting in a tiny inflatable rubber raft in the middle of the North Atlantic waiting for somebody to pick them up.  We positioned C-130 crews with pararescue divers so that the wait should not exceed eight hours in the worst case. 

Eight hours bobbing in a little rubber raft in the north Atlantic.  Ever see the movie “Titanic”?

If all three SSMEs quit prematurely there is real trouble.  Of course, many 3 engines out situations probably were caused by an ‘uncontained’ failure of one of the engines, possibly blowing the back end of the orbiter off, taking out the hydraulics, rudder, aileron control, etc.  Those cases were not survivable.  Assuming that all three SSMEs shut down benignly (not the most likely case) the situation would still be dire.  There is little to no way to control the trajectory and the black zones for those cases were immense on the charts.  For some few lucky cases a successful ECAL might result but then it’s not really a lucky day if all three engines quit, is it?

In the early shuttle days, the 1/3, 1/3, 1/3 statistics of safe engine shutdown with the redline software was pitted against the low likelihood of surviving a multiple engine out abort.  The rule was, launch with the Main Engine Limits switch in auto.  Then, if an engine fails prematurely – assuming it is not in the 33% case where it blows up – the system will inhibit auto shutdown of the two remaining engines.  A sick engine might limp along but sensor failures would be precluded.  The redline software was to be reenabled only very close to the end of the powered phase when the trajectory was acceptable if a second engine shut down.

As time went by, the safety features of flying one of those multi engine out trajectories were improved, the engines were improved and became more reliable.  The statistics changed from 1/3, 1/3, 1/3 to something like 25%, 45%, 30%.  The astronauts wanted to avoid engines blowing up so they proposed putting that dratted switch in the (hard) enable position.  As the senior Ascent Flight Director, I was wedded to the idea that we should leave the limits inhibited until a safe reentry could be reasonably expected.  White papers and presentations proliferated.  Statistical studies that could make your head swim proliferated.  Mathematical shenanigans multiplied.  Tempers began to rise and positions became calcified.

The debate came to a head when the Space Shuttle Program Manager, Tommy Holloway, gave a direction to the Flight Directors and the Astronauts to resolve the issue, period.  He directed the MSFC SSME guys got involved, after all they were the experts.

When we got to MSFC on that special day, we heard a lot of presentations and had several heated arguments.  That is the NASA way, it seems.  But the one presentation that won the day was from a young SSME engineer named Mike Kynard.  He argued, from ground test data, that engine failures followed a ‘bathtub curve’.  That is, the probability of engine failure was highest very early, shortly after start up.  If the engine got to a stable running state (which would occur before liftoff), then the probability of engine failure was low but gradually increased with time.  The probability graph looked like a sideview slice through a bathtub:  high on the ends and low in the middle.  Mike further argued that the redline shutdown was very reliable in flight; explosive failures were rare in the middle part of the curve and the software (again based on ground tests) was very unlikely to cause an erroneous shutdown.

That argument changed most minds.  Together with the improving capabilities to achieve a two-engine out contingency abort, the verdict was as follows: (1) launch with the Main Engine Limits switch in Auto.  Following a first engine failure the software will promptly issued redline inhibit commands to the remaining two engines, at that point (2) the booster office makes a quick assessment of the two running engines and, if operating normally, recommends Main Engine Limits switch action to the Enable position (sending out the enable command) then back to Auto (to allow for assessment if a second engine quits).  And part (3), if multiple redline sensor failures are detected (as in STS-51F) to put the Limits switch in Inhibit (which was what was done for that flight). 

That is how we flew out the remaining shuttle flights.  Not everybody was completely happy, but everybody could live with it:  the very definition of a consensus. 

Tommy was happy.  I think.  The arguments subsided. 

Are you ready to receive your space cadet nerd badge?

And for the record, I still think it was stupid to put all the A/E Flight directors on the same plane.

Flight Rationale

Absolute certainty can never be attained for many reasons, one of them being that even without limits on time and other resources, engineers can never be sure they have foreseen all possible contingencies, asked and answered every question, played out every scenario. – Dr. Diane Vaughn, The Challenger Launch Decision

The term ‘flight rationale’ is not familiar to most people.  Inside NASA, it means a description of the reasons why a particular space mission is considered acceptably safe to carry out. Flight rationale is generally associated with some particular part, or subsystem of a vehicle, or it can be a particular operation that is being considered.  Good flight rationale describes the set of information that manifestly shows the part, subsystem, or operation will perform as required with requisite safety margins intact.  Alternatively, poor flight rationale includes unproven assumptions, incomplete testing, or analysis that contains flaws. 

Unfortunately, in the real world of space flight, there is seldom perfect flight rationale.  Sometimes there is good flight rationale.  More often the rationale proposed for a flight contains ambiguities.  Someone must exercise judgement to determine whether the flight rationale is adequate or not. 

Before getting too far, remember this constant in space flight:  it is not safe by any ordinary consideration of safety.  The energies involved are orders of magnitude above those encountered in almost any other endeavor.  The requirement to make flying vehicles as light as possible leads to factors of safety that are minimal.  The environments encountered in spaceflight are extremely hostile to human life.  Even the best designed, flight proven vehicles, under the best considerations, have analytical probability of failure that is eyewatering in comparison to everyday life.  For example, airliners are supposed to be built to a fatality rate of less than one in ten million.  Automobiles – less their fallible drivers – even safer.  The NASA acceptable failure rate for commercial crew spacecraft is one in about 500.  No sane parent would put their child on a school bus where the probability of a fatal crash is 1 in 500.  So be advised; going into space is a risky business and is not safe under any normal understanding of that word.

I like to quote Dr A. R. Dykes of the British Institute of Structural Engineers:

“Engineering is the art of modelling materials we do not wholly understand,

into shapes we cannot precisely analyze,

so as to withstand forces we cannot properly assess,

in such a way that the public has no reason to suspect the extent of our ignorance.”

Which pretty much sums up ordinary engineering projects like highways and dams but is wildly optimistic for most any product in aerospace engineering. 

Engineers almost always start with analysis:  an attempt to mathematically demonstrate using the laws of physics that a part will do its job.  But there are always assumptions in any analysis; some assumptions are better than others but there are always assumptions in any analysis. 

Test is considered better than pure analysis.  However, as my old mentor, the Manager of the Mission Evaluation Room, Joe Micheley used to tell us: ‘it is better to be ignorant than run a stupid test.’  Of course, Joe used more colorful language.  Over and over again, I have seen some hardware test that was done which did not match the situation really being studied.  It is almost impossible to test, on the ground, all the combined effects of spaceflight:  high vacuum is hard to replicate in the lab, the thermal environments are extreme, there is no known way to include microgravity, radiation may play a role, etc.  But even then, there is always an argument over whether the principal parts of the test are correct.  For example, a metal part tested with a heavy load to see if it bends or breaks.  Did we really understand what the load would be?  Are we sure we got the right values?  Sometimes engineering is all about the argument.

If a failure occurred it is always the goal to discover the ‘root cause’ and correct it.   Following a long investigation with much hardware testing and even more analysis, the team will declare it has found ‘root cause’.  Most likely.  Maybe.  Never 100% sure.  Again and again, I have been party to failure investigations that concluded that root cause had been found and thus corrections could be made.  Except sometimes the corrections didn’t work because the root cause wasn’t accurate.  Always take the determination of ‘root cause’ with a grain of salt.  More salt if the investigation was rushed. 

And in the end, somebody – some one person – designated to be the decision maker; that person has to come to a conclusion.  A good decision maker listens to arguments from all sides of an investigation.  The debate needs to reach a certain status before making a decision.  A good decision maker can ask penetrating questions. A good decision maker will know when to ask for more information before deciding. The engineers will always always always ask for more tests, more analysis, more time to get more information to be more certain of their conclusions.  The decider also has to decide when enough has been done. 

The rub in all of this, having been the designated ‘decider’ in too many situations, is that it always involves the risk to human life. 

These are decisions that do not affect some faceless, nameless, random human.  Rather these decisions will affect the safety of people that you know.  At the very least, these are people we have worked together with, trained together, spent hours in difficult meetings together.  Astronauts have lives like most of us.  We send our children to school with their children; those children participate in the same after school programs:  sports, music, scouts.  Your spouse and their spouse worked, partied, socialized together.  I’ve had to made decisions about the safety of astronauts who were literally my neighbors, some who attended the same church my family does.  Knowing that you will have to look the spouses and the children in the eye after whatever happens, well that does not make deciding any easier. 

As a Flight Director, I had to make calls every single day that affected the safety of the astronaut crews.  Most of the time, I found out how good a decision I had made in a short time:  a few hours at most.  As a Program Manager I had to make decisions every day that would have consequences not realized for months, perhaps years.  That does not ease the burden.

I do not envy today’s decision makers, the ones weighing flight rationale.  My only advice is to listen thoroughly, question effectively, ask for more data when necessary.  But when it is time, a decision must be made. 

A work of fiction which I like has summed it up perfectly: “Life was not safe, and nothing could make it so, neither fashionable dresses nor bank accounts. The baseline of human life was courage.” – ‘News of the World’ by Texas author Paulette Jiles

Breaking the [Flight] Rules

The official NASA history of STS-109 can be found on the agency web page: 

STS-109

The last part of that official account reads: 

“After a successful launch, flight controllers in Mission Control noticed a degraded flow rate in one of two freon cooling loops that help to dissipate heat from the orbiter. After reviewing the loop’s performance, mission managers gave the crew a “go” to proceed with normal operations. The problem had no impact on any of the crew’s activities. Both cooling loops performed normally on de-orbit and landing.”

The official NASA description of what happened on STS-109 is a lie.

I should know.  I was there. 

Marianne Dyson and I worked together in Mission Control in the early days of the Space Shuttle program.  She had been in touch with Jim Newman, a crew member on STS-109.  Jim asked her if she knew how she figured in STS-109 even though she was not working in MCC.  Marianne asked me to fill in the story. 

Marianne: ‘I was in the Flight Activities Branch and was the book manager for the Post Insertion (nominal) timeline, Launch Day Deorbit, Loss of FES Deorbit, Loss of 2 Freon Loops Deorbit, and ‘If BFS Omit’ procedures. I was responsible for developing and validating all those procedures for STS-1, 2 and 3.’

These were serious and complex checklist procedures for the astronauts to use in flight.  Post-Insertion covered the period just after launch when the crew was turning the Space Shuttle into an operational orbital outpost.  The Deorbit procedures were all failure responses.  Of all the checklists in the pantheon of Space Shuttle procedures, the very hardest to perform was the dreaded ‘Loss of Two Freon Coolant Loops’ power down and deorbit procedure. 

To understand, a short discussion of the Flight Rules is necessary.  By training and practice the adherence to Flight Rules was burned into the culture of Mission Control.  Careful consideration prior to any mission went into the development, review and approval of Flight Rules.  During a flight, it was considered a cardinal sin to break a Flight Rule.

Buried deep in the book, page 18-105 (or as my electronic version has it, page 1947 of 2053 total pages), the operative words are found in a table (background rationale following in italics): 

Space Shuttle Operational Flight Rules  Volume A   All flights

Rule A18-1001 Thermal Go/No-Go Criteria

 FCL (2)         

Ascent Abort if:          Invoke MDF if:     Enter NPLS if: 

2 Lost                                  —-                               1 Lost

Loss of one Freon loop requires a PLS because the next failure (loss of other Freon loop) could result in loss of crew/vehicle.

Nominal ascent is continued so that more time is available to reconfigure for a one Freon loop entry and also because loss of one Freon loop is not an emergency. If both loops are lost, an emergency entry (ascent abort) is required because all cooling to the vehicle is lost.

If both Freon loops are lost, an emergency entry is required because the FC stack temperatures will reach the specification operational limit of 250 deg F within approximately 50 minutes. In addition, the electrolyte will reach the 25 percent operational limit in approximately 75 minutes. At this point, continued operation of the FC’s is questionable. This assumes . . .

To decode:  this Flight Rule required an immediate abort during the launch phase if two Freon Coolant Loops fail:  either Return to Launch Site, Trans-Atlantic Abort, or Abort Once Around depending on when the failures occurred.  Loss of only one FCL during launch did not require an abort but entered into the next step.

During the ‘on orbit’ phase of the mission, the failure of one of the two freon loops would result in ending the mission, planning a landing at the Next Planned Landing Site (NPLS) which by definition was within 24 hours.  A PLS landing was always to one of our three primary sites:  KSC’s Shuttle Landing Facility in Florida, the Edwards Air Force Base in California, or the White Sands Space Harbor in New Mexico.  Weather and timing would determine which of the three to use.  NPLS minimized the time exposure to the failure of the remaining good loop balanced with the safety of returning to one of the best landing sites.  A third category for some failures involved something called a Minimum Duration Flight but that was not an option for this equipment. 

As an aside, if both of the freon loops were to fail while on orbit, perhaps while waiting for NPLS, other rules mandated an emergency landing as soon as possible.  ELS sites were identified all around the world but did not have the equipment, long runways, and weather forecasting capability that the PLS sites did.

Since my copy of the Flight Rules dates from late in the program, it documents the history of STS-109. Section 18 contains a definition of ‘loss of a freon loop’ with 3 full pages of background ‘rationale’ (all in italic) describing what happened on STS-109 and the subsequent engineering analysis.

The Space Shuttle was an electric airplane; nothing happened without electricity.  There was no control, no anything without the electrical power generated by the three fuel cells.  Batteries were non-existent.  If the fuel cells did not make electricity, the shuttle was a rock. 

Fuel cells combine hydrogen and oxygen to produce electricity, water, and lots of heat.  That heat had to be removed to condense the water vapor in the fuel cells so it could be removed.  If the water was not removed, the fuel cells would ‘flood’ and the chemical process would stop working, electrical generation would cease.  The Freon Coolant Loops (FCL) circulated freon as a fluid to collect the heat generated in various parts of the orbiter and transport it to the radiators or the flash evaporators where the heat was dissipated out into space.  For redundancy the orbiter had been designed with two loops and each of those had two redundant circulating pumps. 

Mission Operations made sure that there was a crew checklist procedure for each and every single item that could break or otherwise fail on the orbiter.  Starting before the first Space Shuttle flight, the Mission Control team built step-by-step procedures which were documented, tested, practiced, and validated.  Which is to say, proven to work properly with either engineering tests or rigorous numerical analysis. 

In a very few cases, there were procedures written for two failures.  Since the loss of both freon loops could be catastrophic in a very short time, quick but complex action had to be taken by the crew.  This was one of the few checklist procedures to address two like-systems failures.  Marianne and a host of other folks worked diligently to provide a way out of that terrible situation.  The Loss of Two Freon Loops procedure required powering down much of the electrical equipment on the orbiter to both conserve electricity and reduce the heat generated which had to be removed.  The checklist was extremely complex, time consuming, and – worst of all – attempts to validate it were unsuccessful. In other words, working the checklist completely ‘right’ was unlikely to succeed. The probability of LOCV was high. 

LOCV – Loss Of Crew and Vehicle.

That is all background to what happened on March 1, 2002. 

STS-109 was a mission to service and repair the Hubble Space Telescope.  The crew and Mission Control team were well trained, excited about the mission, and dedicated to leaving the Hubble in perfect condition.  The Hubble Space Telescope Operations team was anxious to get their instrument fixed.  Prelaunch had been difficult with launch scrubs due to weather and technical issues.  When STS-109 finally left the ground all of us were pleased.

Ascent Flight Director for the mission was my good friend and colleague John Shannon.  The Lead Flight Director was Bryan Austin.  I was assigned to be the Mission Operations Director.  This was a replay of the team on STS-93, when I got launch fever.  I was determined not to fall into that again.  See https://waynehale.wordpress.com/2013/10/31/keeping-eileen-on-the-ground-part-ii-or-how-i-got-launch-fever/

My position as MOD was to coordinate with the other members of the Mission Management Team.  During the countdown and launch, everybody on the MMT except the MOD was in the Firing Room at the Launch Control Center in Florida.  The MMT included the Space Shuttle Program Manager, the JSC, KSC, MSFC, and SSC Center Directors, the Orbiter Project Managers (and project managers of all the other shuttle elements), the Head of the Astronaut Office, the Chief of Space Flight Safety, and almost all the other senior managers in the Space Shuttle Program.  The MMT was charged with making the most important decisions, if time were available, regarding any Space Shuttle flight. The MMT was the only body that was allowed, after deliberation, to change a Flight Rule.

When the Public Affairs Officer refers to ‘mission managers’ he means the MMT.  The MOD is not authorized to act without their direction.  Flight Rules are always to be followed unless the MMT rules otherwise. 

Since the countdown had gone so well, and the launch had been delayed, the MMT was really anxious to get home – back to JSC, MSFC, or SSC – as soon as the launch was over.  After nominal cutoff of the main engines (MECO), the management team had few short speeches, took part in the ceremony of the beans and cornbread, and quickly headed to the Shuttle Landing Facility to board the Gulfstream II management aircraft for the flights home.  Very limited to no communication was available while they were in flight.

In short, those of us in Mission Control were without senior leadership direction for those hours. 

Mission Control never considered ‘ascent’ to be over until after the OMS-2 burn put the orbiter into a stable, non-re-entry orbit, and completed various other critical tasks.  Among those required was closing the ET umbilical doors on the belly of the orbiter; changing the onboard computer system from launch to on-orbit software configuration; opening the payload bay doors and establishing freon loop cooling through the radiators; checking out the star tracker navigation system.  When all those items were completed, the crew was given a ‘go for orbit ops’.  Their first step after that was usually to get out of the bulky launch/entry pressure suits, activate the toilet, and start putting away the chairs on the middeck.

Sometime after MECO, sometime after the MMT got on the airplanes, but before getting a ‘go for orbit ops’, the EECOM (Environmental Electrical, Consumables Manager) spoke up.  Responsible for the cooling on the orbiter, he pointed out that one of the freon coolant loops was not operating at full flow.  The flowmeter in Freon Coolant Loop #1 was showing a flow of only 200 lbs./hour.

The failure limit defined in the Flight Rules was anything less than 211 lbs./hour. 

Technically, legally, analytically, FCL #1 was considered failed. 

Things got very quiet in the Flight Control Room. 

We all knew what that meant. 

At that point, theoretically, the Flight Director should declare a First Day PLS (Planned Landing) and the crew should start working the procedures to land at Edwards AFB on orbit 3.  The timing of the discussion made that dicey; starting down that path would have required a rush job to be ready to retrofire in about 90 minutes.  Also, theoretically, the crew should be directed to perform the Loss of One Freon Coolant Loop power down which was long and involved turning off quite a bit of the redundant equipment. That would leave the vehicle open to other failures. 

The Ascent Flight Director started doing what any good FD will do – asking a lot of questions of the EECOM.  It was the EECOM’s opinion that the Flight Rule was ‘conservative’, the flow rate was just below the limit, and there was enough flow to at least consider continuing.  Flight strongly wanted to get to a stable situation and sort options out. 

There was a short discussion with the crew about potential power down.  They were told to pull out the loss of one freon coolant loop checklist and review it, but take no action just yet.

Here is the crux of the situation:  if the other freon loop – the good one – were to fail, quit, leak, whatever; would the questionable freon loop provide enough cooling to avoid the dreaded 2 Freon Coolant Loop procedure? 

Maybe.

As John remembers it: “Definitely one of those “is it failed or not” cases and of course being stable on-orbit while you figure things out is not a bad idea.” 

The Flight Director turned around and leaned over the MOD console.  John looked at me and said: ‘better tell the MMT’.  But I couldn’t.  They were in the air. 

A decision was required. 

I punted.  I asked John what he recommended.  He was inclined to continue on rather than terminate.  I told him I concurred and the flight should continue.

Later on, I did get to have a long conversation with the MMT.  Much engineering analysis was turned on and worked on very hard during the entire mission. 

FCL #1 never regained full flow during flight.  So much for ‘Both cooling loops performed normally on de-orbit and landing.’

In the end, all the analysis indicated we made the right decision.  As John recently discussed: “This would be a good case for why you have a flight control team instead of just programming the flight rules into a computer. Human judgment and risk trades are critical to spaceflight operations.”

Indeed. 

In a pinch, the low-flowing freon loop would have provided just enough cooling by itself, with an appropriate powerdown, to avoid disaster.

But that does not change the fact that we broke the Flight rule that day.

Weeks after the flight, after all the engineering analysis was complete and double checked, the flight rule was revised.  The new limit at which a FCL is considered failed was 163 lbs./hour, less than the old limit of 211 lbs./hour. New procedures were written and passed validation.  Much work was done in case the situation should ever happened again. 

It never did.

But the decision on STS-109 launch day wasn’t made by ‘the mission managers’.  It was John, EECOM, and me. 

One final change:  when it came my turn to set the rules for the MMT, I added one more step after launch: the MMT had to stay on station at KSC, where there was data and good communications, until after the ‘go for orbit ops’.   

Putting Atlantis At Risk

Warning: A long story that has a lot of technical details – and a moral.

Reflecting on my career as a Space Shuttle Flight Director, there were many difficult decisions to make and sometimes some terrifying results.  Putting peoples’ lives at risk and potentially destroying complex and expensive vehicles paid for by the US taxpayers was a daily possibility for every decision made.  Sometimes the consequences were only clear in hindsight.

Fifteen years in the NASA Space Shuttle Flight Director’s office, I supported missions during the on-orbit phases which was always fun, sometimes frustrating, but only rarely worrisome. 

Prior to launch, during the countdown, all the problems really belonged to the Firing Room team and the Launch Director down in Florida.  They had to prep the vehicle and make sure every little piece was working as it should or the system would scrub for the day.  I had only to think about the weather at the abort landing sites – an interesting problem but almost always clear cut:  Go or No-Go. 

To prepare for the flying part of a shuttle launches, my team and I practiced very hard.  The training team would throw problem after problem into every launch simulation.  (See my earlier post ‘Nexus of Evil’ https://blogs.nasa.gov/waynehalesblog/2010/02/16/post_1266353065166/).  When we could proficiently handle each individual problem, the trainers would throw in combinations of problems, more and more and more.  Until we cried ‘uncle’ and past that.  Until we could respond almost automatically to any number of combinations of multiple situations.  Frequently the training days ended with sweat and pounding hearts as if we had been at the gym instead of sitting comfortably under the air conditioning; the work was mental and sometimes psychological. 

During the launch phase (a bare 8 ½ minutes) there was precious little time to think, the responses had to be conditioned in, automatic almost.  In the end, every one of ‘my’ launches in reality was uneventful with no significant problems.  At the end of a real flight day, I sometimes felt that our training had been wasted.  But we were prepared. 

On the other hand, being the Flight Director for the Entry phase of a Space Shuttle flight was the most nerve-wracking job that I ever had.  Takeoffs are optional, landings are mandatory.  With the shuttle in flight entry and landing was inevitable.  Planning and evaluation of potential opportunities, starting before flight, went on the entire mission.  While the three-team rotation of on-orbit flight controllers worked to make sure that the payloads were deployed, the science experiments were performed, etc., etc., the Entry team and its Flight director would come into MCC every day to plan, evaluate, discuss, and digest all the possibilities. 

Evaluate the situation if anything on the orbiter was broken – rarely.  Talk with the landing site folks at the primary and backup sites to gauge the readiness of equipment and personnel at the runways – regularly.  Spend a lot of time in the weather office looking at forecasts – always.  Did you know that meteorology is an inexact science?  We had the best forecasters in the world, but when the landing was more than a few days away pondering forecasts was an exercise in futility – they always changed. 

As the day of landing grew closer, the forecasts were scrutinized to an increasingly minute degree.  All the senior NASA management felt that they were amateur meteorologist and felt free to provide opinions.  But only one person had to make the decision.  Pressure grew daily.  A lot of time was spent at the coffee pot in the hallway outside mission control debating the options with various astronauts, managers, schedulers, and other interested parties. 

About 3 days before landing, we had to commit to send the convoy team to one of the three sites in the US we guessed to be best.  If Florida, we could use the Shuttle Landing Facility at Kennedy Space Center with its single long runway.  If California, the Edwards AFB/Dryden Flight Research Center where they similarly had a single long paved runway but many other options on Roger’s dry lakebed.  The least used option was New Mexico with the hard-as-concrete gypsum dry lakebed runways at White Sands Space Harbor aka Northrop strip.  We only landing a shuttle at WSSH once and it was always the lowest priority in our thinking, fewest number of personnel, least facilities.

Edwards, like the SLF, had one concrete runway that was twice as wide as any normal big airport featured, twice as long as the typical airport runway, and specially constructed to bear the weight of extremely heavy aircraft landings.  But the concrete runways were singletons:  they pointed one direction.  The problem was that winds could come from any direction; not just straight down the runway but across it.  A crosswind could sometimes exceed the maximum allowable for the shuttle: 12 (later 15) knots (nautical miles per hour).  Without a runway in a different direction, well, the landing site was unusable.  At the SLF, surrounded by the swamp, you had to have the winds just right.

Edwards famously had Rogers Dry Lakebed.  The Air Force annually scribed out a large number of ‘runways’ on the hard surface of the lakebed.  They ‘painted’ with asphalt side stripes, threshold markings, and – for the shuttle – aimpoint markings.  To be considered for a normal landing (not emergency) we also required various indicator lights to be set up on a runway and, for the best change at landing, microwave beam scanning landing system (MSBLS) huts that gave precise angle information to an approaching shuttle.  Unfortunately, at Edwards, there were far more runway options than we had equipment.  Generally, only a couple of the lakebed runways were ‘instrumented’ which put them on the candidate list for places to land. 

Oh, and just one more thing.  Rogers Dry Lake sometimes was not dry.  In most winters the rains filled the lake, not very deep, but enough.  When wet – or even damp – the lakebed surface was too soft to land a shuttle.  The tires would dig in, perhaps snap off, maybe make the vehicle tumble.  So, among the criterion for using runways on the lakebed, the first was they had to be dry and hard.  The Air force provided us with the services of their geologist who use various instruments to test the load bearing characteristics of the lakebed. 

Who would have thought?

Runways are designated by the compass direction they point:  for example, runway 33/15 runs northwest to southeast.  An aircraft on the runway would show its compass heading as 330 degrees (northwest) or if pointed the other way at 150 degrees (southeast). 

STS-37 flew in early April 1991 and Rogers Lakebed was just drying out.  Since the prevailing winds in California blew from west to east, the waters had been pushed to the eastern side of the lakebed and the western side started drying out earlier.  We had some confidence that the most western of the lakebed runways 33/15 might be usable, but the geologist had to do his testing.

Just as an additional consideration, the shuttle program had never used lakebed runway 33/15.  The pilot astronauts took turns in the Shuttle Training Aircraft – which flew the final approach like an orbiter – and spend many hours – days – weeks – practicing landing at the runways likely to be used.  Edwards 33/15 was never on the list.  Nobody had ever trained to land on that runway.  Think about that as a potential problem.  It was not shown in the Flight Maps and Charts book that the commander on orbit could consult and become familiar with. 

As the day of landing approached, it was clear that weather was going to be a problem, as usual.  KSC had No-Go forecasts of rain, low clouds, and wind for several days in a row.  This forecast turned out to be accurate. WSSH similarly had unacceptable landing weather.  Edwards looked really with clear skies and no thunderstorms or rain of any kind in the forecast.  But strong north winds blew almost directly across the concrete runway situated east-west (22/04), far exceeding the crosswind limit.  Only the lakebed runway 33/15 pointed in the right direction, if it was dry and hard enough to use. 

The first or normally planned landing day, when we could not consider using the lakebed runways, I had to make the call to wave off for 24 hours – it was just too bad everywhere.  This was not a hard call.  Its easy when its clear; the hard part comes when its marginal.  Crews generally appreciated a wave off if it was called early – they got a day in orbit to rest from a hard paced mission, look out the windows and take pictures, and generally enjoy being in space.

There were limits on how long the Space Shuttle could stay in orbit:  oxygen, water, food, and the Lithium Hydroxide canisters which removed Carbon Dioxide from the air:  all were limited and being used at a substantial rate.  We needed to land soon.

On the +1 day as we called it, the stakes were higher. 

During the wave off day, the geologist completed his evaluation and reported that the 33/15 runway was hard enough to bear the weight of a shuttle landing.  The ground crews scrambled to move approach lights and other navigational aids – but not the MSBLS – to the end of the runway we might use:  the southern end which would put the wind right on Atlantis’ nose.  They were NOT able to respray the asphalt markings.  Those markings, vital cues for a pilot to make a good landing, had been dimmed by the winter rains.  We sent a seasoned astronaut commander to fly the STA and evaluate – all in all the 33-runway was deemed acceptable. 

Remember, we had never used runway 33 before.  And, as it turned out, never would again. 

No pressure on the Flight Director.  No sirree, no pressure.  Never let them see you sweat. 

On deorbit day April 11 I ordered the crew wake up early enough to try to land at KSC in case the weather cleared.  No such luck.  An hour and a half later would be the first opportunity at Edwards. I spent my time between sitting on the console, pacing back and forth, getting briefings on the STA runs, visiting the weather office in person for updates, and having those ad hoc meetings at the coffee pot. 

Too much coffee was consumed. 

There was only one runway in the continental United States that would be go for a shuttle landing that day.  The wind was right down the runway 12 gust to 18 knots, no crosswind. 

But wait.  It gets more complicated. 

Every hour or so a weather balloon is launched at the landing site and tracked it measure the winds aloft as high as 50,000 ft.  On that day the upper winds were ugly, high and shifting.  Worse, there was a tremendous shear in windspeed and direction just above 10,000 ft.  The measured winds aloft exceeded anything previously analyzed for a shuttle landing.  

No pressure, right.  Don’t let my cardiologist know. 

The Space Shuttle Orbiter is a 100-ton glider that has only one shot at making a successful landing.  After the deorbit burn which occurs an hour before there was no way to delay, no go-around possibility, no second try.  Theoretically possible to ‘redesignate’ to another runway, there wasn’t even an emergency runway with acceptable conditions anywhere near Edwards.  To say that the shuttle would be committed was an understatement. 

During shuttle landings, the autopilot is in control from the start of reentry down to about 70,000 ft.  At about the point the shuttle drops below the speed of sound, the commander takes manual control and flies the rest of the way down to landing – which is only 2 ½ minutes.  Very shortly after taking control, the commander starts a turn to align with the runway.  An imaginary cone in the sky – the Heading Alignment Cone – is mathematically displayed to the commander.  On this approach on April 11, we planned for him to make a right turn of ¾ of a circle – 271 degrees.  Commanders sit on the left of the cockpit so they prefer a left hand turn to see the runway as they turn; we planned for a right hand turn to conserve energy.  This meant the commander he could only look at the instruments for flying cues. 

Every time we got balloon data, the mission control team ran a simulation of an automated landing.  We used the autopilot model because we had no way to predict what a real man-in-the-loop pilot would do.  Use of the autopilot for the final part of entry was not allowed because it had certain well-known limitations, and it would not perform well without a MSBLS – which we did not have installed on runway 33. 

But doing a computer simulation using the autopilot all the way to landing gave us critical information about the affects of the upper air winds on the one and only landing attempt that the commander could make. 

After rolling off the HAC and onto the final approach at about 12,000 feet, the shuttle dives to gain speed, using one of the ‘aimpoints’ painted on the lakebed the commander puts the nose down to a glideslope of 19 degrees – 6 times steeper than commercial airlines.  Gaining speed to about 290 knots until the commander starts his pull up at about 2,000 feet to set up for landing.    

Normally the commander is diving at the ‘nominal aimpoint’ – in this case a large black triangle painted in asphalt on the lakebed surface about 7,500 feet from the runway threshold.  Speed – which is synonymous with energy at this point – is controlled by the rudder/speedbrake on the tail of the orbiter.  The automated system checks to see if there is enough energy to make the landing at the criteria desired – 2500 feet past the threshold at 195 kts.  The automated system will open the speedbrake if the energy is too high and close it down if the energy is low:  15% is the minimum setting if the energy is low; it is desirable to approach with the ‘boards’ open to about 20 to 30%. 

If the energy is evaluated to be low from the balloon data, the commander will change to the ‘close in aimpoint’ which is 6,500 feet from the threshold, thus picking up 1000 feet of margin to touchdown.  In even lower energy situations, the commander can stretch the landing by going as low as 185 kts – which gains another 1000 feet in equivalent touchdown distance.  All this to try to get to our normal touchdown point 2,500 feet past the threshold.

The simulations predicted that the landing would be ‘low energy’.  Using the forecast winds the simulation needed the close-in aimpoint and the speedbrakes were full closed all the way down yielding a touchdown distance of 2380 feet past the threshold at 195 knots.  All ‘go’ by the flight rules but with yellow flags indicating caution all around. 

The only options left for such a low energy day would be allowing the touchdown distance to go as short as 1,100 feet past the threshold – considered the minimum for safety – and holding off landing for another 10 knots of airspeed to 185 knots.  Those were it.  The coin of the realm in these situations is distance past the threshold and our purse was almost empty. 

Using the data from the first balloon – at 4 hours prior to touchdown – the situation had deteriorated.  Again, using close in aimpoint, closed boards, the touchdown distance was predicted to be 1800 feet at 195 knots.  Still ‘go’ but getting a brighter shade of yellow. 

My blood pressure ticked up. 

At Edwards the Shuttle training aircraft was flying mimicking a shuttle landing.  The first approach the aircraft would not go into simulated shuttle mode so no touchdown data came in.  The experienced shuttle commander at the controls reported that the turbulence was not an issue and it appeared to be a good day to land. 

Using the L-2.5-hour balloon data the touchdown prediction had deteriorated to 1300 feet past the threshold – still ‘go’ but close to the 1100 foot minimum required by flight rules. We reported this result to the crew.

On his last dive at the runway, the STA pilot reported simulated touchdown at 1600 feet past the threshold which gave me some very slight sense of relief. 

Time to decide.  It always gets really quiet in Mission Control and all the senior managers had stopped giving advice.  It was my call. 

Knowing that the only runway on the continent where the shuttle could safely land had marginal energy conditions, I gave a “Go for Deorbit” to the Capcom to pass to the crew.  I knew that we could have safely waved off for a second day, but tomorrow’s weather forecasts were about the same.

A few minutes later the crew executed the deorbit burn and we were committed.  COMMITTED. 

While the shuttle was on its way down, we got our last update from the L-1 hour balloon data, touchdown 1800 feet past the threshold. Better. I began breathing again.

As the shuttle approached the HAC, the commander delayed by a fraction of a second turning onto the cone, something that was not at all uncommon.  However, flying outside the HAC increased the distance the glider would have to fly and thus made the energy situation even worse. 

Unbeknownst to us there was another phenomenon at play; the wind shear altitude had dropped from 13,000 feet to just under 10,000 feet.  This change in the wind speed and direction effectively robbed the orbiter of some energy. 

In the shuttle software, at 10,000 feet, there is a check of speed and distance to go which, if it is good, the guidance scheme transitions to what is called ‘approach and land’.  When the wind shear was at 13,000 feet the software detected the energy difference and directed the commander to fly more aggressively to make up energy and the A/L transition occurred on time.  With the wind shear below 10,000 feet, the A/L transition criteria was not acceptable and the guidance did not direct the commander to fly as aggressively. Rolling out on final the wind shear delayed the software transition which then did not indicate to the commander he should fly a more aggressive high lift/drag profile.

Coming down the energy situation just got worse.  The commander could see it, but we in MCC could not.  He pulled up at the right point, lowered the gear at the right point, but it was a long way to the threshold.  Too long.  The speedbrakes were closed and at the point the only option at that point was to hold off landing by holding the nose higher as the speed decreased.  Touchdown finally occurred at 166 knots – the second slowest in shuttle history.  The main gear slapped the lakebed surface some 623 feet short of the threshold of the runway.

Disaster?

We got lucky.  Every runway the shuttle was allowed to use has a 1000-foot underrun – safe space before the threshold.  On the lakebed where the stripes were rather arbitrary there were miles of safe landing space. 

It was a safe landing. 

I was unaware at how bad it had been.

It was several hours later when I got the call.  The first apology I made was to the commander for putting him in that position. 

We had been lucky when they paid us to be smart. 

We spent the next six weeks going painstakingly through the situation. 

I thought I was going to get fired for putting the crew at risk.  But the senior folks decided I had learned a valuable lesson. We had all learned a valuable lesson. 

We made many changes to the entry planning process.  One was to make sure the STA pilot reported if the A/L transition was delayed (it had been delayed during his dives but it was not reported to MCC).  Another was to steepen the glideslope by an additional 2 degrees to increase maximum speed before pullout to 300 kts to give more energy.

We trained the pilots to fly the HAC turn closer, although that had mixed success; that is partly what you get with a human in the loop.

But most important was to re-emphasize to Entry Flight Directors to use extreme caution when committing the orbiter to a landing.

It is not a job for the fainthearted. 

The next time we fly a winged vehicle down from orbit, I hope they don’t have to learn that lesson again. 

Terminal Count

Probably the most heart stopping moments in spaceflight occur at the final stages of a countdown to liftoff.  Will it go or not?  What happens once the engines start?  Success or failure or just wait for another day?  I lived through a number of shuttle launches – and launch attempts – and every time I watch a rocket launch – of any kind – when the clock ticks down to the final minute my heart starts racing.

For the Space Shuttle there were a series of documents which detailed how launch operations were conducted.  The most famous was S0007 (pronounced ‘Sue Seven’ or sometimes ‘S triple balls seven’).  The entire document came in five volumes.  In the days when we worked in paper, it took five 2-inch-thick binders to hold it all.  Every step was numbered and the responsible party was named for each step.  Most of my career was “Houston Flight” and I would answer to the NASA Test Director or the NASA Launch Director or the NASA Operations Manager on their communications ‘loops’ as required. 

In my Flight Director reference book was a copy – shown above – of Figure 13-3 ‘RSLS and GLS Interaction T-38 Seconds to T-0’.  Chockablock with important stuff because a lot happened in those last seconds.  At the final stages, it was all on automatic with the computers in control.

Two programs, the GLS (Ground Launch Sequencer) and the onboard RSLS (Redundant Set Launch Sequencer) played the final duet.  People were just observers, along for the ride.  There were folks that could stop the countdown if they found necessary, which happened several times for the odd items that were not monitored or controlled by the GLS and RSLS.  Memorably were the launch scrubs caused by the hazardous gas detection system (Haz Gas).  Almost everything required by the Launch Commit Criteria was automatically monitored. 

Onboard the Space Shuttle there were five General Purpose Computers.  For the launch phase, computer #5 was running the Backup Flight System.  The BFS was there to take over if there was a total failure of the ‘redundant set’.  The BFS did not have the command capability to launch the shuttle and was really only in listen mode prior to liftoff.  Computers 1 through 4 were all running the same software at the same time in lock step; they comprised the ‘Redundant Set’.  The RSLS was only one of many programs running in the ‘redundant set’ computers during the prelaunch phase also known as Major Mode 101.  To continue to launch – and fly safely – all four computers had to agree.  If any one got out of sync or gave an incorrect command, or failed to listen to the others, the RSLS would detect that and issue a hold so that no launch would occur. 

On the chart the top half details what the RSLS is doing and the bottom half details what the GLS is doing.  Commands could be given once or ‘continuously’ (every computer cycle) for a given period of time.  Likewise, telemetry of critical items could be checked (verified) once or ‘continuously’ (CFVY) every computer cycle of 40 milliseconds for a specified period of time.

Time across the bottom starting at T-50 seconds with the first item:  a command that the GLS automatically commanded the big Liquid Oxygen and Liquid Hydrogen Fill & Drain Valves to close.  The ET should be completely full of propellant and no more would be added.  Looking a little later on the chart at T-34 seconds, the GLS would verify that those valves actually closed.  As always for any verification that failed, an automatic hold would be issued. 

Getting back to the time scale at the bottom, in big bold letters at T-31 seconds was the notation: “LAST AVAILABLE HOLD POINT”.  To this day, whenever I am watching the countdown of any vehicle, whether it matters to that system or not, my heartrate picks up significantly at T-31 seconds.  Like Pavlov’s dog I am conditioned to respond. 

There is also an interesting note in the box: “Onboard RSLS TBO Clock Stops Decrementing at T-6.6 sec; GND Clock continues.”  The RSLS countdown time displayed to the crew notoriously stopped at the command for main engine start.  After that point things either happened – or not -so quickly that human response did not come into play.

In the RSLS section is the long box “CVFY NO SSME 1,2, or 3 PAD DATA PATH FAIL, CHANNEL FAIL, or CONTROL FAILURE (EH, HL, OR MCF AND LIMIT EXCEED)” Each Space Shuttle Main Engine had two computers – prime and backup – which controlled the functions of the engine.  The Redundant Set had to have good data and command links with the SSME controllers – no data path fails.  Each SSME controller had to report that there were no failures in the control channels to its engine valves, and no detected major component failures (MCF), no redline exceedances either high or low on any engine temperature or pressure.  That is a lot to check on every 40 milliseconds from T-20 minutes all the way to T-0. 

Next, we will look at the detailed commands and verifications second by second. 

The GLS deactivates the SRB joint heaters – which were added after Challenger – at T-50 seconds.  At T-40 seconds the GLS cuts ground power to the Space Shuttle so that all the electricity must come from the onboard fuel cells.  Also, at T-40 seconds the GLS verifies that the Gaseous Vent Arm (GVA or the ‘beanie cap’) is fully retracted out of the way.  That beanie cap is on a long arm that stretches out over the nose of the External Tank to remove any vapors.  On the very last shuttle launch, STS-135, the indicator that the GVA was out of the way failed so we had a scary – but short – hold while people in the firing room manually confirmed it was out of the way. 

At T-31 we pass the last hold point – anything after that will be a launch scrub.  At T-31 seconds the GLS sends “LPS Go for auto sequence start”.  Launch Processing System is a generic term for the entirety of the ground system; the ‘auto sequence’ is the RSLS which takes precedence at that point. 

At T-30 seconds the GLS commands the hydraulic power units in the base of the solid rocket boosters to get ready to start; the actual start command comes at T-28 seconds. 

The RSLS, at T-28 seconds is doing a one-time check to see if it received the LPS Go for auto sequence start.  At T-27 seconds the RSLS starts another software sequence to open the orbiter vent doors.  Why?  As the shuttle launches, the outside air pressure decreases with altitude; there are motor actuated doors all along the sides of the shuttle that open to allow the pressure to equalize.  Otherwise, the structure would pop at some point.  Not be good.  Prior to launch, all the orbiter cavities are flooded with dry nitrogen gas to prevent any flammable leaks from catching fire.  Opening the vent doors too early would allow oxygen to get inside and the fire hazard would increase.  T-27 seconds allows all the doors to be fully opened if both electrical motors on each door function properly, operating on just one of the two redundant motors takes longer to open the doors.  If any motor doesn’t work, that door may not be fully open at liftoff and there was a huge debate early in the program about whether that would be OK.  Down at T-7 seconds the RSLS checks to see if all the vent doors are open.  We decided to scrub the launch if any vent door motor failed rather than start opening the doors before the last hold point at T-31 seconds and risk oxygen intrusion during a hold.  More than one flight held at T-31 seconds, there was never a vent door motor failure, so that proved to be a good decision that avoided potential fire hazards. 

At T-26 seconds the GLS commands the Liquid Hydrogen High Point Bleed valve to closed.  It would be bad to lift off with a leak in the LH2 system.   The GLS verifies that valve is closed at T-12 seconds. 

Then there is a blessed five seconds of calm.  It is the last quiet period before everything starts happening, seemingly all at once.

At T-21 seconds the GLS commands the SRB gimbal test.  With the hydraulics pressured up, those big nozzles are swiveled back and forth to make sure they work properly.  Starting at the same time the GLS begins continuously verifying that the SRB hydraulic turbines are running at the proper speed.  There are two turbines in each booster for full redundancy in flight but we decided not to launch unless both were working properly.  I don’t ever recall an SRB hydraulic unit failing in flight (they only run for about 2 and ½ minutes) but when I was the Shuttle Program Manager and found out that there had never been a test with only one turbine running, I mandated that one of the ground test firings in Utah shut down one of the two turbines and measure the gimbal response.  It worked fine.  That test only cost $2 million – which is a story for another day. 

At T-18 seconds the RSLS commands the safe and arm devices – which prevent an inadvertent pyrotechnic event – to arm.  This is for the SRB ignition and the ground umbilical release.  Hazardous phase truly initiated at this point. 

By T-16 seconds the SRB gimbal test should be complete and the nozzles back down at the null position for liftoff so the GLS starts continuously verifying that position down to T-0.  Also, at T-16 seconds the GLS activates the water deluge on the pad, the so called ‘sound suppression’ system.  This has different modes and the first activation is the ‘pre liftoff’ mode which is a trickle compared with what happens immediately after liftoff.

At T-15 seconds the RSLS begins continuously checking that all the pyrotechnic initiator capacitors are fully charged and ready to function.  Also starting at T-15 seconds the RSLS starts continuously checking that all the onboard computers have good data coming in from all over the vehicle (no MDM Return Word bypass). 

At T-12.5 seconds:  the RSLS starts continuously commanding valves in the Liquid Oxygen system that recirculate fluid to open, this continues every 40 milliseconds down to engine start time.  At T-9.5 seconds the RSLS checks to see if those valves are open. 

At T-12 seconds the GLS is also busy; it commands valves to close to terminate the helium fill going to the orbiter and the GLS also does a one-time check to verify that the rudder/speedbrake is in the launch position, the GLS locks down its command system to the SRBs and immediately removes any inhibits that were set.

At T-11 seconds the RSLS commands another software program in the onboard computers to start:  navigation.  The crew sees this as the ‘Eight ball’ display showing attitude goes to vertical.  At the same time the RSLS commands the main engine throttle settings to 100%.  The engines have not started yet, but when they do, they will aim to run at 100% of the rated power level.  For almost all of the shuttle flights ascent flight used 104% (or 104.5% for later flights) but the pad was only certified to 100% so the command to throttle up to 104% was one of the first things that the computers did after liftoff.

At T-10 seconds the GLS fires the ‘sparklers’ at the base of the launch pad to burn off any stray hydrogen vapors before the engines start.  Scrubbing at this point incurs about a week effort to replace those items.

Also, at T-10 seconds the GLS issues the ‘go for main engine start’ discrete.  We did a long study once that showed this was the last critical action of the GLS; a total failure of the ground computing system after this would not stop the shuttle from launching itself.  The RSLS is totally in charge now.

At T-9.5 seconds the RSLS sends ‘start enable’ to the main engines – get ready! At that same point the RSLS checks one time to see if each main engine indicates it is ready.  This is a complicated process that requires terminating liquid oxygen cooling to the engines – which occurred back at T-50 seconds when the valves were closed or really even earlier than this chart when LOX replenish was terminated.  The engines must be chilled to the right temperature for a smooth start and during replenish operations the engines are actually too cold.  When replenish ends, LOX from the External tank starts flowing into the cooling channels; this LOX is in the big 17-inch pipe coming down the outside of the ET.  Since it is outside, the LOX is actually a tad warmer than the LOX which was coming in from the ground tanks.  This allows the engines to warm up just enough so that the temperatures are in the ‘start box’.  When the main engine controller senses the right temperature, it issues the ‘engine ready’ discrete.  This what the RSLS is looking for.  On several flights where we were holding at T-31 seconds (after replenish was terminated) the engine temperature crept higher – out of the ‘start box’ and the engine ready discrete went away.  SCRUB!

Also, at T-9.5 seconds the RSLS starts continuously commanding the Liquid Hydrogen prevalves for each engine to open.  If they are closed, no fuel gets to the engine.  Continuously commanding those valves to the open position every 40 milliseconds until the engine starts.

A fraction later, at T-9.4 seconds the RSLS starts continuously commanding the Liquid Oxygen overboard bleed valves to close and continues this until the main engines start.  Remember the GLS had closed the equivalent on the liquid hydrogen side at T-26 seconds, an eternity ago.

At T-9 seconds the GLS deactivates the Liquid Hydrogen recirculation pumps and a second later at T-8 seconds the GLS shuts off its capability to command anything on the orbiter. 

At T-7 seconds the RSLS verifies that it has received the LPS go for engine start, and verifies that the LOX recirc valves are open, and starts continuous verification – for 0.4 seconds – that the engines are all in ‘engine ready’ mode, checks the vent door positions, and the prevalve positions and then the big show starts.

At T-6.6 seconds the RSLS issues the start commands to each engine:  engine 3, engine 2, engine 1 with 120 millisecond staggers.  The engine gimbals are commanded to override to prevent failures during the engine start transients.  From now to T-0 the RSLS will verify that no main engine controller has issued a ‘shutdown mode’ or ‘post shutdown mode’ discrete.  All the action turns to the main engine controllers which execute a tightly choreographed sequence of valve operations to safely start each engine and bring it up to 100% throttle.  Describing that sequence would take much longer than this paper. 

The forlorn GLS issues its last command to shut down the ground cooling units at T-6 seconds. 

At T-2 seconds the RSLS starts continuously checking to verify each main engine is reporting that it is running at greater than 90% throttle setting; there after the RSLS resets the engine gimbal commands to allow steering after launch to happen

At T-0 the last RSLS commands go out – fire the umbilical release pyros, fire the hold down post pyros, fire the ET vent arm disconnect pyros, and all in the same millisecond command window, fire the SRB ignition pyros.  Then reset all the controllers (Master Event Controllers – MEC) for the pyros.

As a flight director, I knew the next step – coming less than 40 milliseconds seconds after all those commands:  The onboard redundant set computer programs moded to the flight – first stage – phase; Major Mode 102.  The RSLS was done and turned off. 

A young Flight Director on console with the SOO7 document open on the console shelf – red binder.

The Most Important Thing

I have a good friend who flew helicopters in combat for the US Army.

His number one rule is “Don’t Do Stupid.” 

I think this is a good rule to remember. 

We have too much at stake in space exploration – among other fields – to allow people to do ‘stupid’. 

Maybe leaving critical bolts out of an aircraft fuselage.  That would be stupid.

Maybe ignoring foam coming off the rocket at launch.  That would be stupid.

Not checking your flight software in an integrated test.  That would be stupid.

Running out of propellant because your spacecraft control system fired the engines so many times they began to leak.  That would be stupid.

Flying a crew around the Moon when the spacecraft heatshield is suspect.  That would be stupid. 

Hey, the list is a long one. 

And I have been on the bus trip to Abilene along with other well-meaning but totally foolish people.   I’ve actually done stupid. Trust me, it does not turn out well. 

In the field of rocketry and spaceflight the energies are too large and the craft are necessarily frail that careful attention at every step is required. 

So, when listening to the various NASA officials in a press conference recently say over and over again that “Safety is Number 1” you may be surprised to find that I was totally irritated by that repetition.

You see I was schooled by experts in risk.  One of them, James Cameron, gave us a lesson that I will never forget.  He said:

“It’s absolutely necessary to use all of our accumulated knowledge to be as safe as possible, but safety is not the most important thing.  I know this sounds like heresy, but it’s a truth that must be embraced in order to do exploration: the most important thing is to actually go.”

If you want to be perfectly safe, you don’t go exploring.  I don’t think we should be that safe.

But don’t do stupid. 

Flight Delay

A few days ago, I wrote about ‘Protecting the Bird Sanctuary’.  I hope you read it. 

Because I told you that story just to be able to tell this one. 

I put this story in the ‘mostly true’ category – I was the Ascent Flight Director and can attest to my end of it.  But the real story belongs to NASA Astronaut Richard N. “Dick” Richards. 

It all started after Dick was kicked upstairs to a management job, his space flight flying days were over.  In its inestimable wisdom, the NASA senior management decided that a flown astronaut needed to make an appearance in American Samoa to encourage the children there to study science, mathematics, and eventually engineering.  Every astronaut (and many a Flight Director) has been the recipient of just such a public appearance assignment. 

Having completed the requisite talks with schoolchildren and teachers, Dick proceeded to Pago Pago International airport to take the flight to Honolulu and home.  But . . .  the flight was delayed.

By the Space Shuttle.   

How?

This time the shuttle had a very heavy payload which limited its trajectory.  It did not have a rendezvous mission so the launch window was long (2.5 hours or so), and the planned orbital inclination was low – which meant that the ET disposal area was in the central Pacific.

NASA had issued the Notice to Airmen (NOTAM) showing where the ET debris would fall and bracketed the times for the full launch window. 

An immutable principle for a shuttle launch required a safe place to land if one of the three main engines shut down early.  For a good portion of the powered ascent, this meant an abort landing at a designated site on the west coast of Africa or in Spain.  At a Trans-Atlantic Abort (TAL) landing site. 

For this particular flight we only had one TAL site.  Or rather, we only had one TAL site where all the navigation aids, convoy team, lights, etc., were in place.  The other potential TAL sites existed, but were not augmented with the proper resources.  Several of these other runways were out of reach given the trajectory and performance limitations on this flight.  At one the runway was being repaved:  you really would not want to try to land there!  On this day there was only one choice. 

Read the rules – I have attached a few pertinent extracts of the Shuttle Flight Rules.  Reading them, one might ask, ‘how did you ever get to launch a shuttle?’ 

As Mother Nature would have it, that one TAL site had unacceptable weather.  Not horrific, but clearly in violation of the Flight Rules.  In the rule excepts attached, I did not include the Landing Site Weather rule, A2-6 if you want to look it up.  It is the single longest rule in the book (21 pages) and without a doubt the most convoluted. 

The brilliant weather forecasters from the National Weather Service who staffed the Spaceflight Meteorology Group were the best in the business.  They could to make a forecast for a specific spot at a specific time with better than 95% accuracy.  On this day, the Weather officer held out hope that waiting until later in the launch window conditions might improve. 

We were all suited up and ready to play as it were, and had nowhere else to go, so we waited.  The Shuttle systems were all running perfectly, no concerns; the tank was loaded and topped off, no concerns; the weather in Florida was about as perfect as it could get there. 

Meanwhile, back in the Pago Pago airport, the passengers were restless.  The airline Captain for the flight came into the waiting room and explained about the Shuttle External Tank expected to fall directly in their flight path.  He had observed – from a great distance – the breakup of an earlier flight.  That convinced him not to take any chances!

So, the passengers fretted, and waited, and talked.  One of the passengers, in conversation with Dick Richards, asked if Dick could find out how long they might have to wait.  Dick, having been a Capcom on several flights, had the phone number for Mission Control so he called.  His fellow astronaut serving as the Capcom summarized the news about the weather.

Dick got the passengers together and explained it to them at some length.  When he was done, one of them summarized it this way: 

“Here we are in the South Pacific waiting to take off because our flight is delayed waiting for the Space Shuttle to launch from Florida, and the Space Shuttle in Florida is waiting to launch due to bad weather in Africa.”

Yep, that is about it.

I really don’t remember if we waited to the end of the window to launch or scrub.  I guess I need to check that.  I do know that I – the Ascent Flight Director is responsible for the Go/No-Go for Abort landing weather – I was totally unaware of the crowd in Pago Pago waiting on my decision.  Not that it would have made any difference. 

====================================================================

With apologies to the Disney people:

It’s a small world after all
It’s a small world after all
It’s a small world after all
It’s a small, small world

There is just one moon
And one golden sun
And a smile means
Friendship to ev’ryone
Though the mountains divide
And the oceans are wide
It’s a small world after all

It’s a world of laughter
A world of tears
It’s a world of hopes
And a world of fears
There’s so much that we share
That it’s time we’re aware
It’s a small world after all

It’s a small world after all
It’s a small world after all
It’s a small world after all
It’s a small, small world

=====================================================================================

SPACE SHUTTLE FLIGHT RULE A2-1 PRELAUNCH GO/NO-GO REQUIREMENTS

LAUNCH WILL BE NO-GO IF THE FOLLOWING CONDITIONS ARE NOT SATISFIED:

F. ACCEPTABLE LANDING CONDITIONS FOR REQUIRED ABORT LANDING SITES, REF. RULE {A2-2}, ABORT LANDING SITE REQUIREMENTS. IF AN ABORT LANDING SITE IS NOT MANDATORY, ACCEPTABLE LANDING CONDITIONS ARE STILL HIGHLY DESIRABLE AT THAT SITE. CONDITIONS WHICH MUST BE MET ARE:

1. NO VIOLATIONS OF THE LANDING SITE WEATHER CRITERIA (REF. RULES {A4-2}, LANDING SITE CONDITIONS, AND {A2-6} LANDING SITE WEATHER CRITERIA).

2. FOR ALL LANDINGS BUT FIRST OR SECOND DAY PLS, NO VIOLATION OF ENTRY ANALYSIS LIMITS INCLUDING:

a. APPROACH AND LAND TRANSITION (REF. RULE {A4-156}, HAC SELECTION CRITERIA)

b. NORMALIZED TOUCHDOWN DISTANCE (REF. RULE {A4-110}, AIMPOINT, EVALUATION VELOCITY, AND SHORT FIELD SELECTION)

c. TIRE SPEED/ROLLOUT/BRAKING LIMITS (REF. RULE {A4-108}, TIRE SPEED, BRAKING, AND ROLLOUT REQUIREMENTS)

d. NZ AND Q-BAR CONSTRAINTS (REF. RULE {A4-207}, ENTRY LIMITS)

3. ACCEPTABLE GROUND NAVAIDS (REF. RULES {A3-201}, TACAN REDUNDANCY REQUIREMENTS AND ALTERNATE TACAN SELECTION PHILOSOPHY; {A3-202}, MLS; AND {A2-6}, LANDING SITE WEATHER CRITERIA).  FOR THREE STRING GPS FLIGHTS ONLY, AN ACCEPTABLE GPS CONSTELLATION CONFIGURATION IS REQUIRED (REF. RULE {A3-204}, GPS CONSTELLATION).

4. ACCEPTABLE LANDING AND VISUAL AIDS (REF. RULE {A3-203}, LANDING AID REQUIREMENTS).

5. ACCEPTABLE CURRENT AND PREDICTED RUNWAY CONDITIONS (REF. RULE {A4-111}, RUNWAY ACCEPTABILITY CONDITIONS).

6. ACCEPTABLE COMMUNICATIONS FROM THE MCC TO THE REQUIRED ASCENT ABORT SITES. (REF. RULES {A3-52E}, MCC INTERNAL VOICE, AND {A3-156}, MCC/ASCENT ABORT SITE INTERFACE).

7. DAYLIGHT LANDING UNLESS CREW SPECIFICALLY TRAINED FOR NIGHT LANDING AND RELATED LANDING AIDS FUNCTIONAL (REF. PARAGRAPHS 3 AND 4 ABOVE). EXCEPTIONS ARE ALLOWED FOR AOA (IF NOT REQUIRED) AND FIRST DAY PLS.

SPACE SHUTTLE FLIGHT RULE A2-2 ABORT LANDING SITE REQUIREMENTS

CONTINUOUS SINGLE SSME OUT INTACT ABORT COVERAGE IS REQUIRED THROUGHOUT POWERED FLIGHT. PRELAUNCH ANALYSIS MUST DEMONSTRATE THIS CONTINUOUS SINGLE SSME OUT INTACT ABORT COVERAGE EXISTS OR LAUNCH IS NO GO PER RULE {A2-1}, PRELAUNCH GO/NO-GO REQUIREMENTS. THIS CONTINUOUS SINGLE SSME OUT INTACT ABORT COVERAGE MUST INCLUDE SYSTEMS FAILURE TOLERANCE AND BOTH SYSTEMS AND ENVIRONMENTAL PERFORMANCE DISPERSIONS AS REQUIRED. THIS CONTINUOUS SINGLE ENGINE OUT INTACT ABORT COVERAGE MUST BE TO A FULLY ACCEPTABLE LANDING SITE THROUGHOUT POWERED FLIGHT. THERE CAN BE NO PRELAUNCH PREDICTED SINGLE SSME OUT ABORT GAP BETWEEN LAST RTLS AND FIRST TAL NOR BETWEEN LAST RTLS OR LAST INTACT TAL CAPABILITY (TO A GO TAL SITE) AND FIRST PRESS CAPABILITY, AND NO GAP IN SINGLE SSME OUT PROTECTION FOLLOWING FIRST PRESS CAPABILITY NO MATTER WHICH OF THE FOLLOWING CRITERIA IS USED TO DEFINE PRESS CAPABILITY.

A. AN RTLS LANDING SITE IS REQUIRED FOR LAUNCH COMMIT.

B. A TAL LANDING SITE IS REQUIRED FOR LAUNCH COMMIT UNLESS THERE IS RTLS AND TWO ENGINE PRESS CAPABILITY OVERLAP. THEN A TAL LANDING SITE IS HIGHLY DESIRABLE.

IF AN ASCENT INTACT ABORT LANDING SITE IS REQUIRED, THEN ALL LANDING SITE REQUIREMENTS, AS SPECIFIED IN THE FLIGHT RULES, MUST BE SATISFIED.

Protecting the Bird Sanctuary

SPACE SHUTTLE FLIGHT RULE     A2-62       ET FOOTPRINT CRITERIA

DESIGN MECO CONDITIONS AND ABORT MODE BOUNDARIES MUST PROVIDE AN ET IMPACT POINT (IP) FOOTPRINT, INCLUDING 3-SIGMA DISPERSIONS, WHICH SATISFIES THE FOLLOWING CONSTRAINTS:

A. NOMINAL MECO TARGET: THE ET IP FOOTPRINT IS MAINTAINED AT A MINIMUM OF 200 NM FROM ALL FOREIGN LAND MASSES (EXCEPT THE FRENCH-HELD POLYNESIAN ISLANDS, WHERE THE MINIMUM CLEARANCE MAY BE REDUCED TO 60 NM WHEN DICTATED BY MISSION OBJECTIVES) AND 25 NM FROM THE PERMANENT ICE SHELF. ALSO, THE ET IP FOOTPRINT MUST BE MAINTAINED AT LEAST 25 NM FROM U.S. ISLANDS.

 Some days you protect the free world, some days you just protect the bird sanctuary. 

I was recently reminded of our work in keeping the world safe from falling External Tanks as another, current, organization grapples with upper stage disposal. 

The Space Shuttle system was completely reusable with one exception:  the External Tank.  Conceived to be analogous to the fuel drop tanks used by military aircraft, it was expended every mission.  The ET was designed to be both as inexpensive and light weight.  In its Super Light Weight configuration, it weighed 58,000 lbs. empty but could contain half a million gallons (735 tons) of cryogenic liquid hydrogen and oxygen in two separate tanks.  An intertank structure in-between housed a steel I-beam that held the entire flight vehicle together:  the reusable Orbiter holding the crew and the twin Solid Rocket Boosters. 

Theoretically capable of taking the ET all the way into orbit, some ambitious engineers envisioned making a space station out of empty ETs.  Maybe a topic for another blog post in the future.  But the biggest problem with the ET was how to get rid of it safely so that no one could be hurt by it falling back to earth.  The ET was jettisoned at nearly orbital velocity and traveled more than half way around the globe to reenter in a ‘broad ocean area’ where it would break up at around 250,000 feet altitude (45 nautical miles, or 75 kilometers high).  The very thin wall aluminum and foam insulation vaporizing in the heat, many large pieces would still make it down to the ocean surface:  the steel SRB I-beam, the 17-inch diameter LOX and LH2 separation valves, the sturdy structural attach points, and many more pieces. 

A key aspect of this story is the requirement, by tradition, treaty, and standard practice, is that a sovereign nation generally considers the waters withing 200 n. mi. of its shore to be its area of economic and military interest (their Exclusive Economic Zone – EEZ).  So, we were constrained to keep the pieces outside of 200 n. mi. from a foreign ‘land mass’ or in this case, an island.  For US territory we could come to 25 n. mi. from the beach, much closer. 

While the Orbiter used the smaller OMS engines to raise its orbit, the ET was on a ballistic path to destruction.  Early in the program this was a two-burn sequence (aka ‘standard insertion’) resulting in the ET falling into the Indian Ocean.  Later on, to get more performance out of the ET, the cutoff was changed slightly faster so that the Orbiter only needing one OMS engine thrust (aka ‘direct insertion’) to get to orbit while the ET splashed down in the Pacific Ocean.  Direct insertion (DI) was a way to get more payload weight to a higher orbit. 

Many times, NASA aircraft or other resources observed – at a distance – the breakup of the External tanks.  The films of the reentry breakups were spectacular. 

For reference you can watch:   https://www.youtube.com/watch?v=1fkeTULQAps 

Careful analysis of these videos led to a well-documented ‘debris catalog’ which was used in analytical computer programs to predict how far the parts might scatter.   This pattern was the so called 3-sigma debris footprint.  3-sigma because it statistically encompassed 99.74% of all possible scattering of the parts.  Analytically the footprint was almost 1,400 n. mi. long by 30 n. mi. wide (2,500 km by 55 km.)

Of course, the planned inclination of the orbit, and a variable launch window for rendezvous flights (for ISS assembly) caused different swatches of the Pacific Ocean to be under the footprint. 

We always planned very carefully where the ET would go and issued international Notices to Air Men and Notices to Mariners about the location and time when pieces might fall. 

I was the Ascent Flight Director for one mission where the ET re-entered very near Hawaii just after local sunset.  The spectacular breakup was observed by many on the Big Island and (according to what I heard) resulted in a call from the Governor of Hawaii to the NASA Administrator to find out ‘why is NASA bombing our state?’  Perhaps an apocryphal story, but why check? A good story is better than the truth some times. 

Getting ready to assemble the ISS, it was important to maximize the launch window and one way to allow more launch opportunities was to lower the insertion orbit.  In the arcane world of orbital rendezvous, coming in lower allowed more opportunities to launch.  Take my word for it – maybe another blog post for another day – it works.  So, the trajectory analysts looked to insert – which is to say, cut off the Main Engines – when the apogee of the orbit was 122 n. mi. – down from the previous target of 173 n. mi. 

This caused the ET disposal footprint to move from the Central Eastern Pacific – east of French Polynesia and west of Mexico – to a location farther south and west, between New Zealand on the west and French Polynesia on the east.  And unfortunately, the end of the ET disposal footprint extended into French Polynesia.  

Here it might be useful to be familiar with the islands of the South Pacific . . . which I was not.  (Maybe I need to read Michener’s ‘Tales of the South Pacific’). 

Close to the west end of the footprint (the ‘heel’), Pitt Island (New Zealand, population 38), the Bounty Islands (New Zealand, uninhabited bird sanctuary), and Auckland Islands (New Zealand, uninhabited) were close to the footprint but protected.  That is to say, the 3-sigma debris footprint was outside 200 n. mi. from their beaches. 

On the east end of the footprint (the ‘toe’), the French Polynesian islands of Tematangi (population 58), Hereheretue (population 56), Anuanuraro (uninhabited), Anuanurunga (uninhabited), and Nakutipipi (uninhabited most of the year) were also similarly protected.  (Note:  spelling may vary). 

But I imagine those 100 or so people got quite a light show on occasions! 

Several of these islands were considered bird sanctuaries which, we were told, were periodically visited by scientists conducted bird population surveys.  We never knew when people might be there. 

But we had one real infringement problem.  Two French Polynesian Islands were just inside the footprint:  Raivavae (population 900) and Rapa Iti (population 500); left and right of the ground track respectively.  Our 3-sigma footprint – in the sideways or ‘crosstrack’ direction – came just outside 60 n. mi. from those two islands.  This violated the 200 n. mi. limit that we were obliged to protect.  None of our trajectory tricks worked, we were stuck with a violation or loss of possible launch days. 

The analysts calculated that the possibility of an ET fragment – in the crosstrack direction – exceeding 3 sigma dispersion (27.5 n. mi.) by the additional 60 n. mi. was less than 1 x 10^-10 probability. One chance in a very large number.  You have better odds at the lottery.  And remember, we only protected US beaches by 25 n. mi., 40% less distance. 

In one of their rare helpful instances, NASA HQ talked to the Department of State about the situation.  State encouraged us to write a letter asking permission to come within 60 n. mi. of these two islands rather than the standard 200.  Under their tutelage, we drafted a very nice letter asking permission of the French government for this exception.  The letter – a formal diplomatic note – was sent in December of 1998.  Copies of the letter were sent to the South Pacific Forum and the South Pacific Regional Environment Programme.  In the next year, NASA sent teams to visit Tahiti to explain the situation and also presented a summary to the New Zealand embassy in Washington.   

And we waited for a response from the French.

Which never came.

After a year with no response, the State Department effectively told us that the French had been adequately notified and receiving no objection, we could put the plan into operation. 

So, we did.   

I wonder if French ESA astronauts Leopold Eyharts (on STS-122) or Philippe Perrin (on STS-111) directly benefited from the increased launch window.  I will have to research that. 

If you think this is complicated, next consider off-nominal – underspeed and abort – cases where we also had to protect for safe ET disposal. 

I never heard a report of any sightings or objections from folks in the South Pacific. Or the French government.

Except one story.  For another day.