STS-93: We don’t need any more of those

It was about a quarter past midnight on July 23, 1999 when the Ralph Roe, the Shuttle Launch Director, told Eileen’s crew that they were go for launch and wished them good luck. The launch, which had been scrubbed late in the countdown on two previous attempts, was going to be about 7 minutes late due to some transient communications system problems at the Merritt Island Launch Annex (MILA) tracking station.

 

In Houston, I was sitting on the back row of mission control acting as the Mission Operations Director. Sitting next to me was Chuck Knarr, former flight director and executive in the United Space Alliance. In the hot seat at the Flight Director position was John Shannon with LeRoy Cain sitting beside him to watch the weather and keep track of the checklists. Capcom was Scott “Scooter” Altman.

 

The old joke was: “which F-14 pilot do you want as your wingman? Cobra, Maverick, or Scooter?” The correct answer was Scooter – Cobra and Maverick were just Hollywood movie creations for ‘Top Gun’; in that movie all the real flying was done by Scooter.

 

Down in the trench was Flight Dynamics Officer (FDO) Lisa Shore with Carson Sparks sitting next to her as TRAJ officer. On the Electrical General Instrumentation and Lighting (EGIL) console was Tim North – about to get the ride of his life – and on the back row was Booster Officer Jon Reding, one of the coolest heads to sit at that position. A well seasoned crew to watch over the launch phase for Eileen Collins, the first woman to command a Space Shuttle mission, her pilot Jeff Ashby, and the rest of the unusually small crew of Columbia.

 

STS-93 carried the heaviest payload the shuttle ever launched; the Chandra X-ray observatory (formerly known as the Advanced X-ray Astronomy Facility or AXAF) and it IUS booster. The Boeing build solid rocket upper stage called the Inertial Upper Stage was extremely reliable, but a heavy load. It had an interesting history; originally called the Interim Upper Stage (same acronym), its more powerful replacement was cancelled for budgetary reasons and the IUS lived on.

 

Everything appeared quiet, but problem lurked just beneath the surface.
Post flight we calculated that the LOX load was less than planned; due to slight temperature variations, tank volume, placement of the ‘full’ level sensors and other factors, Columbia was going to launch this night with 897 lbs less LOX in her tank than was intended. Given everything else working normally, there should have been a performance reserve still in excess of 3,000 lbs. But everything else did not work as precisely as planned.

 

Columbia, the oldest shuttle, had flown 25 missions by this time and the tiles and thermal blankets showed it; scars and stains over almost all of them. And hidden, deep down inside, were the flaws and insults that any real world flying machine always carries: some minor, some major, some that never had a consequence, and some that ultimately contributed to her loss.

 

One flaw lived in a stretch of 22 gauge kapton insulated wiring, nearly half way down the payload bay, where a single strand of AC current carrying wire had rubbed and chaffed against a screw head which had a minor burr where the tech had over tightened it long ago, and which other techs had probably stepped on during turnaround refurbishment. In the Right SRB, a hydraulic pressure sensor was not completely connected to its wiring and even though it was showing accurate pressure now, the connection could shake loose under vibration just enough to open the circuit and give a false low pressure reading. In the center main engine the two chamber pressure measurements A&B were reading exactly the same with the engines off, but the B channel had a bias that would only show up when the engine reached full throttle; it would read 12 psi high. This was outside the allowable error but still small compared to the 6,000 psi operating pressure. On the right engine, a deactivated LOX post in the main injector had a gold coated pin that wasn’t quite as tightly seated as it should have been, whether it was installed slightly off or worked loose over a number of firings, in the end it didn’t matter. Columbia had a lot of really minor flaws that didn’t affect today’s story, but some of which would show up later with other consequences. Viewgraph PowerPoint spaceships are always flawless; real spaceships are made and maintained by fallible human beings and are less perfect.

 

Somebody posted the NASA TV video of the launch on YouTube – you can watch it at:

So the countdown started up at T-9 minutes as it always did, seemingly uneventful on the surface, but we all knew what was about to happen and the adrenalin built up as we waited. Ashby started the APUs right on time, pumping high pressure hydraulic fluid to all the valves in the main engines among other things. At T-31 seconds the onboard computers took over, calculating the exact launch parameters, allowing the IMU gimbals to turn freely, opening vent doors so that pressures could equalize during the climb to space. At T-10 seconds the ground computers sent the last command onboard – the electronic equivalent of ‘go for main engine start’. The onboard computers reacted immediately by firing off the ‘sparklers’ called ROFIs (never can remember what that stands for) to burn off the excess hydrogen down by the main engines. At T-6.6 seconds the onboard computer commanded the first main engine to start, and staggered the second start by 140 milliseconds, and then the third by the same delay. The SSME controllers commanded the spark igniters on, and started the complex choreography of valve openings to bring the three main engines safely to roaring life.

 

By about T-3 seconds, all engines were up and operating at 100% of rated power level. Exactly when it happened is not clear, but on the right engine, the gold plated pin from LOX post 32 in row 13 came shooting out. Just like a bullet it went through the narrow part of the converging nozzle and flew out into the nozzle extension.
Now two scary things could have happened. First, the LOX post, which was pinned for a reason, could have failed allowing LOX to pour into the engine cavity where the hot hydrogen was introduced. Remember from an earlier post that this could have caused an explosion, or melting of other LOX posts, or other really bad things to happen. Failure of the LOX post was considered a CRIT 1 failure – loss of vehicle and crew ‘promptly’. Fortunately – luckily – the LOX post held together for the next eight and one half minutes. How close we were to disaster has never been determined.

 

Second, the nozzle extension could have failed. Early on in ground testing, one of them did fail – spectacularly – generating a huge explosion and fire. Those nozzle extensions were one of the weak points of the entire complex engine system. Made of 1080 long narrow stainless steel tubes braze welded together, the nozzle extension was cooled by liquid hydrogen flowing down its length. So cold on the outside that a layer of frost condensed out of the moist Florida air in seconds over the outside of the nozzle extension, the temperature inside was hot enough to melt any metal. These tubes were a bear to maintain and the entire extension had to be replaced periodically. The tubes were prone to split which required complex weld repairs until there were just too many repairs and a new nozzle extension had to be installed. The next upgrade to the SSMEs was to build a more robust channel wall nozzle extension. The shuttle program ended before that was done. Someone had calculated that if 5 adjacent cooling tubes split or were otherwise ruptured, there would not be enough local area cooling and a burn through would occur, causing a cascading failure of the nozzle and . . . a CRIT 1 failure.

 

The bullet shaped LOX post pin hit the side of the right engine nozzle extension about two thirds of the way to the end with great force. Just by sheer luck, three nozzle tubes were breached. Three adjacent nozzle tubes lost cooling and started leaking hydrogen into the hot stream of gas coming out of the engine. Three tubes, not five. The adjacent cooling tubes kept the nozzle from failing during the eight and a half minutes the engines operated.

 

At the time, the LOX post pin and nozzle tube leak went undetected. Look closely in the video and see a blue streak on the side of the nozzle. Nobody in Mission Control noticed it until afterwards during the photo review. The nozzle was leaking 3.5 lbs of hydrogen every second.

STS-93.MissionStatusBriefing.Slide (2)

Several things happened as a result. One might think that this would lead to running out of hydrogen, which, as an earlier post noted, would have been very bad. But the engine reacts in to the loss of hydrogen in this area in a very interesting manner. This leak was downstream of the hydrogen flow meter so to all observers – human and digital – it appeared that the right amount of hydrogen was going into the engine. The real loss of hydrogen flow to the turbines driving the pumps on the right engine was lower mixture ratio – the mixture ratio now closer to stoichiometric – and the turbine temperatures increased by about 100 degrees. Normally there is about a 200 degree margin between the normal turbine temperatures and the redline value that will automatically shut an engine down. The nozzle leak immediately used up half the operating margin to the redline.

 

Since the chamber pressure dropped slightly due to the loss of fuel for the fire in the main combustion chamber, the SSME controller commanded more oxygen be sent to the MCC. That may sound strange but it is the way the SSME computer controls the chamber pressure. Again, the mixture ratio is off and not only is the fire hotter in the MCC, but more oxygen is being consumed. Due to both of these processes, post flight calculations showed that the LOX tank should have been short about 3,000 lbs – translating to about 200 fps short at MECO. That large velocity shortfall did not happen because we were lucky: what happened to the Center SSME corrected that. Hold that thought.

 

Meanwhile, about a minute after launch, the booster officer and his team recognized the fact that the right engine turbine temperatures and speeds were higher than normal. They correctly identified that this might be due to a nozzle leak, but there was another potential anomaly that also had the same signature. If the oxidizer pump started to lose ‘efficiency’ (blades rubbing, pump clogging, etc.) it would look the same. As the SSME controller commanded mixture ratio changes to keep up with the loss of efficiency on the pump, the turbines would reach their temp limit and the engine would have to throttle down to prevent a shutdown: this was called ‘thrust limiting’. Until the SSME went into thrust limiting, the Booster team could not tell the difference between an oxidizer turbine/pump efficiency loss and a nozzle leak. The instrumentation just wasn’t precise enough to know what was going on. Jon and his team correctly identified that the engine was running off nominally (‘off tags’) but could not quantify it. Later on, when the FDO asked him, the Booster officer had to report that none of the engines were ‘suspect’. All these terms were precisely defined in the flight rules and had specific actions for the flight controllers and crew to take to maximize safety. But this leak was too small for any of that.

 

The reason the booster officer took over a full minute to identify problems on the right engine was because of distractions: the AC 1 short – which was real, and had real consequences – and a big red light that went off on his console saying the Right SRB hydraulic system pressure had dropped to catastrophic levels – which was not real. Each of the solid rocket boosters put out over twice the thrust as the three SSMEs put together. The steering mechanism was critical and each SRB had two separate hydraulic systems for redundancy. If both hydraulic systems on one SRB failed there would be no steering on that side; probable loss of control, and another one of those ‘prompt’ CRIT 1 failures – loss of vehicle and crew. So having that red light go off right in front of his nose took a few seconds to sort out. It clearly was a transducer failure, not a real hydraulic system failure since all the other parameters were OK. But it surely got the heart rate higher. Good thing it wasn’t displayed to the crew; they never knew about it.

 

Failed SRB TVC could, in the worst case, result in the shuttle flying directly back to the Launch Control Center. Given worst case reaction time by the Range Safety officer (7 seconds), large chunks of SRB – probably with the propellant still ignited – could land on the LCC. I used to think about that when I sat in the front row of the LCC, up near the glass. Those big louvers in front of the windows? They were for shade; not only no protection from flying debris but likely to become lethal missiles themselves.

 

As Columbia lifted off the KSC public affairs officer, Lisa Malone made her little speech about how this flight advanced X-ray astronomy and women’s authority in the world. In the MCC, we didn’t listen to the PAO commentary so we were shocked to hear Eileen report “Fuel Cell PH”.

 

I’ve written about fuel cell PH failure mode before: https://blogs.nasa.gov/waynehalesblog/2009/01/07/post_1231342021582/

 

To make a long story short, it means that one of the fuel cells might be failing “It’s the Kaboom Case, Flight”.  On board Columbia, the Master Alarm klaxon had gone off and there were two messages on the failure summary page: H2O pump, FC 1 PH. As it turns out, these were symptoms, not the cause of the problem. The Fuel Cell was not breaking down, ready to explode, and the water pump (used for cooling) was not shut down. Instead it was one of the alternating current buses- AC 1 Phase A – which had shorted. About 5 seconds after liftoff, for about half a second, it sputtered along with a short of up to 72 amps. That is a lot of current!

93 short

The automatic protection cut in and shut down the affected part of the circuit. The Fuel Cell instrumentation that monitored for the escape of potassium hydroxide ( a strong base, high PH) used this bus, and this unpowered instrumentation gave an erroneous alarm. The water pump slowed and then resumed after the circuit breaker popped and normal power was returned on all the other AC1 equipment. An avionics bay fan (providing air cooling) had slowed but not enough to trip an alarm. What remained was loss of power to the SSME controllers.

 

The A computer on the Center SSME lost power, never to be recovered. The B computer (DCU B) immediately took control and the engine ran on normally. Well, almost – back to that in a moment. The Right Engine lost its B computer, but the A computer stayed in control and the engine, with its hydrogen leak, hung in there.

 

Loss of DCU A on the center engine caused a couple of interesting consequences. First of all, the MCC lost almost all the telemetry on the center engine; valve positions, turbine speed, some temperatures, almost everything was gone with the lost of DCU A. The folks that programmed the SSME computers put everything on the A side and very little – just Main Combustion Chamber pressure and hexadecimal word indicating any problems in the engine – on the B side. We were on ‘backup data’. The B computer also lost all the sensors that were being read by the A computer. Most importantly, the A chamber pressure transducer went away. The B computer no longer averaged the A and B transducers to get its chamber pressure for the basis of engine control but only had the B transducer – and it was reading 12 psi high! This may not seem like a lot, but it caused the center DCU B computer to command a throttle down on the center engine. All of a sudden, less LOX (and hydrogen) was being consumed. Almost invisibly, the Center engine was making up for the large shortfall of LOX that had been created by the nozzle leak on the right engine.

 

How lucky we were. Instead of being 200 or more fps short at MECO, possibly leading to an abort landing or requiring two tons of OMS propellant to make up, we wound up being only 15 fps short, well within the capability of the OMS budget.

 

Less than a minute off the pad and multiple failures. What were you thinking Sim Sup? Surely that is not a realistic case. Except it was real.

 

During a shuttle ascent, the crew normally has very little to do other than monitor the automated systems. As the velocity builds up, the MCC – the FDO – keeps the crew informed of what ‘abort mode’ they might have to use if one or more main engines were to turn off prematurely. As long as nothing happens, the crew happily gets to sit quietly until they are in orbit.

 

For STS-93, the big action for the crew was to take the AC bus sensors to off. Since the inverters – the devices that change the direct current electricity into alternating current – had a failure mode that could possibly make them over volt and fry their associated equipment, the automated circuit protection was enabled to swiftly react to any voltage issues. For the record, that never occurred in 135 shuttle flights. After losing two SSME computers, another AC bus dropping offline would cause one of the two affected main engines to shut down, probably requiring an abort. There was a slight chance the automated voltage control equipment could erroneously trip off an AC bus. Given the situation, the flight rules directed the crew to disable the automated shutdown. So Scooter told Ashby to take the AC Bus Sensors to off. That was the only action that the crew had to take the entire ascent.

 

They were blissfully unaware of the erroneous R SRB HYD pressure transducer. They did not get the word that the FDO was seeing the affects of . . . .something . . . .which turned out to be the combined effects of the right and center engine anomalies: a ‘6 fps thrust update in the ARD’ – not huge, not entirely atypical, but something going on. Not enough to change any calls: ‘no suspect engines’ ‘no under speed predicted’ ‘nominal shutdown plan’ were phrases – all carefully scripted by flight rules – that were passed between the flight controllers.

 

And of course, the flight controllers were unaware of all of what transpired inside Columbia.  That may be a good thing.

 

As it turns out, there was a LOX shortfall of 405 lbs. The LOX low level sensors detected depletion of the LOX, commanded main engine shutdown approximately 0.15 seconds earlier than guidance would have wanted, and the final shuttle velocity was about 15 feet per second short of what was desired; out of more than 25,500 feet per second. There was enough margin in the Orbital Maneuvering System load to make up about 300 lbs, and the flight proceeded normally.

 

Except Mr. Shannon, when informed of the situation shortly after main engine cut off said loud and clear on the Flight loop: “Yikes. We don’t need another one of those.”

 

If you look at the video of NASA TV just after those words, you can see me on the telephone, talking to the program manager down in Florida, about what this all meant.

 

It was just a couple of minutes later that one of the projectors hanging from the ceiling in Mission Control – the projectors that put up the displays on the front screens – overheated and started smoking. Quick action by the Ground Control officer to shut it off probably prevented a fire in the MCC, which would have lead to an evacuation.

 

That’s really over the top, isn’t it Sim Sup?

 

How much more exciting can you get. Give me a nominal, boring launch any day.

From friends at NASASpaceFlight.com watch the annotated video:

https://www.youtube.com/watch?v=O9WjCyWq-iA

 

Now for the lesson: Be prepared. Spacecraft are complex and can fail in complex ways. Never, ever let your guard down. Practice for disaster all the time.

 

And remember:  Murphy does not play by the rules.

Posted in Uncategorized | Tagged , | 41 Comments

Practicing for Disaster

Even though we sometimes hated them, the training teams that prepared mission control and the astronauts for every flight are real heroes.  Without their efforts, all of us flight controllers would have believed we knew everything there was to know about everything and would have tripped over our own shoelaces at the first sign of trouble.

I can remember after about a year of integrated training for STS-1, just when we were feeling like we were hot stuff, a new capability drop for the simulators came on line.  That first day of training with the new capability, the EGIL sang out “Control Bus AB1 is down” on the Flight Director loop.  We all looked at blankly each other and asked, ‘what the heck is a control bus’.  Turns out that failures in the shuttle electrical system works were not simulated before, and now they were.  All of us learned an awful lot about the shuttle electrical system very quickly.

Training flight controllers and astronauts was a complex job especially when we had “integrated training” which was the closest thing to real space flight available.   There were certain ‘cases’ that each flight controller and crew member were required to experience and master.  These malfunction cases were practiced over and over and over again until they became so routine that we were bored by them; identification and reactions were automatic and swift.

Then there were new cases for new crew members and new flight controllers; hundreds of potential malfunctions that could possibly occur.  Each new person had to demonstrate familiarity and confidence to identify and rectify hundreds of potential shuttle systems failures.  Little failures, big failures, complex failures, simple failures.  Hundreds, if not thousands.

Then the gain was raised with complex interacting multiple failures: this computer and that electrical bus, this IMU and that GPS, this hydraulic system and that aero data set with wind shear, this TAL abort and that leaking tire, and on and on and on.  A seemingly infinite set of dual complications.

Master all the dual combinations?  Get ready for triples!  Shesh.

On a busy day the sim team had to make sure that multiple flight controllers saw multiple failures in each eight and a half minute shuttle launch profile.  We generally did six launches in one day.  Or in the entry sims that simulated the last 15 minutes of entry – four of those cases filled up a day.  The nexus for all problems of course was the Flight Director.  Sometimes it felt like the Flight Director was wading through a class of excited grade school students all calling for his attention at once.  The Flight Director had to recognize and prioritize failure responses very quickly.  Some things just had to wait (“Flight, the cockpit voice recorder just failed, have the crew switch to recorder #2.” ” OK INCO, we will do that as soon as we get the cabin air leak stopped, the fire out, and the abort mode selected”).

Flight Directors – including this one – tended to get testy on days like that.  ‘Not realistic Sim Sup’  or ‘We will never have an ascent with that many failures Sim Sup’ or other brief communications that we cannot reproduce in a family oriented publication.  In the old MCC of Apollo heritage which we used for early shuttle missions, the Sim Control team area overlooked the Flight Control Room – but there were curtains on the MCC side  When the Flight Director really had enough, he would have the curtains closed, blocking off the sim control team’s view.  Later on, the sim team used the remote controlled TV cameras in the Flight Control room to observer their victims, er, trainees during tense moments.

An eight and a half minute ascent simulation with a full malfunction count could leave you breathless and heart racing.  Sometimes the debrief took two hours to discuss what had happened and action items would be assigned that might take weeks of research to finally answer.

When we got to real flight, it was so calm by comparison as to be boring.  Many the nominal ascent I would look back and wonder what we trained so hard for.

Then came STS-93.

After that one, the Flight Directors complained a lot less about busy training runs.

 

 

Posted in Uncategorized | Tagged , , , | 15 Comments

STS-93: Dualing computers

In the early days of rocketry, when subsystems reliability was low, hard experience led designers to add redundancy for critical functions where they could.  Redundancy comes at a cost:  increased weight, increased complexity, unintended interactions, complex schemes to manage the redundancy, etc.  Now days, subsystem reliability is much higher, especially electronic parts.  So today we have sophisticated discussions about dispensing with redundancy and living with single string high reliability systems, even in critical areas.

Whenever I am engaged in one of these discussions I remember STS-93 where a 2 cent screw and a 10 cent length of wire demonstrated the vulnerability of an otherwise highly reliable critical system.  “Dualing computers” is not a misspelling, it is a safety concept.

Each Space Shuttle Main Engine has a computer mounted right to its side to run the complicated functions required for safe operation of that very complex, high energy device.  And every SSME controller is made up of two redundant computers:  DCU A and DCU B (Data Control Unit).  The A computer is always in control while the B listens along – until and unless the A computer fails; then the B computer takes over.  Each computer has its own way to control every valve and its own set of instrumentation required to run the engine:  pressures, temperatures, valve positions, turbine speeds.

However, when both A and B are working, they share data.  So there is a pressure measurement for the main combustion chamber wired to DCU A and another one wired to DCU B, but when both computers are working they share data and make computations based on the average of the two chamber pressure measurements.  If one of the computers fails, the other carries on but then has only one measurement to make calculations from – no more averaging.

Almost all the telemetry that is sent from the engine to Mission Control comes from the A computer; if it fails the B computer sends only a few data points, not nearly as many as the A side.

When STS-93 had its little problem, every engine kept working just fine even though two computers on two separate engines went silent.   There was never another case of SSME computer loss in the entire suite of shuttle flights.  These computers were highly reliable. The computers never failed because of an electronic part problem or a software error.

But in systems design, like warfare, defense has to be made at the most vulnerable point.  For the SSME controllers this was the power source.

If the shuttle designers hadn’t built in redundancy, two of the three engines would have shut down just after lift off.  The results from that would not have been good.  The crew has a procedure to run called “2 out First Stage”.  It is one of those procedures that Capt. Young used to describe as “keeping busy while you wait to die.”

The next time someone tells me they have a highly reliable system that doesn’t need redundancy, I will remember STS-93. I hope you do too.

 

Posted in Uncategorized | 26 Comments

STS-93: Dodging Golden Bullets

 

Calling it Rocket Science is, of course, a misnomer. Science provided the background but today it is definitely Rocket Engineering. Scientists and Engineers mix together like, well, cats and dogs. Friendly détente some days, not so much other days.

But there is one part of building liquid rocket engines that can still be called Rocket Art – making the injectors work. All combustion engines mix fuel and oxygen to make a fire; but for really complete combustion – really good gas mileage – that mixing can be very tricky indeed.

In a large, high pressure, high thrust liquid rocket engine like the SSME, a great deal of art is involved in designing the injector plate at the top of the combustion chamber. In a poor design, spray patterns from the injector can create hot spots on the wall of the chamber, defeating the cooling mechanisms and melting out the side – like letting loose a welder’s cutting torch. In other cases poor mixing can lead to combustion instability. Think of an overloaded out of balance washing machine, but much more powerful. The mighty F-1 engines of Apollo were plagued by combustion instability which was never really quite solved. That made each moon launch more of a gamble at the very start than you probably realized.

In the world of technical and economic secrets, injector design is protected by ITAR and economic espionage laws. We won’t go to that level of course.

In the SSME, the very hot partially burned Hydrogen gas must be mixed with the still super cold liquid Oxygen in just the right way to protect the engine through the start transient, main stage with its varying throttle settings, and the shutdown transient. Did I tell you that the propellant flow is over a half a ton a second? And it burns at over 3,000 degrees F? And passes through the throat of the nozzle – about the size of a dinner plate – at the speed of sound? And in all this, the engine is 99.9% of the maximum theoretical efficiency for this type of heat engine? And that’s not all, its reusable, too. Only one in the world.

LOX posts and pin

Looking at the picture, the liquid oxygen is introduced through a forest of stainless steel tubes called LOX posts. Cooled by the LOX inside, heated by the hot Hydrogen outside, the tubes are both robust and at the same time frighteningly fragile. Hugh forces work on the LOX posts, especially during start up. Vibration forces are high throughout. And if one of those posts breaks off at the root, well, very bad things can happen. In the cool NASA parlance, a LOX post failure is CIL Crit 1. Loss of vehicle, loss of crew ‘promptly’ upon just one failure.

To eliminate the potential of a LOX post failure, inspections of the hundreds of LOX posts is performed with ultrasound. If a post shows signs of ‘fatigue’, the remedy is to plug it at the base. They use a gold pin, about the size and shape of a bullet. In the history of the program, over 200 LOX posts were pinned in this way, and only a couple ever worked loose. STS-93 was one of those.

Back to the case of STS-78 discussed in the last edition of this blog; the ‘overboard mixture ratio’ for the vehicle was changed because of the number of LOX posts that were pinned on the engines of that flight. Instead of being the expected 6.03-ish, it turned out to be more like 6.002-ish. That difference of .028 meant more fuel was used, less oxygen, and, in combination with other factors, resulted in fuel depletion just at the guidance commanded MECO. A bigger difference would have cut the engines off early – safe except for the trajectory implications. We got lucky.

On STS-93, a different set of circumstances was in play and the results were worse. Manageable, but that is due to being luckier than we deserved.

As my old boss used to say: “It’s better to be lucky than smart.” I hate that but it’s true.

Stay tuned.

Posted in Uncategorized | 11 Comments

Understanding STS-93: the key is Mixture Ratio

Some time back I started to tell the story of the most interesting shuttle launch:  STS-93.  I think it is time to return to that topic.  To understand what happened, some background is necessary.

If this is too engineering-geeky for you, well, what are you doing thinking about rockets and space travel?  Consider this part of your education.

Consider the following graph – I certainly spent many hours studying it and its relatives.  I would tell you frankly, I am sure I never completely understood it.  So don’t feel bad if you don’t either.  But it gives a summary of some very complex interactions.

FPRFlight Performance Reserve (FPR) is the mass of fuel (Hydrogen) and Oxygen left in the External Tank when it is jettisoned just short of orbital velocity.  Minimizing FPR is a good thing – every ounce thrown away is an ounce that could have been payload – food, water, experiment, satellite – something useful.  FPR thrown away is . . . .wasted.

At the same time, keeping too little FPR, or making the mistake of not keeping any reserve at all, means that one likely comes up short.  Short of energy, short of velocity, not in orbit, but on a ballistic trajectory that re enters the earth’s atmosphere very soon.  Too soon.  STS-93 was nearly that case.

If you look at the burning of Hydrogen and Oxygen – the second highest energy release possible on the Periodic Table – you would find that the stoichiometric ratio for complete combustion and maximum energy release is 16: 2 hydrogen atoms (atomic mass = 1) attached to one oxygen atom (atomic weight 16) for a complete combustion MR of 8.0.  But the space shuttle’s main engines have a mixture ratio – reminiscent of Avogadro’s number without the exponent – of 6.02.  If you look at the chart above, you will find at that mixture ratio, the unusable masses are just about at minimum.  But anything that drives combustion in the engines away from that optimum point will lead to increased unusable mass.

The reason for such a wretchedly low mixture ratio is that the closer the MR is driven toward stoichiometric, the hotter the fire.  The turbine blades in the turbines that power the pumps feeding the engines can’t take a much hotter fire than results from 6.02.  Blades would melt, casings too, bad things indeed would happen.  Temperature sensors in the turbines should trip the engine to shut itself down before that happened. On STS-51 F in July of 1985, both voting temperature measurements failed and started reading higher than the turbine temps actually were:  51F became the only case of an SSME shutdown in flight and it was caused by faulty temperature readings.  But back to our story:   6.02 is ‘just right’. At least to two decimal points.  We spent 30 years arguing about what the next decimal point should be.

Important Safety Note:  if propellant depletion occurs, it must occur first on the oxygen side.  If the hydrogen runs out first, the last sputters at the turbine will be much closer to stoichiometric, and, well, bad things wil happen.  Did occasionally happen early in ground testings.  Big mess in the bottom of the flame trench at Stennis.  Not what anybody wanted in flight.

So STS-78 was  a real wake up call.  On that flight, the low level sensors in the ET flashed ‘dry’ just a fraction of a second before the engines shut down.  No problem ensued, there was enough fuel in the line to shut down safely, but it scared the bejabbers out of everybody.  At least everybody that understood what that meant.

Bet you never heard about that close call.

There is supposed to be a ‘fuel bias’ (extra hydrogen) of almost 1000 lbs.  But on STS-78, due to another instrumentation failure and some funny mixture ratio business, the engine burned right through that extra thousand pounds of hydrogen and all the other ‘dispersion’ allowances that were loaded in the tank.

It all started with plugged LOX posts.  LOX post plugs played a part in STS-93, too.

All space geeks need to stay tuned.  It really is rocket science.

 

 

Posted in Uncategorized | 13 Comments

O Columbia

When I first heard that the Houston Grand Opera proposed to produce an opera about the Columbia disaster, I was appalled.

 
If HGO wanted a spaceflight opera, it would better be the moon landing of Apollo 11; or the successful story of Hubble repair; or the construction of the ISS. But not Columbia.
I envisioned a maudlin, inaccurate, titillating soap opera treatment of a still too painful event. Something that would be disrespectful of the dead, their families, and all those who worked so hard and yet failed a decade ago. Something that Hollywood might produce.
I resolved to try to shut this down.

 
Then I received an invitation to talk with the HGO leadership, the writers and creative talent who would develop and produce the opera. I drove downtown with the mission burning in my heart to stop this, direct them away, keep this project stillborn.

 
When I met the HGO team, they were respectful, considerate, and affirming. They reached out to more than a few members of the NASA community to gain input. We talked about everything possible aspect and I expressed my concerns in my best program manager brusque style. They listened. I came away grudgingly admitting that it might be barely acceptable. Not accurate, but possibly respectful.

 
Great tragedy, as I am reminded, can focus attention on great human values. Art, music, dance can lead to an emotional understanding which can transcend and reach people who otherwise would be unaware. The most powerful works of the stage – from Shakespeare to Faulkner – use tragedy to highlight values and meaning.

 
After the HGO team spend several weeks at work, we were invited back to hear the reading of the libretto; the words. A crowd of familiar NASA faces joined at the stage door of the Wortham center in downtown Houston to hear what had been created.

 
First we heard from the composer designated to work on the project. A soprano brought some of his previous work into glorious voice.

 
Then we were ushered into a room for the reading of the libretto. Very informal, no sets, no costumes, just a stark rehearsal room where we sat on folding chair while several performers on stools faced us over music stands holding the pages.

 
The various voices read, did not sing, the text, all the while in character.

 
It was not merely good, it was great.

 
It put space exploration in a historical context. It used the point of view of a young person aspiring to be an astronaut. There was nothing neither maudlin nor titillating. At one point there were tears among the audience. It was powerful.

 
After the reading we were asked for comments; the astronauts spoke first, then we all had the chance; praise for the approach, minor corrections to the script, overall an emotional release.

 
The music is in composition, much later the sets and costumes will be constructed, rehearsals will begin. The curtain should go up in October 2015. Just like all space projects, the schedule will take some time.

 
I think it will be great.
You should mark your calendar and go.

Posted in Uncategorized | Tagged , , , | 6 Comments

Manifest Destiny or Pipe Dream?

“I started out as a child” – William H. Cosby, Jr., Ed.D.

I have been ruined by the timing of my childhood. Grew up with the space race; 3 years old when Sputnik launched, 7 when Gagarin and Shepard flew and JFK promised the moon, allowed to stay up late to watch if Ranger or Surveyor were going to be successful, watched Gemini through grade school, Apollo through High School, and somewhere in between read a host of science fiction stories.
SF definitely ruined my life; captivated as I was at an impressionable age by Asimov, Clark, Heinlein, and so very many others. Watched “Destination Moon” a thousand times along with every other B grade SF movie ever made. Then, of course, came Star Trek. After that show went off prime time network programming, one of the local TV stations in my area showed reruns every night. So I watched Star Trek reruns every night – every night! – all through high school. There was a time I could quote the dialog from any episode verbatim. My social life suffered accordingly.

Read about Goddard in his tree reading War of the Worlds and deciding to find a way to go to space.  “It is difficult to say what is impossible, for the dream of yesterday is the hope of today and the reality of tomorrow.”
Long after he was dead, JFK’s speeches were the proof text for the future. Space – its exploration and exploitation – was the future. I read and re-read the works of Gerard K. O’Neill and believed. I had no doubt. It was our Manifest Destiny to explore space.
One of my middle school friends proclaimed in about 1967 that we would never reach the moon, it was impossible. Somehow he and I lost touch over the years but I expect he has become one of those fringe types who believe Apollo 11 was filmed on some sound stage in Arizona. I reveled in the fact that he was wrong.
Both fact and fiction pointed to an inevitable future.
Somehow it hasn’t quite worked out as expected.
2001 came and went without lunar bases, without any excavation of black obelisks in Tycho, without manned spaceships heading out to Jupiter or Saturn. I suspect that Zefram Cockrane wasn’t born last year, and I fear that warp drive might just not be possible in the universe we inhabit.
What happened to those flying cars which we should have by now? At least we should have the hover board. But no.
In a year when one’s birthday ends in a zero, brooding comes easily. Thinking about how there is more runway behind than ahead. Wondering if a career’s worth of work and worry were well invested or wasted. Wondering when – or if – the promise will come true.
It becomes easy to fall into cynicism, to write the whole thing off as a stunt. A one-time geopolitical ploy rooted in a particular time and policy made for reasons which no longer exist. Leaving us with the rump vestige of a dream which lives on merely to siphon off public money into some sort of jobs program. Something which exists merely to exist.
Hardnosed taxpayers want to know what they are going to get for their money. Spending billions to plant a flag on some distant point in the universe merely for prestige does not constitute a business case which shows a return on investment. They insist that the ISS better come up with something more important than videos of astronauts in zero G chasing water globules around with straws.
Space travel could just be a pipe dream. Something like believing in fairies or unicorns. Something for children which adults dismiss.
Certainly makes it hard to listen to idealistic pronouncements of ‘horizon goals’ and the inevitability of our future in space.
Ok then, shut ‘er down, put the pieces in a museum, and tell kids to study finance and marketing because that’s where the big money lies. Gordon Gekko was right after all: get filthy rich by any means necessary, win every contest by any means available. That’s what counts. That’s all that counts.
Hmm.
But you know . . .
Pessimism never improved the human condition. Cynics never accomplished anything positive. Real and lasting satisfaction only comes when heart and strength are given to something bigger than yourself.
Roddenberry showed us the future could be better than the present. Heinlein really got it right; it’s not about money. It’s about freedom, achievement, and the joy that comes from accomplishing something hard and worthwhile. There has to be meaning in life or it is not worth living.
Those who go forth into the world – into the universe – have at least the chance to succeed. Those who stay home in fear or greed inexorably fall into stagnation, into dissolution, decay, and finally destruction. Grow or die, that is the law of the universe.
So what if the timeline runs a little longer than we expected? Does that mean we stay home, let the cynics run the world, and wait for Malthus to say ‘I told you so’?
Today we are chasing our tails because the space experts debate destinations: Moon, Mars, or asteroid? O’Neill and I say do them all. The ‘horizon goal’ isn’t Mars, it is the entire solar system. When we have built colonies on every habitable niche, then maybe we will find a way to go to the stars.
I didn’t say bankrupt the treasury. Don’t squander other people’s money; figure out how to do it anyway.
Nobody said that it would be easy. But what is our choice?

 

It’s been a long road, getting from there to here.
It’s been a long time, but my time is finally near.
And I will see my dream come alive at last. I will touch the sky.
And they’re not gonna hold me down no more, no they’re not gonna change my mind.

‘Where My heart Will Take Me’ written by  Diane Warren

Posted in Uncategorized | 22 Comments