Practicing for Disaster

Even though we sometimes hated them, the training teams that prepared mission control and the astronauts for every flight are real heroes.  Without their efforts, all of us flight controllers would have believed we knew everything there was to know about everything and would have tripped over our own shoelaces at the first sign of trouble.

I can remember after about a year of integrated training for STS-1, just when we were feeling like we were hot stuff, a new capability drop for the simulators came on line.  That first day of training with the new capability, the EGIL sang out “Control Bus AB1 is down” on the Flight Director loop.  We all looked at blankly each other and asked, ‘what the heck is a control bus’.  Turns out that failures in the shuttle electrical system works were not simulated before, and now they were.  All of us learned an awful lot about the shuttle electrical system very quickly.

Training flight controllers and astronauts was a complex job especially when we had “integrated training” which was the closest thing to real space flight available.   There were certain ‘cases’ that each flight controller and crew member were required to experience and master.  These malfunction cases were practiced over and over and over again until they became so routine that we were bored by them; identification and reactions were automatic and swift.

Then there were new cases for new crew members and new flight controllers; hundreds of potential malfunctions that could possibly occur.  Each new person had to demonstrate familiarity and confidence to identify and rectify hundreds of potential shuttle systems failures.  Little failures, big failures, complex failures, simple failures.  Hundreds, if not thousands.

Then the gain was raised with complex interacting multiple failures: this computer and that electrical bus, this IMU and that GPS, this hydraulic system and that aero data set with wind shear, this TAL abort and that leaking tire, and on and on and on.  A seemingly infinite set of dual complications.

Master all the dual combinations?  Get ready for triples!  Shesh.

On a busy day the sim team had to make sure that multiple flight controllers saw multiple failures in each eight and a half minute shuttle launch profile.  We generally did six launches in one day.  Or in the entry sims that simulated the last 15 minutes of entry – four of those cases filled up a day.  The nexus for all problems of course was the Flight Director.  Sometimes it felt like the Flight Director was wading through a class of excited grade school students all calling for his attention at once.  The Flight Director had to recognize and prioritize failure responses very quickly.  Some things just had to wait (“Flight, the cockpit voice recorder just failed, have the crew switch to recorder #2.” ” OK INCO, we will do that as soon as we get the cabin air leak stopped, the fire out, and the abort mode selected”).

Flight Directors – including this one – tended to get testy on days like that.  ‘Not realistic Sim Sup’  or ‘We will never have an ascent with that many failures Sim Sup’ or other brief communications that we cannot reproduce in a family oriented publication.  In the old MCC of Apollo heritage which we used for early shuttle missions, the Sim Control team area overlooked the Flight Control Room – but there were curtains on the MCC side  When the Flight Director really had enough, he would have the curtains closed, blocking off the sim control team’s view.  Later on, the sim team used the remote controlled TV cameras in the Flight Control room to observer their victims, er, trainees during tense moments.

An eight and a half minute ascent simulation with a full malfunction count could leave you breathless and heart racing.  Sometimes the debrief took two hours to discuss what had happened and action items would be assigned that might take weeks of research to finally answer.

When we got to real flight, it was so calm by comparison as to be boring.  Many the nominal ascent I would look back and wonder what we trained so hard for.

Then came STS-93.

After that one, the Flight Directors complained a lot less about busy training runs.



About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

17 Responses to Practicing for Disaster

  1. Kerri Hagee (entry support) says:

    Miss those sims now:)

  2. Beth says:

    Wayne, you are a master of the suspenseful buildup…Alfred Hitchcock would be proud.

  3. Dan says:

    Thank you! When are you going to write a proper book?

  4. walt pinkston says:

    The story slowly unfolds …

  5. Yusef Johnson says:

    HA! I just remember being in the backroom when the controller went down, and everybody looking at me like it was my fault….There was a lot of good analysis done pre-flight with respect to STS-93’s contingency abort issues. Thank God we never had to use it on 93 or any other flight…

  6. Chris Ramsay says:

    Thanks for the words about the training folks. Before I was a Flight Controller, I was a “Sim Guy” working up and executing those “cases” for the controllers. In some ways, I liked that job even more than being a Flight Controller.

    The history of simulations saving missions goes all the way back to Apollo. It was an integrated sim for Apollo XI that exposed Gene Kranz and his team to the type of Software glitch that actually occurred during the real landing. This allowed Jack Garman to be ready.

  7. Darrel says:

    I never felt hated, but I did make sure I knew what cars the Flight Directors drove so I could keep an eye out for them when I crossed the street after a sim.

    There was that ascent sim for STS-47 where we had the OMS leak and the LOS before you could tell the crew what to burn for OMS 2… (there might have been a couple more failures than that, but I’m sure they were minor), I did sense some frustration during the debrief on that one…or maybe a rookie Sim Sup just needed to be put in his place.:)

    I miss those conversations on the Sim Sup-FD private loop or the meeting up the the FD suite after a “tough” run. Maybe there can be more of those private conversations during the commercial crew asc/ent training??

    The best part was when you and your team “nailed” an extremely tough run. It was an awesome thing to see, I hope I get experience that again with commercial crew training.

  8. Susan Bates says:

    I really love to read your blogs. May I share them on FB??? So much wisdom about life through your experiences….

  9. Vince says:

    Yep. That’s training with the big dogs. Had intense training in my AF career but I have to admit, even my most challenging sim block(s) never came that close to those levels of intensity. On the other hand, when the lights did illuminate and they can, I was grateful for the unmerciful beatings in the simulator and in simulation. Good training works well. Thanks for all of that Wayne. Humbling to relive, reconsider, re-examine.

  10. Mike says:

    As another of those evil sim guys I have long since come to understand that Murphy was both an optimist and a sim guy at heart. Too many times early in the early ISS program we’d get hammered how unrealistic and impossible and terrible a set of sim cases was only to have the vehicle decide that a little cautious humility was in order. We need to be rigorous and challenging for our students because spaceflight itself is merciless. Luck sometimes sees us through, but it’s better to be lucky but very well prepared. Of course now days a lot of us sit both sides of the sim landside, doing turns both in the instructor and flight controller roles and I think we’re all better off for the opportunity and experience. Each job helps make me better at the other.

    Deeply appreciate hearing new sides of stories I’ve heard in various forms over the years, thanks for keeping up your blogging since you slipped out the front gate! Only way we avoid repeating history is to study and learn from it!

  11. Lisa says:

    It was tough love, Flight. We just wanted you all to be ready for anything.

  12. mgrabois says:

    One fun case I came up with, I think it was for the STS-116 long sim (rendezvous and dock on 1st day, ISS EVA on 2nd day), was to work with the Space Radiation folks to simulate a potential solar flare in the next 24 hours. So EECOM and Surgeon had to do their homework on radiation shelters, and EVA had to be thinking about what if the flare occurs during the spacewalk, etc. We ultimately had it be a false alarm so that it didn’t mess with the Flight Plan, the objective was to get everyone to dust off their procedures. Then about 6 months later, there was a real solar flare and the Surgeon/BME team thanked us for getting them mentally ready as they hadn’t practiced those procedures in quite a while, and the sim case gave their team the opportunity to update them.

  13. BrianDude says:

    SCE to Aux!

  14. You certainly watched and remember movie The Core. One nerd question: could the shuttle activate its main engines to help correct a wrong landing course?

    • waynehale says:

      Never have watched that movie. The main engines on the shuttle took their fuel from the bid external tank. Once that was jettisoned there was nothing to run those engines.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s