Memory Fault

I’ve been thinking a lot about spaceflight accidents this week and how to avoid them.  One technique is to study lessons from past accidents and near misses to learn what can go wrong and how to avoid being the subject of the next Presidential commission investigating a disaster.

 

Several years ago, when I was running the Space Shuttle program, we contracted with a small firm to help write up about two dozen close call incidents that had occurred during various shuttle flights.  A fairly broad spectrum of folks associated with the shuttle were asked to describe a close call that they remembered, and the contract team would do the hard work of pulling the information out of the archives and putting it into readable form.  These two dozen close call incidents then became didactic tools to teach new members of the space flight team – and remind old ones – of just how close to the edge of the cliff we skate in human space flight – and how to avoid the big fall.

I wish you could read these, I think they would be very valuable to those folks who are contemplating building a spacecraft, for example.  But in the IT Security world at NASA, the decree was made that these lessons are protected by firewall from the outside world.  Something that you all paid for as tax payers is simply not available for your perusal.  But my point today is not the lack of transparency at NASA.

As the word spread, many more than two dozen close calls were identified, so we had to pick the most illustrative.

I, of course, contributed to the list.  In particular there is one flight that stands crystal clear in my memory.

The Spacelab module has an emergency vent valve in case of fire or toxic leak.  The crew would (theoretically) throw the valve open and evacuate the lab shutting the pressure hatch from the crew compartment behind them.  This is a manual valve and cannot be operated remotely.  Of course, it is normally in the closed position.  On one launch attempt of a spacelab flight, the launch was scrubbed for a few days.  During the interim before the next launch attempt, the IFM (In Flight Maintenance) guys spent their time pouring over the closeout photos that were taking of the interior of the shuttle and the spacelab.  Hundreds of photos are taken to document the condition of every possible part prelaunch. The IFM guys are tasked to be ready to fix anything that might break inside the habitable volume, so studying the close out photos has a lot of value in preparing for any eventuality.  To their horror they discovered that the Spacelab emergency vent valve was in the full open/depress position.

So if we had launched, the hatch between spacelab and the crew module was closed, so the crew would have been in no danger; but during the climb to orbit, while the crew was strapped in their seats, the spacelab would have totally depressed and there would be no way to repressurize it on orbit.  Loss of mission, probably early return, certainly equipment damage to the Spacelab and its experiments would have resulted.

But scrubbing the launch and the vigilance of one guy going beyond his usual duty saved the day.  The Spacelab module was opened up, the valve positioned correctly, and the mission launched and was fully successful.

Great story, right?  It is crystal clear in my memory; I’m just a little hazy about exactly which mission number it was.  But no problem there were only about a dozen Spacelab module flights so finding the records would not be hard.

Except they couldn’t find any record of any incident like the one I remembered.

There is nothing in the records, and worse, nowhere in the memory of the IFM guys, the Spacelab guys, or the KSC riggers.  Nobody remembered this except me.  But I knew it to be true – so I sent them back to search again.  Looking for this incident because a priority for me.  But to no avail.

It never happened.

So what am I to say?  Obviously my memory is at fault.  That is a terrible thing to contemplate.  Did I dream it?  Or maybe there was a simulated mission where this was one of the failure conditions that Mission Control was supposed to handle, and that training has gotten mixed up in my mind with the real flights.  Or it could be something worse.  I think that at the very least it says that I’m fallible, at least as fallible as the next guy, maybe more so.

That is a good lesson to learn.  That I (and you) are not as smart as we think we are.  That my (and your) memory is not as good as you think it is.  That there is value in having good documentation and a team to look over the data.

But nonetheless: check your closeout photos before you launch.

 

 

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

37 Responses to Memory Fault

  1. Great story and I think it’s fairly common for intelligent people to pull different pieces of information together and be absolutely certain about an even that never happended. I have done this myself on more than one occasion and so now I do use some NASA like tricks of checklists for certain key system deployments and photographs of network cables and so on. It’s saved me grief on more than one occasion as well so well worth the effort.

    By the way, I do recall reading about an incident in the simulator right near the beginning of the shuttle programme. The story went something about the crew being put through a simulated TAL and all the computers crashing with big green ‘X”s on them resulting in a very upset crew and several changes to how software updates were done. I’ve not been able to find that story again though – Are you aware of it?

    • waynehale says:

      Sorry, that was a long time ago and I don’t remember it. But there have been many training simulations where problems with the flight software were uncovered accidentally.

  2. Safety John says:

    The funny thing is, good lessons can come sprout as well from what could have happened, as from what actually happened. If the potential is real, and the prevention is clear, it doesn’t really matter whether it materialized on a given day or not, the lesson is still valid. Fables like “the tortoise and the hare” are one way this technique has been used for generations. Good teaching moment Wayne.

  3. P. Savio says:

    Slightly unrelated to your story but directly related to one of your previous posts relating to the “Abort Request Command”

    At 11:35 into this clip Hugh Harris refers to the test of the “Abort Advisory System” – this is the only time I’ve heard a reference to it during a launch countdown that I can remember.

    This series of STS-1 Youtube clips is great to watch as it covers hours of the countdown and launch from NASA TV – great memories of that day – including all those sideburns !!!

  4. Bill Miller says:

    @Gary Williams, the document at this link describes the sim run that I think you are referencing. In any case, it’s a fascinating account of the early days of Shuttle DPS.
    http://klabs.org/DEI/Processor/shuttle/shuttle_primary_computer_system.pdf

    • waynehale says:

      Sorry Bill, I read the entire article – and it is fascinating about the shuttle flight software, but I found nothing in it about a simulation run with Spacelab vent valve misconfiguration. Maybe you have the wrong attachment?

      • Bill Miller says:

        Apologies for not being clear (and for hijacking your blog post). I was responding to Mr. William’s question about the pre-STS-2 sim case.

      • Sorry Wayne, that was in answer to my question and was the very story I was looking for. It’s quite fascinating how the simulator systems can run the exact same software as the actual shuttle. As an IT person I’m always interested in the Shuttle/station computer systems and how the software people write and test code.
        Thanks for finding that link Bill.

      • waynehale says:

        The shuttle training simulators located in B5 and B35 at JSC run the flight software in special host equipment that simulates the orbiter’s general purpose computers. Its quite a goood system.

  5. pat b says:

    Seems like the valve should have had a safety wire and flag to leave it normally
    in the closed position and make it obvious if it was pulled.

    Or the valve should have been behind a glass to break to turn.

  6. pat b says:

    could the crew have treated the issue as a in flight depress, done a prebreath,
    put on their pressure suits, depressed the cabin and let someone
    go into the spacelab and shut the valve?

  7. Charley S McCue says:

    Yes, this is certainly time to reflect.

    Also, a different spin on this memory. This made a profound impact on you. Maybe the files are there and for everyone else, it was business as usual. So you remember and they don’t. You understood the problem and they found it acceptable.

    For me, that echos a strong memory from both the Challenger and Columbia investigations. One of the STS project leaders asked the commissions to answer the same question, why was the original requirement of no strikes to the TPS not followed.

    At least one answer given was each flight flew with strikes and damage and the common wisdom was it was acceptable.

    • waynehale says:

      I think you have taken a left turn somewhere. I am convinced, after searching myself, that this particular incident did not occur. It is a real stretch to connect my faulty memory with the problems that brought down Columbia.

      • Charley S McCue says:

        Sorry. Only meant meant that if it occurred, then the reason no one else remembers is that the did not see the same danger as you. The rest was a failed attempt to illustrate that.

  8. Steve T. says:

    Wayne- I have enjoyed your blog posts since they started and particularly this one. Your point is right on about using memory in the way you described, but even more so than you might realize. It’s a lesson learned not only in the way memory can be relied upon, but in how information is passed along. What you described did occur on (or rather, prior to) a Spacelab flight- just not not quite the way you’re thinking about it (I think I can just say “different valve”). I know there are limitations about what can be said here, but you would find it a fascinating story (and one very much akin to when something is said at one end of a line of folks only to morph into something completely different at the end).

    • waynehale says:

      Do tell. I am very interested. Maybe I’m not ready for the old folks home yet.

      • Steve T. says:

        No sir, you’re not ready for that yet! There was a mis-configured valve in one of the experiments in the lab, designed to allow the experiment to vent to vacuum on orbit. It’s this valve you’re thinking of earlier. The question started out asking if anyone had heard of something like this happening and morphed into a discussion of the cabin depress valve, as communications without a lot of details will sometimes do. At one point your name is mentioned, but the conclusion never got back around to you.

      • waynehale says:

        Well, don’t hold back, give us some of the details Steve

  9. Graham says:

    I wonder if an FOIA request could turn up these close call incident reports? Or do you mean that they were classified?

    • waynehale says:

      Oh they are not classified; a FOIA could do it, but I was somewhat hoping that mere embarrassment due to my mentioning the subject might motivate releasing the case studies.

  10. Barbatsalos Dimitris says:

    Hello Mr. Hale and thank you for your wonderful posts!

    There is a small typo in case you would like to correct it. It is in the fifth paragraph from the end: “Looking for this incident because a priority for me.” should read “Looking for this incident became a priority for me.”

    Kind reagrds,
    Dimitris Barbatsalos
    Athens, Greece

  11. Steve T. says:

    It was during the turnaround between the first and second launch attempts of STS-55 (there were three attempts; the third was successful). The experiment valve in question was leak checked and failed. They discovered the valve was not in the fully closed position and a redundant sealing cap was not installed correctly. The misconfig likely happened during CEIT. Due to the dilligence you described, the situation was found and corrected. Over the years, it became a story about the depress valve you alluded to in your blog. Typically, the folks who were discussing this are not involved a lot in nitty-gritty pre-launch work, so that’s likely why it got turned into a depress valve story (so probably a lesson more in communicating to upper management properly!). This does not as far as I know involve any data collected from the case studies.

  12. Dave H. says:

    Wayne, are you really, REALLY sure that it “never happened”?
    Hollywood has tons of fun with scenarios such as you describe here.

    But seriously, this is why people keep workplace journals and save every fax, e-mail, and anything associated with whatever it is that they’re working on.

    Even though it’s been over two years since I left my last employer, I still get e-mails from the techs with problems they cannot fix. For example, I’ve spent most of the afternoon conversing with a tech who is unable to find the cause of a combustion air oscillation in a boiler providing steam to a 50MW generator. The control system is a pneumatic Hagan, and I still remember how those systems were designed.
    But I seem to be the “last of the singing cowboys”. Even the in-house techs don’t understand how this control system works.

    Thanks to Steve T., it would appear that your memory is not “faulty” in the least.

  13. Mike W says:

    Human memory is a funny thing. Its reliability isn’t simply a matter of whether the neurons’ connections were made correctly, but is in fact colored and altered by our perceptions and the stories we build up around those things that we do recall.

    There’s a school of thought around this that states that the stories we tell ourselves about things that have happened actually alter our memories of those events, and eventually we believe our stories even if the original memories were quite different. We can no longer recall the original events, but rely entirely on our stories, because that’s all we can recall. Even worse — because we all filter, via perception and interpretation, everything we experience we can’t even be sure that the original memory of events wasn’t altered as we experienced it; as strange as that may sound, it seems to be a fact of life.

    Physiologically, it may be that the neurons’ connections were simply altered to strengthen connections related to the story and wither or shut down connections related to the original memory, or that our existing connections altered our original perception of the events in question.

    In either case, memory is only truly reliable if we can set aside our “stories” and “filters.” This, I can attest from personal experience, is truly something that requires exceptional awareness and constant vigilance. It doesn’t come naturally… we have to work hard at it.

    • Heather says:

      “the stories we tell ourselves about things that have happened actually alter our memories of those events, and eventually we believe our stories even if the original memories were quite different. We can no longer recall the original events, but rely entirely on our stories, because that’s all we can recall. Even worse — because we all filter, via perception and interpretation, everything we experience we can’t even be sure that the original memory of events wasn’t altered as we experienced it.”

      Nicely stated, Mike W. This is a very good way of explaining the quirks of how our memory works. Thanks!

  14. Ronald Smith says:

    When I was in first grade, I used to read space books in my library that talked about how when flown, SpaceLab would change the world. The Year was 1993, and the book was already 20 years old ……. I was very surprised years later to find out Spacelab had already flown four years before I was born.

    Just wondering, do you have any stories on working with SpaceHab (company and modules), should be a good story to tie in the new commercial partnerships.

  15. Chris says:

    Wayne, I do understand that you are prepared to write the incident off as a form of imagined memory, but STS-55 was the Spacelab D2 mission, in which one of the payload facilities was a German materials science rack.
    Would there not have been a discussion with the German space agency people, once the wrong valve position was discovered? Possibly you could get verification from that corner?

  16. Brian N says:

    Wayne,

    I recently embarked on a similar venture (much smaller scale) to capture lessons from my field during the short time that I have been privileged to work for the Space Shuttle Program. During my research, I found that some of the folks that have worked for decades on the Shuttle program had conflicting memories with other folks on certain incidents. Reconciling these old memories was the most rewarding part of this effort. Many of my findings were never formally archived (except in a worker’s memory banks). Below is a link to my final report (if you care to peruse it):

    http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100042352_2010045106.pdf

  17. Grant Henson says:

    The way an organization reacts to close calls tells you a lot.

    Treating a close call as a learning experience and encouraging open debate about why it happened and how to prevent it from happening in the future, is a pretty good sign of a healthy organization

    Refusing to acknowledge a close call except to say, “This just goes to prove how robust our system is! Yay us!!” is a pretty good sign of impending disaster.

  18. Steve Pemberton says:

    I am always amazed when I listen to airline pilots going through their checklists. If you didn’t know better you would think that this was their first time flying an airplane. In reality they are doing a repetitive task that they have done for years, for perhaps the thousandth time, and yet they refuse (and actually are not allowed) to trust their memory. For me this has been a life lesson in the importance of treating complicated tasks with respect. I remind myself that if those guys need checklists for what they do, then surely I do also when I am doing a task of any importance.

    Human memory is a wonder and a mystery, and its complexity goes far beyond what computers are capable of. However even basic computers handily beat humans at reliably recording details. I don’t relish the idea of having a computer chip implanted in my brain, but in lieu of that humans are always going to have to write down anything that is going to matter to them more twenty minutes from now.

  19. Beth Webber says:

    Wayne,
    After missing your blog on the NASA website for the past several months, I finally had the bright idea of doing a Google search for you; should have done this a couple days after you retired! So glad to find you, and to read once more your fascinating insights in our Space adventure.

    Beth

  20. Richard F says:

    Your memory may be better than you think. In late 2005, I corresponded with the JSC folks compiling your excellent close call histories. I suggested that you were recalling STS-35 which was a Spacelab pallet flight (payload bay telescopes, no pressurized module). The misconfigured valve was in the Shuttle airlock. It was caught in August 1990 and recorded by KSC as IPR 293. At the time, I was at JSC in MOD EVA and was the one who spotted the valve during normal review of airlock/suit closeout photos. To myself and others I know, this is one of many real experiences that the future should heed, but are being lost to posterity due to past/current attrition and gaps in knowledge capture processes. It reminded me that nothing should be taken for granted with space flight, that there is no substitute for attention to detail or technical competence, and checks/balances are essential.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s