STS-121: The Hardest Launch: Part 2 – Electrical Problems

 

Explaining ECO sensors to the Press

Explaining ECO sensors to the Press

Now to return to a subject I have left pending for too long:  The STS-121 launch.

We had three major problems to solve to get approval for the launch.

I hate intermittent electrical problems.  It doesn’t matter whether they are on my car, with my home sound system, or on the space shuttle, intermittent electrical problems are the worst.  Even when you get the repairman to look at the system when it is not working, sometimes it is nearly impossible to find the problem.  As my friends remind me, I’m a Mechanical Engineer by training, I don’t really understand electricity!

So it was with the shuttle in return to flight time.  The big orange external tank does not have a gas gauge similar to what you find or a car or boat or airplane; it simply had a few ‘level’ sensors that tell when the tank is filled up to the top (for loading) and when it is empty.  There were no measurements in between.  Many other rockets are like this.  Even telling when the gas left in the tank is at the ‘full’ or ‘empty’ point is not easy when dealing with liquid hydrogen and liquid oxygen because they are so very cold; it takes a special sensor to indicate ‘wet’ or ‘dry’.

Starting with some tanking tests for the first ‘return to flight’ – STS-114 – we started experiencing some problems with the ‘empty’ sensors, called Engine Cutoff (ECO) Sensors because they were there for a critical safety issue.  If the fuel – hydrogen – tank ran dry while the engines were still running, the ‘fire’ in the engines would get very hot indeed due to the surplus of oxygen and likely the engine would suffer – a great euphemism – ‘an uncontained failure’.  Not what you want.

So, we started troubleshooting:  the first suspect was the electronic box (‘point sensor box’) in the orbiter that deciphered the electrical signals from the sensors.  I was surprised to find out that this piece of equipment was Apollo heritage!  The electronic schematic drawing was signed off in the 1960’s for the upper stage of the Saturn V.  We put a team of experts lead by Ed Mango on the investigation.  After weeks and many tests on various tanks and orbiter point sensor boxes, the conclusions exonerated this old gear.  As the orbiter team members told me:  think outside the box.  They even had a T-shirt made with that phrase.

Next we investigated the little sensors themselves.  A metal cube about an inch on a side; inside was a very fine wire that changed electrical properties depending on whether it was immersed in fluid or not.  We found that the electrical connections inside this little sensor could have some issues.  Ah ha!  Multitudes of x-rays and resistance tests were suspicious but inconclusive.  But that had to be it.

Many long hours were spent in meetings and reviews to develop ways to determine if a particular sensor was good or likely to fail.  New techniques for manufacturing were proposed.  During this time, I elected to make a site visit to the people that build those sensors:  the Goodyear aircraft avionics plant in Vergennes, VT.  It probably scared the dickens out of the factory technicians to have the Space Shuttle Program Manager come stand at their work bench and watch them make tiny crimp connections on the almost microscopic wires.  But we were convinced that was the problem and we were on the road to fixing it.  The ET assigned to STS-121 had the ‘best’ sensor boxes we could find.

That was the status as of the Flight Readiness Review in June of 2005.  With some reluctance, the FRR board accepted our plans including the wacky logic tree for what to do if more than one sensor failed during the countdown.  So, despite all our worry and work, or because of it, ECO sensors were not the reason that there was disagreement over signing off on the CoFR.

 

I wish I could tell you that was the end of the story, but it wasn’t.  Not only did we scrub the first launch attempt for STS-121 because more than one ECO sensor circuit was giving erroneous reasons, but later we found out the real cause:  It wasn’t the Point Sensor Box in the Orbiter; it wasn’t the sensors in the bottom of the External Tank.  It was the pin connectors on the pass through where the wiring went from inside to outside of the hydrogen tank.  Something we thought we had exonerated early on.  We had jumped to an erroneous conclusion early in the troubleshooting and spent over a year working on the wrong problem.  Somebody from a different program pointed out – much later than STS-121 – that the Delta program had a similar problem which was caused by pin connectors in the tank wall pass through and they had solved their problem by soldering the wires together.  Which is what we did.  Which solved the problem.  After almost two years of work.

I wish I had a nickel for every time we misdiagnosed a problem during our days on the Space Shuttle.

 

ECO sensorsPin Connector

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
This entry was posted in Uncategorized. Bookmark the permalink.

13 Responses to STS-121: The Hardest Launch: Part 2 – Electrical Problems

  1. David SUndström says:

    Wayne, you’re getting your mission confused here- STS-121 was scrubbed twice, but that was for weather (July 1 2006, anvil clouds and July 2 2006 due to poor forecast that included active t-storms).

    STS-114 was scrubbed once and that was due to the LH2 LLCO sensor#3 failing it’s mandated OMRS pre-launch test. Spent nearly two weeks trying to track them culprit with the investigation focusing the orbiter PSB in Avionics Bay 5.

    ET-119 had its LH2 LLCO senors R&R’ed in the VAB after a couple had shown questionable resistance checks prior to shipment to KSC from MAF. This along with the need to send Discovery’s specially instrumented RMS back to MDA in Canada for repairs after an OPF bucket banged into the upper composite arm section conspired to delay STS.121 from May 2006 to July 2006.

    STS-115 ended up with a scrub to due a LH2 LLCO sensor#3 failure during it’s September 8 launch attempt. Recycled for an attempt 24hrs later and all sensors worked fine per the plan (mysterious healing of the sensors not yet understood).

    Everything stayed fine with the system for over a year until STS-122 in December 2007. Then during the first launch attempt on December 6 2007, LH2 LLCO sensor#3 once again failed. Launch scrubbed. After a lengthy MMT discussion decision was made to shorten the launch window to just 60 seconds along with requiring all 4 LH2 LLCO sensors to work perfectly and the try again for December 8. Then everything seemed fine again, matching the prior history of the sensor failures until sensor#3 again failed with sensor#1 setting up to fail. And since the launch window for December was running out anyway, the decision was made to stand down and conduct a full investigation.

    That investigation was what lead to the specially instrumented tanking test that showed that the external feedthrough connector on the LH2 tank of the ET was the true source of the problems all along, Then the decision was made to bring in some techs had done a similar fix for the Centaur used on the Atlas (not Delta, Delta didn’t used cryos on the upper stage until Delta III).

  2. Todd Martin says:

    I think of these issues and solutions as being equally important to pure scientific progress. What I found most gratifying in your story was the way you were able to take and use the hard lessons learned from the Delta 2 folks. In my work designing and troubleshooting pharmaceutical tooling, I’ve done my best to capture past people’s work and build a technical library to assist in new projects. Engineering knowledge is precious and you should be proud of your part in building it.

  3. DerekL says:

    Intermittents are the worst… On 655 we had an intermittent memory failure pop up in DCC2 (one of our two main computers) right after we left the shipyards in ’85. It would persist for an hour or two and then vanish in the wind. It would pop up two or three times a year, and we were still working it when I left in ’87. They were *still* working it when the boat was decommissioned in ’93. We replaced the memory modules in toto at least twice. We replaced the backplane (also twice). And still it popped up. When the decommissioning schedule came through they were preparing for a Hail Mary nuclear option – ripping out and replacing the entire related cabling loom.

    Always thought there had to be something snaky in the design myself. The computers on 655, the trainers at Charleston and Bangor, I saw more memory problems than any other class of problem.

    More fun though was when the system failed in a way The Book said was impossible…

  4. lion says:

    Dear god, 2 years of meetings, reviews, & airport security lines. We’ve since heard that the modern technique is a lot less meeting & reviewing, but a lot more prototyping & testing. Wonder if the modern technique would have saved any of the 2 years.

  5. JC Carleton says:

    Ah, remember it well. I’m also a Mechanical Engineer but on the SRB side and we had those kind of times also. Great days!

  6. Thank you for an excellent explanation of a tough problem. I remember the launches and last minute scrubs.

  7. ljwegendt says:

    That is called a bulkhead electrical connector

  8. Dave H. says:

    Wayne,

    From what I remember, the reason the ECO sensors are of vital importance is due to the shuttle’s engine design. The LH2 pump is driven by the O2, the O2 pump is driven by the LH2. Loss of either pumped liquid will result in the now-unloaded pump to exceed its 37,000 RPM design speed and explode. This will result in the “blowing out of the back end of the orbiter and a very bad day for everyone!”. So I’ve been told.

    The ECO sensor is what we instrument geeks call an RTD, short for “resistance temperature detector”. The ECO sensor is a 1/4 inch length of platinum wire, and follows the same temperature/resistance curves as a standard Honeywell RTD commonly found in industry. When its resistance changes as a result of LH2 level transitioning from liquid to gaseous, that change provides the engine cutoff signal to the controls.

    The resistance of the RTD increases with temperature. What was happening with the bulkhead connector is a phenomenon called “cold creep”, where the cold was “traveling” (being mechanically transferred through) through the wires into the pins of the connector. Because of the temperature difference between liquid nitrogen and liquid hydrogen, this was not showing up when tests were performed with liquid nitrogen in the tank.

    When NASA brought in a company from nearby Coraopolis, PA that employed a time-domain reflectometer, and filled the tank with liquid LH2, they discovered that the problem was the bulkhead connector. The cold was causing the water vapor from the air trapped inside the connector to freeze around the male pins, causing the female receptacle to spread and acting as an insulator (pure water has very low conductance). The reflectometer told them the distance from the instrument, and the drawings gave the engineers the distance.

    The cure was to eliminate the bulkhead connector and silver-solder the wiring.

    Everyone in maintenance falls prey to troubleshooting the wrong problem at times. It’s just part of the job. The trick is to listen when someone says “Maybe we’re barking up the wrong tree?” and not dismiss them outright.

    • Wes says:

      It wasn’t just the water. There was a grease that was used to put the connector on and make sure the threaded connector didn’t gall (in case you needed to get it off again). We went through several different test phases testing first the water and grease separately that showed neither would cause the intermittents on it’s own. This was the reason for the initial exoneration. However, when we put the two together, we hit the smoking gun. The grease caused the water to bead locally [sometimes] creating the forces to keep the pins out of contact. We ran a lot of tests in the CryoTestLab watching the chilldown process on many different pin connector types and combinations. At the end there was some fancy statistics and a report which was never released. I think they changed the pin connector type in addition to soldering them down.

  9. BradK says:

    I’ve been a follower of your blog for some time and I just wanted to say thank you for taking the time to do it. I’m glad to see that you at it again.

  10. John Ragley says:

    I remember this one well. I was part of the team at the NSLD tasked with “repairing” the Point Sensor Boxes once this mess started. Although the PSB’s were exonerated, we still had all 5 of them in a state of repair at the depot, trying to get just one back to KSC for a launch. The electronics were fine, but the metal guides that held the circuit cards in place kept de-bonding every time we re-installed the cards. After several months and many different products were tried (with Program approval of course), we managed to ship all of the boxes back to KSC.

    P.S. I actually had one of the shirts that said “It Ain’t the Box!”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s