Now to return to a subject I have left pending for too long: The STS-121 launch.
We had three major problems to solve to get approval for the launch.
I hate intermittent electrical problems. It doesn’t matter whether they are on my car, with my home sound system, or on the space shuttle, intermittent electrical problems are the worst. Even when you get the repairman to look at the system when it is not working, sometimes it is nearly impossible to find the problem. As my friends remind me, I’m a Mechanical Engineer by training, I don’t really understand electricity!
So it was with the shuttle in return to flight time. The big orange external tank does not have a gas gauge similar to what you find or a car or boat or airplane; it simply had a few ‘level’ sensors that tell when the tank is filled up to the top (for loading) and when it is empty. There were no measurements in between. Many other rockets are like this. Even telling when the gas left in the tank is at the ‘full’ or ‘empty’ point is not easy when dealing with liquid hydrogen and liquid oxygen because they are so very cold; it takes a special sensor to indicate ‘wet’ or ‘dry’.
Starting with some tanking tests for the first ‘return to flight’ – STS-114 – we started experiencing some problems with the ‘empty’ sensors, called Engine Cutoff (ECO) Sensors because they were there for a critical safety issue. If the fuel – hydrogen – tank ran dry while the engines were still running, the ‘fire’ in the engines would get very hot indeed due to the surplus of oxygen and likely the engine would suffer – a great euphemism – ‘an uncontained failure’. Not what you want.
So, we started troubleshooting: the first suspect was the electronic box (‘point sensor box’) in the orbiter that deciphered the electrical signals from the sensors. I was surprised to find out that this piece of equipment was Apollo heritage! The electronic schematic drawing was signed off in the 1960’s for the upper stage of the Saturn V. We put a team of experts lead by Ed Mango on the investigation. After weeks and many tests on various tanks and orbiter point sensor boxes, the conclusions exonerated this old gear. As the orbiter team members told me: think outside the box. They even had a T-shirt made with that phrase.
Next we investigated the little sensors themselves. A metal cube about an inch on a side; inside was a very fine wire that changed electrical properties depending on whether it was immersed in fluid or not. We found that the electrical connections inside this little sensor could have some issues. Ah ha! Multitudes of x-rays and resistance tests were suspicious but inconclusive. But that had to be it.
Many long hours were spent in meetings and reviews to develop ways to determine if a particular sensor was good or likely to fail. New techniques for manufacturing were proposed. During this time, I elected to make a site visit to the people that build those sensors: the Goodyear aircraft avionics plant in Vergennes, VT. It probably scared the dickens out of the factory technicians to have the Space Shuttle Program Manager come stand at their work bench and watch them make tiny crimp connections on the almost microscopic wires. But we were convinced that was the problem and we were on the road to fixing it. The ET assigned to STS-121 had the ‘best’ sensor boxes we could find.
That was the status as of the Flight Readiness Review in June of 2005. With some reluctance, the FRR board accepted our plans including the wacky logic tree for what to do if more than one sensor failed during the countdown. So, despite all our worry and work, or because of it, ECO sensors were not the reason that there was disagreement over signing off on the CoFR.
I wish I could tell you that was the end of the story, but it wasn’t. Not only did we scrub the first launch attempt for STS-121 because more than one ECO sensor circuit was giving erroneous reasons, but later we found out the real cause: It wasn’t the Point Sensor Box in the Orbiter; it wasn’t the sensors in the bottom of the External Tank. It was the pin connectors on the pass through where the wiring went from inside to outside of the hydrogen tank. Something we thought we had exonerated early on. We had jumped to an erroneous conclusion early in the troubleshooting and spent over a year working on the wrong problem. Somebody from a different program pointed out – much later than STS-121 – that the Delta program had a similar problem which was caused by pin connectors in the tank wall pass through and they had solved their problem by soldering the wires together. Which is what we did. Which solved the problem. After almost two years of work.
I wish I had a nickel for every time we misdiagnosed a problem during our days on the Space Shuttle.