The recent success of SpaceX in sending their Dragon capsule to the International Space Station reminded me of the first Space Shuttle ground up rendezvous and the difficulties we had during that mission. Of course we were planned it on the 13th shuttle mission rather than the third, like SpaceX, but you would have to have triskaidekaphobia to believe that was the cause of our problems.
Rendezvous in space was not a new subject in 1984; it had been pioneered by the Gemini program nearly 20 years earlier, and practiced by necessity during Apollo. Skylab and ASTP required ground up rendezvous. The Russians accomplished ground up rendezvous capability shortly after the US did. Even during earlier shuttle flights there had been rendezvous with deployed satellites, the successful SPAS deploy and retrieval on STS-7 for example. But for STS-13 we planned the first ‘ground up rendezvous’ for the shuttle program with a target that was already in orbit; this was a more complex mission than the deploy/retrieve on shuttle flight 7.
Jay Greene, Emerald Flight, was the lead for the mission, a veteran FDO from Apollo who knew a few things about rendezvous in space; Gary Coen, Gray Flight, was to be the ascent flight director on this auspicious mission; John Cox, Granite Flight, was the EVA specialist; I was the lead Propulsion systems officer; the holder of the “bingo” number which would terminate the rendezvous; Ron Dittemore was Prop officer on the Granite team which would monitor the grapple and EVA repair of the ailing Solar Maximum Mission (SMM)-the objective of the flight.
Launching to catch up with an orbiting satellite requires complex mathematics, precise timing, and variable steering from the launch vehicle. Monitoring the launch from Mission Control requires complex computations; being able to correct errors during the launch loads even more computational requirements on the MCC. Back in those ancient days, there were 5 IBM 370/65 mainframe computers which drove the MCC. Those behemoths collected all the telemetry, processed it for display on small black and white monitors, made all the trajectory calculations, basically did everything. The prime computer was designated the Mission Operations Computer (MOC) and the backup, in lock step with the MOC, was the Dynamic Standby Computer (DSC). The other three computers were held in a ready state but not active. In theory, if the MOC crashed (not unheard of) a select over to the DSC could be accomplished in seconds with no loss of monitoring.
Somewhere during the preparation of the mission, senior NASA managers decided that the shuttle mission numbering scheme was . . . troublesome. The STS-10 mission had been scrubbed out of the manifest, STS-12 greatly delayed, and the STS-13 mission designation apparently gave some people . . . concern. So a very complex, almost incomprehensible, mission numbering scheme was invented: two digits and a letter designation. First digit, the fiscal year the mission was originally planned (1984 became the “4”), second digit, the launch site (1 for KSC, 2 for Vandenberg), the letter was the sequence of the mission as originally planned for that year. So STS-11 became STS-41B and STS-13 became STS-41C. Really complex; really bureaucratic, and in my opinion, really dumb. I still don’t know how that would have worked when we wrapped around the end of the decade. Later, of course, we resumed the old fashioned numbering scheme and if flights flip-flopped on the manifest we just lived with it.
As usual, for flight STS-41C, we practiced the mission dozens of times with the crew in the simulators and the full flight control team in the MCC. Sim Sup gave us hundreds of problems to solve to save the mission. We were ready. And on April 6, the orbits aligned, the weather was good, and Challenger and her crew launched right on time. Immediately after liftoff, the MOC promptly died. Select over and . . . the DSC was dead. First time in the shuttle history, the MCC was useless during launch. A common software problem had stopped processing in both machines. MCC was blind and the crew was on their own. Gray Flight was not happy. The Prop console was located just in front of the Flight Director console. We heard some choice words from the Flight Director that morning as he spoke with the ground computer operators. On the row behind the Flight Director console was the Mission Operations Director console, and the legendary White Flight, Gene Kranz, added to the discussion in unforgettable terms.
But Challenger was flawless that day, and the climb to orbit went according to plan. Within an hour, a new MOC was up and running and we tried to forget the helpless feeling we had listening to the crew radio calls during the data-less ascent. Problem number 1 of the mission appeared not to have any lasting impact.
The first objective of the mission, deployment of the school-bus-sized Long Duration Exposure Facility, went off without incident. Then we set our sights on the crippled SMM. Bob Crippen was the Commander, a veteran of the STS-7 rendezvous, and everything went exactly according to plan. For the first time in shuttle history, we put a crew out the airlock during a rendezvous to be ready to grab the SMM. George ‘Pinky’ Nelson would be flying over to the SMM using the Manned Maneuvering Unit backpack; he had a device called the TPAD to lock onto the SMM strapped to the front of his space suit. Handover between the Emerald Flight and Granite Flight teams occurred while we were stationkeeping with the SMM. Even though I was technically off shift, I wanted to watch the retrieval so I scooted over to the left seat on the Prop console during Pinky’s short flight to the SMM.
Then everything went to worms. The TPAD device would not attach to the SMM. Worse, when Pinky bumped the SMM, it started tumbling and translating away from Challenger. On the Prop console was saw lots of thruster firings and the gas gauges heading down toward the “bingo”. Crip lost sight to Pinky and started to maneuver the shuttle to keep him in view. We were using propulsion gas at a fantastic rate. Granite Flight leaned over the console and asked Ron if we could keep this up “overnight” since the sun was about to set for that orbit. Ron replied he didn’t know (we had never simmed this!), and in an instant that will forever be etched in my memory, Gene Kranz leaned over the console and said “you better figure that out, Prop!” I wouldn’t say chaos reigned, but it was close. Calculators in the Prop back room began smoking as new bingos were calculated. The potential for catastrophe was high: the orbiter could run out of gas and not be able to safely deorbit; or the orbiter could run into the SMM or into Pinky, still free flying, or the orbiter could get so far away that Pinky’s MMU couldn’t get him back to the shuttle. Let’s just say that our heart rates were high. Crip somehow kept Pinky in sight – the definition of cool in a crisis – and moments later Pinky gave up, flew his MMU back to Challenger, and the shuttle backed away.
In American football, they call it broken field running; when the play doesn’t go as planned, and the team has to improvise while the clock is ticking. The next 24 hours were a blur. With little propellant reserve, we had to plan another rendezvous, stationkeep, and Terry Hart had to try to grapple the restabilized SMM with the shuttle arm. Somehow the team pulled all that off. SMM was repaired, nobody died, Challenger landed safely, and the flight went into the history books.
We learned a lot about rendezvous that flight.
So when a ground up rendezvous goes smoothly, as almost all the shuttle flights to the ISS did decades later, or as the SpaceX flight did last week, I remember how hard it can be. I smile, and remember how we nearly lost it, on a flight that was originally planned to be #13.
Triskaidekaphobia? No. But some days I do wonder. As my old boss, an Apollo veteran, would often remind us “It’s good to be smart, but it’s better to be lucky.”
That was a very good story Wayne, I love to hear these stories and all that went into the shuttle, it was an amazing machine and I feel lucky to have finally senn a launch….. the last launch! what a memory! I sure do love hearing your memories of the shuttle! NOTHING beats a great memory!!
I also love hearing the war stories from Mission Control. As someone who has always been fascinated with orbital mechanics, I especially enjoyed this post, even if this mission turned into a trajectory analyst’s nightmare.
Thanks for the kick to memory vault.
Your story reinforces a thought that that occurred during the LIDAR troubles that slowed the approach. Everyone was calm, every call was clear, and everything fell into place. I expect that there were plenty of words and work in the background back the professional effort that ended in success has given me the greatest sense that we have a great future in space.
Too bad it’s happening decades too late. I would have liked to seen footprints on the moon again.
Whew, what a story. I wonder how much, if any of this, goes on today. What is seen from NASA TV is as Charlie McCue observes: cool professionalism.
We always wanted it to look cool, calm, professional. Under the surface it was rarely that way!
It’s like what they say about ducks, isn’t it? Calm and serene on the water’s surface, but paddling like mad underneath. You guys did an incredible job, and as an American and a taxpayer, I’m proud of what you all did.
Thanks for juxtaposing 41-C with Dragon’s first rendezvous, Wayne. Although decades apart, these mission illustrate how, with sufficient training and preparation, two different teams having minimal flight experience with their respective vehicles could overcome adversity and accomplish an initial ground-up rendezvous. History is proving NASA has no monopoly on success in space operations, and it will undoubtedly do likewise for failures. The folks at SpaceX deserve congratulations, but they’d also be the first to admit rendezvous isn’t as easy as they’d first thought. Hopefully, that revelation won’t significantly affect the $s per kg delivered to ISS!
If I may digress into space trivia, I don’t believe it’s necessary to qualify 41-C as just the first ground-up rendezvous in the Shuttle Program. In my opinion, it was also the first rendezvous. On STS-7, Challenger maneuvered no more than 1000 ft from SPAS at very low rates, and that’s proximity operations; not rendezvous. In a post-flight presentation accessible at http://www.nss.org/resources/library/shuttlevideos/shuttle07.htm, the STS-7 crew calls the SPAS activity “prox ops” too. On 41-B, I believe some rendezvous operations tests were performed on a target balloon ejected from the payload bay (which failed to stay inflated), but no attempt to rendezvous with it was made.
You mentioned gas gauges, how reliable were they? And was it possible to do some propellant calculations based on the number of and length of thruster firings?
The Shuttle reaction control system was a hypergolic, pressure fed system. The gaging was using the PVT method on the pressurization tanks. Stated accuracy was 10% but in practice if thermal equalibrium was achieved, it was much more accurate than that. Counting thruster firings could be used as a backup measure but it required offline analysis and was not a real time tool.
In many pictures of the apollo-era houston and whatnot I can see displays with nice trajectories drawn. How were these drawn? you said that in shuttle times a bunch of IBM’s did it, but what about apollo times?
Before my time. I’ll have to check with my elders
Fascinating stories we never heard before. Thanks Wayne. I hope Dan is right and SpaceX realizes how difficult, not just Rendezvous is, but how difficult spaceflight is. My impression during their press conference following capture and docking of Dragon, was that they since they had proven they could re-enter and land safely, they weren’t worried about Dragon’s re-entry and landing. I am just praying with their continued success, they do not let their guard down and learn from the mistakes of the past. Thanks again Wayne!
I am reminded each time I read one of our posts, that spaceflight is one of the most amazing and challenging things we do as humans. I am thankful for the great minds and dedicated folks that have made our space program possible.
What SpaceX has done is nothing short of amazing. Do you think this opens the door, at least to LEO, for the general public within the next 10 years?
Thanks again for another great post.
Time will tell!
The success of COTS 2 was all the more remarkable after reading the Hale blog, but it took 18 months for that 1 launch. It’s not an economically viable launch rate, but they did it. That they succeeded without a strict policy of hiring exclusively engineering degree holders should give a boost to all the hobbyists.
As an engineer, I can firmly say that some of the best and most competent and innovative people have never seen the inside of an engineering school.
You’ve talked about a number of your colleagues (most recently John Muratore) in the past. Any chance of writing a bit about working with Gene Kranz?
Very difficult to write about the legends of our times . . .
Interesting story, Wayne. Most people tend to think of how NASA does things in the same terms as “Armageddon”‘s Harry Stamper…backups upon backups, people thinking stuff up…and we are disappointed when our heroes are as human as the rest of us.
During the timeframe you describe, computers and distributive control systems were in their infancy. What we today call “Stuxnet” was a standard operating “feature” of the era. Like enerything else, processors got better and programmers became more proficient. Scenarios as you describe are today the exception, not the rule.
As you well know, success has two companions: arrogance and complacency. SpaceX’s success will lead to greater things, but it will inevitably lead to cost-cutting until the thin margin of safety is breached.
Those who fail to learn from history are doomed to repeat its mistakes.
I refuse to believe in the eventual triumph of evil; arrogance and complacency can be overcome. Success comes from humility and constant vigilance.
As for processors getting better and programmers becoming more proficient – I had to rebuoot my MCC distributed workstation often; matter of fact, I have to reboot my PC often.
Wayne, may I quote your “Success comes from humility and constant vigilance” at our weekly safety meeting? It dovetails nicely with something I said a few weeks ago: the price of safety, much like that of freedom itself, is eternal vigilance.
Out in the industrial world, many processes from electric power to wastewater treatment are being controlled by distributive control systems and programmable logic controllers. While it’s true that the first generation of these were subject to various failure modes, there are many systems whose MTBF ratings are measured in years. Trust me, they’ve learned and gotten better.
I run two Dells and a Gateway at home, both running Windows XP, and I really can’t recall rebooting being an issue. I can’t say I share your problems here.
“Evil” is a word only applied to engineering when something unforeseen or unintended occurs. A good example would be certain two-stroke Japanese street motorcycles of 40 years ago. The engines would easily overpower the frame and suspension, with truly evil results.
Still, Edmund Burke’s statement about evil triumphing if good men do nothing comes to mind.
Great blog post Wayne. You need to write a book about your days at NASA. PLEASE!
Fascinating as always! I was wondering what the launch coverage was like after the computer failures you described, so I found the launch on YouTube:
Contrary to my expectations, the PAO ends up calling out data, especially after 1:45 in the video. How was he able to do that? Was he listing to information being read down by the crew that isn’t being broadcast, was there some data still making it to mission control, is he just basing the numbers on pre-launch expectations for certain times in the ascent, or is there another mechanism that I’m not thinking of? With the issues you described, that was a really calm public face on the launch; watching the video you’d never guess that anything was going wrong!
Public Affairs officer was reading off the pre-flight predicted numbers at that time. The launch was nominal; just MCC had a problem.
I seem to recall that the RMS grapple on the second day was possible because the SMM was restabilized using magnetic torquers. But there was some reason the original plan was to use EVA rather than to grapple the SMM with the RMS directly, I cannot remember why, perhaps the attitude was wrong?
The EVA “grapple” attempt failed, the mechanism didn’t work
The TPAD adapter failed to engage because of an undocumented fiberglass pin holding on the multilayer thermal insulation (MLI) on the outside of the MMS bus section of the SSM spacecraft. That kept the TPAD from getting far enough onto the trunnion pin on the MMS and automatically engaging.
I was the Flight Dynamics Officer that guided Challenger to Solar Max and devised the maneuvering plan that enabled us to return for the final rendezvous while staying with fuel redlines. Wayne – great to hear (read?) from you after so many years!