Miracle Workers

Scotty : Oh, you didn’t tell the captain how long it would really take, ya?  Oh, laddie. You’ve got a lot to learn if you want people to think of you as a miracle worker.

In July of 2005 we returned the shuttle to flight on STS-114.  I wrote about it earlier:  https://waynehale.wordpress.com/2012/04/18/how-we-nearly-lost-discovery/   We had made a terrible mistake that had to be rectified before the shuttle could be flown again.

At the end of August 2005, Hurricane Katrina slammed into New Orleans and nearly submerged the factory at Michoud – the only place in the world was that we could make or modify the External Tanks.  Even though the factory was saved, the nearly 2000 people that did the work to make those precision tanks lost their homes, their cars, and anything else that could not be evacuated in time.  It was weeks before we could even get in contact with all the evacuated workers and it would be months before their soggy houses could be made livable again.

Bill Parsons was the Program Manager and I was his deputy.  But Bill had close ties to the region and especially the hard-hit Stennis Space Center just a few miles again.  NASA tapped Bill to lead the recovery efforts for SSC and the other affected NASA facilities.  They promoted me to take his place in the program.

It was not long into September when headquarters started asking how soon we could get the Michoud factory going, fix the problems with the tank insulation in a definitive way, and when could they expect us to fly the shuttle again.

There were so many unknowns; first we had not yet begun to diagnose the foam problem or formulate a solution.   While the facility was largely undamaged, managing the human tragedy for the workforce was a huge question.  Infrastructure was in shambles; roads and bridges, electrical power and drinking water.  No grocery stores – or any other kind of store – was open anywhere nearby.  Having a dry, functional factory was worthless without people to operate it.

During the initial return to flight period when we sought to retrain the management to avoid poor communications and decisions, we hired Dr. Tufte to educate us on how better to communicate.  He gave (still does) a stirring indictment of the dangers of powerpoint presentations and highly recommended that writing white papers – requiring the use of complete sentences and organized paragraphs rather than bullet points – was superior.

So, wanting to demonstrate my new management tools, I gathered all the available information and wrote a white paper on the problems, potential solutions, and most critically about the schedule regarding flying again.  By shear dumb luck and overestimation of the problems my conclusion was that the shuttle would likely not fly before September 2006 – basically a year away.  Rather than having the typical NASA briefing, I mailed all the principal leaders – like the Administrator – my white paper.  Initially the response was gratifying; no significant questions, all issues and problems understood.  And while the gap in flights was disappointing, it was understandable, and we should proceed.

One of the most important qualities about NASA that you need to realize is that NASA cannot keep a secret.  Before we had any press releases or media briefings, like magic it was being reported that the shuttle would not fly before September of the following year.  Mike Griffin kept asking about the basis for the media reports.  The answer:  Wayne Hale’s white paper.  Somebody had passed it along.

Oh well.  The cat was out of the bag and even though we didn’t get to frame it as we would have liked, it was an accurate report.

Meanwhile, the real miracle occurred.  The dedicated workers down in New Orleans came back.  They came back to soggy houses that had to be gutted and rebuilt.  They came back to long commutes to the grocery store and any other kind of store.  They came back to the mess and in spite of it all, they came back to work.  Sooner and faster than anyone imagined that they would.  Lead by Wanda Sigur, the Lockheed-Martin workforce came back to work, and the factory came back on line months earlier than I had estimated.  In the meantime, John Muratore and his band of aerodynamics geniuses did the hard work to prove the fixes necessary to the ET would be safe.

Those are the real miracle workers, not us upper management dummies.

In the end, Discovery flew again on July 4; about two months faster than I had predicted.  It was hailed as a great success, both because we finally fixed the foam problem and because we flew sooner than we had predicted.

I learned a lot in those days.  Not the least of which was how to build a practical schedule to solve complex technical problems.

These days the space program is facing huge technical and schedule challenges.  There are all kinds of questions about deadlines and plans.  It seems to me that many of the schedules and launch dates that are published are invariably optimistic and don’t allow enough time to solve the inevitable problems that arise.

Sorta makes you want to give those folks the ‘Scotty’ advice.  Rather than building the quickest, shortest, most optimistic schedule, put some extra time on it.  Then, if you are good, or lucky – or perhaps a miracle occurs – everyone will be pleased.  The other way is not so pleasant.

When the Captain absolutely positively had to have warp drive back on line before death and destruction, Scotty always had a plan which got the job done – and sooner than required.   Somewhere there is a model to follow:  even with a demanding deadline, build a plan that can be done in less time than the deadline.

Just think about it.

 

Launch Fever

My entire professional career I have heard warnings about ‘launch fever’.  When a team works on a space project for a long time – months, years sometimes – pouring heart and sweat into it, launch day is the worst time.  Best and the worst, because the plan all comes together – or worst because it doesn’t.

It is so easy to want – with a capital W – want to see the vehicle fly that any obstacle becomes a serious frustration.  Temptation is very high to wave away issues, ignore anomalies, squeeze the rules, or brush aside inconsistencies just to GET ON WITH IT.  Unfortunately, in the rocket business, any wee error can result in catastrophe.  So, wanting to launch must be tempered with the discipline to deal rigorously with situations even if that means waiting to fly another day.

I’ve been a victim – and written about it before – https://waynehale.wordpress.com/2013/10/31/keeping-eileen-on-the-ground-part-ii-or-how-i-got-launch-fever/

But I’d like to address a little known but different inciter of the dread disease.  One of the greatest traditions in NASA’s human space flight program – and how it puts launch fever into the head of every person in the firing room.

Beans and cornbread.

You either like them or you don’t.  For those of us of a certain heritage, there is nothing better.  Comfort food at its zenith.

Norm Carlson was the legendary leader of the KSC Launch Processing team; head of the NTDs [NASA Test Directors].  Being a good southern boy he especially liked beans and cornbread.

At JSC, in the Mission Control Center where I spent much of my early career, you had to stay on console for long periods of time.  Leaving was simply not an option.  Food had to be with you.  Eating on console was de rigour.  No other way.  We even had the coffee pot in the Flight Control Room and those with long enough headset cords could reach the pot without unplugging.  Sometimes it got messy – I’ve written about the time we thought it was a good idea to have an ice cream party during LOS – it did not go well.  But food, good or bad, messy or simple, was part of being in the MCC.

Don’t even ask about the Atomic Fireball jar at the FDO console.

Or the time that the sniffer dogs brought in before the President came to speak, the dogs which almost ate my bologna sandwich brown bag lunch.

But at KSC, in the Firing Rooms, food and drink are strictly not allowed.  No coffee, no ham sandwiches, no chewing gum, no nothing.  When I started serving countdown duty in the Management ‘Bubble’ room at KSC this was a rude realization.

Not that there wasn’t food in the building – there was lots and lots of food in all the offices, downstairs in the little cafeteria, all over the place.  But you had to leave console and exit the Firing Room to go to it; eat it away, and potentially miss whatever happened in the count while chowing down.  Obviously not to do during the critical parts of the countdown.

One unnamed senior NASA manager – you know who you are – would sneak a bag of candy into the management bubble and surreptitiously pass the illicit bag down the row of managers – underneath the bookshelf of the console.  A whole line of senior managers trying to hide the fact they were chewing chocolate candies was a sight to behold.  I’m positive they fooled absolutely nobody.

Norm Carlson instituted a grand tradition of having beans and cornbread after every successful launch.  Sometime around L-3 or 4 hours, someone would fire up the cookers in the hallway outside the firing room and start cooking the beans, warming the cornbread.  The ventilation system in the Launch Control Center would distribute the warm delicious smell throughout the building.  Sitting on console in the firing room you could hear the stomachs growl all over the place.  It was torture.  But the worst part was the knowledge that if the launch scrubbed, those beans and cornbread went into the freezer to wait for another countdown.  No beans on scrub days!!!

Talk about inciting mass launch fever.  If some sociologist writes a paper about why the NASA launch team got in a hurry and launched when we shouldn’t have – and doesn’t mention the beans and cornbread – well, that paper just won’t be complete.

Just a few days ago, I got to witness the first commercial crew demonstration launch from KSC.  I was stuck with the VIP crowd in another building, separated from the launch team.  I just must wonder if they served beans and cornbread over there after the launch.  Or is there a new tradition?  Or maybe everybody just went home.

Traditions can be a good thing.  But consider the unanticipated consequences if you try to institute a new one.

Armando sampling the beans!

Careful What You Ask For

Roy Estess was one of the smartest and best that I have ever met.  He spent most of his career testing large rocket engines at the NASA installation now known as the Stennis Space Center.  In fact, he became the Center Director for SSC later in his career.  Everybody who knew Roy came to appreciate his insight, his high level of integrity, and his managerial skill.

In 2001, Roy was appointed acting Center Director at Johnson Space Center where I really got to know him.  On one business trip together, I heard him tell the following story, full of wisdom, which you should appreciate.

Roy said that as Stennis Center Director he got a call from the NASA Administrator about once a month.  In between these calls, his pride would suffer.  ‘Aren’t I a Center Director?  Isn’t Stennis a NASA Center?  Don’t we deserve more attention than one call a month?’  He asked.  Then with a grin, Roy said, ‘Now, I’m at JSC and every day the NASA Administrator calls two, three, four times.’

“And I’ve decided that once a month was just fine.”

Working as a Flight Director on the overnight shift was at once a hardship and a delight.  After all the senior managers made their last phone calls, generally before midnight, and before they all started showing up for their morning pre-briefings about 6:30, the Flight Director was master of all.  Or so it seemed.  From about midnight to 6 AM you could decide what the shuttle team would do without any oversight or interference.  Except of course that Mr. Abbey always came in right about 2 AM.  But he generally did not direct, he just wanted to say hello.  Every morning.  You could set your watch.  But Mr. Abbey never stayed very long and never ever gave any direction – at least not to me.

Being the master of your own fate is a great thing, until you need advice or don’t know what to do.  A very important fact that I learned early in my career:  calling people at 3 AM is not a good way to get advice.  I did that exactly once.  Thereafter I decided that there were no troubling issues that could not wait until 6:30.   And in the meantime I could decide to do anything – anything that would be over by 6:30.

For decades, folks at NASA have wished that they could get the attention of the President, like Webb had with JFK.  Occasionally that happened, GHWB proposed a Mars mission in 1989.  That did not work out well.  Reagan finally approved a Space Station, but Clinton almost cancelled it.  So on and so forth.  Good and bad but nothing spectacular and not like JFK.  Probably never happen again.

Now, NASA has the attention of the President, the Vice President, and a whole bunch of Very Important People.

It will be interesting to see how it turns out.

STS-121: The Hardest Launch: Part 2 – Electrical Problems

 

Explaining ECO sensors to the Press

Now to return to a subject I have left pending for too long:  The STS-121 launch.

We had three major problems to solve to get approval for the launch.

I hate intermittent electrical problems.  It doesn’t matter whether they are on my car, with my home sound system, or on the space shuttle, intermittent electrical problems are the worst.  Even when you get the repairman to look at the system when it is not working, sometimes it is nearly impossible to find the problem.  As my friends remind me, I’m a Mechanical Engineer by training, I don’t really understand electricity!

So it was with the shuttle in return to flight time.  The big orange external tank does not have a gas gauge similar to what you find or a car or boat or airplane; it simply had a few ‘level’ sensors that tell when the tank is filled up to the top (for loading) and when it is empty.  There were no measurements in between.  Many other rockets are like this.  Even telling when the gas left in the tank is at the ‘full’ or ‘empty’ point is not easy when dealing with liquid hydrogen and liquid oxygen because they are so very cold; it takes a special sensor to indicate ‘wet’ or ‘dry’.

Starting with some tanking tests for the first ‘return to flight’ – STS-114 – we started experiencing some problems with the ‘empty’ sensors, called Engine Cutoff (ECO) Sensors because they were there for a critical safety issue.  If the fuel – hydrogen – tank ran dry while the engines were still running, the ‘fire’ in the engines would get very hot indeed due to the surplus of oxygen and likely the engine would suffer – a great euphemism – ‘an uncontained failure’.  Not what you want.

So, we started troubleshooting:  the first suspect was the electronic box (‘point sensor box’) in the orbiter that deciphered the electrical signals from the sensors.  I was surprised to find out that this piece of equipment was Apollo heritage!  The electronic schematic drawing was signed off in the 1960’s for the upper stage of the Saturn V.  We put a team of experts lead by Ed Mango on the investigation.  After weeks and many tests on various tanks and orbiter point sensor boxes, the conclusions exonerated this old gear.  As the orbiter team members told me:  think outside the box.  They even had a T-shirt made with that phrase.

Next we investigated the little sensors themselves.  A metal cube about an inch on a side; inside was a very fine wire that changed electrical properties depending on whether it was immersed in fluid or not.  We found that the electrical connections inside this little sensor could have some issues.  Ah ha!  Multitudes of x-rays and resistance tests were suspicious but inconclusive.  But that had to be it.

Many long hours were spent in meetings and reviews to develop ways to determine if a particular sensor was good or likely to fail.  New techniques for manufacturing were proposed.  During this time, I elected to make a site visit to the people that build those sensors:  the Goodyear aircraft avionics plant in Vergennes, VT.  It probably scared the dickens out of the factory technicians to have the Space Shuttle Program Manager come stand at their work bench and watch them make tiny crimp connections on the almost microscopic wires.  But we were convinced that was the problem and we were on the road to fixing it.  The ET assigned to STS-121 had the ‘best’ sensor boxes we could find.

That was the status as of the Flight Readiness Review in June of 2005.  With some reluctance, the FRR board accepted our plans including the wacky logic tree for what to do if more than one sensor failed during the countdown.  So, despite all our worry and work, or because of it, ECO sensors were not the reason that there was disagreement over signing off on the CoFR.

 

I wish I could tell you that was the end of the story, but it wasn’t.  Not only did we scrub the first launch attempt for STS-121 because more than one ECO sensor circuit was giving erroneous reasons, but later we found out the real cause:  It wasn’t the Point Sensor Box in the Orbiter; it wasn’t the sensors in the bottom of the External Tank.  It was the pin connectors on the pass through where the wiring went from inside to outside of the hydrogen tank.  Something we thought we had exonerated early on.  We had jumped to an erroneous conclusion early in the troubleshooting and spent over a year working on the wrong problem.  Somebody from a different program pointed out – much later than STS-121 – that the Delta program had a similar problem which was caused by pin connectors in the tank wall pass through and they had solved their problem by soldering the wires together.  Which is what we did.  Which solved the problem.  After almost two years of work.

I wish I had a nickel for every time we misdiagnosed a problem during our days on the Space Shuttle.

 

Memory Overload

Digging through my files I recently found an email which caught my attention:

=====================================================================

From:  Larry A.

Sent:  Thursday October 21, 2004 7:18 AM

TO:  A whole bunch of people who are probably retired now

Subject:  SMS Issue – SMS AR 018459 – Unexpected GPC Errors – 1141A T.L. – OI30 STS-114 BASE2 FSW (OF03.01) – Closure

After additional runs in the SMS and SPF and analysis of the data collected on those runs, FSW DR 121227 has been opened on the issue.  The SMS AR will be closed in reference to the FSW DR.  The description of the FSW DR reads as follows:

“In MM 104-105, 202, and 301-302 when an Item 22 (Load) is executed on the XXXXX YYYYY MNVR display, it is possible for 2 SQRT of negative number GPC errors to be generated.  This will occur when the TIG entered is not consistent with the PEG 4 burn targets.  The Orbital Altitude Time Task in the DIP computes the inverse of the mean orbital rate and the sine of the eccentric anomaly as a function of the square root of the semi-major axis.  Given the inconsistency of the TIG with the targets and the fact that the guidance converged to a solution, the semi-major axis is computed as a negative number.”

===========================================================================

So here is my problem:  almost two decades later I completely understand EVERY BIT of this email!

How many of my memory cells are taken up with holding on to obsolete and totally useless information?  Is this why I can go down the hall to another room in my house and, upon arriving there, wonder what it was I came to do?

Sigh.

I imagine that there are dozens, if not hundreds, of former flight controllers, astronauts, trainers, software programmers, and trajectory analysts who were nodding their heads reading that email and saying ‘Yes!  I understand the problem!’

Some neuro-psychologist needs to explain to me how to clean out my memory cells of unwanted and useless information so that I don’t get confused looking for my car keys.

Oh well, its fun to remember the old days.

And my wish for you today is that your semi-major axis is never a negative number.

 

 

 

 

 

Fifteen Years

Two weeks ago, I participated in the NASA Remembrance Day and lessons learned activities which happen every year around February 1.  I believe it is very important to remember Dick Scobee, Gus Grissom, and Rick Husband and their crews; to remind ourselves of their sacrifice and what we need to do to safeguard future space travelers.  But this post is not about those brave crews or the lessons should remember.

What really blew me away was the realization that fifteen years have passed since the loss of Columbia and her crew.  It seems like yesterday.

Isn’t that what all the old folks say; it seems like just yesterday.

That tragedy sparked changes in my professional life that I had neither foreseen nor desired.  Reviewing the events in my life over the past fifteen years brings so much to mind:  being selected to leadership roles in the Space Shuttle Program, working feverishly to fix the problems, change the culture, and get flying again, making sure that the program would operate safely – or rather, more safely – all that in just the first five years.  Changing jobs inside NASA to build partnerships with other organizations, supporting the President’s commission to plan the future of human spaceflight, helping to develop the initial stages of NASA’s commercial crew program – that took the next three years.  Then retirement from the government and the start of a new career providing support to commercial spaceflight on the industry side – where I am today.  Katrina, Ike, Harvey.  And in between it all seeing my son and daughter graduate from college; experiencing the wedding of my son and his beloved; the blessing of grandchildren into our lives; the passing of my mother, the passing of my wife’s parents, and the passing – oh how hard to write this – the passing of my daughter.  The last fifteen years have been packed full of happiness and tears.

Just like the song: “sunrise, sunset; I don’t remember growing older, when did they?

Fifteen years earlier, at the end of February 1988, my class of flight directors was selected.  Becoming a NASA Flight Director had been my top career goal for years.  It proved to be the best job I ever had.  There is no feeling like leading a team of highly trained and highly motivated folks to achieve great things in a difficult environment.  It was tremendously exciting and equally frightening.  There were tremendous possibilities and “always be aware that suddenly and unexpectedly we may find ourselves in a role where our performance has ultimate consequences.”   Making tough calls about the weather and priorities and all the little things that go wrong; participating in Hubble and ISS assembly and Chandra and on and on.   I got to work hand in hand with astronauts, senior NASA executives, scientists; meet numerous VIPs that were interested in our work – everybody from movie stars to US senators to members of foreign royalty.  And through all that watching my children grow, coaching them through all kinds of activities, actively participating in local civic affairs, taking long family vacations.  Its hard to see how I could have done so must; I must have been much younger.

Fifteen years earlier, an astounding 45 years ago, I was a college freshman just trying to figure out who I was – even though I knew who I wanted to become.  In the following fifteen years I met and married my beautiful wife, graduated from college both undergrad and grad school.  I got my dream job at NASA:  Mission Control and STS-1 and Challenger, working with Sally Ride, John Young, and working for Gene Kranz.  And we bought our first and second houses, and in between had babies.  Claudette, Alicia, Allison.  How could all of that be packed into just 15 years.

Fifteen years earlier – can it be 60 years ago?  It was 1958 and I was three.  I don’t remember – who remembers when they were three – but according to family legend I was totally captivated by space; sputnik in October 57, Vanguard (kaboom!), Explorer 1 at the end of January 1958.  Then hanging breathlessly on the adventures of Mercury, Gemini, a hundred robot explorers, and the Moon landing.   It seems I was destined to participate in the great adventure from my earliest years.  Growing up in the 60’s was sometimes a surreal experience and it is filled with memories even of people I knew that never came back from southeast Asia and others we lost.  ‘Has anybody here seen my old friend John, can you tell me where he’s gone?’  It was an unbelievable time.

Sunrise sunset, swiftly pass the years.

Fifteen years is simultaneously a long time and the merest instant.  So much to be learned. So much to experience.  So much more to come.

I just pray the next fifteen years will have more happiness and less tears.

Game Changing Technology

I’ve spent all week at the Langley Research Center in Virginia; there is much good work going on here.  We had no small number of discussion about ‘game changing technology’ without a good definition of what that phrase really means or how such a change takes place.

At the same time I am keenly aware of the surrounding area’s historical sites.  Jamestown, Yorktown, and Williamsburg are just up the road.  There are Civil War sites all around.  But the one that jumped out at me may have some lessons for those of us in the space business to consider. Just a few minutes from the Langley gates is the Mariner’s Museum with relics from the battle between the USS Monitor and the CSS Virginia, formerly Merrimack.  There is a lesson in game changing technology there.

In the winter of 1861 a technology revolution was brewing in the midst of the U.S. Civil War. Naval technology had progressed incrementally for a long time, centuries actually.  Wooden ships with sails carrying broadsides of carronades.  Recently steam engines had started appearing onboard oceangoing ships but these were auxiliary power, sails were still the prime mover.   In the Crimean war around 1856 some innovative designs were partially tried out but the results were inconclusive.  Tradition and the conventional wisdom of the day did not look to radical change.

The Confederacy knew they could never match the number and size of the Union fleet but they were desperate to break the naval blockade that was strangling their war effort.  Something radical must be tried to overcome the Union advantage.  A plan was developed to use a partially burned frigate hulk – the USS Merrimack- as the platform for something very radically different; a solely steam powered, iron clad floating battery that might just turn the tide of the war.  This was the basis for the CSS Virginia; hull and steam engine from a conventional frigate with an upper deck unlike anything seen on the water; sloped and reinforced sides and top covered with iron sheeting.

They didn’t pay much attention to military secrecy; the entire activity was written up in the local newspapers.  Maybe that was an early attempt at psych ops – fill the enemy with fear.  Construction of the Virginia was widely reported and the news traveled north.

When the CSS Virginia became operational on March 8, 1862, she was nigh invincible.  Steaming to the Union fleet she created total destruction leaving flaming and sinking wrecks in her wake as their cannon shells bounced harmlessly off.  Only the outgoing tide and drawing sunset abated the Virginia’s destructive path.

The North was in a panic at the report.  An invincible terror weapon had been unleashed on them and there was no defense.

Well, not exactly.  At the earlier reports, the admirals turned to a truly wild man with crazy ideas:  John Ericsson.  He was the Elon Musk of his day; promising radical and unbelievable change.  His design was unlike anything ever seen; a practically submerged main body of a ship with a rotating iron turret housing two cannons on top.  With the southern warship well under construction, the admirals gave him 100 days to build this new vessel which was not much more than a concept.  He did it in 118 days.  (Note:  18% schedule overrun).  It was ready just one day after the Virginia’s first foray.

You probably know the story; the two iron-clads slugged it out without doing any real damage to each other.  Military historians call the action of March 9, 1862 a draw.  Except, of course, that the blockade was not broken.  Neither ship saw action again and within months both were destroyed.

But their encounter changed everything; in the blink of an eye naval design would never be the same.  The age of wooden sailing warships was over.  Every ship in every fleet in the world was instantly obsolete.  “Monitor madness” ensued while nations around the world started building copies of Ericsson’s design.

Game changing technology.

So what is the lesson for us? Here are the first three that I can think of:

1. There are harbingers out there.
2. Only truly crazy revolutionaries make game changing inventions
3. When the time is right, change will happen overnight

Can you think of more lessons?