Careful What You Ask For

Roy Estess was one of the smartest and best that I have ever met.  He spent most of his career testing large rocket engines at the NASA installation now known as the Stennis Space Center.  In fact, he became the Center Director for SSC later in his career.  Everybody who knew Roy came to appreciate his insight, his high level of integrity, and his managerial skill.

In 2001, Roy was appointed acting Center Director at Johnson Space Center where I really got to know him.  On one business trip together, I heard him tell the following story, full of wisdom, which you should appreciate.

Roy said that as Stennis Center Director he got a call from the NASA Administrator about once a month.  In between these calls, his pride would suffer.  ‘Aren’t I a Center Director?  Isn’t Stennis a NASA Center?  Don’t we deserve more attention than one call a month?’  He asked.  Then with a grin, Roy said, ‘Now, I’m at JSC and every day the NASA Administrator calls two, three, four times.’

“And I’ve decided that once a month was just fine.”

Working as a Flight Director on the overnight shift was at once a hardship and a delight.  After all the senior managers made their last phone calls, generally before midnight, and before they all started showing up for their morning pre-briefings about 6:30, the Flight Director was master of all.  Or so it seemed.  From about midnight to 6 AM you could decide what the shuttle team would do without any oversight or interference.  Except of course that Mr. Abbey always came in right about 2 AM.  But he generally did not direct, he just wanted to say hello.  Every morning.  You could set your watch.  But Mr. Abbey never stayed very long and never ever gave any direction – at least not to me.

Being the master of your own fate is a great thing, until you need advice or don’t know what to do.  A very important fact that I learned early in my career:  calling people at 3 AM is not a good way to get advice.  I did that exactly once.  Thereafter I decided that there were no troubling issues that could not wait until 6:30.   And in the meantime I could decide to do anything – anything that would be over by 6:30.

For decades, folks at NASA have wished that they could get the attention of the President, like Webb had with JFK.  Occasionally that happened, GHWB proposed a Mars mission in 1989.  That did not work out well.  Reagan finally approved a Space Station, but Clinton almost cancelled it.  So on and so forth.  Good and bad but nothing spectacular and not like JFK.  Probably never happen again.

Now, NASA has the attention of the President, the Vice President, and a whole bunch of Very Important People.

It will be interesting to see how it turns out.

STS-121: The Hardest Launch: Part 2 – Electrical Problems

 

Explaining ECO sensors to the Press

Now to return to a subject I have left pending for too long:  The STS-121 launch.

We had three major problems to solve to get approval for the launch.

I hate intermittent electrical problems.  It doesn’t matter whether they are on my car, with my home sound system, or on the space shuttle, intermittent electrical problems are the worst.  Even when you get the repairman to look at the system when it is not working, sometimes it is nearly impossible to find the problem.  As my friends remind me, I’m a Mechanical Engineer by training, I don’t really understand electricity!

So it was with the shuttle in return to flight time.  The big orange external tank does not have a gas gauge similar to what you find or a car or boat or airplane; it simply had a few ‘level’ sensors that tell when the tank is filled up to the top (for loading) and when it is empty.  There were no measurements in between.  Many other rockets are like this.  Even telling when the gas left in the tank is at the ‘full’ or ‘empty’ point is not easy when dealing with liquid hydrogen and liquid oxygen because they are so very cold; it takes a special sensor to indicate ‘wet’ or ‘dry’.

Starting with some tanking tests for the first ‘return to flight’ – STS-114 – we started experiencing some problems with the ‘empty’ sensors, called Engine Cutoff (ECO) Sensors because they were there for a critical safety issue.  If the fuel – hydrogen – tank ran dry while the engines were still running, the ‘fire’ in the engines would get very hot indeed due to the surplus of oxygen and likely the engine would suffer – a great euphemism – ‘an uncontained failure’.  Not what you want.

So, we started troubleshooting:  the first suspect was the electronic box (‘point sensor box’) in the orbiter that deciphered the electrical signals from the sensors.  I was surprised to find out that this piece of equipment was Apollo heritage!  The electronic schematic drawing was signed off in the 1960’s for the upper stage of the Saturn V.  We put a team of experts lead by Ed Mango on the investigation.  After weeks and many tests on various tanks and orbiter point sensor boxes, the conclusions exonerated this old gear.  As the orbiter team members told me:  think outside the box.  They even had a T-shirt made with that phrase.

Next we investigated the little sensors themselves.  A metal cube about an inch on a side; inside was a very fine wire that changed electrical properties depending on whether it was immersed in fluid or not.  We found that the electrical connections inside this little sensor could have some issues.  Ah ha!  Multitudes of x-rays and resistance tests were suspicious but inconclusive.  But that had to be it.

Many long hours were spent in meetings and reviews to develop ways to determine if a particular sensor was good or likely to fail.  New techniques for manufacturing were proposed.  During this time, I elected to make a site visit to the people that build those sensors:  the Goodyear aircraft avionics plant in Vergennes, VT.  It probably scared the dickens out of the factory technicians to have the Space Shuttle Program Manager come stand at their work bench and watch them make tiny crimp connections on the almost microscopic wires.  But we were convinced that was the problem and we were on the road to fixing it.  The ET assigned to STS-121 had the ‘best’ sensor boxes we could find.

That was the status as of the Flight Readiness Review in June of 2005.  With some reluctance, the FRR board accepted our plans including the wacky logic tree for what to do if more than one sensor failed during the countdown.  So, despite all our worry and work, or because of it, ECO sensors were not the reason that there was disagreement over signing off on the CoFR.

 

I wish I could tell you that was the end of the story, but it wasn’t.  Not only did we scrub the first launch attempt for STS-121 because more than one ECO sensor circuit was giving erroneous reasons, but later we found out the real cause:  It wasn’t the Point Sensor Box in the Orbiter; it wasn’t the sensors in the bottom of the External Tank.  It was the pin connectors on the pass through where the wiring went from inside to outside of the hydrogen tank.  Something we thought we had exonerated early on.  We had jumped to an erroneous conclusion early in the troubleshooting and spent over a year working on the wrong problem.  Somebody from a different program pointed out – much later than STS-121 – that the Delta program had a similar problem which was caused by pin connectors in the tank wall pass through and they had solved their problem by soldering the wires together.  Which is what we did.  Which solved the problem.  After almost two years of work.

I wish I had a nickel for every time we misdiagnosed a problem during our days on the Space Shuttle.

 

Memory Overload

Digging through my files I recently found an email which caught my attention:

=====================================================================

From:  Larry A.

Sent:  Thursday October 21, 2004 7:18 AM

TO:  A whole bunch of people who are probably retired now

Subject:  SMS Issue – SMS AR 018459 – Unexpected GPC Errors – 1141A T.L. – OI30 STS-114 BASE2 FSW (OF03.01) – Closure

After additional runs in the SMS and SPF and analysis of the data collected on those runs, FSW DR 121227 has been opened on the issue.  The SMS AR will be closed in reference to the FSW DR.  The description of the FSW DR reads as follows:

“In MM 104-105, 202, and 301-302 when an Item 22 (Load) is executed on the XXXXX YYYYY MNVR display, it is possible for 2 SQRT of negative number GPC errors to be generated.  This will occur when the TIG entered is not consistent with the PEG 4 burn targets.  The Orbital Altitude Time Task in the DIP computes the inverse of the mean orbital rate and the sine of the eccentric anomaly as a function of the square root of the semi-major axis.  Given the inconsistency of the TIG with the targets and the fact that the guidance converged to a solution, the semi-major axis is computed as a negative number.”

===========================================================================

So here is my problem:  almost two decades later I completely understand EVERY BIT of this email!

How many of my memory cells are taken up with holding on to obsolete and totally useless information?  Is this why I can go down the hall to another room in my house and, upon arriving there, wonder what it was I came to do?

Sigh.

I imagine that there are dozens, if not hundreds, of former flight controllers, astronauts, trainers, software programmers, and trajectory analysts who were nodding their heads reading that email and saying ‘Yes!  I understand the problem!’

Some neuro-psychologist needs to explain to me how to clean out my memory cells of unwanted and useless information so that I don’t get confused looking for my car keys.

Oh well, its fun to remember the old days.

And my wish for you today is that your semi-major axis is never a negative number.

 

 

 

 

 

Fifteen Years

Two weeks ago, I participated in the NASA Remembrance Day and lessons learned activities which happen every year around February 1.  I believe it is very important to remember Dick Scobee, Gus Grissom, and Rick Husband and their crews; to remind ourselves of their sacrifice and what we need to do to safeguard future space travelers.  But this post is not about those brave crews or the lessons should remember.

What really blew me away was the realization that fifteen years have passed since the loss of Columbia and her crew.  It seems like yesterday.

Isn’t that what all the old folks say; it seems like just yesterday.

That tragedy sparked changes in my professional life that I had neither foreseen nor desired.  Reviewing the events in my life over the past fifteen years brings so much to mind:  being selected to leadership roles in the Space Shuttle Program, working feverishly to fix the problems, change the culture, and get flying again, making sure that the program would operate safely – or rather, more safely – all that in just the first five years.  Changing jobs inside NASA to build partnerships with other organizations, supporting the President’s commission to plan the future of human spaceflight, helping to develop the initial stages of NASA’s commercial crew program – that took the next three years.  Then retirement from the government and the start of a new career providing support to commercial spaceflight on the industry side – where I am today.  Katrina, Ike, Harvey.  And in between it all seeing my son and daughter graduate from college; experiencing the wedding of my son and his beloved; the blessing of grandchildren into our lives; the passing of my mother, the passing of my wife’s parents, and the passing – oh how hard to write this – the passing of my daughter.  The last fifteen years have been packed full of happiness and tears.

Just like the song: “sunrise, sunset; I don’t remember growing older, when did they?

Fifteen years earlier, at the end of February 1988, my class of flight directors was selected.  Becoming a NASA Flight Director had been my top career goal for years.  It proved to be the best job I ever had.  There is no feeling like leading a team of highly trained and highly motivated folks to achieve great things in a difficult environment.  It was tremendously exciting and equally frightening.  There were tremendous possibilities and “always be aware that suddenly and unexpectedly we may find ourselves in a role where our performance has ultimate consequences.”   Making tough calls about the weather and priorities and all the little things that go wrong; participating in Hubble and ISS assembly and Chandra and on and on.   I got to work hand in hand with astronauts, senior NASA executives, scientists; meet numerous VIPs that were interested in our work – everybody from movie stars to US senators to members of foreign royalty.  And through all that watching my children grow, coaching them through all kinds of activities, actively participating in local civic affairs, taking long family vacations.  Its hard to see how I could have done so must; I must have been much younger.

Fifteen years earlier, an astounding 45 years ago, I was a college freshman just trying to figure out who I was – even though I knew who I wanted to become.  In the following fifteen years I met and married my beautiful wife, graduated from college both undergrad and grad school.  I got my dream job at NASA:  Mission Control and STS-1 and Challenger, working with Sally Ride, John Young, and working for Gene Kranz.  And we bought our first and second houses, and in between had babies.  Claudette, Alicia, Allison.  How could all of that be packed into just 15 years.

Fifteen years earlier – can it be 60 years ago?  It was 1958 and I was three.  I don’t remember – who remembers when they were three – but according to family legend I was totally captivated by space; sputnik in October 57, Vanguard (kaboom!), Explorer 1 at the end of January 1958.  Then hanging breathlessly on the adventures of Mercury, Gemini, a hundred robot explorers, and the Moon landing.   It seems I was destined to participate in the great adventure from my earliest years.  Growing up in the 60’s was sometimes a surreal experience and it is filled with memories even of people I knew that never came back from southeast Asia and others we lost.  ‘Has anybody here seen my old friend John, can you tell me where he’s gone?’  It was an unbelievable time.

Sunrise sunset, swiftly pass the years.

Fifteen years is simultaneously a long time and the merest instant.  So much to be learned. So much to experience.  So much more to come.

I just pray the next fifteen years will have more happiness and less tears.

Game Changing Technology

I’ve spent all week at the Langley Research Center in Virginia; there is much good work going on here.  We had no small number of discussion about ‘game changing technology’ without a good definition of what that phrase really means or how such a change takes place.

At the same time I am keenly aware of the surrounding area’s historical sites.  Jamestown, Yorktown, and Williamsburg are just up the road.  There are Civil War sites all around.  But the one that jumped out at me may have some lessons for those of us in the space business to consider. Just a few minutes from the Langley gates is the Mariner’s Museum with relics from the battle between the USS Monitor and the CSS Virginia, formerly Merrimack.  There is a lesson in game changing technology there.

In the winter of 1861 a technology revolution was brewing in the midst of the U.S. Civil War. Naval technology had progressed incrementally for a long time, centuries actually.  Wooden ships with sails carrying broadsides of carronades.  Recently steam engines had started appearing onboard oceangoing ships but these were auxiliary power, sails were still the prime mover.   In the Crimean war around 1856 some innovative designs were partially tried out but the results were inconclusive.  Tradition and the conventional wisdom of the day did not look to radical change.

The Confederacy knew they could never match the number and size of the Union fleet but they were desperate to break the naval blockade that was strangling their war effort.  Something radical must be tried to overcome the Union advantage.  A plan was developed to use a partially burned frigate hulk – the USS Merrimack- as the platform for something very radically different; a solely steam powered, iron clad floating battery that might just turn the tide of the war.  This was the basis for the CSS Virginia; hull and steam engine from a conventional frigate with an upper deck unlike anything seen on the water; sloped and reinforced sides and top covered with iron sheeting.

They didn’t pay much attention to military secrecy; the entire activity was written up in the local newspapers.  Maybe that was an early attempt at psych ops – fill the enemy with fear.  Construction of the Virginia was widely reported and the news traveled north.

When the CSS Virginia became operational on March 8, 1862, she was nigh invincible.  Steaming to the Union fleet she created total destruction leaving flaming and sinking wrecks in her wake as their cannon shells bounced harmlessly off.  Only the outgoing tide and drawing sunset abated the Virginia’s destructive path.

The North was in a panic at the report.  An invincible terror weapon had been unleashed on them and there was no defense.

Well, not exactly.  At the earlier reports, the admirals turned to a truly wild man with crazy ideas:  John Ericsson.  He was the Elon Musk of his day; promising radical and unbelievable change.  His design was unlike anything ever seen; a practically submerged main body of a ship with a rotating iron turret housing two cannons on top.  With the southern warship well under construction, the admirals gave him 100 days to build this new vessel which was not much more than a concept.  He did it in 118 days.  (Note:  18% schedule overrun).  It was ready just one day after the Virginia’s first foray.

You probably know the story; the two iron-clads slugged it out without doing any real damage to each other.  Military historians call the action of March 9, 1862 a draw.  Except, of course, that the blockade was not broken.  Neither ship saw action again and within months both were destroyed.

But their encounter changed everything; in the blink of an eye naval design would never be the same.  The age of wooden sailing warships was over.  Every ship in every fleet in the world was instantly obsolete.  “Monitor madness” ensued while nations around the world started building copies of Ericsson’s design.

Game changing technology.

So what is the lesson for us? Here are the first three that I can think of:

1. There are harbingers out there.
2. Only truly crazy revolutionaries make game changing inventions
3. When the time is right, change will happen overnight

Can you think of more lessons?

STS-121: The Hardest Launch – part 1

“The hardest part of any launch is getting the first foot off the ground” – Tommy Holloway, Space Shuttle Program Manager

I was doing some online research recently and ran across this document which I had not seen in over a decade:  https://www.nasa.gov/pdf/150505main_121_CoFR.pdf

The picture shows Associate Administrator Bill Gerstenmaier signing the STS-121 CoFR document following the Flight Readiness Review.  Over his shoulder is NASA Administrator Mike Griffin who played a pivotal role at the FRR.  He is deep in discussion with Chief of NASA Safety, Bryan O’Connor (back to the camera)

A good, gray, boring government document comprising the signature pages from STS-121 Flight Readiness Review.  Signatures showing that all the senior management agreed to the launch of STS-121, except, if you go to the last couple of pages, not all of them agreed!  This is the only case I know of where such dissent happened regarding a shuttle launch.  The entire story of STS-121 has multiple lessons that may apply today as well.  Buckle in and I will tell you the story – but remember it is my perspective and I’m probably biased and sometimes forgetful.

It has been five years since I wrote about how we nearly lost Discovery on the return to flight after Columbia:  https://waynehale.wordpress.com/2012/04/18/how-we-nearly-lost-discovery/

After two and a half years of trying to correct the problem that caused the loss of Columbia, the launch of STS-114 proved we had failed.  Before we could even start to make corrections, Hurricane Katrina smashed into New Orleans and complicated everything.

Setting aside the serious issues of the PAL ramp and recovering from Katrina, there were at least three other major problems associated with returning the shuttle to flight status (again!) which were not completely resolved at the time we were ready to launch STS-121:  foam loss from the ‘ice/frost ramps’; thermal scan indications of manufacturing anomalies in the RCC leading edge panels; and the nagging issue of intermittent failures in the engine cut off sensor system.  I plan to write a separate background post on each of these issues and then a summary of the drama and decision that occurred in mid-June 2006.

Returning again to the major anomaly on STS-114, the loss of a large section of foam insulation from the ‘protuberance air load ramp’ (called the PAL ramp) was a surprise.  The Shuttle senior leadership team had considered and discussed removing that 17 pound sprayed on foam barrier which was the PAL ramp, but decided against removing it.  John Muratore was one of the leading proponents of removing the PAL ramp and we probably should have listened to him.  The data in 2004/5 showed historic foam insulation loss from the PAL ramp was extremely rare and limited, and the defect/divot foam loss mechanism did not apply to the PAL ramp.  The major error in the logic, found after STS-114, was that the defect/divot foam loss mechanism was not the real problem; differential coefficient of thermal expansion between foam layers was the real problem.  Go back to the earlier Discovery post for more details.  After STS-114 it was absolutely obvious that removal of the PAL ramp had to be done.

The PAL ramp was a late addition to the External Tank prior to the first flight of the shuttle back in 1981.  Aerodynamic modeling was not very sophisticated in those days and there was an indication that the ‘protuberances’ might be ripped off the outside of the tank in some extreme cases.  The ‘protuberances’ – shown in the attached picture – consisted of the 17 inch diameter pipe carrying the liquid oxygen from the front part of the ET, a rectangular cable tray that carried critical instrumentation signals, and two smaller metal pipes carrying gaseous hydrogen and gaseous oxygen (in different pipes!) heated by the engines in the aft end of the Orbiter, to maintain pressure in the External tank as the LH2 and LO2 were pumped out to feed the engines.  All of these ‘protuberances’ were attached with metal fittings to the aluminum skin of the ET.  The attachment fittings were chilled by the cold fluids inside so those fittings were insulated with foam; these insulated foam fitting were called ‘ice/frost ramps’ since they had to prevent the formation of frost or ice and had to be ramp shaped due to the aerodynamics.

Nothing is easy.

To remove the PAL ramp and eliminate the danger of chunks of foam coming off and damaging the orbiter heat shield, we had to first prove that the aerodynamics concerns identified back in 1981 did not exist.  This required an extensive wind tunnel test campaign to measure forces followed by analysis using computational fluid dynamics models to check the cases the wind tunnel testing could not do.  Attached is a picture from the Glenn Research Center wind tunnel test article showing the sub scale protuberances that were tested there.

Building models, scheduling wind tunnel time, checking the results all took months.  Most of the work wrapped up in May with a few loose ends.  Removing the PAL ramp was proven to be safe; but in the mean time there were other problems.  More on that next time.

Administrivia

 

It has been some time since I have gone over the purpose and rules for my personal blog site.  Recently some folks have demonstrated their confusion about this site.  For the record, let me repeat some of the ‘rules’ associated with my blog site for your awareness.

First of all, this is my personal blog.

As the disclaimer on the home page says, these posts and comments represent my person thoughts, insights, and concerns.  This blog site is definitely not the views of any other organization or person.  I write about what I am interested in and what I want to say, and literally nobody tells me what topics I should address or how to approach them.  What I write here is subject only to my own approval.  Not even my wife gets a vote. (she doesn’t read it).

This blog site is NOT representative of my current employer, Special Aerospace Services, LLC.  It is my hope that I do not offend SAS management or their many clients, but again, this is my personal site and they neither direct me nor necessarily agree with me on what I post.  This is my personal blog site.

This blog site is NOT representative of my former employer, the National Aeronautics and Space Administration, the U.S. Federal Government.  Long ago, I used to have a blog on the NASA web page that did represent many times the official position of the US government but THIS IS NOT IT.  Again, it is not my intention to offend my old colleagues, but they do not direct me nor do they necessarily concur with my thoughts and positions.  This is my personal blog site.

This blog site is NOT associated with any of the volunteer organizations with which I am (or was) a member or officer.  This is NOT an official site for the NASA Alumni League, the Coalition for Deep Space Exploration, the Commercial Spaceflight Federation, the American Institute of Aeronautics and Astronautics, the Rice University Engineering Alumni Board of Directors, the Rice Space Institute, the Baker Institute at Rice University, etc., etc., etc.  None of those organizations give direction for my blog nor do they influence me on topics or how I address them.  This is my personal blog site.

This blog site does not represent the advisory roles that I have with the Federal Government under the Federal Advisory Committee’s Act.  This blog site is not an official site for the NASA Advisory Council nor for the FAA Commercial Space Transportation Advisory /committee.  They do not direct me on what to blog or how to address issues.  In general, I avoid discussing NAC or COMSTAC issues in my blog; after all those FACA Committee meetings are by law public and are well reported by the media.  While I attempt not to offend those organizations, I will say what I want and in the way I want on this blog site.  This is my personal blog site.

In Texas, we do not register as members of a political party, we merely register as a voter.  While I have definite political leanings, I am not an active member of any political party.  Political organizations do not direct my topics on this blog site nor do they influence how I address topics.  This is my personal blog site.

Now that we have that clarification over, let me address some of the rules for commenting on my posts.

I have sole discretion over what comments are posted and what are not posted.  No one else influences me on what will or will not be posted.  There is no appeal.  I will refuse (disapprove) comments in the following categories:

1. Comments that I find offensive.  Keep your comments civil.  No slander allowed.  No profanity allowed.    You get the picture.
2. I will not post comments that lead to other web pages or other blog sites. It’s just my policy.
3. I will not allow comments that are clearly advertising gimmicks.
4. I will only allow comments that I deem to be ‘on topic’ for the post. Nothing else.  This is at my sole discretion.
5. I will disapprove ‘comments’ that are longer than the original post. Get your own blog site.

 

I think that is about it.

 

As a final note, please be aware that I write mostly from my faulty memory.  After I retired from NASA I was dismayed to find that I would not be able to access the extensive archives that the agency maintains on the space shuttle and other programs.  In spite of my repeated and courteous requests, the agency shows no signs of letting me or any other retiree access that data for our memoirs or scholarly papers or other objectives.  So what I write is subject to what I remember.  I feel confident though, that my old colleagues – both active and retired – will set me straight if I mis-remember some facts.  I will try to correct any errors that are pointed out to me, but it may take a while.  After all, I’m mostly retired