Recently I visited old colleagues at NASA’s White Sands Test Facility. The Orbital Maneuvering System tanks, removed from Discovery, were sitting in the dirt, ready to be cut up and disposed of. The toxic rocket fuel they held for most of their lives makes cleaning those tanks impossible, and it would be a hazard to museum goers to be exposed to fumes from residual rocket fuel hiding in those tanks. So disposing of those high tech, precision built tanks is sad but necessary.
At an early point in my career at NASA I cared a great deal about the function of those tanks. Because I was responsible to know how much gas was in the tank.
The gas gage on the shuttle never worked. Think about it; the gas gage on your car depends on gravity to keep the fluid in an orderly manner in the bottom of the tank. In space, in microgravity, blobs of fluid just float around inside the tank. How do you measure that? Even more important; how do you get the fuel to the rocket engine? In the Orbital Maneuvering System that question was answered by building in a fine mesh screen: double dutch twill weave. The surface tension of the liquid would keep the gas out and fuel in where the pipe exited from the propellant tank. But reusable tanks brought the concern that those screens could be damaged by vibration or shock during flight. Periodically between flights the screens were tested using a “bubble point” device. A good pressure check with no bubbles meant the screens had no holes and were working properly. Those bubble point tests were done at White Sands. Thereby hangs a tale.
Removing the OMS pods and shipping them to WSTF was a real logistical nightmare. Somebody had the bright idea to move the bubble point testing equipment to KSC so the test could be performed without moving the pods. Brilliant! Until we ran into the standards issue.
For space flight hardware, the Shuttle program specified the standards used in the design, development, testing, and production. But for ground test equipment, the space center where the equipment was used was responsible for the standards. You might think that NASA would have a set of standards for things like welding a pressurized metal tank used in ground checkout of space flight hardware. But if you thought that you would be wrong. Much of the time NASA appears to be a loose confederation of 10 quasi independent fiefdoms, each pretty much in charge of their own business. People often ask me what would I do if I were king of NASA for a day. They expect me to say something like: build this rocket, launch that satellite. Rather I think how I would standardize the procurement processes, or the human resources procedures, or the engineering standards used across the agency. But then I always was a dreamer, tilting at impossible windmills. Launching rockets is easy; getting engineers to agree on standards is hard.
Back to our story . . .
The WSTF had build the OMS tank screen bubble point testing equipment to the center recognized standard for welding pressure vessels. I don’t remember whose standard that was, but it was a nationally recognized standard; let’s say it was a standard of the American Society of Mechanical Engineers. A good standard. A recognized standard. But not the only standard.
At the Kennedy Space Center, they use a different standard; I don’t remember which one, but let’s just say it was the standard of the American National Standards Institute. Now I’ve probably gotten it wrong, but that is just the illustration. Another standard; equally recognized, equally good.
Unless you are a welding standards expert. At KSC, as at every center, a Technical Expert is responsible to see that all hazardous equipment is built to their standards
Moving an ASME certified piece of hardware to the ANSI requiring Kennedy Space Center turned into a herculean job. The KSC Technical Expert would NOT approve its use. Some obscure, arcane difference existed between the standards that he never completely explained to me. A poor program manager could not overrule the Independent Technical Expert, only the Center Director could do that. So no matter how much we cajoled, persuaded, pleaded, accepting the equipment was a non-starter. The Center Director said it was important to show support to his Technical Expert.
So what were we to do? Continue the costly and somewhat dangerous practice of sending OMS tanks half way across the country? Stop doing a necessary safety check of the screens and risk stranding astronauts in orbit some day? Build a new set of test equipment with the KSC required standard which would be a complete waste of the taxpayer’s money? Or. . . accept a waiver.
Yuck.
The program manager had to sign a waiver to the requirement saying that we, the program management, accepted the risk of using non-standard equipment. Yes, we were evil, blind to the risks involved, interested in only schedule and cost. Or so you would think if you read the waiver description.
Small price to pay to get on with business, save money, eliminate other hazards. Honestly, I never understood what the argument was about. I think it was really about control.
So these days I read the NASA procurement request for the new Commercial Crew vehicles, and see that there is a long list of specifications that the companies must use. Almost all of them have the notation that the bidder could propose using a different standard if they are willing to prove to the NASA Technical Expert that the proposed standard “meets or exceeds” the NASA requirement.
And I think about that OMS bubble point equipment and wonder why we are making companies tilt at impossible windmills. Building the rocket is easy. Getting agreement on standards between engineers is hard.
Wayne Hale's Blog 
In my experience supporting Constellation for a few of its early years, this problem was greatly exacerbated by having all of the centers involved in building the flight system, for political reasons, not just the original four centers.
Is this perhaps an important reason SpaceX has achieved its success to date – having a chief engineer who really is a chief?
Wayne, a very interesting behind the scenes look at one aspect of the nuts and bolts of the Shuttle program. I’ve often wondered how well the different NASA entities play together. Probably, like the company I worked for, some play better than others together. Turf is a fact of life, and almost always an inefficiency.
Beth
Lord we miss you Wayne!
Hank
I was the TRW Development Engineer for the Peacekeeper Stage IV surface tension propellant storage assemblies, built by the very same people (not just the same company – the same people, e.g. Dale Stone, Dan Anglim, Dave File, etc.) who built the OMS tanks. MDAC developed the bubblepoint standards themselves, and we at TRW (the System Engineering and Technical Assistance contractor for ICBMs) bought off on them. This is the first time I’ve heard that we were not bigger jerks than NASA….
It’s too bad they think they have to cut them up. Decontaminating an N2O4 tank is a piece of cake. An MMH tank is only slightly more difficult, but by “slightly” I mean a tad above trivial. Since the tanks probably cost north of $1 million each in 1980 dollars, I would hope that someone would reevaluate this move. At least save the N2O4 tanks. They won’t hurt anyone, honest!!!!
Well, that is quite different than the decontamination story I have been told. You should contact WSTF and explain your “only slightly difficult” way to decontaminate those tanks. Could save the government a lot of money!
It’s probably all in the standards – i.e. the standard approach is to cut them up rather than risk a decontamination…..
The reason the tanks, along with the rest of the components, were removed from the OMS pod/FRCS modules was due to the soft goods in the system. The concern was the soft goods off gassing and over the years somebody at the Smithsonian decides to pull a flange on tank there could be an accumulation of vapors that would be detrimental to their health. It was way too expensive to decontaminate, replace the soft goods, reassemble and reinstall the tanks and other components into the pods/mods.
Believe me, if it was a “trivial” effort to decontaminate hypergolic components, WSTF would have figured it out. You can passivate the tanks and make the residual propellant less apt to react. But decontaminate??? You can’t ever be sure how the tank will be used or who will get access to it in the distant future. And it is virtually impossible to get at all the softgoods unless you do some destructive disassembly.
soft goods are usually meant to be replaced on a regular basis. Are you saying there are soft goods, welded in, that are not accessible?
Yep. Not required to be replaced for the lifetime of the system .
Laf
When we were going to fly our small satellite off of the Hitchiker cross bay truss using a small tether to toss it to a higher altitude (supposed to fly on STS-85 but did not due to another similar issue) we ran into exactly the same problem.
Supposedly all of the structural standards for the specification for the maximum dynamic load on a payload flown on the Shuttle are derived from a single JSC standard (NSTS-1700-7B or something like it). However, MSFC and GSFC got to “interpret” that standard how they liked. The numbers that I remember were that a payload had to have a structural margin of 1.25 using the MSFC standard and 1.75 using the GSFC standard.
The problem was that the Hitchhiker that was a GSFC responsibility in the 1990’s was derived from the design of the MPESS carrier (exactly the same actually) that was originally developed at MSFC in the 1970’s. Due to political wrangling the MPESS turned into Hitchhiker at GSFC. Well, there was one problem with that. The structural design parameters were developed using the MSFC numbers but they were being interpreted by GSFC in the 1990’s by, you guessed it, the GSFC standard.
When we were looking at flying our payload GSFC and Swales had developed the Hitchhiker Small Satellite Dispenser (don’t remember the exact acronym). Since the Hitchhiker goes across the cargo bay, and since our satellite is an appendage on that cross bay structure, someone got concerned about structural margins. Then GSFC ran the design numbers on the dynamic loads (yep) with the GSFC numbers and came up with dynamic loads as high as 55 g’s. The folks at MSFC said that this was preposterous as they had run the same numbers using the MSFC standards and came up with a much lower 12-14g.
GSFC argued that since it was going to be an MSFC payload (SEDSAT and an MSFC tether (SEDS), flying on a GSFC Hitchhiker payload carrier that was their responsibility, that MSFC had to use the GSFC numbers. MSFC then said that this was silly because they had built the hardware in the first place and that their numbers were derived from not only the perfectly fine MSFC standards but that these standards for “their” MPESS had come from actual measurements that they had made on the hardware when it was originally flown before GSFC appropriated the hardware.
There was no resolution to this by appealing to the original JSC documents as these were “downstream” derived standards that, as long as they adhered to the original JSC requirements (and both systems had been validated by JSC), then GSFC and MSFC had to fight it out. This was appealed to the NASA Chief engineer at the time and his solution was……
Split the difference!
Which satisfied no one because at the end of the day the equations can’t just be split up and since this satisfied neither party, GSFC and MSFC agreed to use the original MSFC numbers and that as long as the GSFC numbers came out the same after calculations to show the same margins, that all would be well. Somehow they got the numbers to come out the same and on we went….
Before getting killed by some equally arcane structural numbers from another group out of MSFC.
Ah, the memories you bring back Wayne.
Then there is the story of Harold Battiglia and the safety panel and the Mars Observer mission where we almost lost an orbiter on orbit…
but that is another story…..
Yep, Hitchhiker and the MPESS. There must be more than a dozen stories there. Every one of them illustrate how hard it is to work across centers at NASA.
You know it amazes me how much friction that one single piece of hardware caused between the centers (as well as wasted money). Marshall thought that they had pulled a fast one by getting the LMC (Lightweight MPESS Carrier) qualified and flown in place of the Space Hab LCC. Then GSFC got a huge contract to build an equivalent carrier that flew on only one Shuttle mission!
Interesting stuff–pretty funny, but in a serious way.
As a watched the video last week on NASA TV showing the USA crew removing the OMS pods from one of the orbiters, I flashed back to 1975 when I saw these same pods under construction in Bldg 101 at McDonnell Douglas in St. Louis. MDC has been long gone. Now the OMS units are being disassembled. And I just turned 70 two days ago. It’s been a long haul.
Hi Wayne, this sort of thing illustrates why I’ve argued that the structure of the system is more important than the leadership.
You have got to have both. Good leadership and good structure. They go hand in hand. Both are required.
The KSC design cabal treats program requirements as mere “suggestions” to be ignored upon the whim of a few inner circle manipulators, waivers be damned. Since they are smarter than program people, there is no need to comply with program standards. They do whatever the hell they want, when they want. Your argument about fiefdoms is spot-on. The question is, how will the agency deal with the fiefs and still ensure success of future programs?
My experience with KSC is quite the opposite of yours. I found them to be persons of high integrity striving to do the right thing every day.
The vast majority of the employees at KSC are indeed men and women of character who just want to get their jobs done in timely and excellent fashion, so my experience in the wider sense doesn’t differ from yours one iota. Please infer “well-established small group” from the word “cabal”, that being those who put their own personal ambition and their overriding, twisted desire to imprint and create some Von Braun-like legacy ahead of, and at the expense of, their assigned mission. It is a cultural infection that oozes down and brings pain to the entire Engineering Directorate. The new Ground Processing directorate will hopefully provide some counterweight to the cabal’s previous near-monopoly on collective managemental mass, and the cabal will have to be more selective on how they wield their political daggers and suppress dissenting technical opinions. They now have organizational equals by whom they can finally be held accountable, God willing, and the KSC development process will perhaps be able to do better than merely “muddle through”.
At the risk of going against the trend here… we should be careful what we wish for. In some cases, one org’s standard is another org’s boat anchor. Your OMS example is a good example of a problem worth fixing. But there are also examples of standards misapplied.
A few random examples: a) Uniform standards almost brought the elimination of Mac, Unix, and Linux at JSC. In the 90s, JackG told us we’d get more Macs “over his dead body”. He came close to success. There were many orgs scrambling for waivers from the Windows mandate, so they could continue to accomplish important things that Windows didn’t do. b) Standardized human resources has turned a task that’s supposed to be about _people_ into a task that more closely resembles outsourced support for consumer electronics. You dial a call center, get a case number, and hope they document enough about your very human issue. A few days or weeks after you called, you get in touch with the right person, only to discover they’ve been answering the wrong question.
So again, the OMS testing is a good example, but let’s not forget that the windmill of bureaucracy (in most large orgs) can turn the most noble of quests upside down.
[I, too remember the corner of B101 where the OMS pods were being made. I walked past them every day on my way into 105/106/107/101M. Sadly, it isn’t just the tanks that are being scrapped, it’s the lines/valves/etc. A friend was involved in some of the negotiations with the museums. The Smithsonian, in particular, wanted all the original stuff. But NASA had to pay for the prep, and NASA couldn’t afford the detox. It makes me sad, but some of us don’t even have a charge code for FY12 and perhaps it’s best we focus on the basics in this age of austerity.]
Dunno why I always get a smile of my face when I see those goofy pictures of Wayne with his grandkid(?s).
When he was running the ship I always felt–safe? Is that the right word?. Like someone was in charge, someone with a bit of common sense. Like the program was in good hands, no matter who was the Administrator.
There are millions out there who feel the same.
I feel it’s also important to say that there are also hundreds of NASA folks exactly like Wayne. People in government, generally, people who believe, people with some common sense. This is how Wayne presents his stories: people doing a good job.
Let’s not forget Wayne and his minions next time everyone goes gaga over some sort of apparent blunder. These are honest people trying their damnedest to do a great job in something they believe in.
Too bad we just threw them away.
I hope Wayne finds more time to update this darn blog! And a book would be well-received, too.
“People often ask me what would I do if I were king of NASA for a day. They expect me to say something like: build this rocket, launch that satellite. Rather I think how I would standardize the procurement processes, or the human resources procedures, or the engineering standards used across the agency. But then I always was a dreamer, tilting at impossible windmills. Launching rockets is easy; getting engineers to agree on standards is hard.”
True enough. That said, after 6 years working on shuttle ground systems, and another 8 in airborne and spaceborne payloads, I’d say you’re headed in the wrong direction. Time for NASA to change radically. In fact, time for it to mostly go away. When Christopher Columbus embarked on his initial journey’s to the Americas he and his ships needed state sponsorship. We’re way past that point with manned spaceflight (and unmanned for that matter). What we don’t need is a bureaucracy and a marching army of contractors all with their hands out. Should our national policy require a mission, then we need to buy it on a firm fixed price basis and get the hell out of the rocket development business. Ditto on payloads. Maybe we retain a launch range or two, (think port facilities for an analogy). Our government needs enough expertise to be a competent buyer of space services, and maybe we own and operate a space fleet of satellites. Meanwhile let the private sector do most everything else. The problem of engineering standards mostly goes away on that basis.
Your position has been adopted by the national leadership. Now we are engaged in a great experiment to see if that approach will work.
The US tried that experiment with Voluntary Consensus Standards (OMB A-119) and it pretty much failed for the customers of large acquisitions (ex: Airforce SMC). SMC instituted a Sys. Eng. Revitalization program to try and develop good VCS documents, a work I think still in progress. Who won I believe, are the product developers who can avoid the complexity of the old Mil-Spec process, I suspect that’s where the lobbying effort would stem from to create the less powerful VCS system. So we are stuck with the VCS Open Standards process, still could work but the gov’t would probably have to commit about as much money to it as it did to creating the mil specs.
NASA’s budget is increasing, not decreasing, Wayne.
http://en.wikipedia.org/wiki/Budget_of_NASA
That is not ” going away” is it?
Just saw the numbers coming from the congressional appropriators today and that doesn;t look like an increase to me. Then again, I was never very good at math.
To clarify what Jeff said, the SMC activity in open standards development (aka VCS) is still alive but at a much reduced rate from what used to be put into Mil-Stds. We can do better. Where we should see the action is by the primes and the sub-tier companies participating in what they see as “best practices”, instead of being told what to do by the USG. Look at Wayne’s story on the cafeteria and you will understand why.
Writing standards and only looking in your own house is sort of a self-licking ice cream cone. Writing standards in collaboration with external (but still qualified) partners/competitors keeps the lawyers and bureaucrats out of yor shorts.
Can’t say as I follow appropriations day to day Wayne, I have a job that doesn’re rely on government funding – at all. Tell me when NASA sustains a cut that is greater than the fraction of the federal budget we are currently borrowing to pay for will you? That would begin to meet my definition of “going away.” Until then you still need to improve your math skills.
Currently the Federal budget borrows more money than is spent on all non-discretionary spending. So to get rid of borrowing we would have to close down everything but social security, medicare, and the department of defense (which would have to be cut). All the rest – from the post office to the EPA to the courts and the FBI – all would have to go away to preclude no borrowing. Or, the entitlements programs would have to be cut significantly. Write your congressman; they could use your advice on where to adjust things.
Wayne,
I almost hate to admit it… But I was the lead quality engineer at WSTF at the time those Bubble Point carts were designed and constructed. It’s not that they met a different standard… They didn’t meet ANY standard other than good engineering practice and construction. Which basically meant that they would work for a while, but not to any predictable standard of wear or cyclic performance. Given their infrequent use to support the Shuttle Program, there was little question that they would do the job quite well for the foreseeable extent of the program. Did WSTF or NASA do anything wrong? I don’t think so. But what needed to happen from the start was WSTF state the limitations of use and a knowlegible customer at KSC understand the limitations and accept the risk. It is that simple, but with the variety of people involved in the project it is and was difficult for the right people to have both the technical expertise and the authority to make it happen.
As a colleague of mine has said on a few occasions ‘That’s the beauty of standards, there are so many to choose from’ and he is right. Of course, it’s easier to use a different standard here.
I do love the fact that your wavier was because you were non standard even though you were standard – just a different standard to the accepted one but then I look at the flip side and think that having *something* that highlights a deviation from the accepted standard is no bad thing.
Almost funny, in a sickening way. So…did anyone point out to the standards guys at these centers that the National Technology Transfer Advancement Act (NTTAA) from 1996 requires all government organization to participate in the development, and use of, industry standards? Only if there are outstanding reasons can this be waived. Waivers have to come from the agency head and reasons for the waiver explicitly stated. Each year all agencies are required to report to Congress on waivers and reasons. This is not a space organization requirement, it is a requirement for all US government organizations.
The last time I checked, the two outstanding “bad boys” on the non-compliance list to Congress were …wait for it….NASA and DoD Space.
Another “almost funny” part of this issue is that, once you get the right people in the room, developing a common standard isn’t that hard. It is true that getting the right people in the same room can be very difficult. Experts (almost by definition?) are the people who understand the whole playing field in their domain. You don’t get to that level by refusing to interact with other experts in your area. Defending rice bowls can be an issue, but the technical details are generally not.
For more on space industry open standards at a global level check out ISO TC20 subcommittees 13 and 14. In the US check out the AIAA standards program. In Europe check out the ECSS activities. In Japan… well, you get the picture.
Just what we need, another level of bureaucracy to navigate!
That’s a pretty standard first response.
Actually, you only need to navigate the bureaucracy if you insist on “going your own way”. While you do have to plan ahead, data show that cost, schedule and reliability are impacted in a strong positive manner when you follow the open standards route.
I’m reminded of an old University joke that went something like
A Philosopher, Mathematician and Engineer were asked “What’s 2 plus 2”?
The Philosopher answered by saying “Well I have to think about it but I think it’s between 3 and 5”
The Mathematician said “2 plus 2 is 4”
The Engineer said “What would you like it to be”………
I’ve never heard that story told about an engineer. The punch line I’ve always heard was directed at accountants.
Interesting conversation. Similar problem with Delta IV second stage. It was designed to meed the design load margin required by the DOD, 1.25. NASA however requires 1.4 for “man-rating”. That was, as I understand it, the figure used for the Shuttle, when loads could only be predicted on paper, that had to carry people on its very first launch and go through an anticipated 100 flight cycles. With the Delta there were numerous unmanned launches where the actual loads could be precisely measured with strain gauges, and the service life of an ELV is much to short for crack propagation, corrosion, and similar random failure processes. There was no rationale for the figure of 1.4, in my opinion it was simply arbitrary.
But no one will view the problem as a whole and find the best solution. Excessive cost makes program failure inevitable. That may be a good reason why it would be better to continue the Space Act Agreements for Commercial Crew and allow the developers to exercise broader control over their systems.
The aviation standard for factor of safety is 1.5; exactly how the shuttle got to 1.4 is a long and involved discussion. The standard for expendible launch vehicles has been 1.25 for some years; again a long discussion on why that figure was picked. All load calculations are “paper” (actually most are electronic these days). That is because you hope to never fly the maximum load case during ascent (or entry). All of the Delta IV and Atlas V flights have been right down the middle of the trajectory envelope and have not measured the limiting case when the situation is at the design limiting load. Even if the actual flight experience had approached the 3 sigma load case, there are dozens of different cases which stress different parts of the structure; and instrumenting all the critical points would be a huge engineering job. Structures folks live by their analysis; don’t discredit it.