“Arm chair generals study tactics; real generals study logistics” – attributed to General Norman Schwwarzkopf
Many of my old friends and colleagues are asking me a question these days: “If you were NASA Administrator, what would you have the agency do?” I know what they want to hear: Moon, Mars, or Asteroid – what is the next destination for human spaceflight? But that is not the answer I would give. Whatever ‘horizon goal’ is established, without significant organizational and cultural changes at NASA, the chance for success is in doubt.
To make NASA into the extraordinarily effective organization it once was and could be again will require significant work to transform it. NASA is filled with extremely smart, highly motivated individuals who are the experts in their fields. They can do amazing things. Measured against any other organization – government or commercial – the NASA civil service and contractor work force is outstanding in terms of inherent capabilities and the desire to make their projects successful.
But success in NASA’s endeavors is hobbled by three structural and cultural problems: (1) inter-center rivalry, (2) mind numbing bureaucracy, and (3) a paralyzing cultural requirement for perfection in all things.
These are the problems I would propose must be improved for any large scale program to be effective. And frankly, resolving these issues exceeds the NASA Administrator’s authority. Solutions will require not just concurrence from the President, but action by the Congress would be required. And given that somebody somewhere would probably file a lawsuit regarding some of the directions, the Judicial branch would have to concur as well. Rapid, coordinated concurrence from all three branches of government? What are the odds of that? So my title: King for a Day.
So in the Libyan fable it is told
That once an eagle, stricken with a dart,
Said, when he saw the fashion of the shaft,
“With our own feathers, not by others’ hands,
Are we now smitten.” – Aeschylus, Choephoroe 59
Topic 1: break down inter-center rivalry. NASA was established in 1958 as a collection of 10 loosely federated fiefdoms and it has never broken out of that paradigm. If you ask a typical NASA employee who they work for, the response will be their center, not the agency. Can’t blame them; they are hired through a center, promotions and career advancements come through their center, the very culture of the organization enforces loyalty to a center. Every center has its local politicians and politics centered on local interests, every center has its own history and area of expertise, and every employee is inculcated with the beliefs and norms. Centers sometimes seem united only in their disdain for NASA Headquarters. Not that anybody openly works to sabotage direction from Headquarters, they just bend the direction toward what their individual project and center would like to do. Competition for scarce resources drives rivalries between centers. In addition, there is a huge ‘not invented here’ problem everywhere. Not just with any idea from an organization outside NASA but also with any idea from another center. It makes the workforce ready to find fault, slow to see the advantages of any new thing not born from within their own organization. Secretive, competitive, and ultimately destructive of the larger purpose, these behaviors have been worse in the past but are still present. My solution: make people move. Many organizations both government and industry do this as a matter of course. Move not just the senior leaders, but the journeyman workers. Take the center name off the badges. Develop a ‘Bureau of Personnel” to centralize promotions, bonuses, and career advancement. No small tasks these.
“A system under which it takes three men to check what one is doing is not control; it is systematic strangulation.” – Admiral Hyman G. Rickover
Topic 2: mind numbing bureaucracy. The organization has evolved, as all bureaucracies do, to the point where too many people can say ‘no’ to any action. In the early days of NASA, this was not so. It is good to have checks and balances and oversight, but the pendulum has swung too far in the direction of (electronic) paperwork, diffuse responsibility, and inaction. The system now has watchers watching watchers watching doers – and always with criticism for the doer. Corrective action will take serious attention from any leader. Achieving the proper balance may well be impossible and the best we can hope for is to swing decision making back to the lowest level possible. Gibbs Rule #13 applies here: Never involve the Lawyers.
“The desire for safety stands against every great and noble enterprise.” – Gaius Cornelius Tacitus
Topic 3: the cultural imperative to make everything perfect. This is a very sensitive topic for me. I have personal been involved with decisions that were made with too little information, riding roughshod over the experts in the field. But these days, after Columbia, the agency is paralyzed by requiring too much: too much data, too many tests, too much analysis. In the Apollo days, this was not so. We – and I am a guilty party in this – have trained the work force to make everything perfect before any project can proceed. In this business, nothing is ever perfect. Space flight involves risk, it can never be completely eliminated. But real space flight is actual flight, not studies and ground tests. It is difficult to find the balance of having done enough to be reasonably sure of success and safety and to get on with a project and actually fly. I hate the term ‘risk averse,’ but as much as it makes my teeth grate, the effect of wanting to make every detail perfect has the same outcome as cowardice: never flying.
So when folks ask me that question: “If you were NASA Administrator, what would you have the agency do?” I have a rueful look on my face and tell them any destination – or all three – are good; the tougher job is what we must do to ensure that we get there.
“Come, my friends,
‘Tis not too late to seek a newer world.
Push off, and sitting well in order smite
The sounding furrows, for my purpose holds
To sail beyond the sunset, and the baths
Of all the western stars, until I die.
It may be that the gulfs will wash us down;
It may be we shall touch the Happy Isles,
And see the great Achilles, whom we knew.”
- Tennyson’s Ulysses
Wayne Hale's Blog 
Very True, The system now has watchers watching watchers watching doers – and always with criticism for the doer, SpaceX is now at a spot where this may start dealing with the Federal Goverment who will never supply “Doers”,,,,
Wayne,
From my perspective, you “nailed it.” I worked 26 years as an aerospace engineer and manager at two Naval Aviation Depots, before moving to a NASA center as a systems engineer and project manager for almost 9 years. There are a lot of similarities between the two organizations, including the inter-center/inter-depot rivalry and the bureaucracy. Both space and naval aviation rely on “doing it right,” as flight safety has to be job 1. However, the most significant thing I saw at NASA was it was risk-averse to the extreme. There is never a 100% safe solution. However there is a fine line between doing enough analysis and testing and doing more to get that last small fraction of a percent, which can be a significant amount more. It is the job of management to determine when enough has been done. NASA has a severe case of “analysis paralysis.” There was a sign in one of the production shops at a Naval Aviation Depot where I once worked that summed it up well: There comes a time in the life of every project when you have to shoot the engineers and get into production. NASA is a great organization and it was a privilege to have worked there; I look forward to it continuing to do great things.
Regarding your Topic 3, Wayne, there are contexts in which I find NASA to be so risk-tolerant, it’s downright irresponsible. The classic example is landing astronauts on Mars for 500-day stints while knowing nothing about human adaptation to Mars gravity (38% of Earth’s). This “plan” becomes irresponsible when NASA’s exploration roadmap contains no study of such adaptation in LEO using artificial gravity, where the experiment can be promptly ended if it goes awry. On Mars, astronauts are months to years from a return to Earth.
When discussing risk tolerance, I think a disservice is done when “risk” isn’t qualified. Back in our Mission Control days, we prepared for Shuttle flights by managing risk. We studied threats to crew and mission at least one failure deep in advance and came up with plans/procedures to address them, thereby creating *calculated* risks we could at least hope to manage should they arise in flight. Sending humans to Mars without studying their adaptation to conditions there is a *blind* risk and irresponsible planning IMHO.
Great piece, Wayne. These aren’t new problems or issues; we saw them in Downey in the 80s and 90s, but it’s always worth repeating them for those unaware, particularly in the midst of policy turmoil of a new administration.
I’m currently most struck by the double standards and hypocrisy of continuing to delay Commercial Crew because “safety is the highest priority,” keeping us dependent on and shipping millions to Putin indefinitely, while planning (or at least considering) a repeat of Apollo 8 half a century later as a political stunt to try to preserve programmatically insane programs.
NASA and Russia have researched astronauts and studied their biology in zero-G for a year or more, so it’s of course feasible to allow astronauts in .38G for much less than 2 years. As has been documented many times over, artificial gravity assets would be cost prohibitive and not provide much benefit over what has been already researched in zero-G.
I moved to Houston from Huntsville in ’75 (worked for a contractor). In Huntsville, there was the Saturn/Apollo program, but in Houston, it was the Apollo/Saturn program.
Strong points, Wayne. One would hope that whoever takes the reins at NASA will work to overcome these obstacles.
Excellent piece, Wayne. It’d be great if the next administrator could address some of the systemic issues that hinder NASA’s efforts, but sadly, he or she will be serving in an era where a majority of elected officials don’t believe in science or a reasonable NASA budget.
Good words.
I spent 9 years in MOD working both Shuttle and visiting vehicle missions, then went to check out the O&G industry for about 3 years, and recently returned to the NASA/FOD world. While in O&G, I was doing technical integration of a global engineering team to support ongoing drilling operations and failures. The path I’ve taken has made for an interesting perspective on the strengths the NASA workforce has and what holds us back from a repeat of the “glory days” that everyone hearkens back to. And, I 100% agree with you on the 3 points you’ve discussed. Having a mission that we all (NASA family and U.S. citizens) rally behind would be great but, even if that were to happen, we’ve become our own worst enemy.
While O&G obviously has a very different objective, one great thing about working there was feeling unshackled by the insane level of bureaucracy that has become the normal expectation in the NASA world. O&G leaves much to be desired in their safety culture (if we’re calling NASA risk-averse, I’d almost call them blindly risk-encouraging). I knew what I was coming back to but it has definitely made it harder to be fully engaged when I know that the level of bureaucracy and inter-agency pissing matches (for lack of a better expression) has the ability to suck the fun out of everything. I’ve been in team meetings in which someone said they didn’t trust anything coming from another center, so we’d need to make sure we dedicate time and resources to redo it. How is this a useful allocation of resources? The time would be better spent learning how the other center does the work/analysis for a product they’re delivering, and discussions on where our concerns might be, so that we can develop a better level of trust with each other.
Another great thing the O&G company I worked for did was an annual review of the skills of their workforce and where it might work to rotate in a new leader. This created the opportunity to be able to grow and learn outside of the specific area you worked in while also benefiting the company. This gave the management team the chance to communicate with upper level leaders people that were high performers and potentially looking for rotational type opportunities or would be a good fit in other positions around the company. Coming back as a contractor, I’ve been quickly reminded of the limited advancement and rotational opportunities for me in the NASA world without becoming a Civil Servant.
I agree that the workforce at NASA is, “is outstanding in terms of inherent capabilities and the desire to make their projects successful.” You can’t motivate a team with money the same way the inherent desire to be a part of something bigger will motivate a team. But, the really motivated employees will only stick around for so long in the face of constantly changing missions AND feeling hindered by everything you’ve discussed.
A good example: spending vast amounts of time and money to “man rate” rockets that have never exploded, and hardly ever even under-performed, in the course of a long career launching, among other things, critical national assets worth a billion dollars or so each (without, of course, LAS capability). NASA is now mulling flying crew on the very first SLS flight. THAT is a juxtaposition truly worthy to behold.
I Think this is your best work Wayne
Yes.
Further that that, It seems that inside and outside of NASA, the whole space community fights over the budget crumbs instead of working together for a bigger cookie. “My program (Center) is better than yours!” arguments make me think of Daffy Duck, “Mine, mine, mine, mine, mine, MINE!!”.
Reading this article was akin to reading my thoughts for the last year. Coming from a fellow government acquisition agency it amazes and pains me to see the extraordinary variations in the methods by which each government agency procures equipment. Much like the inter-agency rival (which is as noticeable as a pink battleship), the lengths taken to “do things different” when acquisition and procurement policy is US Code, not “well here are some guidelines, just do it the way you want”. When that happens, government agencies get taken advantage of and feel they are held hostage by the big industry players. I say, hold them to a cost reduction driver and if they don’t like it scrap the program. Nothing would be more interesting than providing a vendor a “Termination for Default” (T4D) letter based on cost overruns. Talk about an industrial complex wake-up call. Also, know when to tag out. Stop worrying about what someone is going to think about an opinion you have if you see something that needs to be mentioned. On that note, senior leadership stop taking things so personal. Just because you were the author of a document several years ago and the project has progressed past the usefulness of that document, don’t be shocked if it gets an update that is more of a re-write. Let the project grow in all facets and people won’t be so gun shy of telling you what they think.
Ok, enough bantering. I enjoyed this article and hope the recommendations made and ideas within are carried out to some degree.
I remember back in the 70’s hearing about some air traffic controllers, don’t remember if it was at a tower or center, who were beleaguered with endlessly increasing new rules, many of which they knew had no positive impact on either efficiency or safety. But try telling that to the supervisor who came up with the latest rule.
Finally in exasperation they decided to follow all of the rules, each and every one to the letter. The result was planes circling around and the traffic jam was a disaster. I don’t remember how long this went on but their supervisors become so panicked that they essentially told the controllers “Whatever you were doing before, go back to that.”
Maybe apocryphal but it seems believable.
By the way hopefully I’m not too aliterate, but I can’t remember ever coming across the word “inculcated” before. Enjoyed looking that one up.
“Working to rule” is a standard tactic in labour disputes.
This discussion is about hindrances to success, not labor disputes about wages, benefits, etc. The air traffic control story as I remember reading it was about controllers who wanted to do the best job that they could, but were hamstrung by unnecessary rules and procedures. They finally started following all of the rules just to prove this point. Whereas work-to-rule in labor disputes is just a method to gain leverage against the employer for some other purpose.
Great points. Totally agree. I wonder about the best way to fix them though. My gut tells me that the best way to get past these things is to get them working on a goal with a date. “Achieving the goal, before this decade is out, of landing a man on the Moon and returning him safely to the Earth”, can pretty quickly get past all three of the issues you mentioned. You pretty quickly get to 8 hour shifts, 24 hours a day, and no time for excessive risk aversion or over-bearing bureaucracy. I wonder if the budget and the administration directed NASA to have fuel depots in LEO and lunar orbit with an ambitious deadline… well, i think it might be the best medicine for the ails you mentioned.
Absolutely endorse your “cross-pollenation” idea anent employees and the NASA centers. Back in its glory days, employees at IBM used to say the three initials actually stood for “I’ve Been Moved.” IBM was hardly unique among large business enterprises in moving its people around on a regular basis. In the government employee sphere, only the military and the FBI seem to do this.
The question of exactly how many centers NASA should optimally have is endlessly arguable. I’m personally inclined to think that number should probably be at least one or two fewer than at present. But I also think implementing your suggestion about moving the game pieces around the board would likely improve things so much over the current state of affairs that avoiding the base closing-type political fights that would ensue were any centers to be targeted for closure might be okay. There would certainly be internal resistance to implementing a policy of routine inter-center transfers every, say, three years, but I don’t see any of the major political players involved having any obvious motivation to “go to the mattresses” to prevent its implementation. The pols are interested mainly in, as Tom Heppenheimer famously put it, “keeping the parking lots full.” Exactly whose cars are parked in said lots at any given point is at least a couple orders of magnitude less interesting, politically.
The multiplicity of NASA centers, and their geographical dispersion, was a product of the politics of the time of the agency’s creation. Most other government agencies are much more DC-oriented in their staff distribution than NASA. Pushing more executive branch agency employees of all kinds outside the Beltway, and then rotating them regularly around the country, would likely have just as salutary a probable effect.
Wayne,
I think the problem you see with inter-center rivalry is overstated. Some competition is healthy. But much of this inter-center rivalry atmosphere you detect, has developed because many high-level folks at JSC, supported by accident investigation boards and media reports, have blamed MSFC managers for decisions they made related to our Shuttle Challenger and Columbia accidents, without more carefully considering the decisions that were made by Level II Shuttle Program Managers officed at JSC, that in my view, are the primary root causes of both Shuttle accidents. Level II Shuttle Program Managers (officed at JSC, but reporting to NASA HQ) made decisions to continue to fly the Shuttle without fixing the SRB O-ring and ET foam shedding design deficiencies, identified in flight anomalies that were known at Level II during the Shuttle flight test program, to be violating Level II System Design Requirements that the Level II Program Manager controlled. If the design deficiencies had been rectified at that time (as they eventually were many years later after two long Shuttle flight grounding periods) for violating Level II System safety requirements, as is common in commercial aircraft programs, the two Shuttle accidents would not have occurred. (Consider how Boeing dealt with its 787 Dreamliner battery fires…..that one slipped though the entire flight test program without being identified.)
During Space Shuttle design and development, I led the Level II Shuttle Pogo Integration Panel from my office at JSC under a Section Head in the JSC E&D organization, but with a Co-Chairman at MSFC who was an Assistant Division Chief at MSFC, and who served as a single point of contact for me, because MSFC had many more civil servants and hardware contractors working to prevent pogo instability on Space Shuttle than we did at JSC. One of the most important mentors to me during that time, was Dr. Robert S. Ryan at MSFC, who was the MSFC Division Chief our Pogo integration Panel Co-Chairman worked for. Our panel goal was to always have a consensus position on Level II decision issues from our panel of civil servants from JSC, MSFC, KSC and three NASA research centers, reps from all Shuttle elements of Orbiter, ET, SRB and SSMEs and advisors from The Aerospace Corporation. We understood, that at times, the various Shuttle Level III Project Offices at MSFC, or Level III Orbiter Project at JSC, would have a different decision position on a Systems level issue, than an optimal decision position as viewed by the Shuttle Systems Engineering and Integration Manager and Level II Program Manager who officed at JSC, but were responsible to NASA HQ, not the JSC Center Director.
In such cases, our panel Co-Chairman at MSFC pulled together the briefing that explained the MSFC Level III Project Office position on the issue (usually the SSME Project Office) and I pulled together the opposing position that the remainder of our panel recommended to Level II integration and program managers. The issue was discussed pro and con at the Level II Program Requirements Control Board (PRCB) and the Level II Program Manager, with recommended input from the Shuttle Systems Engineering and Integration Manager that I reported through, made the final decision for the Program. The biggest decision issue our panel had was the Level II decision to locate the Shuttle Pogo Suppressor inside of the SSME. That was a significant impact to the SSME Project, which it understandably resisted from their point of view. But from an overall Systems level point of view, where that unorthodox suppressor location was optimal for Shuttle System pogo suppression performance, the Level II Program Manager allocated the identified cost and schedule relief to the SSME Project to accommodate an optimal System design decision, and we all went away happy without any inter-center rivalry to impede implementation of that decision.
This is the management structure NASA HQ preferred for the Space Shuttle Program and decreed to the NASA Centers, because they realized the engineering expertise needed to integrate the complex Shuttle System was resident at JSC, MSFC and KSC, and could be more effective if they were more closely located with day to day decision-makers, than if such day to day program management were located at NASA HQ. In these intimate inter-center working relationships, that occurred in all of the Shuttle Level II Technical Integration Panels, we did not experience difficulties with the inter-center rivalry you see as a big problem for NASA. It wasn’t a serious issue during Shuttle design and development, and doesn’t have to be that way today.
It’s my belief that the “primary root cause” of both shuttle disasters was schedule pressure. Everything else was mechanical.