My good friend and colleague from the Flight Director’s office, Jeff Bantle, left the agency several years ago to head up Lockheed-Martin’s Presidential Helicopter project. He now has a great presentation talk about how incrementally added requirements can sink a project. I suppose many folks could tell a similar story about a home renovation project that got out of hand. Indeed, one of the first lessons at project managers’ school is the need to control requirements. A successful project always has a concise, well considered, thorough list of what must be done to be successful, and a successful project manager is one that manages to avoid “requirements creep”. (Couldn’t we just wallpaper the kids’ bathroom while we are redoing their bedroom? How about added a comm system to the helicopter that can communicate anywhere with anybody?).
Put in simple terms, if you want to be successful, it is good to have the “must do” list of things and not add to it half way through. While I can’t cite specific studies, my observation of several major NASA projects that have gotten in trouble over the years shows a high correlation with new, added, late, or poorly defined requirements which caused technical issues, increased the costs, and delayed the schedule. Put simply, a good program manager has got to have the gumption to just say no to changes in requirements – even when they are really good ideas.
I have frequently quoted the old adage: “given unlimited resources and unlimited schedule, great engineers will produce exactly nothing.” That is because we engineers can’t leave it alone; there is always a slightly better way to do it if we just junk the design we have in hand and start with a clean sheet of paper. Successful program and project management requires many talents and abilities, but critical among them is having a leader who can just say no to creeping requirements.
So the Space Shuttle Program – lead by some really terrific managers over the years – really clamped on to the notion that having good, well defined, concise requirements was absolutely necessary and new, nice-to-have ideas were very carefully scrutinized and generally turned down. Part of that is due to financial concerns – a topic for the next blog post in this series. But mostly, that culture was inculcated because requirements control is the cornerstone of excellent program management.
Many people inside the shuttle program – as well as hordes of armchair commentators – never understood this. The need to keep strict division between doing what was mandatory, what must be done – the ‘requirements’ – and those really good ideas that might make the system better, safer, more economical, etc., is difficult. And all experienced project managers have been burned by “good ideas” that cost more money than expected, have unintended consequences, and frequently just didn’t work out.
So, over time, the shuttle program did some things – like upgrading the Space Shuttle Main Engines – and did not do other things – like building advanced solid rocket boosters or liquid strap on boosters. Working on a real program with real deadlines and a real budget means that you have to choose. Mostly, shuttle program managers over the years just had to say no.
In space repair of the shuttle thermal protection tiles was considered a requirement at the start of the program in the 1970’s. The tiles were fragile and could be easily damaged. A great deal of money, time, and effort was spent in trying to develop a repair material and techniques that could be applied in space. It didn’t work out. Technology wasn’t there. The requirement was deleted. After the Columbia accident, we reinstituted tile repair as a requirement. Many experts reminded us that repair was a failure before and we should avoid repeating the mistakes of 30 years earlier. But after 30 years, materials science had advanced enough so that the project was successful. Successful at great cost, operational complexity and overhead, and significant schedule impact. Was it worth it? Tile repair plus other improved capabilities gave us the confidence to fly another 20 shuttle flights. We never needed to repair a tile in space, of course.
So every item, every requirement, every suggestion was reviewed – and repeatedly reviewed as the financial constraints tightened – and the phrase “is this really a requirement” came to mean, ‘is this really something that we have to do or can we argue our way out of doing it’. So “requirements” became a tyranny. Justifying something as a requirement became a herculean task. For all but the most obvious mandatory changes, the leadership just said no in the name of “good program management” in a “mature program”. There is another topic to discuss in that last sentence too.
And so the day came, when an operation that many people wanted, and that I championed, one that bears directly on the loss of Columbia, was denied because “nobody has a requirement for that”.
It probably won’t have helped anyway.
The story will continue.
Wayne Hale's Blog 
I can only relate to lousy project manglement. I once worked for a computer software company that developed a loan approval system for financial companies, i.e., banks. The first product was deployed to kiosks and later, a web interface was developed.
The client provided their criteria for approving loans. After the client signed the contract with the company, work would begin with their specifications.
However, the downfall of the company, my employer, was that they had a problem with saying “Okay, but this change will cost this much and delay the project by this much..” They had a problem with saying “No”; Anytime a client request for a change or enhancement came through, they said “Sure! Do you want to supersize your order with a biggie drink and super fries?” In an environment such as this with creeping featurism, it is no wonder that the company never delivered a completed project.
I worked with great, dedicated programmers, but there were some problems at the executive level. It is no wonder that the company went bankrupt even though they had patents for the process.
A post that really speaks to me as an engineer. I worked as a software test engineer, and our requirements documents were the test bible. Keeping the software ‘to the requirements,’ was difficult but necessary. Almost all of our requirements creep came from marketing, though, and not engineering. Lots and lots of great ideas, but only so much time, money and manpower to make it happen.
how much of this frustration is due to the shortcomings of the waterfall project management approach? I understand that for items as critical as flight software, or a space vehicle, some rigidity is necessary. But we as humans have only been dealing with the technologies available to us today for a very short while. As such, project management approaches must change, and unfortunately, NASA often lags the industry in this regard. Rightly or wrongly.
I think you are out in left field. Good project management begins with small projects that are not high tech; the process is the same. NASA is not lagging the industry in this area.
I was pretty unpopular in the early ’90s for making changes cost something (time, money, or both) when the customer (a NASA division) came to my group (another division doing the software work), and had a great idea. Left to their own devices, my programmers would have tried to implement each and every idea: Virtually all of them were excellent, beneficial, and useful. I had to say “No” if we were going to deliver on schedule with the limited fiscal and personnel resources we had, though. It wasn’t fun, it wasn’t easy. It was necessary.
On another JSC project, I flew some hardware on Spacelab-J. We saw a lot of requirements thrown at us I couldn’t understand. We were required to demonstrate safety issues, and then told to mitigate them after we’d complied with the requirements (additional requirements layered on top). We had to add testing because MSFC EMI/EMC was, on paper, different from the JSC EMI/EMC testing, although the procedures and specs were identical. That MSFC had a different document and didn’t simply reference JSC specs, which were the originals, didn’t matter, we had to test them twice and document the different test sessions to prove we’d not just pencil-whipped the result.
Requirements are necessary for safety, reliability and quality, but are NOT something that should be waved around to get someone to just do extra work or to try to get a project to fold because it’s on someone’s list. Well-written requirements are wonderful. Poorly written, conceived or forced requirements can lead to all sorts of problems.
You mentioned that the technology for tile repair wasn’t there at the beginning of the Shuttle program, only later was it possible. Was it a similar story with on-orbit inspection using OBSS?
No, the technology that would have been necessary to inspect visually was available much earlier. We just didn’t put the parts together because there was no requirement to inspect the heat shield before re-entry . . .
Great topic, Wayne. My experience with requirements management during the Shuttle years taught me several lessons. First, such management is doomed to chronic failure without a well documented and current ops concept. Without this documentation there is no coordinated basis for requirements rationale or prioritization. Second, an otherwise valid requirement which cannot be immediately funded for implementation should NOT be deleted. These requirements, and the rationale for implementing them, should be carried and prioritized for the life of the program or until they’re no longer valid. Without this requirements backlog, how can effective arguments be made against excessive funding cuts? Third, a program which cannot fund at least some low-priority requirements for implementation is symptomatic of an endeavor whose integrity is seriously at risk.
As an ops-savvy guy, I hope you’ll agree that not all requirements creep is bad over the life of a program. The more a complex system is operated, the smarter engineers get about how to operate it more efficiently and safely. The Shuttle Program had multiple processes with which to report, assess, implement, and reward ideas equating to new requirements reducing operational costs and risks.
I’m not sure I made my point well. Avoiding requirements creep is absolutely essential, but so is good judgement in approving those new things that are worthwhile. Life is full of choices, making good ones is often very complex.
During my days as a shuttle engineer on the OMS/RCS subsystem at KSC, I led a team of folks who were able to navigate the requirements change gauntlet and rewrite the requirement for the ground servicing of the RCS helium tanks (V42BF0.020 to be exact). There was an inherent problem with the requirement that led to many arguments on console during servicing and potentially out-of-spec helium loads. To make a long story short, we turned an arduous task that included dozens of pages of hand calculations in a procedure that utilized graphs developed decades before (and had been copied multiple times so that you could barely extract the necessary data) into a simple entry of a few pieces of data into a spreadsheet that output a consistent result. This change along with a few others resulted in a simplification to the process that turned a ~30 hour pad clear operation into a ~16 hour (or better) pad clear operation-a fairly significant improvement.
Requirements changes are not always a bad thing. With programs that last decades you are almost certainly going to have improvements in technology (as you stated with respect to tile repair) that will allow you to streamline processes or make a “better” design, but in my new job as subsystem lead on a new NASA project, I now realize the ripple effects of requirements changes through a project. It is certainly a delicate balance.
Changed ’em? I think Ralph Roe and I developed those graphs back in 80 or 81; how could you change them?
OK, not serious – obviously they needed update and you guys did a great job in making the operation more efficient while preserving the overall intent.
Best of luck in the new era.
Wayne,
In my career, there have been several kinds of “requirements creep”. As others have mentioned, sometimes it’s the stuff of the product buyers’ dreams. Sometimes it’s technology-driven, especially in the world of computers and microprocessors.
Moore’s Law renders today’s best obsolete tomorrow.
Most often, it’s safety that causes this effect. While a Model A will do the same job as a Prius, how many people would drive that venerable antique to work every day?
STS-107 didn’t have the ET-mounted camera that the flight preceding it had. Every mission after did, and when combined with HDTV we were all treated to some simply astounding views!
I recently rode my 2005 Harley-Davidson 1200 Roadster from Pittsburgh to Kokomo, IN for the annual gathering of Vietnam War veterans. Although I never needed to open the box of tools I brought along, they would have been sorely missed if I had needed them. Better to have it and not need it than to need it and not have it.
Spaceflight and motorcycle touring have this in common: weight and space limitations. You learn to choose wisely.
I am an experienced project manager, and in my experience requirements changes are a fact of life. It’s the job of the project manager to work with his team to understand what is the impact of a proposed change on cost, schedule, risk and anything else of interest, as well as the impact of not making the change. Then it’s the job of the project sponsor to decide if the change is worthwhile. Approving or disapproving changes without understanding the impact is the road to project failure.
I am SO not involved with anything as heavy as the space program, but I well remember having arguments with people in one of my old companies about how we can’t jerk around revisiting every single decision whenever the mood strikes us. That at some point we had to be willing to plant out foot in a certain spot, leave it there, and start thinking about where the next step goes. I got blown off and dismissed constantly.
You know what the sad thing is?
I am in marketing. All the guys who blew me off when I said this were the engineers.
Their company was successfully bought out, and they all got rich. Throw enough money at a worthless company, and even it can make someone rich.
I work in a nonprofit now. Much, much nicer. Here, the CXOs actually have brains.
Anyhow, I can sympathize with the concept of just making a decision and going with it without wasting endless time revisiting it over and over … even from way over on the marketing side of the building.
Of course the intent here is to be a history lesson for others so that they can learn how to avoid the mistakes we made.
A little introspection is a good thing; but like most things it can be overdone. You can’t grow marigolds if you keep pulling them up to examine the roots
The really sad thing is that they never wanted this level of rehashing AFTER something had been completed to determine how it went, where the wins and losses were, and what we might have wanted to do differently next time. >_<
I think most engineers with more than 10 years of experience should be able to say “been there, done that” by now, though perhaps not to the scale of the Shuttle program. It makes me think of the ill-fated but enormously expensive Future Combat System. At its peak, its budget actually exceeded that of the Shuttle program, and given its ultimate lack of results, should make taxpayers cringe. The teeny bit left at the end (Brigade Combat Team Modernization) was finally put out of its misery last year. I was involved with it at a subcontractor; fortunately, I got out a couple of years before Congress finally pulled the plug, so I was able to avoid the inevitable layoff. It was bad. We were, I kid you not, delivering more proposals in response to customer RFPs than we were delivering actual SDRLs. And yes, this slowed us down, for entirely obvious reasons — we were diverting resources to doing the proposal work, and the constant churn meant that engineers couldn’t really ever get ahead. It was the most surreal work environment I’ve ever been in, because it was so difficult to see where we were going.