I’m in my 11th year of IT. During this time I have been exposed to most of the software development methodologies around. At least, I’ve been exposed to numerous interpretations of these methodologies since no firm I’ve ever worked at or consulted for ever designed software the same way as any other firm. The strange thing about software development is its strangeness. One would figure that after forty+ years of programming and software development we would be able to be able to manufacture software with the same aptitude we build cars or skyscrapers. But the dirty little secret is that software development remains more of an art than a science.
Such a comparison really isn’t fair to software development though. When we consider car manufacturing, Henry Ford is credited with mass manufacturing the automobile. But the story goes that when asked about the color choices for his first cars available to Americans, he said “They can have any color they want – as long as its black.” Ford became successful because he limited consumer choice.
That’s something that software consumers refuse to accept. While companies like Microsoft, Oracle, and others manufacture mass-produced software, much of this software is modified by each company that implements it. Many of these changes aren’t just cosmetic user preferences; they are significant changes to the functionality of the application. There’s even a name for it: COTS - customized off-the-shelf software. The average individual might be able to pick up a copy of a game or an application, install it on her machine and have her do something useful, but businesses can’t do that. COTS takes an off-the-shelf product and customizes it, often using programmers and developers who work solely with that product.
Businesses are as individualized as the people that work for them. While they can use MS-Office and some other apps straight out of the box, when it comes to the software that runs their infrastructure they need something that is COTS, home-grown, or more likely a blend of both. Add in legacy software and you end up with a huge mess needing someone like me to come to help figure out a better, more efficient and cost-effective way out.
To service these situations various software methodologies have come into being. Waterfall and Agile are the two most common, with Scrum and RUP as common flavors of Agile. All of these have their benefits and drawbacks. Waterfall design is perfect for large scale projects having static goals, constraints and scope. An OS or a single application can be done quite well under waterfall since the goals, constraints, scope and the requirements that come out of the interplay between the three will not change (or will not be allowed to change.) Use that same methodology in an environment where any of those elements change and mayhem results. In such a situation any of the agile methodologies works well, but costs more since they tend to need more resources who must be onsite and cannot be offshored to save funds. Conversely using an agile team to build an application that isn’t going to be subject to extreme time pressure or change is a waste of money.
The art of management comes in by knowing how to meet the needs of the company using the resources at hand. Take for example a company with an agile culture and staff whose new management has worked on large scale, waterfall projects. The management comes in promising the shareholders to save money from offshoring and outsourcing by retooling the culture and staff to the “new” waterfall methodology. It spends a year designing and debating a new “SDLC” and rolls it out with great fanfare to the developers, analysts, project managers, and testers. The management specifies a cutoff day by which all development must be done following the new waterfall SDLC.
There’s just one slight problem: The nature of the work inside the company lends itself to agile methodology. All the existing systems, applications and databases have been built to service tight deadlines, many of which are constrained by service level agreements (SLA)s to third parties that cost the firm money when they are down. In this environment requirements change quickly since the complexity in such a system is so great that they often cannot be known until the application – or a part of it – has been designed and tested. RUP and other agile methodologies take this fact into account by the use of design iterations in which requirements are gathered, designs based on them are developed and tested to meet the goals of the project.
Waterfall cannot handle this situation. Because it cannot rely on design iteration it forces the attempt to capture all possible requirements beforehand. To capture all possible requirements, the business analyst must venture down every possible path, envision every possible scenario that the software must account for, and interview every resource that knows anything about the system or a component. To do this takes time and results in a requirements document set that is so large it threatens the position of Ambien as the country’s leading sleep inducer, and is so thick it could stop bullets.
Because the requirements set is so large, the architect’s design is bloated, which inevitably leads to bloated code. Bloated code wastes the time of developers as they invariably chase solutions to problems that may not even arise in the usage of the application. Then there’s all the testing that bloated software requires – not to mention what results if by this stage the UAT team discovers that the application itself doesn’t work.
This happened to a long-ago employer who after 5 years and $15 million pulled the plug on a project. I was brought in to do the post-mortem and help determine if anything could be salvaged from the effort. I determined that the business conditions, scope, requirements and technology had changed so much over the 5 year period that there was really nothing to reuse.
The developers had valiantly attempted to build a project with shifting scope and requirements that changed by the week. There were network drives filled with folders stuffed with documents that had version numbers in the double digits. Continuity was lost as teams were fired, methodologies were tried and discarded, consultancies were brought in to salvage the project and fired, and project managers quit in disgust. Through all that the developers had produced a product that actually worked and covered 80% of the core business functions.
Why had the project failed? Because management – or rather the CIO - refused to listen to the project managers and allowed schedules to slip, scope to vary, and functionality to be added. When work stopped one Friday and the developers were shown the door that morning, the board of directors announced that the firm would right-off the $15 million.
The CIO did not lose his job amazingly enough. A $15 million failure under his belt, and he was still employed – IS to this day as far as I know. The firm started from scratch on another, smaller project that completed quickly and has been a success.
There is no single way to design good software, just like there’s no good way to treat people’s ailments. Medicine is as much an art today as it was in the days of Galen. Sure the drugs are better and there’s much less fussing about with goat entrails, but the bottom line is that good doctors know their patients – and their treatment options – well. Good management knows while a waterfall methodology worked well for one project, it would be disastrous on another in the same way that a good family doctor knows that chemotherapy may save a cancer patient’s life but be completely wasted on someone having a heart attack.
Methodologies are simply tools, and IT managers need to learn that there is no such thing as an “agile shop” or a “waterfall shop”. The IT needs of firms are much broader than a single tool. Doctors don’t solely prescribe aspirin, nor do they treat every ailment with cholesterol lowering medication. IT managers need to learn that they need to prescribe the correct solution for a problem – or better yet hire and rely upon the people who know this.
