How can Agile kill a project

I'll continue to talk about software engineering from previous post (How can Waterfall kill a project). People can learn from the failure of many waterfall projects in the wrong way. Let's see how agile can go wrong in the same situation.

A company has come up with this great idea to build car, boat, and truck (assuming they never exist before) to get to the emerging market. Being an innovative company, they really want to push the technology boundary.

Learning from the failure of the waterfall process, they decided to adopt agile methodology and build cars first without worrying about boats and trucks. With this simplified requirement, the project went smoothly. They delivered cars to the market on time within budget. Everyone is happy.

When the market starts to heat up and competition pour their products into the same market, manufacturing only cars is no longer profitable. So the company decide to continue their adventure by building the boats. With the success in delivering cars, one would reasonably think building boats should be much easier until one developer points out that they require about the same amount of the budget for building cars. Product manager asks why and the developer provides this answer: "Developing the engine is the part that consumed majority of the resource. The engine built for the cars is embedded into the car frame and cannot be reused for the boats since the car frame is not suitable for building the boats. To develop the boats, we'd have to develop the engine again." The company did their calculation. If they invest in building the boats with about the same cost, by the time their boats hit the market competition may also deliver similar thing. Since manufacturing cars is no longer profitable, manufacturing boats with the same cost may suffer the same fate. So the decision is made not to build boats less to say trucks. Since manufacturing cars is no longer profitable, the manufacturing line is closed. Everyone is sad.

In retrospect, what could have been done differently? Why didn't they design an engine that works for cars, boats, and trucks? Wait a minute! Are we going back to the failed waterfall process?

I'd learn the lesson differently. Agile tends to give developers tunnel vision and leads to under-engineering of the product. Taking away the design step in the engineering process only makes this situation worse. Agile or not, you still should follow the same design/engineering principle: separation of concerns. Had the engine been designed in such a way that it is not concerned about how it is mounted (in this case embedded in the car frame), it could have been reused for the boats even though it may not be capable of driving a truck yet.

How can Waterfall kill a project?

Waterfall engineering process has been used in many industries with great success. We've used it to build airplanes, railways, buildings, and many other things. But in software industry "Agile" has become THE software engineering process everyone is following. What is wrong with waterfall?

I often use this simple story to explain my understanding of it.

 A company has come up with this great idea to build car, boat, and truck (assuming they never exist before) to get to the emerging market. Being an innovative company, they really want to push the technology boundary.

The project is following a waterfall model. They started by doing a good design of the things they want to build. During this process, they realize that they need a piece named "engine" that can be reusable for all three products. They start to design this perfect engine. Nobody has ever built something like this before. Invent an "engine" is hard and invent an "engine" that can fit into all three products is harder than that. A lot of resource is put into the effort. The resource required is so immense that the project runs out of budget before they can deliver a single car.

In retrospect, what could have been done differently? Why didn't they design an engine that only works for cars and deliver some cars before worrying about boats and trucks? One would reasonably believe designing a simplified "engine" just for cars should not have consumed so much resource. In terms of software engineering, the lesson learnt is waterfall tends to over-engineer a product or a part of it. The effort put into this over-engineering effort may cause delay for accessing the market.

Some people learn the lesson differently. They think the "design" piece in waterfall is to be blamed for the failure. Why do you spend so much resource on generating these useless design documents before writing code? Coding is design after all!

 I believe lessons learnt this way is part of the reason that "agile" is born. In the name of "reducing documentation overhead", the design step is removed from the engineering process so that no design documents are generated. But I don't agree with the way this lesson is learnt. Can Agile kill a project? That'll be my next post.

Agile and Design Evoluation

Agile is right in at least one way in that no matter how hard you try, your requirements do change over time. There are good reasons for requirements to change. Market condition changes, technology advances, and product scope changes are all good reasons for it.

Let's admit that requirements are changing and you are always shooting at a moving target. To handle requirements effectively, you need a good strategy to evolve your design to deal with the changes.

In agile practice, developers are often given a limited view of the product to start their implementation. The best design they can come up with will unavoidably only cover the variations within that scope. You may blame the initial designers for lack of abstraction in their design to handle future variations. But in reality there is nothing wrong with it since they are limited by the knowledge about future requirements. It is hard, if ever possible, to determine how abstract their design should be to cover possible future variations without knowing what will come up. A more practical solution is to evolve your design from concrete to more abstract.

One way of handling requirement changes is to constantly revisit your conceptual models in your design in every iteration of your agile process. Before you add more if-else statements to your functions, more enum values to your switch statements, or boolean flags to your data structures, you should really think about your conceptual models and your state machines. Are they still valid given the new requirements?

If-else statements are really behaviour changes. Theoretically, behaviour changes are supposed to be handled by polymorphism in good OO design. The need for more if-else statements is a good indication that you need more derivations to handle behaviour differences. More of your base classes should become abstract (pure virtual in C++). If-else statements may bandage your obsolete design but will lead to more grief for the future.

More enum values or boolean flags are really amendments to your existing state machines, exponentially unfortunately. As state space of your state machines expands so dramatically, it is unlikely your last state diagram will hold. Revisit these state diagrams to see how it should really be under the new requirements.

Given a set of requirements, come up with a good OO design is already a challenging task in most cases. Shooting a moving target only makes it harder. Without a good strategy to deal with requirement changes, it is unlikely your software project will end up with great success, at least from software design's perspective.

Agile: What it should be and what it should NOT be

Agile as a software engineering pratice is gaining popularity. However, it doesn't always meet people's expectation. Like everything else, it comes with Pros and Cons.

When it all comes down to it, what Agile is trying to against is so-called "overhead" in "traditinoal waterfall" software engineering approach. We may want to take a closer look at the "overhead". In traditional waterfall approach, we may have documentations for requirements and analysis, design at different levels, and code. What is considered "overhead" is the time spent to create and maintain these documents. This is clearly marked by the advocates for "face-to-face" communication in Agile. However, there exists certain documentations that are required throughout your project that worths the effort to maintain. An obvious example is architectural design and component design guidelines. Theoretically, the information in these documents can be communicated by "face-to-face" communication but in reality that is not the efficient way to communicate them in entirety. How can you talk about your architectural design precisely the same for many times? Development teams are often very dynamic. If you want your team to have a consistent understading of the architectural design, documentation is more than likely a more efficient way of communcation than face-to-face. Without architectural design documentation and component design guidelines, the pressure exercised by Agile approach often lead to broken architectural integrity and sometimes broken encapsulations at lower level. When the skill level of the developers differ, the problem can eaily spread through the enire code base. The consequence is not obvious in current and near future itertions. However, as they accumulate, your code base becomes harder and harder to maintain. The cost to add new features will increase. For Agile to be efficient, it should be guided by architectural design and component design guidelines.

The good side of Agile is to advocate "Test-Driven-Development". This is what it should be. If a project is started with a good architecture design and good test coverage, it is unlikely for broken architectural integrity to happen. Component design guidelines may be reflected in the design and implementation of test suites.

Like anything else, Agile may be abused as well. It is just like a knife. You may cut your fruit or your finger depending on how you exercise it.

Agile should NOT be used as an excuse for lack of long term vision of product management. Agile advocates "embracing changes". However, it does not warrant product management the freedom to change their mind at free will often as a result of the lack of vision. In software development, changes are costly regardless what software engineering approach you take. You waste the time to put the code in and the time to remove unused code less to say the test code that comes with it.

Agile should NOT be used against the pursuit of a relatively stable architecture. Among all the costly changes, architectural change is probably the most costly. Without architectural design and component design guidelines, your code base is unlikely to be designed for changes since developers are rushing to finish current sprint without any consideration for things outside current sprint. Without architectural design and component design guidelines, you do not have the right tool to really embrace changes. That is exactly against what Agile has advocated in the first place.

For Agile to be sucessful, you really need to be agile. Learn from the lessons in your pratice and make changes as mistakes or negative effects are observed. That is the real spirit of the word "agile". Religious belief is just the opposite of it.

More Design Patterns?

Since the GoF, there are quite a few more publishments on Design Patterns. However, none of them can be as influncial as the GoF. Don't we need more design patterns?

In my opinion, the GoF revealed some of the fundamental principles/criteria for a good OO design. The key is how to distribute information/data and responsibility/code among different classes/modules in such a way that
1. foreseeable changes are made easy and
2. conceptual behaviour and actual implementation are decoupled.

Design patterns are examples where these principles are applied properly. Once you understand these principles and are able to apply them to real world design issues, more examples/patterns will certainly help you to enhance the understanding of these principles but the impact is becoming marginal. That is probably why none of the later publishments can be as influncial.

Agile without Architecture?

Being Agile is good but without a collective architectural view of the product, it may easily become cowboy development where each developer just does what he/she thinks is right. When that happens, architectural integrity can be easily lost. In this sense, the function of an architect is essential to the success of Agile development. This function may be performed by an individual or by a collective effort.