In this chapter we will review the history of Agile, where it came from and discuss its key principles. We will get to the core of what it means to produce software in an Agile way and most importantly we will focus on where the lessons of this book fit in.
A significant portion of software businesses are organized to provide a useful application to their customers for a regular fee, typically delivered through the browser or mobile web. These companies, in general, began by solving a specific problem for their target users and over time added new features and improved their product.
A key concept to understand is that, often, when these software products are first conceived, the entire scope of their solution is not known. As the solution is used by customers and tested by the market, new requirements emerge that the designers may never have anticipated. These features can then be added on to the existing application, driven by the organic emergence of customer needs.
We aren't used to "building" things this way, because typically the more we build, the less the per-unit cost is. Consider a house: it needs a big hole in the ground, a foundation, plumbing, wiring, framing, a roof, and various finishing treatments. Once you get the work crew and the backhoe and the cement trucks out to the site, the difference in a foundation that is 1000 square feet and one that is 1500 will not be that much. But to add a 500 square foot foundation to an existing house would be really expensive.
Thus when we build a house we often consider our future needs based on some reasonable assumptions. We might assume that we will have 2 kids and get another car, so we plan to build a 3 bedroom house with a 2 car garage. This seems reasonable, but as we all know, life is not always so predictable.
What if we could grow our house? If it was cost effective, it would make perfect sense. Start out with a single bedroom house with a one car garage. Then we could add bedroom for the first child. Suddenly flying cars are invented! We can then add a landing pad on the roof. Oh my! We just had triplets, let's add 3 more bedrooms. It sounds crazy, but if it were cost effective, this would be the best way to "grow" a house, to adapt to changing needs.
With software we can do this. We can start very simple and only do the minimum needed to solve the user's problem. Then we can add features driven by market need, always ready to adapt. In short we become very "agile" in our ability to respond to changing needs and requirements. The trouble starts when we build too much, too soon based on a lot of assumptions.
When you hear the word "Agile" with regards to a software development team or project, it refers to a specific set of formalized concepts that came about as an answer for the need to adapt to changing requirements. In essence, Agile is about an incremental approach to building, such that a team is in a position to change direction quickly.
Agile development as we know it today was synthesized in 2001 with the publication of the "Manifesto for Agile Software Development" as a collaborative effort between 17 influential software developers. This document continues to serve as the most succinct and authoritative source on what it means to practice agile software development.
Why did these developers write this manifesto? To what were they reacting?
During the latter half of the twentieth century, software development grew significantly from its military and academic roots into the major force we see in the economy today. At first, the same management techniques were applied to software projects as would have been applied to other industrial endeavors. Specifically, these techniques involved a process that favored up-front design, a detailed plan and schedule, and an assembly line production process.
For software this became known as waterfall due to the way work would flow from each phase of development to the next. For example, design specifications would flow to software engineers to build and then to a quality assurance team to test. It's important to recognize that just like water can't flow back up a waterfall, it was difficult for feedback to flow back up the process. Once the spec was created it was set in stone, and by the time the product was testing, timelines usually dictated that changes could not be made, even if they were ultimately beneficial to the end user. In order to formalize a way to improve the old ways of doing things, Agile was born.
The values of agile software development are contained in the Manifesto for Agile Software Development, which is composed of 4 core principles. We will list and discuss each of these below:
Individuals and interactions over processes and tools
While processes and tools are often used to make collaboration more efficient, the focus must be on the interaction, rather than the tool. We can get bogged down by formalized process, for example, when perhaps a quick conversation between a developer and a tester is all that is required. A huge industry of tools and workflow processes has grown up around supporting agile development, but understand that these are still secondary. A highly functioning Agile team can be more effective with a white-board and post-it notes than a dysfunctional team can be with the most expensive project management software.
Working software over comprehensive documentation
The core of agile development is incremental delivery through development iterations. Specifically, small portions of a total deliverable will be completed, end-to-end, with customer or user feedback folded in to the next iteration. A focus on the "most important" features first requires a minimum of up front planning, with future features remaining flexible to change. Working software can self document through both its design (such as API specifications) or through its associated acceptance or system tests. This reduces the need for documentation that must be updated in parallel to the software.
Customer collaboration over contract negotiation
We want to constantly encourage deep interactions in the development process. A contract is instantly an adversarial stance, creating separation between the customer and the developer (this can be an internal customer, end user, or external client). Agile teams involve the customer in regular feedback cycles in-sync with development so as to reduce the need for rework and avoid going down the wrong path. This can take many forms including direct engagement with the final end users of software such as with usability tests, regular discussion with a "product owner" who serves as a representative of the end users and other stakeholders, or client review meetings where the working software is demonstrated.
Responding to change over following a plan
Continuous feedback and iterative development are central to the agile approach. The goal is to build in a structured way to embrace change, as the feedback is essential to quickly hone in on what the customer really wants. It is the old waterfall style approach that starts with following a plan. If at the end of the day, the original plan resulted in a bad solution, it may be too late to change the product. Many processes that support agile development build in buffer room to accomodate changing requirements.
In addition to the core values listed above, there are also a further 12 principles of Agile Development that further expand the manifesto. They are all worth reading and understanding but one of the principles is important enough to warrant further discussion and that is number seven, the idea that "Working software is the primary measure of progress." It is so important because adhering to this principle helps unify the efforts of multiple supporting teams and creates a measuring stick for assessing the value of work tasks.
Let's consider an example with our fictional company, PunchStarter, and discuss a more traditional approach and then the agile approach to developing a feature set.
PunchStarter wants to give inventors the ability to create a rich mini website in order to promote their fundraising campaign. Up until now, they have only allowed a title and description. The traditional, or waterfall, approach to this feature set might follow these steps:
This seems reasonable but notice that in step one, a whole bunch of features were scoped. In fact there may be a lot of assumptions as to what is important. Maybe marketing suggested a rich editor because of one customer request for image galleries. Maybe the scope tried to creep with each stakeholder adding more features that could be built "because we are in there anyway." In step three, development may consider all of the features together, first creating all of the back end components, updating the database once, moving on to the front end. Finally, not until the last step do we have working software, with a chance for the users to try it out, and offer feedback.
Now consider a more agile approach with an eye toward iterative delivery of working software:
With the Agile approach, a subset of functionality is delivered, to production, and in total, before moving on the next feature. This is valuable because it shortens the time between idea and validation. What if users love the feature and swamp the website with image uploads? Well that is a good thing to learn before they might do a similar thing with even larger videos; the team would have a chance to upgrade their web servers. What if no one ever asked for a power user markup language? Well since it was never implemented no development time is wasted.
The common fear is that doing things "one-at-a-time" will lead to disjointed, piecemeal development. Agile does not discourage thinking holistically about an application design in light of what might be developed, rather it advocates delivery in working increments so as to remain flexible to constant feedback.
The Pareto Law in the context of software development suggests that 80% of users will only use 20% of features, and so why not develop those 20% of features first? (Source: http://www.mnl.com/ourideas/opensource/8020_rule_in_software_developm_1.php). Agile methodologies help reduce the risk of delivering an application bloated with features that hardly anybody uses.
It is therefore essential, when striving for an agile process in software development, if you ever are mired in discussion and debate, ask the question, "how does this quickly get us to working software?" This can help steer the focus of a team back on course.
If our goal is to continuously deliver working, valuable software, then how do we get in a position to do that? How can we take a great idea for the next social network, CRM software, or photo sharing site, and deliver it in useful chunks, piece by piece to our user base, all the while being ready to change directions?
Well the answer of course is to start with a plan. And that is the part of the Agile process on which the remainder of this book will focus. At a high level what we must do is strategically break down our big idea into bite-sized chunks of functionality, where each of these chunks has some value to our customer or user. When we have a large block of these chunks then we can begin the building process!
It may sound a little like up-front design now, but trust us, it's different, and along the way we will show you how taking this approach to planning will maintain flexibility, letting you respond to changes in needs.