Agile software development

Template:Software-development-process Agile software development is a conceptual framework for software engineering that promotes development iterations throughout the life-cycle of the project.

There are many agile development methods; most minimize risk by developing software in short amounts of time. Software developed during one unit of time is referred to as an iteration, which may last from one to four weeks. Each iteration is an entire software project: including planning, requirements analysis, design, coding, testing, and documentation. An iteration may not add enough functionality to warrant releasing the product to market but the goal is to have an available release (without bugs) at the end of each iteration. At the end of each iteration, the team re-evaluates project priorities.

Agile methods emphasize face-to-face communication over written documents. Most agile teams are located in a single open office sometimes referred to as a bullpen. At a minimum, this includes programmers and their "customers" (customers define the product; they may be product managers, business analysts, or the clients). The office may include testers, interaction designers, technical writers, and managers.

Agile methods also emphasize working software as the primary measure of progress. Combined with the preference for face-to-face communication, agile methods produce very little written documentation relative to other methods. This has resulted in criticism of agile methods as being undisciplined.

History

The modern definition of agile software development evolved in the mid 1990s as part of a reaction against "heavyweight" methods, as typified by a heavily regulated, regimented, micro-managed use of the waterfall model of development. The processes originating from this use of the waterfall model were seen as bureaucratic, slow, demeaning, and inconsistent with the ways that software engineers actually perform effective work. A case can be made that agile and iterative development methods are a return to development practice seen early in the history of software development. Initially, agile methods were called "lightweight methods." In 2001, prominent members of the community met at Snowbird, Utah, and adopted the name "agile methods." Later, some of these people formed The Agile Alliance[1], a non-profit organization that promotes agile development.

Early agile methods—created prior to 2000—include Scrum (1986), Crystal Clear, Extreme Programming (1996), Adaptive Software Development, Feature Driven Development, and DSDM (1995).

While it was not the first agile method, Extreme Programming (usually abbreviated as "XP") established the popularity of agile methods. Extreme Programming was created by Kent Beck in 1996 as a way to rescue the struggling Chrysler Comprehensive Compensation (C3) project. While that project was eventually canceled, the methodology was refined by Ron Jeffries' full-time XP coaching, public discussion on Ward Cunningham's Portland Pattern Repository wiki and further work by Beck, including a book in 1999. Elements of Extreme Programming appear to be based on Scrum and Ward Cunningham's Episodes pattern language.

Principles behind agile methods — The Agile Manifesto

See also: Agile Manifesto

Agile methods are a family of development processes, not a single approach to software development. In 2001, 17 prominent figures in the field of agile development (then called "light-weight methodologies") came together at the Snowbird ski resort in Utah to discuss ways of creating software in a lighter, faster, more people-centric way. They created the Agile Manifesto, widely regarded as the canonical definition of agile development, and accompanying agile principles.

Some of the principles behind the Agile Manifesto are:

• Customer satisfaction by rapid, continuous delivery of useful software
• Working software is delivered frequently (weeks rather than months)
• Working software is the principal measure of progress
• Even late changes in requirements are welcomed
• Close, daily, cooperation between business people and developers
• Face-to-face conversation is the best form of communication
• Projects are built around motivated individuals, who should be trusted
• Continuous attention to technical excellence and good design
• Simplicity
• Self-organizing teams
• Regular adaptation to changing circumstances

The publishing of the manifesto spawned a movement in the software industry known as agile software development.

In 2005, Alistair Cockburn and Jim Highsmith gathered another group of people — management experts, this time — and wrote an addendum, known as the PM Declaration of Interdependence.

Comparison with other methods

Agile methods are sometimes characterized as being at the opposite end of the spectrum from "plan-driven" or "disciplined" methodologies. This distinction is misleading, as it implies that agile methods are "unplanned" or "undisciplined". A more accurate distinction is to say that methods exist on a continuum from "adaptive" to "predictive". Agile methods exist on the "adaptive" side of this continuum.

Adaptive methods focus on adapting quickly to changing realities. When the needs of a project change, an adaptive team changes as well. An adaptive team will have difficulty describing exactly what will happen in the future. The further away a date is, the more vague an adaptive method will be about what will happen on that date. An adaptive team can report exactly what tasks are being done next week, but only which features are planned for next month. When asked about a release six months from now, an adaptive team may only be able to report the mission statement for the release, or a statement of expected value vs. cost.

Predictive methods, in contrast, focus on planning the future in detail. A predictive team can report exactly what features and tasks are planned for the entire length of the development process. Predictive teams have difficulty changing direction. The plan is typically optimized for the original destination and changing direction can cause completed work to be thrown away and done over differently. Predictive teams will often institute a change control board to ensure that only the most valuable changes are considered.

Agile methods have much in common with the "Rapid Application Development" techniques from the 1980's as espoused by James Martin and others (see RAD).

Contrasted with other iterative development methods

Most agile methods share other iterative and incremental development methods' emphasis on building releasable software in short time periods. Agile development differs from other methods in that its time periods are measured in weeks rather than months and work is performed in a highly collaborative manner. Most agile methods also differ by treating their time period as a strict timebox.

Contrasted with the waterfall model

Agile development does not have much in common with the waterfall model. As of 2004, the waterfall model is still in common use. The waterfall model is the most predictive of the methodologies, stepping through requirements capture, analysis, design, coding, and testing in a strict, pre-planned sequence. Progress is generally measured in terms of deliverable artifacts—requirement specifications, design documents, test plans, code reviews and the like.

The main problem of the waterfall model is the inflexible nature of the division of a project into separate stages, so that commitments are made early on, and it is difficult to react to changes in requirements. Iterations are expensive. This means that the waterfall model is likely to be unsuitable if requirements are not well understood or are likely to change radically in the course of the project.

Agile methods, in contrast, produce completely developed and tested features (but a very small subset of the whole) every few weeks or months. The emphasis is on obtaining the smallest workable piece of functionality to deliver business value early, and continually improving it/adding further functionality throughout the life of the project.

Some agile teams use the waterfall model on a small scale, repeating the entire waterfall cycle in every iteration. Other teams, most notably Extreme Programming teams, work on activities simultaneously.

Contrasted with "cowboy coding"

Cowboy coding is the absence of a defined method: team members do whatever they feel is right. Agile development's frequent re-evaluation of plans, emphasis on face-to-face communication, and relatively sparse use of documents sometimes causes people to confuse it with cowboy coding. Agile teams, however, do follow defined (and often very disciplined and rigorous) processes.

As with all methodologies, the skill and experience of the users define the degree of success and/or abuse of such activity. The more rigid controls systematically embedded within a process offer stronger levels of accountability of the users. The degradation of well-intended procedures can lead to activities often categorized as cowboy coding.

Suitability of agile methods

Although agile methods differ in their practices, they share a number of common characteristics, including iterative development, and a focus on interaction, communication, and the reduction of resource-intensive intermediate artifacts. The suitability of agile methods in general can be examined from multiple perspectives. From a product perspective, agile methods are more suitable when requirements are emergent and rapidly changing; they are less suitable for systems that have high criticality, reliability and safety requirements, although there is no complete consensus on this point. From an organizational perspective, the suitability can be assessed by examining three key dimensions of an organization: culture, people, and communication. In relation to these areas a number of key success factors have been identified (Cohen et al., 2004):

• The culture of the organization must be supportive of negotiation
• People must be trusted
• Fewer but more competent people
• Organizations must live with the decisions developers make
• Organizations need to have an environment that facilitates rapid communication between team members

The most important factor is probably project size. As size grows, face-to-face communication becomes more difficult. Therefore, most agile methods are more suitable for projects with small teams, with fewer than 20 to 40 people. Large scale agile software development remains an active research area.

Another serious problem is that initial assumptions or overly rapid requirements gathering up front may result in a large drift from an optimal solution, especially if the client defining the target product has poorly formed ideas of their needs. Similarly, given the nature of human behaviour, it's easy for a single "dominant" developer to influence or even pull the design of the target in a direction not necessarily appropriate for the project. Historically, the developers can, and often do, impose solutions on a client then convince the client of the appropriateness of the solution, only to find at the end that the solution is actually unworkable. In theory, the rapidly iterative nature should limit this, but it assumes that there's a negative feedback, or even appropriate feedback. If not, the error could be magnified rapidly.

This can be alleviated by separating the requirements gathering into a separate phase (a common element of Agile systems), thus insulating it from the developer's influence, or by keeping the client in the loop during development by having them continuously trying each release. The problem there is that in the real world, most clients are unwilling to invest this much time. It also makes QAing a product difficult since there are no clear test goals that don't change from release to release.

In order to determine the suitability of agile methods individually, a more sophisticated analysis is required. The DSDM method, for example, provides a so-called ‘suitability-filter’ for this purpose. Also, the Crystal family of methods provides criteria on how to select the method for a given project. The selection is based on project size, criticality and priority. However, other agile methods do not provide such explicit instruments to assess their suitability for a project.

Some agile methods, like DSDM and Feature Driven Development (FDD), are claimed to be suitable for any agile software development project, regardless of situational characteristics (Abrahamsonn et al., 2003).

A comparison of agile methods will reveal that they support different phases of a software development life-cycle to varying degrees. This individual characteristic of agile methods can be used as a selection criterion for selecting candidate agile methods.

Agile development has been widely documented (see Experience Reports, below, as well as Beck pg. 157, and Boehm and Turner pg. 55-57) as working well for small (<10 developers) co-located teams. Agile development is expected to be particularly suitable for teams facing unpredictable or rapidly changing requirements.

Agile development's applicability to the following scenarios is open to question:

• Large scale development efforts (>20 developers), though scaling strategies and evidence to the contrary have been described.
• Distributed development efforts (non-co-located teams). Strategies have been described in Bridging the Distanceand Using an Agile Software Process with Offshore Development
• Mission- and life-critical efforts
• Command-and-control company cultures

It is worth noting that several large scale project successes have been documented by organisations such as BT which have had several hundred developers situated in the UK, Ireland and India, working collaboratively on projects and using Agile methodologies. While questions undoubtedly still arise about the suitability of some Agile methods to certain project types, it would appear that scale or geography, by themselves, are not necessarily barriers to success.

Barry Boehm and Richard Turner suggest that risk analysis be used to choose between adaptive ("agile") and predictive ("plan-driven") methods. The authors suggest that each side of the continuum has its own home ground:

Agile home ground:

• Low criticality
• Senior developers
• Requirements change very often
• Small number of developers
• Culture that thrives on chaos

Plan-driven home ground:

• High criticality
• Junior developers
• Requirements don't change too often
• Large number of developers
• Culture that demands order

Agile methods and method tailoring

In the literature, different terms refer to the notion of method adaptation, including ‘method tailoring’, ‘method fragment adaptation’ and ‘situational method engineering’. Method tailoring is defined as:

A process or capability in which human agents through responsive changes in, and dynamic interplays between contexts, intentions , and method fragments determine a system development approach for a specific project situation.

Potentially, almost all agile methods are suitable for method tailoring. Even the DSDM method is being used for this purpose and has been successfully tailored in a CMM context. Situation-appropriateness can be considered as a distinguishing characteristic between agile methods and traditional software development methods, with the latter being relatively much more rigid and prescriptive. The practical implication is that agile methods allow project teams to adapt working practices according to the needs of individual projects. Practices are concrete activities and products which are part of a method framework. At a more extreme level, the philosophy behind the method, consisting of a number of principles, could be adapted (Aydin, 2004).

In the case of XP the need for method adaptation is made explicit. One of the fundamental ideas of XP is that there is no process that fits every project as such, but rather practices should be tailored to the needs of individual projects. There are also no experience reports in which all the XP practices have been adopted. Instead, a partial adoption of XP practices, as suggested by Beck, has been reported on several occasions.

A distinction can be made between static method adaptation and dynamic method adaptation. The key assumption behind static method adaptation is that the project context is given at the start of a project and remains fixed during project execution. The result is a static definition of the project context. Given such a definition, route maps can be used in order to determine which structured method fragments should be used for that particular project, based on predefined sets of criteria. Dynamic method adaptation, in contrast, assumes that projects are situated in an emergent context. An emergent context implies that a project has to deal with emergent factors that affect relevant conditions but are not predictable. This also means that a project context is not fixed, but changing during project execution. In such a case prescriptive route maps are not appropriate. The practical implication of dynamic method adaptation is that project managers often have to modify structured fragments or even innovate new fragments, during the execution of a project (Aydin et al, 2005).

Agile methods and project management

Agile methods differ to a large degree in the way they cover project management. Some methods are supplemented with guidelines on project management, but there is generally no comprehensive support.

PRINCE2 has been suggested as a suitable, complementary project management system.

Agile methods

Some of the well-known agile software development methods:

• Extreme Programming (XP)
• Scrum
• Agile Modeling
• Adaptive Software Development (ASD)
• Crystal Clear and Other Crystal Methodologies
• Dynamic Systems Development Method (DSDM)
• Feature Driven Development (FDD)
• Lean software development
• Agile Unified Process (AUP)
• Essential Unified Process (EssUP)

Other approaches:

• Software Development Rhythms
• Agile Documentation
• ICONIX Process
• Microsoft Solutions Framework (MSF)
• Agile Data Method
• Database refactoring

Agile beyond software development

Agile software development depends on some special characteristics possessed only by software, such as object technologies and the ability to automate testing. However, related techniques have been created for developing non-software products, such as semiconductors, motor vehicles, or chemicals. For more on them, see Flexible product development.

Measuring agility

While agility is seen by many as a means to an end, a number of approaches have been proposed to quantify agility. Agility Index Measurements (AIM)[2] score projects against a number of agility factors to achieve a total. The similarly-named Agility Measurement Index [3], scores developments against five dimensions of a software project (duration, risk, novelty, effort, and interaction). Other techniques are based on measurable goals [4]. Another study using fuzzy mathematics has suggested that project velocity can be used as a metric of agility.

While such approaches have been proposed to measure agility, the practical application of such metrics has yet to be seen.

Criticism

Agile development is sometimes criticized as cowboy coding. Extreme Programming's initial buzz and controversial tenets, such as pair programming and continuous design, have attracted particular criticism, such as McBreen and Boehm and Turner. However, much of the criticism is believed by Agile practitioners as misunderstandings about agile development.

In particular, Extreme Programming is reviewed and critiqued by Matt Stephens's and Doug Rosenberg's Extreme Programming Refactored.

Criticisms include:

• lack of structure and necessary documentation
• only works with senior-level developers
• incorporates insufficient software design
• requires too much cultural change to adopt
• can lead to more difficult contractual negotiations

The criticisms regarding insufficient software design and lack of documentation are addressed by the Agile Modeling method which can easily be tailored into agile processes such as XP.

Agile software development has been criticized because it will not bring about the claimed benefits when programmers of average ability use this methodology, and most development teams are indeed likely to be made up of people with average (or below) skills.

Post-Agilism

In software engineering, post-Agilism is an informal movement of former "Agilistas" (Agile Software Development evangelists) who have chosen to draw from a much wider range of methods and schools of thought on software development, preferring to avoid being constrained by what they consider to be "Agile Dogma". The term Fragilism is sometimes used to mean the same thing.

Much of the debate around Post-Agilism centres around the meaning of the word Agile - with a capital 'a' - vs. "agile" (the dictionary definition of the word). In late 2005, Jason Gorman argued that the meaning of Agile was ambiguous and was being inappropriately applied to a very wide range of approaches like Six Sigma and CMMi. He also argued that "Agile", "evolutionary", and "lean" (as in Lean software development) did not mean the same thing in practice, even though they are all lumped under the banner of "Agile" - possibly for marketing purposes. Gorman argued that process-oriented methods, especially methods that incrementally reduce waste and process variation like Six Sigma, have a tendency to limit an organisation's adaptive capacity (their "slack"), making them less able to respond to discontinuous change - i.e., less agile. He also argues in later posts that "agile", "lean" and "evolutionary" are strategies that need to be properly understood and appropriately applied to any specific context. That is, there is a time to be "agile", a time to be "lean" and a time to be "evolutionary".

The debate continued on various discussion groups, and transferred into the blogosphere in December 2005. In June 2006 the debate widened and the term Post-Agilism was coined by Jonathan Kohl to describe the growing - but still very loose - association of people extolling "post-Agile" sentiments in their work.

Much of the post-Agile thinking centers around Nonlinear Management, a superset of management techniques that include many Agile practices.

Experience reports

Agile development has been the subject of several conferences. Some of these conferences have had academic backing and included peer-reviewed papers, including a peer-reviewed experience report track. The experience reports share industry experiences with agile software development.

As of 2006, experience reports have been or will be presented at the following conferences:

• XP (2000, 2001, 2002, 2003, 2004, 2005, 2006)
• XP Universe (2001)
• XP/Agile Universe (2002, 2003, 2004)
• Agile Development Conference (2003, 2004) (peer-reviewed; proceedings published by IEEE?)
• Agile (2005, 2006) (peer-reviewed; proceedings published by IEEE)

See also

• Software Engineering
• Extreme programming
• Collaborative software development model
• Software Development Rhythms

Further reading

• Fowler, Martin. Is Design Dead?. Appeared in Extreme Programming Explained, G. Succi and M. Marchesi, ed., Addison-Wesley, Boston. 2001.
• Riehle, Dirk. A Comparison of the Value Systems of Adaptive Software Development and Extreme Programming: How Methodologies May Learn From Each Other. Appeared in Extreme Programming Explained, G. Succi and M. Marchesi, ed., Addison-Wesley, Boston. 2001.
• Tomek, Ivan. What I Learned Teaching XP
• M. Stephens, D. Rosenberg. Extreme Programming Refactored: The Case Against XP. Apress L.P., Berkeley, California. 2003. (ISBN 1-59059-096-1)
• D. Rosenberg, M. Stephens. Agile Development with ICONIX Process. Apress L.P., Berkeley, California. 2005. (ISBN 1-59059-464-9)
• Beck, et al., Manifesto for Agile Software Development
• Larman, Craig and Basili, Victor R. Iterative and Incremental Development:A Brief History IEEE Computer, June 2003

• Abrahamsson, P., Warsta, J., Siponen, M.T., & Ronkainen, J. (2003). New Directions on Agile Methods: A Comparative Analysis. Proceedings of ICSE'03, 244-254.
• Abrahamsson, P., Salo, O., Ronkainen, J., & Warsta, J. (2002). Agile Software Development Methods: Review and Analysis. VTT Publications 478.
• Aydin, M.N., Harmsen, F., Slooten, K. v., & Stagwee, R. A. (2004). An Agile Information Systems Development Method in use. Turk J Elec Engin, 12(2), 127-138
• Aydin, M.N., Harmsen, F., Slooten van K., & Stegwee, R.A. (2005). On the Adaptation of An Agile Information Systems Development Method. Journal of Database Management Special issue on Agile Analysis, Design, and Implementation, 16(4), 20-24
• Cohen, D., Lindvall, M., & Costa, P. (2004). An introduction to agile methods. In Advances in Computers (pp. 1-66). New York: Elsevier Science.
• Karlstrom, D., & Runeson P. (2005). Combining agile methods with stage-gate project management. IEEE Software, 22(3), 43-49
• Highsmith, J. Agile Software Development Ecosystems. Addison-Wesley Professional, 2002 (ISBN 0-20176-043-6)
• Waldner, JB. Nanocomputers and Swarm Intelligence. ISTE, 2008, (ISBN 9781847040022)

External links

• Manifesto for Agile Software Development
• The Agile Alliance
• Post-Agilism: Process Skepticism by Jonathan Kohl (This is where the term "post-Agilism" first appeared in the blogosphere.)

• Interview on Lean for Software Methods
• Agile Methods and Other Fairy Tales - One of the few Anti-Agile Papers (this is a link to a company)
• Why Agile Software Development Techniques Work: Improved Feedback
• Risk based selection for agile iterative lifecycle methods
• "The Demise of the Waterfall Model Is Imminent" and Other Urban Myths
• Agile, Multidisciplinary Teamwork by Gautam Gosh
• Agile Requirements by Rachel Davies
• Two Ways to Build a Pyramid by John Mayo-Smith
• Levent Gurses (2006-11-01). "10 Mistakes in Transitioning to Agile: Slow down the transition in order to go fast". Dr. Dobb's Journal. {{cite news}}: Check date values in: |date= (help)
• Breaking the Major Release Habit - Why are long iterations a hard habit to break?

• Agile Toolkit Podcast - Conversations and Interviews related to Agile Software Development

• Video - Why does Agile Software Development pay? - by OutSystems
• Agile Modeling: Effective Practices for XP and RUP by S.W. Ambler.

• TechBookReport An archive of book reviews devoted to a range of software methodologies, particularly Agile (including Scrum, XP and other agile processes).

• Template:Dmoz
• Agile Methodology Overview - OutSystems Agile Methodology Overview
• What's Wrong With Agile Methods by Tom Gilb