Agile Zone is brought to you in partnership with:

Kief is a software delivery consultant with ThoughtWorks in London, specializing in tools, practices, and processes for the continuous delivery of software. Kief is a DZone MVB and is not an employee of DZone and has posted 19 posts at DZone. You can read more from them at their website. View Full User Profile

Iterations Considered Harmful

12.14.2012
| 7367 views |
  • submit to reddit
The iteration is a cornerstone of agile development. It provides a heartbeat for the team and its stakeholders, and a structure for various routine activities that help keep development work aligned with what the customer needs. However, the way many teams run their iterations creates serious pitfalls which can keep them from delivering software as effectively as they could.

The orthodox approach to the iteration is to treat it as a timebox for delivering a batch of stories, which is the approach most Scrum teams take with sprints (the Scrum term for an iteration). In recent years many teams have scrapped this approach, either using iterations more as a checkpoint, as many ThoughtWorks teams do, or scrapping them entirely with Kanban and Lean software development.

For the purpose of this post, I will refer to these two general approaches to running iterations as the “orthodox” or “timeboxed-batch” iteration model on the one hand, and the “continuous development” model on the other hand. Although orthodox iterations have value, certainly over more old-school waterfall project management approaches, continuous development approaches which do away with timeboxing and avoid managing stories in batches allow teams to more effectively deliver higher quality software.

Orthodox iteration model: (or “timeboxed-batch” model). Each iteration works on a fixed batch of stories, all of which must be started and finished within a single iteration. Continuous development model: Stories are developed in a continuous flow, avoiding the need to stop development in order to consolidate a build containing only fully complete stories.
The anatomy of the orthodox iteration

In the classically run iteration or sprint, the Product Owner (PO) and team choose a set of stories that it commits to deliver at the end of the iteration. All of the stories in this batch should be sufficiently prepared before the iteration begins. The level and type of preparation varies between teams, but usually includes a level of analysis including the definition of acceptance criteria. This analysis should have been reviewed by the PO and developers to ensure there is a common understanding of the story. The PO should understand what they can expect to have when the story is implemented, and the technical team should have enough of an understanding of the story to estimate it and identify potential risks.

The iteration begins with an iteration kickoff meeting (IKO) where the team reviews the stories and confirms their confidence that they can deliver the stories within the iteration. The developers then choose stories to work on, discussing each story with the PO, Business Analyst (BA), and/or Quality Analyst (QA) as appropriate, then breaking it down into implementation tasks. Implementation takes place with continual reviews, or even pairing with these other non-developers, helping to keep implementation on track, and minimizing the amount of rework needed when the story is provisionally finished and goes into testing.

The QA and BA/PO then test and review each story as its implementation is completed. This is in addition to the automated testing which has been written and repeatedly running following TDD and CI practices. Only once the story is signed off do the developers move on to another one of the stories in the iteration’s committed batch.

As the end of the iteration approaches, developers and QAs should be wrapping up the last stories and preparing a releasable build for the showcase, which is typically held on the final day of the iteration. In the showcase, the team demonstrates the functionality of the completed stories to the PO and other stakeholders, the stories are signed off. The team holds a retrospective to consider how they can work better, then on the next working day they hold the IKO to start the following iteration.

When the iteration ends the team has a complete, fully tested and releasable build of the application, regardless of whether the software actually will go into production at this point.

The start and end dates of the iteration are firmly fixed. If there are stories (or defect fixes) which aren’t quite ready at the end of the iteration, the iteration end date is never slipped. Instead, the story is not counted as completed, so must be carried over to the next iteration.

The benefits of the orthodox iteration

This style of iteration offers many benefits over traditional waterfall methodologies. A short, rigid cycle for producing completely tested and releasable code forces discipline on the team, keeping the code in a near-releasable state throughout the project, and avoiding the temptation to leave work (e.g. testing) for “later”, building up unmanageable burdens of work, stress, and defects to be dealt with under the pressure of the final release phase.

The timeboxed iteration also forces the team to learn how to define stories of a manageable size. If stories are routinely too big to complete in one iteration this is a clear sign that the team needs improve the way it defines and prepares stories.

This demonstrates another benefit of the iteration, which is frequent feedback. The team is able to evaluate not only the quality of their code and its relevance to the business by getting feedback quickly, they are also able to evaluate how effectively they are working, and try out ideas for improving continually throughout the project.

Fundamental problems with the orthodox itertation

The timeboxed-batch approach to iterations has value, particularly for teams inexperienced with agile. However, it has fundamental problems. At core, this approach is waterfall written small, with many of the same flaws, albeit with a small enough cycle that issues can be dealt with more quickly than with a full waterfall project.

To understand why this is so, let’s flesh out the idealized anatomy of the iteration from above with some of the things which often happen in practice.

  • At the start of the iteration, no development is taking place, because everyone is working on preparing the new batch of stories. The BA’s are extremely busy now, because they have a full working set of stories to hand over to developers (i.e. however many stories the team can work on at once, that’s how many stories the BA’s must hand over all at the same time). The QA’s are less busy now, although they may be helping the BA’s out, and planning their testing for the iteration’s stories.
  • Actually, I lied. Development is taking place, and QA’s are extremely busy. Testing and bugfixing stories left over from the previous iteration is still going on. See points below to understand why. As a result, preparation and starting of the stories for this iteration is sluggish because of work carried over from the previous iteration. This actually isn’t necessarily bad, since it helps to stagger the story preparation work, preventing the BA’s from becoming a bottleneck. However, depending on whether carryover work was factored into the number of stories chosen for the current iteration (business pressures often make this difficult), this may mean the team is already at risk of failing to meet its commitment.
  • Once the previous iteration’s work has settled down, QA’s have little to do until the end of the iteration approaches, at which point they come under enormous pressure. Developers are humping to get their stories done in time, leaving QA’s with a pile of stories to be tested in time for the showcase. Any defects they find increase this pressure even more, with very little time to get the fixes in and then re-tested (maybe needing even more fixing and re-testing!)
  • If developers complete a story with a bit of time left in the iteration, they aren’t able to start new stories because the stories for the following iteration won’t be ready to work on until the IKO.
  • In the end, some stories don’t get fully tested during the iteration. They may be tested in the following iteration, after having already been signed off as “complete” by the unsuspecting Product Owner. If so, developers need to be pulled away to fix the defects found, or else the defects are added to a backlog to be fixed “later” (also known as “probably never”). Other developed code is left completely untested or under-tested, with the vague hope that any defects will be found in later testing phases, or that maybe there aren’t any important bugs anyway. In fact, these defects will be found, but not at a time more convenient to the team.
  • If any serious issues are raised by stakeholders during the showcase there is not time to fix them until the next iteration, which means it will take the full length of an iteration before a truly releasable build is created.
The root problem of the orthodox iteration

At the end of the day, the orthodox iteration suffers from two problems which are inherent in its very definition: it organizes work into batches, and it enforces a timebox.

Batching work is the antithesis to flow. The Lean approach to working aims to maximize the flow of work for the members of a team, which in software development translates to getting stories flowing easily through creation, analysis, implementation, validation, and release. When a developer finishes one story and it is signed off, she should have another story ready for her to pick up and start on. This shouldn’t need to wait on an arbitrary ceremony, and certainly shouldn’t have to wait for everyone else on the team to finish their stories and get them all signed off.

The batching focus of orthodox iterations doesn’t only cause developers to block, it also turns BA’s, QA’s, and the PO into bottlenecks. As described above, the start and end of the iteration each put a full working set of stories in the same state at once, all needing the same activity carried out on them at once.

Imagine an assembly line which starts up to assemble twenty cars, then stops while they are all inspected at once. Only once all of the cars are inspected and their defects fixed does the line start up again to begin assembling another twenty cars.

Timeboxing is also a source of problems for iterations. The main problem is the arbitrary deadline creates pressure to get stories “over the line” so they can count towards the velocity for the iteration. Unless management is enlightened (or uninterested) enough to avoid focusing on fluctuations of velocity from iteration to iteration (and even the most enlightened managers I’ve worked with do get worked up over velocity) this leads to the temptation to rush and cut corners, or to play games with stories.

Rushing obviously endangers the quality of the code, which almost certainly leads to delays down the line when the defects surface. Playing games, such as closing unfinished stories and opening defects to complete the work, or counting some points towards an unfinished story, undermines the team’s ability to measure and predict its work honestly. These bad habits will catch up one way or another.

Expecting code to be complete at the end of the iteration, fully tested, fixed, and ready for deployment, is unrealistic unless the iteration is structured with significant padding at the end. This padding must come after all reviews, including the stakeholder showcase, to allow time to make corrections, unless those reviews are mere rubber stamp sessions, with no genuine feedback permitted. This then means the team will be underutilized during the padding time. Otherwise, if there is so much rework done during this period that the entire team is fully engaged, then the risk of introducing new defects is too high to be confident in stable code by the end.

The alternative is to break the strict timeboxed-batched iteration model by interleaving work on the next iteration with the cleanup work from the previous iteration. This turns out to not be such a bad idea, and leads to evolving away from the timeboxed-batch iteration model towards the continuous development model.

The continuous development model

The continuous development model may be purely iteration-less, e.g. Kanban, or it may still retain the iteration as a period for measuring progress and for scheduling activities such as showcases. Once development is underway stories are prepared, developed, and tested using a “pull” approach, being worked on as team members become available, so that stories are constantly flowing, and everyone is constantly working on the highest value work available at the moment. This requires some different approaches to managing work flow than are used with other approaches. For more information, look into Kanban and Lean software development.

Since joining a year or so ago I’ve found that although no two ThoughtWorks projects run in exactly the same way, there is a strong tendency to use iterations which look a lot like Kanban, but retaining a one or two week iteration. Iterations are used to report progress (including velocity), and to schedule showcases and other regular meetings, but stories are not moved through the process in batches. Teams don’t start and stop work as a whole other than the start and end of a release. If the showcase is two days away, nothing stops a developer pair from starting on a new story knowing full well it will be incomplete when the codebase is demoed to the stakeholders, and possibly even deployed to later stage environments.

Although we do make projections and aim to have certain stories done by the next showcase, the team doesn’t promise to deliver a specific batch of stories. If it makes sense, stories can be dropped, added, or swapped as needed. This gives the business more flexibility to adapt their vision of the software as it is developed. It also reduces the pressure to mark a given story as “done” by a hard deadline, since there is no disruption from letting work carry on over the end of an iteration.

I’ve seen a Scrum team become ornery and rebellious when a PO made a habit of asking to swap stories after a sprint had started, even though work hadn’t been started on the particular stories involved. This was made worse because bugfixes were scheduled into sprints alongside stories, meaning that any serious defect found in production completely disrupted the team. Another factor that aggravated the situation was that the stories for each sprint were agreed before the end of the previous iteration. So if the showcase raised ideas for improvements to the functionality completed in iteration N, new stories could only be started in iteration N + 2 at the soonest. This hardly created a situation where the PO or the business felt the development team was responsive to business needs.

Also see Oren Teich’s post Go slow to go fast, which points out the problems with deadlines, and that iterations are simply a shorter deadline.

Challenges and rewards of continuous development

There are certainly challenges in moving to continuous development over the timeboxed-batch model. There is more risk of stories dragging on across multiple iterations. This can be mitigated by monitoring cycle time and keeping things visible, so that the team can discuss the issue and make changes to their processes if it becomes a problem.

For teams which are new to agile and still struggle to create appropriately sized stories, the timeboxed model may be more helpful to build the discipline and experience needed before being able to move to a continuous model. However, for experienced teams, timeboxing and batching stories simply has too many negative effects.

Continuous development, with a looser approach to iterations, maximizes the productivity of the team, avoids pitfalls that put quality at risk, and offers the business and the team more flexibility.

Published at DZone with permission of Kief Morris, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Jilles Van Gurp replied on Fri, 2012/12/14 - 4:08am

I totally agree with you. I have worked in scrum teams (with thought workers embedded in them) and it was a near total disaster, pretty much for the reasons you outline. 

I think there are a few things to add though. The formal roles in Scrum are very clear. However, when brought into practice, office politics and managerial relations become a huge factor and the rules go out of the window. 

My biggest beef with scrum is that the role of a PO is a junior management position. That means it taken by inexperienced people who don't have a lot of leverage in conversations with their more senior peers. The role of a scrum master on the other hand is not a management role but a technical role (typically). That means he/she is completely powerless and hopelessly dependent on the PO for actually getting stuff done. Typically the scrum master is a senior developer. Babysitting junior management is typically not a great use of the skills such people have. 

The way this plays out is quite predictable and the net results is that you have long winded meetings where debate tends to get heated on completely irrelevant issues about e.g. how many points a story is, what the definition of done is, etc. I find scrum and its meetings to be tedious. I've noticed most developers share this feeling.

Another big issue with scrum is that it is designed for teams in isolation. Most companies that are not small start ups have multiple interdependent teams that need to deliver together. The above managerial cluster fuck in combination with this is ugly. Scrum is completely inappropriate for this. None of the Agilists or scrum fan boys I've met (and I've met a lot) have a good answer for how to organize in a big company. The book on how to apply agile in big corporate development teams has yet to be written. Any claims to the success of any approach in this situation are at best unvalidated speculation (i.e. baseless bullshit).

Allow me to describe a scene not so long ago. Me and about 150 colleagues spread over three continents traveled to a big meeting room in North America. I was there as an observer from another large team to make sure our interests were covered. The teams goal: decide on the sprint planning for the next four two week sprints of the 13 or so scrum teams in the room (!!!!). We had a self proclaimed agile coach/ cheer leader to coach the whole thing. This involved a little pep talk at the beginning, breaking the room into little corners and basically gathering up the results in a powerpoint at the end of the day. It was surreal to see corporate insanity at this scale. 

Management loved it. They were doing agile and got to assign themselves new titles and congratulate each other with their mutual success. Developers were hopelessly confused talking about stories, points and velocity. I recognized it for what it was: classic waterfall: emperor running butt naked through the streets. Needless to say that this was one of the most dysfunctional teams I've worked with, ever. Nothing got done. Every decision was political. The quality of the little that did get done was appalling. 

The reason continuous development works is because technology has solved some of the issues that used to make this hard. It used to be that branching was hard, which meant you had to have a lot of ceremony (aka process) for things being integrated on the main branch. Technology has solved this. With Git you can achieve separation of concerns and juggle multiple ongoing stories in parallel and integrate the output of these stories when ready.  Take a good hard look at the way linux is developed: anything in the corporate world pales in comparison. Massive velocity, massive team size, massive amount of lines of code changed, regular as clockwork & rock solid releases. 

The linux process is extremely simple: do work; convince others your work is done and ask them to integrate your work (pull request). If it is not done, they'll say no. This combines very well with continuous deployment. Either it is ready and it can go live or it is not and needs more work. Nothing in between. Each story is a new story and you can organize for just that story and have a separate process for delivering it. That's the real beauty: you separate organizational concerns from functional concerns and from software quality. There is no need for everyone to follow the same process they only need to agree on policies for integrating into the main line. 

I'll take technology over process any day. 

Spencer Dunfee replied on Fri, 2012/12/14 - 12:14pm

 Hi Kief,

I was wondering if you guys ever cover tools like web conferencing software that can be used for software developers? At Mikogo we've had a lot of positive responses from the software development sector - our screen-sharing platform is often used for both live collaboration between developers and online presentations with clients.

Does this sound like something that could be of use to you? Or something that would interest your readers? Please let me know your thoughts. You can find more info here - http://www.mikogo.com/solutions/software-development/.

Thanks!

Spencer Dunfee
Mikogo Team
sdunfee@mikogo.com

Post Agilist replied on Fri, 2012/12/14 - 3:15pm

 I agree that iterations are harmful; as well I agree with Jilles and I have a number of postings on my website (postaglist.wordpress.com) that go into these aspects.

 However, I do not believe that

 1) One has to use pull based methods to be iterationless -- they can be push based as well. If you don't believe me look at the digestive tract of nearly every organism

 2) One should be developing/changing the product while figuring out the feature set. One word. Prototyping. A big huge miss for agile


 Also on my site I have a post about refactoring the role of the PO to solve many of the problems widely recognized with that role


PA

Willie Wheeler replied on Thu, 2012/12/20 - 8:26pm

Nice post Kief.

One force driving toward batching is a costly regression testing process, such as when regression testing is largely manual. Batching changes spreads the cost of that regression testing over a bunch of small changes.

To move away from iterations, it's important to drive down the cost of that regression testing by using fast, comprehensive test automation.


Brad Appleton replied on Mon, 2013/02/11 - 7:22pm

Hmmn - what you are describing to me looks like it is not a question of whether or not one does timeboxed+batched iterations, but of whether or not one does frequent incremental integration of stories during the iteration. Extreme Programming uses timeboxed iterations that deliver batches of stories and doesnt have the problems you describe if done properly using continuous integration of stories more-than-daily in a continous flow of working, tested stories.

If you do development iteratively in time-box iterations, but without incremental development and frequent integration *during* the iteration, you're still really just doing big-bang integration (albeit at the end of every iteration rather than only at the end of the release).

If however you do technical agile practices for incremental development (such as those prescribed by eXtreme Programming), then you still have one-piece flow of stories going on during the iteration, and working tested code is integrated and built, and in a potentially-releasable state daily or more often. That is still continuous *development* (just not continuous *deployment*).

Where the difference between one-piece-flow and timeboxed iteration come into play is not whether or not the development+integration is continuous (it can be for either of them) but in whether or not the deployment/delivery is continuous. True one-piece-flow would not just build+integrate but also release+deploy each story as soon as it was built+tested. Whereas with time-boxed iterations, even though I may have continuous development+integration of stories during the iteration, I still release them as a batch (even though I developed them continously).

It is the associated cost+effort of releasing and deploying the (already integrated+tested) results that determine if you are better off releasing/deploying in batches or a story at a time. The more you can decrease the cost+effort of the release-and-deploy transaction (using things like "continuous delivery" and deployment pipelines and DevOps for example) then the more frequently you can deliver to production and the closer you can get to one-piece-flow (but that still assumes the customer organization is able to reveive those changes and their impact on their business-processes as quickly as you can deliver them - which isn't always the  case.)

So the harm you are describing is most definitely real, but the root of the problem isn't the use of timeboxed iterations of batches of stories, it is the lack of incremental development and frequent/continuous integration -- something that is all too often common in shops that attempt to adopt Scrum without also adopting the most fundamental technical practices of agile development (and which Scrum itself described, at least in the very first book on Scrum, as at least daily builds, and automated tests).

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.