Agile Zone is brought to you in partnership with:

Mike is a certified PMP project manager and a certified ScrumMaster. Mike was involved with the creation of the DSDM Agile Project Leader certification, holds this certification at the Foundation, Practitioner, and Examiner levels. Mike was named an honorary member of the DSDM consortium and served on the board of APLN and the Lean Software and Systems Consortium. He currently co-leads the PMI Agile Community of Practice. Mike is a DZone MVB and is not an employee of DZone and has posted 137 posts at DZone. You can read more from them at their website. View Full User Profile

Should You Use Agile To Build Your Next Home?

02.12.2014
| 6355 views |
  • submit to reddit

Before we start our conversation about governance in the structure-governance-metrics framework we’re building out, I want to take a minute and see if I can finally tell you guys about the house my wife and I were planning to build this past summer. It’s an important conversation because we need to establish a shared understanding around what it means to plan on an agile project.

Scenario One – Team Agile Home Builder

My wife and I decide we want to build a new home. We find a piece of property we like in a nice neighborhood. We find a builder we like and think we can trust. We have a really good idea of the type of house that will work for us and our family. We decide that we want to spend somewhere around $500K to get the house completed but it needs to be finished sometime before our kids leave school for the summer.

Next step is to reach out the builder and see what he can do.

The next day we show up at the builders office and tell him about our plan for our new home. The builder is really excited, seems competent, and is very confident he can get the house built within the time and cost constraints that we’ve established. We ask him what’s the next steps for planning construction on our new home. To our surprise, he tells us that he is an agile builder and doesn’t do any planning.

The builder proceeds to explain to us that as an agile builder, he doesn’t do any planning. He will meet with us every two weeks to determine our priorities and figure out the next most important thing to build. Every two weeks we can inspect what he’s built and change our mind about anything anytime. He tells me this is really in my best interest because I won’t know exactly what I want until I see it.

When I ask if I’ll have the house I want when I run out of time and money, he doesn’t know, but assures me I’ll have the most valuable parts of the house early. Furthermore, I could actually move into the house anytime during the build process, and if I decide I’ve built enough house at any time, I can cancel the contract early and pay him for only the weeks he worked plus 20% of the remaining contract.

So here is my question… would you spend your money this way?

Most people when they are spending their own money, want to have some idea of what they are going to get when the time and money runs out. They want some assurance that they’ll have enough bedrooms and enough bathrooms. They want to know the acreage of the lot and if they are going to get a one story or a two story house and how many cars they’ll be able to fit into the garage.

People understand they might have to make some tradeoffs later in the build process. They might go over on carpet and have to make some tradeoffs with the lighting. They might decide they want fancy cabinets and go with a less expensive hardwood floor. They might want to defer decisions about the colors, or the finish, or the bushes they want to put in the front yard. They aren’t willing to leave out bedrooms.

We wouldn’t spend our own money this way, but this is the way we are asking our customers to spend their money when they build software with us.

Scenario Two – Scaled Agile Home Builder

Fortunately for us, this wasn’t the way that our builder asked us to build our house. We did meet with him and describe our dream home. We talked about the property we wanted to build on, the number of bedrooms and bathrooms, and the number of cars we’d be able to fit into the garage. We talked about the quality of the finishes and flooring and how we wanted to yard landscaped. That took about an hour.

After that, we spent a week or two having meetings with the architect. We got pretty specific about the room layout, the number of floors, the layout of the basement, and the placement of the house on the lot. We made high level decisions about floors and cabinets, but didn’t pick any specifics, decide any colors, or pick and specific appliances, light fixtures or door knobs. There was way more left open then decided.

What we did decide was the stuff that would drive the cost of the house. The stuff that would be expensive to change later. We went into the process understanding that a ton of stuff was going to change, but there was also stuff that would be really difficult to change, and those decisions had to be made early. We talked about potential risks once they broke ground and buffered in case anything went wrong.

The result was a blueprint and elevation for the house. We had a line item budget for construction costs, material costs and miscellaneous tools and supplies. Every line item factored in a profit for the builder. Every line item represented some feature or attribute of the house that I would probably care about when the house was complete. We did all that in a few weeks and made no detailed decisions about the home.

It was interesting, the first cut of the plan was about $200K more than we wanted to spend and we were aware that there was some risk and should probably buffer another $50K just in case. Even though I trusted the builder, I had my CPA and advisor review the plan line-by-line just to make sure all the costs were reasonable and we were making a sound financial decision before we decided to move forward.

As we were reviewing the plan… the builder said something to me that was pivotal for how we were going to manage this project. He told me that I had a 20K budget for hardwood floors. For that price, he assured me that I would be able to get a high quality floor, extra long and extra wide boards, and a very high quality wood with a very high quality finish. That said, my budget was 20K for the hardwood floors.

When it got time to build put in the hardwood floors, it was up to me to choose what kind of floors I wanted to put in. If I decided I wanted a less expensive floor, I could save money and use it for something else. If I wanted an exotic wood like Brazilian Cherry, that would cost more and I’d pay extra. It would ultimately be up to me, but he would work with me to understand my trade-offs and any associated costs.

At Enterprise Scale, Estimates Become Budgets

I instantly made the connection that this was exactly how I coached teams to define, estimate, and plan software projects. You have to have a high level plan. You have to make key decisions early. You have to have time and cost estimates to present to the customer. What you don’t have to do is define anything and everything before you even get started. You have to establish budgets to bound your uncertainty.

At the point the builder made the estimate, he was bound to manage the project to that estimate… the estimate became a budget. The builder had to work with me as the primary stakeholder to help me make tradeoffs to live within that budget. As we break down software requirements, focusing on minimally marketable features and maximizing value, we are effectively doing the exact same thing.

Okay… let me drive this one point home. Just like you wouldn’t build a house with an open ended scope, your stakeholders don’t want to commit to an open ended software project. We have to do enough work up front to put together a credible plan, mitigate risk, and understand costs. We have to estimate the work, set budgets, and make tradeoffs to ensure the product is done on time and on budget.

It is important to be able to tell people what the project is going to cost and when it’s going to be done. It’s important to tell people what they are going to get for their time and money. It’s also important they know the risks and can buffer some contingency. They also need to know what’s locked and what’s open and the kinds of tradeoffs that they might have to make along to way to ensure success.

Why Is This Such a Difficult Problem to Solve?

The problem is that many software teams are trying to build products in less than ideal conditions. They are constantly realizing issues and risks they didn’t anticipate. They are working across too many projects at one time. They have external dependencies that can’t be managed or controlled. People are not dedicated to the team. The result is that people are coming to the conclusion software can’t be estimated.

Going back to the house analogy, it’s as if we are building on rocky soil and are constantly finding boulders that have to be broken up and hauled away. It’s as if we have contractors that never show up when they say they will. It’s as if we are constantly changing our mind between a one story house and a two story house or deciding we want a basement after the foundation has already been poured.

Summing It All Up

The answer can’t be throwing our hands up and refusing to make a commitment. We might have to pay a few thousands dollars to determine if the build is feasible. We might have to pay another few thousand for soil tests to make sure the land can accommodate our new home. You can spend money to mitigate risk. Sometimes you need to do some work to learn what you don’t know.

No estimating, no planning, and not committing won’t be the answer for most organizations. It’s not a starting place anyone can live with.

You have to ask yourselves what it is going to take to stabilize the delivery process. Forming teams that stay together helps, that was my previous post on structure. Having a governance model to control the flow of work helps too, that’s my next post. Having a means to mitigate risk, invest in discovery, establish baseline metrics, and the ability to measure improvement over time all play a part. More posts to come!

#noestimates is a #nonstarter

One final note… Kimi and I never ended up building this house. The neighborhood was beautiful. The plan and elevation were beautiful. We really liked our builder and trusted he could deliver our dream house within the time and cost constraints we agreed on. The problem was that housing prices were depressed in the county we planned to build in and the appraisal came in at 60% of the build cost of the home.

We could have moved forward anyway, but would have been instantly in a house that was worth less than we paid for it, with no indication it would recover in the foreseeable future. So sure, we spent a bunch of time and mental energy, and even some hard cash to find out we shouldn’t build the house. Had we built the house using mainstream agile thinking, we’d have found out too late it wasn’t worth what we’d actually paid for it.

We spent a few weeks of time and a few thousand dollars but avoided what would have been a very, very expensive mistake. That’s not waste in my opinion… that’s good risk management.

 

Published at DZone with permission of Mike Cottmeyer, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Nicolas Frankel replied on Wed, 2014/02/12 - 9:15am

I am also working in a large entreprise, and have found myself in front of the same problem "estimates becoming budget" you mention.

However, I must disagree on your analogy, which seems very strange to me. Houses are somewhat older than software and people on both sides of the fence speak the same language. Moreover, product owner have a clear understanding of requirements, and know they cannot change during building. Things are very, very different in the software industry. Do I need to detail?

Worse, following your analogy, it seems to me that large construction works regularly (if not always) are finished behind schedule and over-budget. Sure, everything is planned from the ground up, and they still fail... By the way, construction has more than 5,000 years history. As software engineers, we're expected to provide clear and precise estimates even if our industry is less than 50 years old!

IMHO, that's just laughable.

Serguei Meerkat replied on Wed, 2014/02/12 - 10:48am in response to: Nicolas Frankel

It is the fault of those people in IT who sell and run the projects that we can't emulate what building industry does and it has nothing to do with the age of the industry.

I used to work for a company that run projects well. We knew the industry for which we developed software. Before the client signed on a dotted line we would go to their side and spend time collecting information (BA and Development leader, not sales people as it often happens) and we would write a report where it would list what the client needed and how much it would cost.

If during the development the client decided to add/remove something, it would be discussed and the price would be given the client who would make a choice whether to go with new requirements or to stick with the original plan.

And everybody were happy - we did not have too many changes of plans, the client knew upfront the costs and we were on time and on budget.

The trick is for developers to know the business domain and been able to talk to the client in the language of their business.





Mike Cottmeyer replied on Wed, 2014/02/12 - 11:40am

Nicolas... maybe in the context of the projects or products you work in software building and house building are very different. The contexts I work in there are many parallels that can be drawn. My post was to explore some of those parallels. You may want to consider asking some clarifying questions before you so readily dismiss my ideas on this.  

The notion of having a high level architectural plan on a software project is necessary and reasonable. The idea of committing to a high level scope, delivery date, and budget is necessary and reasonable. The idea of negotiating fine grained scope and implementation details within the context of the high level plan is necessary and reasonable. The idea of buffering for risk, driving out uncertainty, and spending money early to prove the viability of the product is necessary and reasonable. 

If you are not doing these things, I find it hard to believe you are delivering enterprise class software in a large organization, which is my context.  

Sure many of our building projects suffer from the same problems as our software projects and are late and over budget as well. Unanticipated changes, too much WIP, poor estimates, not enough negotiation with the client, not enough trust and respect between the parties involved etc. The solutions to many of these problems, across both domains, are very much the same. 

So, I'm sorry that you find my exploration here laughable, but I stand behind my comments.  If you'd take a step back, consider the merits of my argument, and maybe start using some of these ideas in your work... I bet you'd have a much better chance of delivering software on a regular schedule and maybe even be able to convince the rest of your organization to give agile a shot.  This whole movement will shut down if we cannot figure out how to get better at doing what we say we are going to do.  

Agile ideas are not for the privileged few, building certain kinds of products, for certain kinds of customers.  These ideas can be applied everywhere.  We have to get better at answering what am I going to get for my time and money when I run out of both.  Not knowing is not an answer.  If you concede that, which maybe you don't... give me a more credible approach to solving the problem. 

Thanks for you post... look forward to your reply.  


Nicolas Frankel replied on Thu, 2014/02/13 - 2:19am in response to: Mike Cottmeyer

 Hello Mike,

Sorry if you thought that the laughable adjective applied to your post, I only disagree with it. That's perhaps due to the reason I'm not a native English speaker.

What I find laughable is expectations regarding our industry. Those are facts:

  • As software engineers, we are to provide accurate estimates, despite most projects having different requirements, contexts, domains, etc.
  • Our industry is new, with technologies changing drastically in a couple of years, bringing with them structuring architectural changes
  • Many (if not all) projects I've worked with or even heard of finished over-budget, and either late or with large scope chunks removed (when not both)

So, my take is that the current strategy of estimating, planning and budgeting is failing. I'm not a an agile guru, far from it, I'm just saying that we should either keep the same strategy and fail over and over or change our ways... Because I'm a tech guy, I cannot provide a solution, but it doesn't prevent me from looking around. What I noticed, is that smaller projects are generally more successful than large one, however.

Now, considering your points...

  • I've never seen a customer spending money on a prototype and canceling the project, whatever the prototype findings.
  • I've never seen anticipated changes. Changes are unanticipated by nature. Funny how the business just changes its mind over and over.
  • I've never seen trust that can be decreed.
  • I've never seen accurate estimates, regarding projects that require budgeting.

The root of the problem, however, is that your experience and mine must be quite different. In that regard, you (and I) cannot make universal rules out of them.

Mike Cottmeyer replied on Thu, 2014/02/13 - 8:48am

Thanks for the reply to my comment.  I actually try really hard not to imply I'm making universal rules... only sharing my experiences and trying to help rationalize some of the nonsense we see on a daily basis.  To your last set of comments... I seldom see many of those things either. That said, we try to empower our customers with rational data around their software development process.  I do believe that in the presence of rational data, we will make rational decisions.  I think the trend toward no estimates is harmful, every bit as harmful as beating people over the head with estimates that we know are wrong.  Constant collaboration through out the process, in the context of a rational plan, with both parties making tradeoffs as they go, IMO, is the only answer.  Sorry if I sounded snarky last time.  Probably just trying to reply in a hurry.  I really do appreciate the dialog!

Serguei Meerkat replied on Thu, 2014/02/13 - 3:18pm in response to: Nicolas Frankel

Nicolas Frankel wrote:I've never seen accurate estimates, regarding projects that require budgeting.

Yes, many software companies and teams don't know what they are doing and can't estimate (or sales people doing the estimates for them). The fact that they are incompetent does not mean that everybody else are incompetent too and that it is normal state of things.

When you thinking of your home improvement, you normally need to know how much it might cost in order to make a decision whether to start in the first place. The same is true for an IT project. The client or management need to know how long it would take and how much it would cost before committing the budget.

Arguments about "new industry" could have worked in 1980s. Today they simply show the incompetence of those who think that someone is going to pay them money without expecting to know what the final amount might be.

I also believe that a lot of the projects are late and above the budget because those who sell their services are lying while knowing perfectly well that their "estimates" are inaccurate. It would be true for sales people, who are trying to get commission, and it would also be true for teams of programmers who fancy a nice project preferably with technology that looks good on CV.

Nicolas Frankel replied on Fri, 2014/02/14 - 6:57am in response to: Serguei Meerkat

I look forward to working with you to learn how you estimate successfully, then.

Lund Wolfe replied on Sat, 2014/02/15 - 9:00pm

It's a risky business, even for the home builder.  The home builder has to get the design exactly right up front.  He has to know everything except furnishings and details, like flooring, cabinets, paint colors but will have to provide estimates for that, too, based on quality of materials.

Ideally, software could be built the same way.  Since it was designed for all the expected functionality the form naturally follows the function.  The difficulty/risk is in the big bang estimate.

The reality is that the application customer doesn't know the finished product up front and the developer doesn't know how much (how long) it would cost even if he did know all the requirements up front.  Software is much more flexible to build or change than a house, but it is also much harder to estimate.

The more practical approach is a compromise of estimating a much smaller initial deliverable with increments of additional functionality.  The customer can then back out at an increment if unsatisfied with the value provided or the quality.

The challenge of agile is in getting the initial design/infrastructure right based on the core functionality and expected incremental functionality such that it still naturally plugs into the design.  The customer and developer should know these requirements up front.  The developer doesn't want to end up in the situation of needing to refactor it into something it wasn't designed/expected to do in the first place.  Sorting the significant requirements from the less relevant details and add-on requirements is very difficult.  Software varies much more from app to app, which makes it harder to build and harder to estimate than a house.

John Lewis replied on Thu, 2014/02/20 - 12:02pm

I've always found the home-building analogy to software development to be fundamentally flawed. Software development is not construction -- it's design. From your description, your process of working with the architect to come up with the detailed design of your home sounds completely agile and iterative to me. You gave him some high-level requirements, he starts to design, he shows it to you (i.e "demo"), you provide feedback, he makes changes, lather-rinse-and-repeat, until you have the complete detailed design for your home that can now be constructed. But in the software world the "construction" phase is essentially free, since once we are down to source code as the final design of the software we assemble/compile/build/package/etc via automated tools. I've long found the "Code as Design " essays by Jack Reeves to be really enlightening in this area.


Darrell Freeman replied on Thu, 2014/02/20 - 5:19pm

I have often heard it said that building and software construction are not comparable because software can be changed. But at my company they are throwing out $100M of software, written in an agile manner, that cannot be changed. If someone were to build a building as in scenario one, then there would be a lot of custom joinery, wiring and plumbing routing, etc. The very same thing can happen in an agile project (any methodology actually). Particularly so if project teams are spread over China, India and your home country. All programmers have is user stories, test conditions and the imperative to get their tasks done within the iteration. This is important for management as all they care about is measurement, velocity and controlling cost. So without proper software governance it is possible to create the conditions where there is a lot of custom code, different styles and components created just to get the job done, creating just as much inflexibility as in a building. This is not important in small projects, after all, agile was partially created to avoid the massive design costs in waterfall, but for enterprise class software where the business is investing tens of millions, they don't want to throw it out every time the business changes. I'm for the author in scenario two.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.