Mr. Mahal has worked his way through all aspects of software development. He started as a software engineer and learned how to design and architect software systems in different industries. He has been a director of engineering, product management, and professional services; mid-career he worked for a company dedicated to training engineers in collecting requirements and understanding OO and conducted research into why software projects fail; and before starting Accelerated Development he rescued a technology startup in Vancouver as their COO. Accelerated Development is dedicated to the production of high quality software through implementing light weight engineering practices for all size organizations up to Fortune 500. Dalip is a DZone MVB and is not an employee of DZone and has posted 37 posts at DZone. You can read more from them at their website. View Full User Profile

Are We There Yet?

09.29.2013
| 585 views |
  • submit to reddit

We associate "are we there yet?" with kids asking incessantly if a long trip is almost over.  It is generally funny, however, it is less funny on a project that should be complete; it is even less funny if it is your project. 

Projects follow distinct phases:

  1. Basic requirements are collected
  2. Project plan and end date are established
  3. Development starts
  4. Projects track to the project plan

Often, tasks start off well until they all level off at 90-95% complete and get stuck.  Management was satisfied with progress until the project stalls, they see frantic activity picking up and they start asking  "are we there yet?."

Like a family vacation, this trip is sometimes not even close to being finished.  You  expect that if 9,000 hours have been spent on a project estimated at 10,000 hours that you would be 90% done.

Surprisingly this is not the case for 7 projects out of 10 -- have you ever worked on a project where a 90% complete project plan meant the project was 90% done?

Project Plans can give the Illusion of Control


 Estimating project completion using the project plan is valid if there is a direct correlation between the project goal and the project plan. You often discover that the goal and the plan differ late in a project. Project plans and results differ because: 

  • Requirements and tasks are missing
  • The project is incorrectly estimated
  • Work is performed on tasks that do not advance the project

Requirements and Tasks are Missing

Clearly missing requirements mean that more work will be necessary to get to the goal, but the time for this work is rarely added to the project deadline.

A relative of missing requirements is missing tasks, this occurs when the work breakdown structure is incomplete and more subtasks are necessary to complete a task than estimated.

In both cases, if there are 2,000 hours of missing requirements and tasks then a project initially forecasted for 10,000 hours should move the deadline to 11,000 hours.  Therefore if 9,000 hours are done then you are only 75% complete.

Project plans that show 90% complete when there are 20% of the tasks and requirements missing are really 75% complete.

Unfortunately, weak IT leadership, internal politics, and embarrassment over  poor estimates will not move the deadline and teams will have pressure put on them by overbearing senior executives to get to the original deadline even though that is not possible. 

The Project is Incorrectly Estimated

There is much literature about how accurate estimates are possible and necessary to successful projects. Weak and uniformed IT leadership will cave in to senior management demands for project deadlines without formal estimates.

A typical CEO and VP Engineering interaction looks as follows:

CEO: We need feature X, how much will it cost and how long will it take to built?

VP Engineering: Well we need to define feature X properly, see how it will be implemented, determine if we have the necessary skill sets, and see what the impact to our other operations will be.  It will take time to do do this work.

CEO: We don't have time for formal estimates.  How hard can it be to add feature X?  By next Friday, I will need a ballpark estimate for time and cost.


VP Engineering: I'll see what I can come up with for next week.

Weak IT executives allow themselves to be bullied all the time by other executives that have no idea what is involved in IT projects.  The end result is an underspecified project that will be underestimated in time and cost (see Why Executive Declared Deadlines lead to Disaster)

The more inaccurate the requirements the more extra work there is to do to get to the target.  That is why short requirements processes lead to strongly shifting requirements and canceled projects (see Shift Happens).  In fact, the degree of requirements shift is equal to the chance of a project being canceled.

Work is Performed that Does Not Advance the Project

Even if a project is correctly specified, there are several activities that will be performed that will not advance the project: 

  • Some requirements can not be implemented as specified and time will be spent researching and implementing work arounds
  • Some requirements will be ambiguously specified and be implemented incorrectly and need to be redone
  • Some requirements will be inconsistent and require time and analysis to establish consistent requirements
  • Infrastructure might need to be refactored when you discover that it will not support the code created later

Work executed on these activities will not advance your project and should not be counted in the total of completed hours. 

So if 2,000 hours have been spent on activities that don't advance the project then if 9,000 hours have been done on a 10,000 hour project then you have really done 7,000 hours of the 10,000 hour project and you are only 70% done. 

Project plans that show 90% complete when  20% has been spent on unproductive tasks are really 70% complete.

Unfortunately...

On projects you will have both missing tasks and unproductive activities.  So if 2,000 hours are unproductive and 2,000 hours are missing in a 10,000 hour project where 9,000 hours have been done then you have only done 7,000 hours on a 12,000 hour project you are only 58.3% done.
Project plans that show 90% complete when there are 20% missing requirements and 20% time spent unproductive tasks are really 53% complete.

Many of you have been in this situation before, you know that the project plan states that you are 90% done but you know that you are not even close to finishing the project.

 Not Changing the Deadline Leads to Worst Practices

Whether a project is off course because of missing activities or non-productive activities, not changing the project end date will lead to schedule pressure as the project advances and people slowly start to realize that you are not going to make it. 


When it becomes clear that a project can't make it's original deadline many organizations will start common but deadly practices.  Excessive schedule pressure often leads to the following bad practices (see Stop It! No... Really stop it.):
  • Friction within the team
  • Friction amongst the managers
  • Inadequate communication with  stakeholders
  • Layoffs of key personnel

Solutions

There are only a few cures for the 'Are we there yet?' problem: 

  1. IT management with the intestinal fortitude to hold out for the creation of proper work breakdown structures and formal estimates
  2. Proper requirement processes that yield complete, consistent, and concise requirements
  3. Proper change management processes to alter the project deadline when missing tasks and non-productive activities are encountered
Failure to comply with these 3 principles means that you will continue to be subject to chaotic environments where 7 out of 10 projects fail (see Executives: Understanding your Chances of a Successful Project)
Published at DZone with permission of Dalip Mahal, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Lund Wolfe replied on Sun, 2013/10/06 - 11:45pm

How often projects fail at step 1, requirements.  Sometimes 95% of the work is in the main processing structure which is fairly obvious and the input and output specifications can be thrown in near the end of development.

It is often assumed that only the developer matters.  Whether it is new development or maintenance, a talented developer will compensate for all other deficiencies and everything else is just a "nice to have".  This does seem to work far more often than it should.

There may be a timeline and a process going through the motions of requirements and paperwork with the customer and development to make it look good, but the actual developer is not involved until day 1 of the coding stage.

So far so good.  The project is on schedule and the checkboxes are all ticked.  We just need to satisfy the % complete, so start on the easy items first.  At some point you get to 50% or 90% and progress nosedives.  It is time to clarify the requirements, the hardest part of any project.  Maybe the developer isn't allowed to contact the customer because the development group is afraid of the customer.  This is even worse than the customer not being sufficiently competent to clarify the requirements at any point in time.

Often the requirements turn out to be simple enough or are so loose (we just need to do it and call it done) that the project can't fail.  Hopefully some lessons will be learned, eventually.  Do the project iteratively and agile.  Work closely with the customer.  Have the actual developer involved in requirements.  Flesh out the core and difficult requirements first.  Implement the core and difficult requirements first (do a spike/architectural test project if needed for new development).  Seeing is believing for the developer and the customer.  The % complete should start out slow and then speed up (or fail early, or at least take a step back and fill in the gaps).

Dalip Mahal replied on Mon, 2013/10/07 - 9:20am in response to: Lund Wolfe

I agree, most projects fall apart at the requirements stage - but why?

Most organizations don't have the correct people to capture requirements.  Most product managers and business analysts don't understand what is necessary to collect complete requirements and distill them into a consistent and concise document.

If organizations tracked requirement defects (see Bug Tracker Hell and how to Get Out!) then they would realize that requirements are to blame for projects falling apart.  Because this is not tracked, requirements defects are not seen in the chaos when a project falls apart.

For some reason senior management assumes that it is a development problem when a project falls apart, but you are correct that most of the time it is the requirements that are to blame.  That also means that most failed projects will fail before they even start!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.