I run a small development firm which specializes Microsoft technologies. Ted has posted 12 posts at DZone. You can read more from them at their website. View Full User Profile

Technical Debt - Part 1: Definition

01.13.2011
| 5402 views |
  • submit to reddit
Technical debt is a subject that is getting a lot of attention lately. Gartner estimates global technical debt at $500 Billion and projects it to reach $1 Trillion in five years. Surprisingly, however, technical debt is a concept that many people in business and technology fail to fully understand. The reason this lack of understanding is so surprising is the magnitude of the problem and the fact that almost every business has (or will have) technical debt and regularly pays interest on it. On the surface, the concept of technical debt is very simple. The term was coined by Ward Cunningham in 1992 to describe the costs associated with maintaining and enhancing a platform where shortcuts were taken in the design or implementation in order to meet financial or timeline objectives. Examples of these shortcuts include failure to do the following:

· document system architectures

· provide adequate code coverage in QA processes

· implement standards and conventions

· properly understand the technologies leveraged

· properly understanding the processes the technology is designed to support

· refactor code base or architecture to meet changing needs

Cunningham followed up with a brief video to further describe and clarify the concept in 2009. The debt metaphor he describes fits surprisingly well. Just as in finance, debt can provide a valuable tool to expand more rapidly than you would be able to otherwise. However, if you accumulate too much debt the majority of your resources will need to focus on servicing that debt. One can think of the principal amount of the technical debt as the cost associated with fixing the problem and the interest as any business cost associated with the shortcomings. The loss in productivity a programmer might experience enhancing an undocumented application is an example of interest.

The key difference between financial and technical debt is that the former is usually very visible and easy to quantify while the latter is not. There aren’t any financial calculators that allow you to input a few parameters and have it calculate a number that tells you how much technical debt you have. When you assume technical debt you are acquiring an asset (working software) at a lower expense (in terms of time or capital) than the true cost of the asset. The gap between the price paid and the true cost represents the technical debt.

Organizations with mature risk management practices will often track these types of technology gaps in a centralized way. However, few have formal governance processes to identify, quantify, prioritize, and resolve technical debt in an effective and coordinated way. Even if those processes exist they usually reside in different areas of the organization (i.e. internal audit, operational risk management, etc.) and coordination becomes difficult. I will talk about each of these processes in detail in subsequent posts.

Strategic Debt

There are several types of technical debt and some have even attempted to create a taxonomy to classify it. However, at a high level the most important distinction is the difference between strategic and nonstrategic technical debt. Strategic debt describes situations where tradeoffs are made in a proactive and measured way. One might wonder why anyone would choose to make compromises that will have consequences down the road. The two most common reasons debt is accumulated is to save money and/or time. It can make perfectly good sense to take shortcuts to get across the goal line when there is a limited budget or time to market is a big concern.

What makes debt strategic is the fact that the pros and cons were evaluated and an informed decision was made by the appropriate stakeholders to assume and manage the debt to help meet strategic objectives. It is important to recognize that just because someone was aware of the technical debt beforehand and made a conscious decision to assume it does not make it strategic. For example, the debt created when a development manager decides on his own to postpone documenting the system his team is building until after implementation cannot be considered strategic to the organization. It may be strategic to that particular manager’s objectives but may be in conflict with the interests of the organization as a whole.

The decision to assume the debt must be explicit and made by key stakeholders based on a reasonably accurate assessment of the risks and rewards contained therein. Additionally, process must exist to manage technical debt to ensure that it maintains visibility and is paid back at the appropriate point in time. Any debt that does not meet these criteria cannot be considered strategic. Below is a scenario which describes strategic technical debt.

An emerging opportunity has been identified and the ability to fully capitalize is directly tied to the organization’s ability to deliver a timely solution. There is also a limited amount of budget allocated to providing a production ready solution. To meet project goals, an open source platform which allows rapid application development is chosen. However, the platform has not been proven to support the volume which would be eventually required if best case projections materialize.

The potential gap the platform brings to the table is discussed with stakeholders and without the reduced cost and shorter timeline the architecture provides it will not be possible to move forward. To mitigate the risk, the company assigns resources to closely monitor usage and routines are put in place where senior leaders meet twice a month to evaluate revised projections. If and when the projections indicate that the platform might not be able to support usage, the plan to migrate to the long term architecture will be executed.

Nonstrategic Debt

While assuming strategic technical debt to help meet objectives can be in the organization’s best interest, assuming nonstrategic debt almost always is not. Nonstrategic technical debt describes situations where gaps are created without stakeholder approval and the implications to strategic objectives aren’t properly considered. The sources of nonstrategic debt are numerous but often result from process deficiencies.

One frequent source of nonstrategic technical debt is low quality code. Sometimes code is written poorly right out of the gate because standards and conventions aren’t enforced and QA processes don’t exist. A more common reason code quality suffers is because shortcuts often must be taken to meet the time constraints usually associated with enhancement requests.

In software development, system requirements change often. This change can be due to real changes in the environment or our understanding of it. As the environment changes or our understanding of it evolves, the code we write must be modified to support that. These modifications can make systems complicated and difficult to support if routines aren’t put in place to refactor code when necessary. It is natural that systems are complex but real challenges can arise when they become complicated. There is a distinct difference between complexity and complicatedness and much has been written on why the distinction is important.

I will dive deeper into governance in a subsequent article but it makes sense to point out that that many sources of nonstrategic technical debt such as poor code quality can be managed or avoided. By adopting development processes that embrace change and put an emphasis on code quality is critical. Many Agile processes implement controls to mitigate the risk of poor code quality. Extreme programming (XP), for example, leverages pair programming where developers write code in pairs which increases code quality by having the benefit of two different perspectives on how to solve technical challenges. The Scrum development methodology requires that a “product owner” who represents the business and QA resources play active roles directly in the development process. Alignment to business objectives and quality are baked directly into the development process instead of being an afterthought.

Organizational Agreement

It is important to point out that technical debt means different things to different people. How much agreement you need to get across your organization depends on what you plan to do with the inventory of technical debt once you go about compiling it. Regardless if you are creating an inventory to manage it discretely within a small development team or creating an enterprise wide governance initiative, it is important that business stakeholders buy into the definition.

The ultimate goal of defining and identifying debt is to manage it. Including business partners early in the process is key because once the technical debt is identified and measured, remediation efforts will need to be prioritized along with other enhancement requests. Business partners are much more likely to trade off implementing new features for paying back the debt if they’re included early in the process. Vetting out the definition with the business will help you stay on the right path as you go about turning over rocks.

Gartner defines IT debt as the cost of clearing the backlog of maintenance that would be required to bring the corporate applications portfolio to a fully supported current release state. While this definition is broad a lot is left to interpretation. A critical question becomes what does “fully supported current release state” really mean? If the application itself is free of technical debt but there are quality issues in the underlying data does that qualify as technical debt using the Gartner definition? What if those data quality issues are from upstream systems and you’re paying interest on someone else’s debt?

Something else to be aware of is that most definitions of technical debt available today focus on workarounds that were implemented during the development process. Your business partners may want to cast the net wider than the traditional definition and include technology risk related items such as control weaknesses. Coming to a consensus on a definition with the folks who are writing the checks (i.e. business partners) and prioritizing the backlog will pay dividends during the remediation process.

Implications

The debt metaphor is an important tool in understanding and communicating the implications of technical debt. Using this metaphor, free cash flow can be thought of as the amount of resources available to maintain and enhance systems. Cash flow is a finite resource that must be allocated carefully. Failing to understand how much of that cash flow is being spent servicing technical debt can lead to poor decisions on where to most effectively invest. This could eventually lead to technical bankruptcy where all an organization’s resources are spent dealing with the inefficiencies created by the debt they have accumulated and they can no longer keep up.

While reaching the point of bankruptcy is an extreme case, I have seen very real examples of this scenario first hand. Once an organization comes close to having this level of debt there can be devastating consequences. Even if systems are completely internal, impacts to customer service usually arise. Just like financial bankruptcy, sometimes the magnitude of the underlying issues are so severe that it becomes impossible to emerge. In the worst case scenarios, systems have to be scrapped and rewritten, entire departments are outsourced, customers and market share are lost and shareholders lose confidence. Therefore, it is critical that organizations create a plan to proactively deal with this phenomena.

In my next article I will discuss how to identify technical debt within an organization. Subsequent articles will discuss the rest of activities in managing the lifecycle including quantifying, resolving and governing technical debt.

From http://blog.acrowire.com/technical-debt/technical-debt-part-1-definition/

Published at DZone with permission of its author, Ted Theodoropoulos.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Artur Biesiadowski replied on Thu, 2011/01/13 - 5:56am

I think that you have failed to mention most important difference between financial and technical debt. If you are not paying financial debt for many years and ignore it, you are going to be in deep troubles and borrowing more money is not going to help you much. With technical debt, very often you can just default on given debt and carry on with almost no ill effects.

If there is a legacy system in the company, it is very often a lot cheaper to create (or buy) new system. While you spend some extra effort for reverse engineering obscure behaviours of old system, this is still many times cheaper that properly refactoring/documenting old one.

In finance world it would be like borrowing  $1M debt and having to pay $100k yearly in interest. Then, at some point you decide to repay the debt - but you can just say "Forget about old debt, I will get a new one for $250k".

So, if you are talking about technical debt, remember that default is always often an option - and it is often a lot cheaper and better in longer run. So before you start complaining to your manager that that COBOL system has to be retrofitted with thousands of unit tests, make sure it will be still used one year from now...

Cloves Almeida replied on Fri, 2011/01/14 - 9:29pm

Artur,

as a side note defaulting on financial debt is an alternative and is often used under the euphemism "renegotiation".

I really liked this "debt" analogy. It will be useful on explaining a few entrepreneurs why they should invest on management capabilities before their companies grow into chaos. They spend more resources (money, smart people) paying their "management service" then improving the company.

Ted Theodoropoulos replied on Thu, 2011/02/03 - 12:57pm

Artur,

Thanks for the comment. You make a good observation. I think what you're referring to is "expected interest." There are a number of ways to quantify debt but the method I tend towards is breaking the debt down into three fundamental components: principal, interest, and interest probability. Principal and interest are self explanatory but interest probability is not as commonly understood.

As you point out, sometimes the principal and interest payments associated with technical debt need not be repaid. Interest probability is the likelihood that the interest will actually come due. An example might be where you have a gap in code coverage in your unit test library. Let's say the gap is in a very stable part of the application where the code base rarely changes and has limited dependencies. The probability that the gap will have negative financial implications (i.e. interest) is lower than coverage gaps in more dynamic areas. If you multiply the interest by the interest probability you essentially have your "expected interest cost."

There are also scenarios like the one you describe where applications are due to be retired and remediation efforts would not generate an acceptable ROI. I discuss prioritization in a Part 4 of the series of articles. I don't get into interest probability because this series was intended to be a 101 level overview. I treat "interest" like "expected interest" to keep things simple. I am working on a more advanced series now and should have the first installment up within a week if you're interested.

http://blog.acrowire.com

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.