Mr. Lott has been involved in over 70 software development projects in a career that spans 30 years. He has worked in the capacity of internet strategist, software architect, project leader, DBA, programmer. Since 1993 he has been focused on data warehousing and the associated e-business architectures that make the right data available to the right people to support their business decision-making. Steven is a DZone MVB and is not an employee of DZone and has posted 133 posts at DZone. You can read more from them at their website. View Full User Profile

Great Lies: "Design" vs. "Construction"

03.12.2010
| 9331 views |
  • submit to reddit
In reflecting on Architecture, I realized that there are some profound differences between "real" architecture and software architecture.

One of the biggest differences is design.

In the earliest days, software was built by very small groups of very bright people. Alan Turing, Brian Kernighan, Dennis Ritchie, Steve Bourne, Ken Thompson, Guido van Rossum. (Okay, that last one says that even today, software is sometimes built by small groups of very bright people.) Overall architecture, both design and construction where done by the same folks.

At some point (before I started in this business in the '70's) software development was being pushed "out" to ever larger groups of developers. The first attempts at this -- it appears -- didn't work out well. Not everyone who can write in a programming language can also design software that actually works reliably and predictably.

By the time I got my first job, the great lie had surfaced.

There are Designers who are distinct from Programmers.

The idea was to insert a few smart people into the vast sea of mediocre people. This is manifestly false. But, it's a handy lie to allow managers to attempt to build large, complex pieces of software using a a larger but lower-skilled workforce.

Reasoning By Analogy

The reasoning probably goes like this. In the building trades there are architects, engineers and construction crews. In manufacturing, there are engineers and factory labor.

In these other areas, there's a clear distinction between design and construction.

Software must be the same. Right?

Wrong.

The analogy is fatally flawed because there is no "construction" in the creation of software. Software only has design. Writing code is -- essentially -- design work.

Architecture and Software Architecture

Spend time with architects and you realize that a good architect can (and often does) create a design that includes construction details: what fastenings to use, how to assemble things. The architect will build models with CAD tools, but also using foam board to help visualize the construction process as well as the final product.

In the software realm, you appear to have different degrees of detail: High Level Design, Detailed Design, Coding Specifications, Code.

High Level Design (or "Architecture") is the big picture of components and services; the mixture of purchased plus built; configuration vs. constructions; adaptation vs. new development. That kind of thing. Essential for working out a budget and plan for buying stuff and building other stuff.

Usually, this is too high-level for a lot of people to code from. It's planning stuff. Analogous to a foam-board overview of a building.

Detailed Design -- I guess -- is some intermediate level of design where you provide some guidance to someone so they can write programming specifications. Some folks want this done in more formal UML or something to reveal parts of the software design. This is a murky work product because we don't have really formal standards for this. We can claim that UML is the equivalent of blueprints. But we don't know what level of detail we should reveal here.

When I have produced UML-centric designs, they're both "too technical" and "not detailed enough for coders". A critique I've never understood.

Program Specifications -- again, I'm guessing -- are for "coders" to write code from. To write such a thing, I have to visualize some code and describe that code in English.

Let's consider that slowly. To write programming specifications, I have to

  1. Visualize the code they're supposed to write.
  2. Describe that code in English.

Wouldn't it be simpler to just let me code it? It would certainly take less time.

Detailed Design Flaws

First, let me simplify things by mashing "Detailed Design" and "Specification" together, since they seem to be the same thing. A designer (me) has to reason out the classes required. Then the designer has to pick appropriate algorithms and data structures (HashMap vs. TreeMap). Then the designer has to either draw a UML picture or write an English narrative (or both) from which someone else can code the required class, data structure and algorithm. Since you can call this either name, the names don't seem to mean much.

I suppose there could be a pipeline from one design document at a high level to other designs at a low level. But if the low-level design is made difficult by errors in the high-level design, the high-level designer has to rework things. Why separate the work? I don't know.

When handing things to the coders, I've had several problems.

  1. They ignore the design and write stuff using primitive arrays because they didn't understand "Map", much less "HashMap" vs. "TreeMap". In which case, why write detailed design if they only ignore it? Remember, I provided specifications that were essentially, line-of-code narrative. I named the classes and the API's.
  2. They complain about the design because they don't understand it, requiring rework to add explanatory details. I've gone beyond line-of-code narrative into remedial CS-101. I don't mind teaching (I prefer it) but not when there's a silly delivery deadline that can't be met because folks need to improve their skills.
  3. They find flaws in the design because I didn't actually write some experimental code to confirm each individual English sentence. Had I written the code first, then described it in English, the description would be completely correct. Since I didn't write the code first, the English description of what the code should be contained some errors (perhaps I failed to fully understand some nuance of an API). These are nuances I would have found had I actually written the code. So, error-free specifications require me to write the code first.

My Point is This.

If the design is detailed enough to code from -- and error free -- a designer must actually write the code first.

Indeed, the designer probably should simply have written the code.

Architecture Isn't Like That

Let's say we have a software design that's detailed enough to code from, and is completely free from egregious mistakes in understanding some API. Clearly, the designer verified each statement against the API. I'd argue that the best way to do this is to have the compiler check each assumptions. Clearly, the best way to do this is to simply write the code.

"Wait," you say, "that's going too far."

Okay, you're right. Some parts of the processing do not require that level of care. However, some parts do. For instance, time-critical (or storage-critical) sections of algorithms with many edge cases require that the designer build and benchmark the alternatives to be sure they've picked the right algorithm and data structure.

Wait.

In order for the designer has absolute certainty that the design will work, they have to build a copy that works before giving it to the coders.

In architecture or manufacturing, the construction part is expensive.

In software, the construction part does not exist. Once you have a detailed design that's error-free and meets the performance requirements, you're actually done. You've created "prototypes" that include all the required features. You've run them under production-like loads. You've subjected them to unit tests to be sure they work correctly (why benchmark something that's incorrect?)

There's nothing left to do except transition to production (or package for distribution.)

Software Design

There's no "detailed design" or "programming specifications" in software. That pipeline is crazy.

It's more helpful to think of it this way: there's "easy stuff" and "hard stuff".

  • Easy Stuff has well-understood design patterns, nothing tricky, heavy use of established API's. The things where the "architectural" design can be given to a programmer to complete the design by writing and testing some code. Database CRUD processing, reporting and analysis modules, bulk file processing, standard web form processing for data administration, etc.
  • Hard Stuff has stringent performance requirements, novel or difficult design patterns, new API's. The things where you have to do extensive design and prototyping work to resolve complex or interlocking issues. By the time there's a proven design, there's also code, and there's no reason for the designer to then write "specifications" for someone to reproduce the code.

In both cases, there are no "coders". Everyone's a designer. Some folks have one design strength ("easy stuff", well-known design patterns and API's) and other folks have a different design strength.

There is no "construction". All of software development is design. Some design is assembling well-known components into easily-visualized solutions. Other design is closer to the edge of the envelope, inventing something new.
Published at DZone with permission of Steven Lott, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Jose Maria Arranz replied on Fri, 2010/03/12 - 8:46am

I think you has got into problems with this article because you have disclosed some dirty secrets which have been hidden for years... monkeys cannot be developers, developers cannot be like monkeys.

(end of irony) 

Nice article, plenty of common sense.

 

David Day replied on Fri, 2010/03/12 - 8:53am

Having been the manager of a specialized team of Application Architect/Designers, I can agree heartily with Mr. Lott.  In addition to the points he makes, the developers who depend on this specialized team end up feeling powerless to make design decisions and begin to slack off.  If/when I get the chance to do it again, I'll make architecture/design the expected responsibility of every developer, and use the senior folks on the team to resolve disagreements.

Senthil Balakrishnan replied on Fri, 2010/03/12 - 9:46am

Hi Lott,
I sort of liked your argument of "design" & "no
construction".
But In a practical world
1.  Of software engineering, the reusability is still
limited to the design patterns and libraries and not the level of reusability
we would expect as coders. I talked about this in my article - http://java.dzone.com/articles/code-template-repository#comment-27073\
2.  Not all the coders are matured enough to understand the
significance of design and reusability.

Josh Marotti replied on Fri, 2010/03/12 - 9:17am

Great article!  I've always said I refuse to code a design by someone who doesn't code.


Too many "enterprise engineers" work more in Word and Vizio than an IDE!  Maybe it works at a 1000 foot level, but once you get to the design of the code, if you haven't coded in the past 2 or 3 days, you are already out of touch with coding enough to need help with design...

David Robison replied on Fri, 2010/03/12 - 10:12am

An interesting article. I think software has suffered by being viewed as engineering rather than art. When creating a significant work of art and artist may elect to allow other artists of lesser talent to contribute to simpler parts of his work while retaining the most significant and intricate parts for himself. They are all artists but their contribution is according to not only their skill but also their talent. Our western world view predisposes us to think that we can educate people to become software "engineers" and, while we can teach the skills of programming, good programmers possess an inherent talent that is not and cannot be taught. You can teach someone to sculpt but that will not make them a great artist, it takes something that is unique and intrisict within them. Thanks for your article, it echos with my own experience and ideas. David

Andrew Arrigoni replied on Fri, 2010/03/12 - 10:17am

Simply put, there aren't enough good designers/developers/etc. to make this work. There is a greater demand for software than there is supply of competent, smart and passionate developers. Therefore, those companies who cannot afford, don't know how or refuse to attract the CSP Developers will end up trying to make do with larger groups of more mediocre developers.

Chao Liu replied on Fri, 2010/03/12 - 10:28am

Back to 6 years ago when I just graduated from U and start as a java developer, I read detailed design doc and think: dude, if you have time to write this, why don't you just code it!

Eran Harel replied on Fri, 2010/03/12 - 10:52am

I love this article :D

Since we can't do ALL the work, what  I tend to do is to:

  • Prototype as much as I can
  • Make sure the HL design, is well understood.
  • Let the “coders” write their own detailed design, but carefully guide them towards the right direction.


I can’ say it always work, especially the last item. If we were all SW craftsmen, then there wouldn’t be a problem. Right?

Andy Leung replied on Fri, 2010/03/12 - 2:53pm in response to: Chao Liu

So what do you think now? Honestly, if you still think this way now, I guess you are still enjoying your fresh grad. moment 6 years ago. If you take a look at it, it's different. Design takes more time in thinking, not others. It makes sense when you work on your own but not in a team of 10+ developers. Designers don't check in/out code (of course including merge branches), don't debug, don't open IDE (may take 30 seconds to do so with all plugins), don't alt-tab whole day (between IDE and web browser if you are doing web with cheap boss who doesn't give you dual monitors), don't retrofit code, don't run any functional (e.g. unit) and non-functional (e.g. profiling) testing, don't fix bug, and finally don't spend 10 days to setup 20 different environments with 200 dependencies on local computer (and found out your cheap boss, who is cheap enough, not allowing you to upgrade to 4GB of RAM so your program hits OutOfMemoryException all the time). So how much does it take you to do all these when you are involved in more than 1 project at a time? So what do these designers (or technical lead sometimes) do? They do the following: 1. Spend time on telling your boss how much time they estimate you need to finish all iterations (code, deploy, debug, fix, re-deploy). 2. Create these so called Not-So-Useful-Documents-When-You-Can-Spend-The-Same-Time-Coding-This documents. Don't blame them, this is compliance issue. 3. Walk through these with architect to make sure their design meets the architectural goal. 4. Walk through these with developers to make sure they agree because developers may be able to think of some gaps. 5. Spend time on fixing documents and re-iterate the entire process. 6. Walk through these with QAs to make sure they understand his design, and think about any gaps from QA perspective. 7. Walk through QA's test plan, including strategy, and test cases to make sure these test cases can cover all areas to thoroughly prove his design is flawless. 8. Spend time on understanding the architecture (sometimes, it depends on who the architect is, someone may be drawing kindergarden quality pictures). 9. Involve in all sorts of status meeting with PM and QA sessions. 10. And finally, they need a life too!!! (excluding those who smoke 10 packs a day with VIP pass to Starbucks since they buy 20 coffee a day). Don't get me wrong, I don't disagree with many of you that, many technical lead, architect, and designers don't really know how to do their jobs. But I think this is more of a personality issue than "Role" issue. I bet you, even these people go do some other jobs, they still can't do it right.

Jilles Van Gurp replied on Fri, 2010/03/12 - 3:25pm

There is one point you are overlooking. Development is iterative. Iterations touch all development artifacts, including architecture and design. That's why developers tend to be very reluctant to maintain architecture and design documents: the overhead of maintaining those is unacceptable. 

I did a Ph. D. on software architecture and object oriented framework before I became a developer (oddly most academics in this area lack development experience). Being an academic, I had the opportunity to look around in several cool software companies. I discovered a few patterns:

- Everybody was really interested in UML, Rational Rose, and Rational Unified which were the silver bullets of the momen. There was a lot of BS floating around about how those were going to revolutionize the industry. 

- While interested, people didn't seem to have gotten very far using the associated tools and processes. I'm talking major big software companies here. They were telling me they were doing UML and Rational Unified. What I saw instead was waterfall (or at best vaguely iterative), a lot of unused Rational Rose license and a handful of word documents with not terribly informative UML diagrams pasted in them.

- The way to get details about whatever system I was studying at the time was a good old 1 on 1 with a lead developer or architect. I met some really good software engineers this way. You recognize a good engineer instantly: they speak with authority, they know their stuff, and everybody else mentions you should talk to them. Forget about design documents, talk to the guy.

I'm now in industry for several years practicing what I used to preach. I see a lot of bad code around me: people coming from universities still can't program but think they can (absolutely the worst combination, guilty of this myself at one point in my career). Software development is a craft you learn on the job and good developers quickly adapt. Pair programming is definitely a great way to help people get up to speed. On the other hand, I strongly believe people need the room to make mistakes as well. I know I learned a lot this way. You learn by making mistakes and studying other people's code and mistakes. There's nothing like hands on experience. By showing people the right way to do things, you get them to make more interesting mistakes. Eventually they are making similar mistakes as you do. That's the flip side of the coin: you are never done learning. Lead by example, always be open to being corrected on your own mistakes.

Thierry Milard replied on Fri, 2010/03/12 - 4:24pm

Yes,

I must agree that I was ten years ago a completly  believer of UML, design, ect....

But now, I changed a lot.Those tools are perhaps overated...

Nothing beats the sweet of the code.

 

I think coder is like a craftman.

Making a software is a lot like art. 

UML drawings, design coding should be use by the same guy. 

 You make a good point.

Endre Varga replied on Fri, 2010/03/12 - 5:20pm

Software development IS engineering. Just a different kind. Also, don't think that bridges and buildings are developed according to the original plan! Just search your local newspaper for constructions that turned out to be more expensive and now being late.

Nicolas Bousquet replied on Fri, 2010/03/12 - 6:38pm

From my understanding, there is distinct important part for a software project :

- ANALYSING 

Understand and detect the use cases, find the users and what they really need. Don't hesitate to interview them, even many of them if possible.  Build iteratively a business model. Make sure you understand the business rules. Make some class and acitivity diagram. Don't use an UML tool to do this (or if you really want, use UML let or something like that). Instead, draw them on paper, or on board. Discussing with your users and other software engineers. To include theses documents into your GED or wiki, use a scanner or a camera, don't waste your time make some beautifull diagram that you'll have to completly change two days later !

Don't do it alone. You have  to discuss your choice with users, domain experts and experimented software designer. You need no more than a few peoples for most of the work, but you need them. If you do all by yourself, you may miss some important things.

You need to understand the limit of the software, what must be done, what is a waste of time... And what could be done later.

You can also make GUI sketch on paper show them to your users. It will help a lot for the ergonomics. But it will also help the user and you see what is "missed", or understand what you'll really need to do

Keep all interviews, diagrams (sketch by hand), meetings note, design decision or discussion. And of course GUI sketchs. Don't waste time doing this "beautifully". Just keep all of you work and make sure it's correctly indexed (subject, involved peoples, tags , date) so that youor your team mates can easily find them.

When you have all of that, make some fondamuntal design documents :

- Major and critical use cases

- Description of domain model

- Major business processes

- Major business rules

- ...

 Then make a specification of what the software will have to be. Be consise but precise. Major requirements both technicals (the task has to be done withing 1 second, the system must integrate withing existing IT) or functionnal have to be included. Name them with number and be sure they are unique. Say what it do, never how. Be sure everybody involved with the project agree and validate the document.

CONCEPTION

Now that you know what you'll have to do, you sketch the major design of your software. You'll have to be sure your major design fit with all the defined requirements. Again you CAN'T do it alone.

You need other experienced designer, discuss... But then you shall only include global design major pattern, convention, frameworks to use.

Do not write the code or algorythms. It's a waste of time and it is not your work ! The conception has to be lightweight consise and help dispatch the work. So you'll have "modules" or "services" each dealing with some part of the software. Determining essential API design is a good thing.

 It should also help every one understand what they will have to do... And how everything interract.

CODING AND DETAILLED CONCEPTION

Then goes the coding... It's only the developper that will do the fine design. If they 'll implement this with pattern X or Y and all. If they prefer Array, Linked list, graphs or HashMap. If they do using procedural programming, relying mainly of OO, data oriented design or functionnal coding... 

And it's them that will ensure that the code can be easily understood, maintened and extended. So they include comments, some document with the exact specification of interchanges beetweens modules and all.

Use "free" document as far as possible : javadoc, generated graph call or class diagram etc. Use unit test as exemple of usage. Don't document everything, just the main things.

Most of the time the design shall not try to do the fine design except maybe for the most tricky part and if your software has something bery complex and unusual or innovative. Instead, YOU NEED, great developpers. Not super hero or brilliant. Just good. Feed them with good global design, help them using the right tool for the jobs and focussing on best pratices and methods.

But let them do they work.

 

AND LOOP

Yes it's itterative, don't do waterfall. Do it iteratively. Implement the most important and fundamentals things first. Take feedback, adapt, evolve, refactor.

 

 

 

 

João Martins replied on Fri, 2010/03/12 - 8:54pm

Nice Article.

Building, Maintaining and Administrating Software is no picnic (its actually hard).

In some scenarios the hard part is keeping software running smoothly, knowledge on software is volatile... and having documentation, at various detailed levels, may help.

In construction, building, maintaning and administrating is easier, there physical materials and characteristics that are easier to discover and understand. It seems allot easier to ensure a building keeps "running" smoothly than some software product/component/part...

On a crazy non related topic: Long live the Cork! When in work you see something that's wrong, if does not make your live allot easier and is not you responsibility, just shut up, or it will become your responsibility! :)

Ed Lee replied on Sun, 2010/03/14 - 5:42pm

It's a very interesting article. From my experience, what ended up happening is the implementation is usually different to the design, unless you specify every little details down to class names, method names, and argument names ... etc, and the developers cannot deviate from it. However, in this case you're assuming the developers have no creativity at all and they cannot think themselves.

What I've done in recently is to only allow myself as an architect to design the components, the design patterns and principles that the developers must implement. It is their responsibility to produce APIs, Sequence diagrams, Javadoc throughout the development phase. The team then goes through an iterantive peer-review process to validation that the code does match the "documents" that the developers produced and is inline with the design principles. 

This will help the developers to think about design while they code, as well as given them the opportunity to grow and understand why certain design patterns was created. 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.