Plan to Write Big Software - and You Have Already Lost
This idea came to me as a tweet, but was way too long for a tweet. As it turns out, it’s also something I’ve been saying a lot lately, since it’s the answer to one of the very common arguments against dynamic languages.
The argument usually goes like this: “Dynamically typed languages are fine for smaller programs and simple web applications, but if you’re building something big, something that will be several millions of lines of code, you really need all the tools you can only get from a statically typed language”. Where the statically typed language mentioned is typically Java. The argument is still common enough without any specific language name mentioned though.
Interestingly, there are several problems and fallacies in this seemingly simple argument. I’m not going to tackle all of them here, but just focus on the beginning. Namely the idea that you are building something big. First of all, how do you know that? Have you done anything like it before? And how do you know it would be big in a different language? How do you know you can’t decouple the application in such a way you won’t have to build anything big? There are so many assumptions that can be questioned here. But at the end of the day, my glib answer that summarizes this usually goes something like this:
“Optimizing for your software project becoming big is the same as optimizing a car to hit a rock wall - you are optimizing for failure”
I firmly believe that becoming big is really failure. Once you have a big enough project you have lost. It might still work, but the cost will be extreme, and maintaining it will be a large burden too.
This is the reason I like agile. It emphasizes small, working pieces all the time. If you work with code this way, you can’t really become big. Instead, your project will be forced to be modularized and divided into smaller, more logical components that are highly cohesive and decoupled from each other.
(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)
Comments
Liam Knox replied on Tue, 2009/10/13 - 8:04am
Please clarify ?
Of course there have been a lot of falacies about what you need to build a 'big' system, but to the think you could some how develop say a multi asset trading system with a few Agile moves and a splash of JRuby is absolute madness.
Nathan Dolan replied on Tue, 2009/10/13 - 8:10am
Gilbert Le Blanc replied on Tue, 2009/10/13 - 8:33am
Software ought to be as big, as complicated, as extensive as necessary to solve the problem.
But no bigger, no more complicated, or no more extensive.
Nicolas Bousquet replied on Tue, 2009/10/13 - 8:55am
What your point here ? That nobody has big or huge product to manage ? Nobody has made Oracle DB ? Nobody has made MS Office ? Linux Kernel ? Amadeus plane réservation system ? Or Amazon web site ?
The point of staticaly typed language is that many bug are detected has you type the code. You dont have to run the program / unit test or anything to detect them. You simply detect them instantely and correct them in no time.
Plus languages like JAVA / C# / C++ have far better IDE support allowing many thing to be automated and fast navigation into the code.
And so what ?
Tony Siciliani replied on Tue, 2009/10/13 - 9:25am
You seem to be stating that any complex project is deemed to failure, and that becoming "agile" somehow take that complexity away.
Which "complexity" you are referring to?
Essential complexity is the difficulty inherent to the problem at hand. Can't be really avoided if you want to solve the problem.
Accidental complexity OTOH is the complexity you introduced in designing your solution. An over-engineered solution will introduce more complexity than it solves. You could in theory limit that, but I fail to see how "going agile" helps here. Programers didn't wait for the agile movement to design complex systems in modules. It's just sound software design principles that guide them.
Andrew Arrigoni replied on Tue, 2009/10/13 - 10:00am
Jonathan Curran replied on Tue, 2009/10/13 - 10:06am
Let's not forget that software is inherently complex regardless of language (some more than others). Second, coupling is not bound to a particular type of language. It simply boils down to how you have desgined your application. Third, you don't define what you are measuring (big project). What does it consist of, why is it so big, can it be made simpler?
If you had used a concrete example (e.g. a particular domain or application) instead of generalizing you might havea point.
Fab Mars replied on Tue, 2009/10/13 - 10:25am
Shaw Gar replied on Tue, 2009/10/13 - 11:14am
Shaw Gar replied on Tue, 2009/10/13 - 11:20am
in response to:
Liam Knox
Alex Santoro replied on Tue, 2009/10/13 - 4:08pm
The author seems to state that big = unmanageable and therefore big = failure, and equates that with milliions of lines of code, without any regard for code organization, modularity, etc.
But if we stick with big = unmanageble, I will make the following unverified assertion: "big programs in dynamic languages are at least one order of magnitude smaller (in lines of code) than big programs in static languages."
Granted, you usually can do more with fewer lines of code in such languages, but the lack of typing will introduce confusion and complexity much earlier on.
Gilbert Herschberger replied on Tue, 2009/10/13 - 4:18pm
I appreciate what the author is trying to say. It makes me think. And I fully embrace, support and advocate the idea that big is bad. It is like building a car from one block of steel. It can be done, but it is going to be very expensive. It is like building a house from one piece of wood. It can be done, but why? It is like building a high-rise office building from one prefabricated piece of concrete. It can be done, but what kind of heavy equipment would it take to put it in place? With real world objects, we see that big is bad.
In software, some are at this very moment building a operating system with all its drivers and applications as a single executable file. It can be done, but it is a very bad idea. Big means monolithic, or everything is made of the same stuff. Big is bad. Big is ugly. Big means that you choose a programming language for this feature based on the needs of some other feature. Big means that dependencies are unmanaged. A feature is allowed to depend upon any other feature.
Time passes while we are building software. At the end of a two year project with two million lines of code, we produce a number of different versions of each feature. Resolving dependencies on versions in a strongly typed language requires much effort. OSGi is required, along with all of its complexity, to defeat the strongly typed Java language in a big project.
Programming in the large is exponentially more complex than programming in the small. So, why program in the large when we don't have to? Again, I point to OSGi as an example. It enables us to build a very large application--without thinking big. Each module manages part of the complexity so that complexity can be managed. OSGi provides an environment for thinking small, while building big. And the Java platform makes OSGi possible. Amazing, isn't it?
How should we build software? We should choose the best language for each feature. We should choose the best from a collection of languages, one feature at a time.
How do we build software? We think big and fail.
Our way of thinking is the problem. We must stop thinking big.
Liam Knox replied on Tue, 2009/10/13 - 5:27pm
in response to:
Shaw Gar
Basically I cant understand the authors point. If you approach things in such a black and white non specified way we wouldnt of got out of the caves let alone been able to post to this forum.
Similarly if it wasnt for the success of certain 'big' projects we wouldnt be in this situation either.
I think the author should really clarify his points in greater detail. Is this a discussion about appropriate tools for given jobs or something else like how certain big projects fail or go massively over budget ( commonly public sector )?
Both of these I think are sensible discussion subjects, 'big == crap' , is not.
Liam Knox replied on Tue, 2009/10/13 - 5:50pm
in response to:
Gilbert Herschberger
I would say the Spring framework is a pretty 'big' framework, though it is the complete anit-failure in every sense from technical excellence to comercial success.
Why it works is down to good design and modularisation but that does not reduce its overall size, it just means it is more maintainable and extensible
But regardless big designs and big systems exist and will always. Some succeed better than others, some are componentized better than others.
Basically I think if the autor simply said, 'break a problem into sub tasks', 'choose the right tools for the job', it would be more appropriate. We should all know this and it would be a complete waste of an article, but none the less altogether more correct and less wasteful than this one.
I really dont know how some articles manage to get posted, dont they get vetted for some level of sanity?
Mrmagoo Magoo replied on Wed, 2009/10/14 - 12:01am
I think there is some value you in this is you look beyond the simple minded and somewhat nasty retorts of some of the posts above.
I have witnessed several projects that became big ugly failures in the end for PRECISELY this reason. Every time they came across a business requirement they immediately went and coded the most generic solution they could come across.
They coupled that with horribly complex class hierarchies just to save a few lines of repeated code or to tie two similar objects together or allow extensibility in the future if required.
The result was the ugly nightmare you would expect. They always assumed things would get "big" and thus made everything more complicated in the process. An extreme example(pun not intended), but I am sure we have all seen less extreme versions of this.
Shaw Gar replied on Wed, 2009/10/14 - 12:24am
Karl Peterbauer replied on Wed, 2009/10/14 - 3:29am
Liam Knox replied on Wed, 2009/10/14 - 6:02am
in response to:
Mrmagoo Magoo
Furthermore different people will assume different meanings in the unspecific nature of the article and then the whole discussion becomes a complete waste of time, money, bytes, evolution etc, etc.
Sure I've projects go horrendously wrong but in no way would I mention 'big' as the reason for this. It infers absolutely nothing.
Shaw Gar replied on Wed, 2009/10/14 - 10:40am
Mario Rabe replied on Wed, 2009/10/14 - 12:10pm
I know the mental trouble you're in, if you refer those planing-problems. Actually it's a mental problem between an structured and an unstructured approach. There are extremes in both directions and that are the people who get trouble.
It is -usually- wrong to say "I don't define anything, as it turns out while I write it", chances are good your encapsulation is bad. It is -usually- wrong, too, to say "I define the _whole_ software upon front", chances are good you never start to implement, cause you continue to engineer ;)
As most times in life you need a way between. So most people prefer to think of manageable tasks and group them to "services". These services _have_ an interface, no matter how it is coded. You cannot get around the "how to invoke/request/...".
If you keep that knowledge in your head, you actively make it harder to others and you self to do something with your software and to reuse it.
Tests and interfaces do a great job here. In java we have both. And as the software (which wasn't planed as a big one) grows, it keeps manageable. So I think my personal way to build software is very similar to yours, but I use more machine-readable documentation inside my code to assist my own rusty brain while thinking about the real problems.
If you like the other way, you are free to do so. Only experience can show up in which cases which way is better.
Remember: All our problems are in our head, not the language, not the tools, nothing else. We get paid to solve a specific problem, and we should do that as good as we know.
Michael Mccray replied on Wed, 2009/10/14 - 12:19pm
Jeroen Wenting replied on Thu, 2009/10/15 - 6:27am
Gilbert Le Blanc replied on Mon, 2009/10/19 - 7:30am
in response to:
Gilbert Herschberger
I like your reply better than the article. :-)
Managing complexity is an important topic. When you read books on the subject of managing complexity it usually comes down to high module cohesion and loose coupling.
I've not built huge projects with Java. The biggest Java project I've worked on is a 120 class Eclipse plug in. Most of the code extends existing Eclipse classes.
I have built huge projects with Cobol, CICS, and DB2. This class of project is mostly data entry, management, and retrieval.
I would like to read of the experiences of a Java development in the order ot 1,000 to 10,000 classes or more.