Ola Bini is a Swedish developer working for ThoughtWorks. His daily job includes working on JRuby, starting up a Swedish ThoughtWorks office and mucking around with Java and Ruby. In his spare time he spends most time on his language Ioke, working on one of several other open source projects or reading science fiction. Ola has presented at numerous conferences, such as JavaOne, Javapolis, JAOO, RailsConf, TheServerSide Java Symposium and more. He is the author of APress book Practical JRuby on Rails Ola is a DZone MVB and is not an employee of DZone and has posted 45 posts at DZone. You can read more from them at their website. View Full User Profile

Plan to Write Big Software - and You Have Already Lost

10.13.2009
| 7103 views |
  • submit to reddit

This idea came to me as a tweet, but was way too long for a tweet. As it turns out, it’s also something I’ve been saying a lot lately, since it’s the answer to one of the very common arguments against dynamic languages.

The argument usually goes like this: “Dynamically typed languages are fine for smaller programs and simple web applications, but if you’re building something big, something that will be several millions of lines of code, you really need all the tools you can only get from a statically typed language”. Where the statically typed language mentioned is typically Java. The argument is still common enough without any specific language name mentioned though.

Interestingly, there are several problems and fallacies in this seemingly simple argument. I’m not going to tackle all of them here, but just focus on the beginning. Namely the idea that you are building something big. First of all, how do you know that? Have you done anything like it before? And how do you know it would be big in a different language? How do you know you can’t decouple the application in such a way you won’t have to build anything big? There are so many assumptions that can be questioned here. But at the end of the day, my glib answer that summarizes this usually goes something like this:

“Optimizing for your software project becoming big is the same as optimizing a car to hit a rock wall - you are optimizing for failure”

I firmly believe that becoming big is really failure. Once you have a big enough project you have lost. It might still work, but the cost will be extreme, and maintaining it will be a large burden too.

This is the reason I like agile. It emphasizes small, working pieces all the time. If you work with code this way, you can’t really become big. Instead, your project will be forced to be modularized and divided into smaller, more logical components that are highly cohesive and decoupled from each other.

From http://olabini.com/blog

Published at DZone with permission of Ola Bini, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Liam Knox replied on Tue, 2009/10/13 - 8:04am

I think this is one of the most absurd articles I have seen posted. What do you clasify as 'big' ? An operating system, the Appolo program, DB2 ?

Please clarify ?

Of course there have been a lot of falacies about what you need to build a 'big' system, but to the think you could some how develop say a multi asset trading system with a few Agile moves and a splash of JRuby is absolute madness.

Nathan Dolan replied on Tue, 2009/10/13 - 8:10am

I firmly believe that becoming big is really failure.
This is a rather stange belief... is Oracle database a failure? Or MS Office? Maybe they should be re-written in JRuby ;)

Gilbert Le Blanc replied on Tue, 2009/10/13 - 8:33am

Software ought to be as big, as complicated, as extensive as necessary to solve the problem.

But no bigger, no more complicated, or no more extensive.

Nicolas Bousquet replied on Tue, 2009/10/13 - 8:55am

What your point here ? That nobody has big or huge product to manage ? Nobody has made Oracle DB ? Nobody has made MS Office ? Linux Kernel ? Amadeus plane réservation system ? Or Amazon web site ?

The point of staticaly typed language is that many bug are detected has you type the code. You dont have to run the program / unit test or anything to detect them. You simply detect them instantely and correct them in no time.

Plus languages like JAVA / C# / C++ have far better IDE support allowing many thing to be automated and fast navigation into the code.

And so what ?

 

Tony Siciliani replied on Tue, 2009/10/13 - 9:25am

You seem to be stating that any complex project is deemed to failure, and that becoming "agile" somehow take that complexity away.

Which "complexity" you are referring to? 

Essential complexity is the difficulty inherent to the problem at hand. Can't be really avoided if you want to solve the problem.

Accidental complexity OTOH is the complexity you introduced in designing your solution. An over-engineered solution will introduce more complexity than it solves. You could in theory limit that, but I fail to see how "going agile" helps here. Programers didn't wait for the agile movement to design complex systems in modules. It's just sound software design principles that guide them. 

 

Andrew Arrigoni replied on Tue, 2009/10/13 - 10:00am

Maybe I just haven't looked hard enough but I haven't found an IDE that can auto-complete all my code and display docs inline as well as NetBeans does for Java for a dynamic language like Ruby or Groovy. It works, some of the times, but not to the extent that I like. I simply am too dumb to memorize all the methods and parameters for all the libraries I use on a day to day basis such that I can type them without looking them up. Maybe I just need a third monitor for Dynamic Programming language so I can have the docs up in front of me all the time. Do you think I could convince my boss to get me a third monitor this way? :D

Jonathan Curran replied on Tue, 2009/10/13 - 10:06am

Let's not forget that software is inherently complex regardless of language (some more than others).  Second, coupling is not bound to a particular type of language. It simply boils down to how you have desgined your application.  Third, you don't define what you are measuring (big project). What does it consist of, why is it so big, can it be made simpler?

If you had used a concrete example (e.g. a particular domain or application) instead of generalizing you might havea point.

Fab Mars replied on Tue, 2009/10/13 - 10:25am

...absurd...

Shaw Gar replied on Tue, 2009/10/13 - 11:14am

@All Impatient commentators who asked what is "big" I think the author has clearly mentioned it in the article "but if you’re building something big, something that will be several millions of lines of code" I'm not sure if statically typed or dynamically typed project has ever had several millions of lines of code? If indeed such a project exists, it would be very strange assumption to think the project itself would not have been modularized into separate sub projects or even possibly sub products. I don't get how being agile will automatically make a project more modularized? I thought that would have to be a design decision? If agile approach makes projects more modularized as you say, then of course the number of lines of code may not hit several millions, but then my argument is why would you assume that only agile can do that? Maybe that could be from your limited world view? There is a saying "When the cat closes its eyes, it thinks the world has gone dark". Such is your position!

Shaw Gar replied on Tue, 2009/10/13 - 11:20am in response to: Liam Knox

If the OS/Appolo/DB2 had several million lines of code, then yes, it's considered big.. as per the author... If I try to build "multi asset trading system" with EJB 2.x, that could be regarded as madness too ;-) wouldn't it Liam? I mean, it basically boils down to choosing the right stack for the problem.. static/dynamic isn't really the way to determine. That said, it's possible that majority of the big projects were somehow linked to static language !!!

Alex Santoro replied on Tue, 2009/10/13 - 4:08pm

The author seems to state that big = unmanageable and therefore big =  failure, and equates that with milliions of lines of code, without any regard for code organization, modularity, etc.

 But if we stick with big = unmanageble, I will make the following unverified assertion: "big programs in dynamic languages are at least  one order of magnitude smaller (in lines of code) than big programs in static languages."

 Granted, you usually can do more with fewer lines of code in such languages, but the lack of typing will introduce confusion and complexity much earlier on.

 

 

 

 

Gilbert Herschberger replied on Tue, 2009/10/13 - 4:18pm

I appreciate what the author is trying to say. It makes me think. And I fully embrace, support and advocate the idea that big is bad. It is like building a car from one block of steel. It can be done, but it is going to be very expensive. It is like building a house from one piece of wood. It can be done, but why? It is like building a high-rise office building from one prefabricated piece of concrete. It can be done, but what kind of heavy equipment would it take to put it in place? With real world objects, we see that big is bad.

In software, some are at this very moment building a operating system with all its drivers and applications as a single executable file. It can be done, but it is a very bad idea. Big means monolithic, or everything is made of the same stuff. Big is bad. Big is ugly. Big means that you choose a programming language for this feature based on the needs of some other feature. Big means that dependencies are unmanaged. A feature is allowed to depend upon any other feature.

Time passes while we are building software. At the end of a two year project with two million lines of code, we produce a number of different versions of each feature. Resolving dependencies on versions in a strongly typed language requires much effort. OSGi is required, along with all of its complexity, to defeat the strongly typed Java language in a big project.

Programming in the large is exponentially more complex than programming in the small. So, why program in the large when we don't have to? Again, I point to OSGi as an example. It enables us to build a very large application--without thinking big. Each module manages part of the complexity so that complexity can be managed. OSGi provides an environment for thinking small, while building big. And the Java platform makes OSGi possible. Amazing, isn't it?

How should we build software? We should choose the best language for each feature. We should choose the best from a collection of languages, one feature at a time.

How do we build software? We think big and fail.

Our way of thinking is the problem. We must stop thinking big.

Liam Knox replied on Tue, 2009/10/13 - 5:27pm in response to: Shaw Gar

To build anything in EJB 2.x or even 3.x I would deem madness, but that is beside the point ;-).

Basically I cant understand the authors point. If you approach things in such a black and white non specified way we wouldnt of got out of the caves let alone been able to post to this forum.

Similarly if it wasnt for the success of certain 'big' projects we wouldnt be in this situation either.

I think the author should really clarify his points in greater detail. Is this a discussion about appropriate tools for given jobs or something else like how certain big projects fail or go massively over budget ( commonly public sector )?

Both of these I think are sensible discussion subjects, 'big == crap' , is not.

Liam Knox replied on Tue, 2009/10/13 - 5:50pm in response to: Gilbert Herschberger

I think you are mixing and argument that componitisation as a better way of implementation and the question of what is 'big' and how does a big system/design fail.

I would say the Spring framework is a pretty 'big' framework, though it is the complete anit-failure in every sense from technical excellence to comercial success.

Why it works is down to good design and modularisation but that does not reduce its overall size, it just means it is more maintainable and extensible

But regardless big designs and big systems exist and will always. Some succeed better than others, some are componentized better than others.

Basically I think if the autor simply said, 'break a problem into sub tasks', 'choose the right tools for the job', it would be more appropriate. We should all know this and it would be a complete waste of an article, but none the less altogether more correct and less wasteful than this one.

I really dont know how some articles manage to get posted, dont they get vetted for some level of sanity?

Mrmagoo Magoo replied on Wed, 2009/10/14 - 12:01am

I think there is some value you in this is you look beyond the simple minded and somewhat nasty retorts of some of the posts above.

I have witnessed several projects that became big ugly failures in the end for PRECISELY this reason. Every time they came across a business requirement they immediately went and coded the most generic solution they could come across.

They coupled that with horribly complex class hierarchies just to save a few lines of repeated code or to tie two similar objects together or allow extensibility in the future if required.

The result was the ugly nightmare you would expect. They always assumed things would get "big" and thus made everything more complicated in the process. An extreme example(pun not intended), but I am sure we have all seen less extreme versions of this.

Shaw Gar replied on Wed, 2009/10/14 - 12:24am

It is interesting that those who agree with the author, tend to use the same word big without attaching a context to it. For me, something that is big, could be very small for someone else. It all depends on your skill/capability, choosing the right solution etc. However, I wouldn't still call it big, may be the the right term for it would be mess/bad solution. Mr Magoo quotes a typical case of poor solution choice and execution, based on ignorance of the reality. That does not make the whole thing as getting big, would it? I don't think so.

Karl Peterbauer replied on Wed, 2009/10/14 - 3:29am

With a dynamically typed language, even a few hundred lines of code can get you into a mess.

Liam Knox replied on Wed, 2009/10/14 - 6:02am in response to: Mrmagoo Magoo

I don't think the retorts are in anyway simple minded or nasty. If people need to invent some context into what the words actually mean in the article then the article is fundamentally floored in the first place. It does not cleary set out what it is trying to convey.

Furthermore different people will assume different meanings in the unspecific nature of the article and then the whole discussion becomes a complete waste of time, money, bytes, evolution etc, etc.

Sure I've projects go horrendously wrong but in no way would I mention 'big' as the reason for this. It infers absolutely nothing.

Shaw Gar replied on Wed, 2009/10/14 - 10:40am

Plan to bash up "Big Software" with out a context - and You Have Already Lost!!!

Mario Rabe replied on Wed, 2009/10/14 - 12:10pm

Olabini,

I know the mental trouble you're in, if you refer those planing-problems. Actually it's a mental problem between an structured and an unstructured approach. There are extremes in both directions and that are the people who get trouble.

It is -usually- wrong to say "I don't define anything, as it turns out while I write it", chances are good your encapsulation is bad. It is -usually- wrong, too, to say "I define the _whole_ software upon front", chances are good you never start to implement, cause you continue to engineer ;)

As most times in life you need a way between. So most people prefer to think of manageable tasks and group them to "services". These services _have_ an interface, no matter how it is coded. You cannot get around the "how to invoke/request/...".

If you keep that knowledge in your head, you actively make it harder to others and you self to do something with your software and to reuse it.

Tests and interfaces do a great job here. In java we have both. And as the software (which wasn't planed as a big one) grows, it keeps manageable. So I think my personal way to build software is very similar to yours, but I use more machine-readable documentation inside my code to assist my own rusty brain while thinking about the real problems.

If you like the other way, you are free to do so. Only experience can show up in which cases which way is better.

Remember: All our problems are in our head, not the language, not the tools, nothing else. We get paid to solve a specific problem, and we should do that as good as we know.

Michael Mccray replied on Wed, 2009/10/14 - 12:19pm

I think that you are saying don't make something bigger than it should be, but seriously, you can't mean that big  software applications are failures?  any operatig system, any office suite, any IDE, the world is not that simple of a place, we hav eto stop being afraid of big.

Jeroen Wenting replied on Thu, 2009/10/15 - 6:27am

Big is no problem, monolythic is the problem. A big system made up of many smaller parts that work together without having many hard dependencies on one another is no problem whatsoever (though its architecture may need more planning and care than your average "agile" coder is capable of envisioning let alone executing, given that the majority of them are essentially hackers who jump right into the code without any foresight or analysis, claiming anything more methodical isn't "agile" and therefore heretical).

Gilbert Le Blanc replied on Mon, 2009/10/19 - 7:30am in response to: Gilbert Herschberger

I like your reply better than the article.  :-)

Managing complexity is an important topic.  When you read books on the subject of managing complexity it usually comes down to high module cohesion and loose coupling.

I've not built huge projects with Java.  The biggest Java project I've worked on is a 120 class Eclipse plug in.  Most of the code extends existing Eclipse classes.

I have built huge projects with Cobol, CICS, and DB2.  This class of project is mostly data entry, management, and retrieval.

I would like to read of the experiences of a Java development in the order ot 1,000 to 10,000 classes or more. 

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.