Gilbert has posted 6 posts at DZone. View Full User Profile

Complexity is the Problem

04.18.2010
| 5740 views |
  • submit to reddit

Who really believes that complexity is not a problem? The complexity of modern software is a problem. Some say that it might be THE problem.

Let's say that you are building an Internet facing web application using a Java-based technology stack. (Pick one.) Building a modern application with a Web 2.0 front-end and web services back-end, add connection to a database and object store. Add security and testing and source control management, poly-oriented methods of development. Who thinks that this is easy?

And then, we have the latest recommendation for how we ought to think about the problem. We have been hardware-oriented, operating system-oriented, platform independent-oriented, language-oriented, procedure-oriented, dynamic-link-oriented, process-oriented, management-oriented, user-oriented, data-oriented, information-oriented, test-oriented, function-oriented, version-oriented and now module-oriented. In general, we keep on trying to figure out how to deal with overwhelming complexity.

Think of the short history of the software industry. The individual, teams, companies and the whole industry looks for ways to reduce the complexity of some problem. Do you remember a time when we were unable to run the same program on two different versions of the same operating system? Backward compatibility became an important aspect of the operating system itself. Younger ones might take it for granted. Or, do you remember when we were unable to compile the same program on two different operating systems? Platform-independent source code became a tradition for compiling a program. Younger ones might think that the solution is obvious. Do you remember when two computers could only communicate with a point-to-point connection? Packet-switched networking became an important aspect of simplification. Younger ones may not be able to imagine a world without networking. Do you remember when binary-identical programs were unable to run on two different operating systems? An abstract machine, such as i5/OS and Java, became an important aspect of simplification. Remember when a database library was embedded within a program? Remember when a browser could only display static pages?

Our goal is to reduce complexity to a level we can manage. Complexity must be managed. Managing it is an aspect of all construction. Techology must be applied on a case by case basis. Adding a technology when it is not required increases complexity. We might say that every development effort is an effort to simplify something, just as every tool is an effort to simplify something. But simplifying one thing likely makes other things more complex.

Complexity is a problem that cannot be "solved" in the same way that a puzzle can be solved. Over the years, software in general has become increasingly complex. When software becomes so complex that we can no longer manage, we look for ways to reduce its complexity back to something managable. Therefore, complexity is never going to go away. It will always be THE problem.

OSGi is "the dynamic module system for Java." It tries to solve one basic, recurring and complex problem for a Java-based application: the java.class.path property (set by the classpath and cp parameters). The problem is basic because the standard java.class.path mechanism does not scale. It is not possible to put thousands of jars on the java.class.path in a platform-independent way because there are hard limits to the length of a command line. (I work with an application that has more than 4,000 jars and an operating system that has a hard limit of 4,000 characters on the command line. It just doesn't fit.)

The problem is recurring because every developer must eventually deal with multiple versions of an application and its dependencies. The standard java.class.path mechanism does not account for different versions of a class, package, resource or framework within one instance of a virtual machine. When java.class.path does not work, where do we go from here?

The problem is complex because some well-known Java mechanisms do not even work with multple class loaders. In other words, a method that works when running with java.class.path may not work when loaded by a custom class loader. A developer must learn to stop using mechanisms that do not work in the new environment.

Going from monolithic (non-modular) to modular is inherently complex. When you have decided to go modular, the question becomes: Are you going to adopt an existing module system, such as OSGi, or build your own? A module system with less complexity than OSGi is possible. I myself have built not one but three. But OSGi offers shared knowledge with an industry standard, multiple implementations of the framework, off-the-shelf collection of pre-compiled modules. Besides, smart people are working to make OSGi less complex.

Published at DZone with permission of its author, Gilbert Herschberger.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Chris Treber replied on Sun, 2010/04/18 - 6:14am

"4.000 JARs in one application"

Please elaborate - who would do that, and why?

 

Chris

Mohammad Juma replied on Sun, 2010/04/18 - 6:38am

Complex Article !!

Fab Mars replied on Sun, 2010/04/18 - 6:38pm

Title proposition: "complexity as an excuse". If this aticle said "My calculator doesnt have the multiplication sign, so I can't do 8*3", that would be no different.

Anyway, eveytime I saw something called "complex" it was because people had made it complex. Complexity is so relative for each person by the way.

There may be tough functional requirements, I have seen some, but they can always be split and simplified so they can be coded, provided you're competent of course. If a requirement remains "complex" for you after careful investigation, then find another job. Any customer who knows his business will agree with that. From that point of view, I can understand the article focusses on the technical side of things. But here again, we're here to find solutions, not to howl at the moon.

Then, after it's built, some software is said complex because it would take a long time for an average engineer to understand how it works and communicates. The reason for it in 99.9999% of the cases is the result of a) the lack of understanding of the business b) overengineering by a hurd of wannabe-architects c) infamous design by incompetent developers, and d) the wish to justify a very expensive SLA.

Take OSGI: it can be used to solve a particular issue and decrease complexity (of building a plugin system for example). But in most of the projects, it was unneeded in the first place, and eventually included so that yet-another crazy-wannabe could have fun playing with a new toy and put a buzzword on his CV.

In other words I see "complexity" as an umbrella word used not to apologize on oneself's inability to build software tailored to the requirements. Let's not swap causes and consequences please.

Caspar MacRae replied on Mon, 2010/04/19 - 7:48am

 

I think the main point missed in this article is that dealing with complexity is often, if not always, a trade-off.   Eg. IoC trades complexity of tight-coupling for byte code manipulation debuggery / what did the container just try and inject?, another eg. ORM trades manually marshalling with mapping shenanigans.  Both have reduced complexity in often repeated and mundane but not without an associated increase in complexity elsewhere.  (Thought surely any successful endeavour should reduce overall complexity).

 

@Fab -  howling at the moon?  Did you just search for articles containing OSGi and comence ranting?

 There isn't a single line whining about complexity, in fact the opposite; it discusses the fact that complexity is a different beast from say, scalablity, and highlights the fact that we'll face complexity in many guises.

 OSGi - crazy-wannabe, new toy, buzzword etc  - seriously how much of the article did you read before putting your pap to paper?

The very last paragraph clearly implies you should ask the question do you need something as complicated as OSGi to achieve modularity? (and then goes on to point out that the complexity of OSGi is and will be reduced over time).

 

 

Tony Siciliani replied on Mon, 2010/04/19 - 8:13am

I would first clearly separate essential complexity from accidental complexity.

Essential complexity is inherent to the problem at hand, and practically unavoidable.

Accidental complexity is created by the solution itself, so it could in theory be minimized with a more optimal approach. Unfortunately, an optimal approach is also subject to software entropy, and, even with regular code refactoring,  can evolve "naturally" overtime into a non-optimal solution.

 

Igor Laera replied on Mon, 2010/04/19 - 10:22am

 

Tony says it all. +1.

In all of my projects, the underlying complexity is not avoidable. If you have to care
for files in 10 different xml structural dialects to import, process, export plus an
every changing database-layout you have to keep concurrent with three different
databases, you simply can't change anything about it.

But if you have slow build computers that it takes one day to test some db-changes
on all variants of the product; if you need to use hacks to get certain types of
db-updates working because the dbadministrator@customer has no clue what he
doing, when you have to use an 1 1/2 year old eclipse with endless bugs in a zillion
absolute necessary plugins - this adds up stupid complexity which could be avoidable.

The real underlying problem is, that the customers/bosses think "that's the reason
I hired a specialist - to 'suffer' trough such real life, organically grown problems -
I wouldn't need him/her if everything was easy and clean and good."

It's like a doctor who has to perform surgery in a far away village every day. But instead
of building a small hospital with better work environment, alots of people have to continue
working everyday with abysmal tools and obscure production processes.

Since many projects exploded in their essential complexity, you simply can't sustain this
type of "fix". Some tried beeing more agile, some tried to reduce complexity by model-
driven approaches. But the inherent problem is often, that too many people try todo too much
upfront, without knowing the real enviroment at the end.

And then you need to "heavy fix" the wildy unrealistic assumptions someone made years
ago who probably isn't in the project anymore - and, pardon me and my experience in
+10 larger projects, probably even doesn't cared when s/he was the head of project. 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.