I've been a zone leader with DZone since 2008, and I'm crazy about community. Every day I get to work with the best that JavaScript, HTML5, Android and iOS has to offer, creating apps that truly make at difference, as principal front-end architect at Avego. James is a DZone Zone Leader and has posted 639 posts at DZone. You can read more from them at their website. View Full User Profile

The Code Reuse Myth

07.22.2008
| 12056 views |
  • submit to reddit

I am of the firm belief that code reuse is a personal responsibility, and should never be a corporate endeavor. Unfortunately, thousands of CIOs and project managers disagree. There is a pervasive attitude among out-of-touch higher-ups that the distribution and reuse of code among developers will instantly reduce man hours and costs. On the surface, the logic is sound, but in the trenches this attitude stifles the emergence of greatness.

There are instances, especially in the creation of a software product, where practical standards must be introduced. No programmer would argue that some process or design protocol should guide and influence code. If you must connect to a database, then a single method for maintaining the connection should belong to the application. If you must traverse an XML document, then a single library capable of serializing nodes into objects should be considered.

Yet, beyond these singular utilities that enhance productivity finitely, there should not be an instance where an entire set of processes are continually re-purposed. Do not misunderstand — a framework that constitutes the backbone of an application is obviously a set of processes that are continually re-purposed. However, these features are common enough to meet the general requirements of a variety of applications, and are ambiguous by nature.

Programming uniqueness

This is probably the most difficult point to communicate to any individual concerned only with numbers and budgets, but programming uniqueness into an application is an obligation. There is no need to architect a solution that is creative for the sake of creativity. Yet, unless you are in the business of product support, clients expect you to drive a wedge between themselves and competitors. Although you can filter code from previous projects as a starting point, changing specifications and interface designs requires a custom approach.

This uniqueness is beneficial, not only because it serves your clients, but because it forces programmers to think about solving unique problems. This is essential to the professional growth of any development staff. Without it, there is little room to learn, and complacency is implicitly encouraged.

Taking code out of context

The simple fact is that code reuse also puts little emphasis on context. Even if your client’s needs are astoundingly similar to the needs of many of your clients, the context will be different. As an organization, a client will have data structures unlike any other, environments that are configured inefficiently, and legacy applications that are downright ridiculous. Any experienced programmer knows that the time it takes to adapt a piece of code out of context will take longer than a fresh approach.

Given a piece of code that was a hand-me-down, it is rarely beneficial to reuse it unless tweaks will be sparse. Even then there are dangers associated with porting large chunks of code, including the introduction of security vulnerabilities, bugs and the loss of optimization.

Prohibiting optimization

Optimization is one of the few measurable characteristics of code quality. Programmers can argue about design patterns, the fine details of OO, and language-specific syntax, but the benefits of optimization can be backed up with hard data. Code reuse does not foster optimization, or even invite explorations into the possibility. More than anything, this prevents entry-level programmers from being trained by senior programmers because it is assumed their code is perfectly refined.

Although we do have a tendency to scoff unrelentingly at the majority of code crossing our path, scrutiny in limited quantities is a good thing. This critical approach to development produces a better programmer, which is what in turn actually reduces man hours and cost.

From http://blog.reindel.com/

Comments

Aaron Digulla replied on Tue, 2008/07/22 - 9:51am

The main problem with code reuse is that our programming languages don't support it. We sacrificed this to the gods of efficiency three decades ago and, while a few people dared to question the practice, everyone was struck down by unexpected lightning out of the blue, so far.

As you said, "context" is the keyword. What we'd need is a programming language where you can adapt concepts to a context that goes beyond "object instance" or "class" or "application". What we need is an efficient way to say "collect all objects and sort them by ID" where the context defines what "ID" is. What we need is an efficient way to describe a model (relations of basic data types) and then have some tool map that efficiently to reality so we can reuse parts of the model in different contexts.

OO can't do that because it's limited to factories and inheritance. Preprocessors can't do it, because they know too little. Having closures in a programming language is the first, tiny step in the direction to be able to push external context into existing code. But to be a real new paradigm, we need "closures" in data types as well. This means being able to reuse fragments of code and data structure definitions in a new context while it pulls along all the algorithms and structures the fragment needs without the developer having to pay close attention what is going on until the point where the result needs peep hole optimization because of performance issues.

Aaron Digulla replied on Tue, 2008/07/22 - 10:32am

I've posted a longer and more elaborate analysis in my blog

Andrew McVeigh replied on Tue, 2008/07/22 - 10:42am in response to: Aaron Digulla

The main problem with code reuse is that our programming languages don't support it.

I agree, but I have a different view of the solution. I feel that expressiveness (closures etc etc) is more or less orthogonal to reuse, and in some cases may hamper it.

Current languages have very poor (nonexistent?) support for component/class customisation or evolution which is vital to reuse. We layer abstraction on abstraction, and then when we find that we need a change to the abstractions at the bottom of the pile, we have to unravel and start again. How often have you seen a bit of code/framework/class/system that was close to being what was wanted but didn't have the appropriate hooks for your particular extension?

Having closures in a programming language is the first, tiny step in the direction to be able to push external context into existing code. But to be a real new paradigm, we need "closures" in data types as well.

I'm of a different mindset. I believe that adding deep support for evolution in a language is one particular way to go that holds promise. I've been working in this area, and have found that a formal component model to be important rather than an OO model, as it is explicit about connections and object allocation, which allows change to be fully expressed directly in the language. I've also found that some more powerful features (e.g. higher order functions etc) to actively work against expressing evolution (and hence reusability) in this model. This is not to say that my model is correct, or that more expressiveness is bad, it's just an interesting perspective that reuse and expressiveness may sometimes conflict.

Cheers,
Andrew

 

Andrew

Andrew McVeigh replied on Tue, 2008/07/22 - 10:55am in response to: Aaron Digulla

The main goal would be to allow to use the compiler as a cut'n'paste tool which checks the syntax and allows me to say "copy that method over there and replace xxx with yyy". Because that's why we think that code reuse could work: We see the same code over and over and over again and each time, the difference is just a tiny little bit of code which we can't "patch" because the language just doesn't allow it.

Interesting -- i completely agree with the last issue. I feel however that it isn't enough to just change methods or a bit of code. I wanted to remake the structure of things in a principled way against a full component model. AYour "copy that method over there and replace xxx with yyy" becomes "copy that component structure over there, but change these connections and these properties" in this model --> I call this "resemblance". A further interesting construct turned out to be "change that method/connection in its current place, affecting all existing instances" --> I call this "redefinition".

It's the basis of the phd research I'm finishing at the moment. Here's an early paper on the constructs I created to do this. An interesting part of the work turned out to be allowing multiple changes from independent developers to be combined automatically.

http://portal.acm.org/citation.cfm?id=1181195.1181206&coll=&dl=GUIDE

Andrew

Aaron Digulla replied on Tue, 2008/07/22 - 1:51pm

My point of view is currently limited to what I see every day. In Java, you often need to say "copy this method and replace the 'int' parameter with a 'double' parameter", etc.

So right now, I'd be happy with something that would give me the AST and allow me to manipulate it to do thinks like copying code and replacing the type of certain variables or copy a piece of code and replace a few calls or lines.

Ideally, this should work with bytecode and source code so I could get debug information from the generated code. Groovy can some that to some extend with the AST Transformers which you can plug into the compiler. I haven't had time to give this a whirl, though.

Andrew McVeigh replied on Tue, 2008/07/22 - 2:14pm in response to: Aaron Digulla

 So right now, I'd be happy with something that would give me the AST and allow me to manipulate it to do thinks like copying code and replacing the type of certain variables or copy a piece of code and replace a few calls or lines.

Here are some things I've come across in my work that are related, you may (or may not!) find them interesting:

Subtext is a programming environment that starts with your premise, and designs a language around it.  The basic idea is that it offers principled support for copy/paste, without losing the link back to the original.
http://www.subtextual.org/
Quite an interesting editor.  Unfortunately, it uses its own language.  

MixJuice allows arbitrary methods in classes to be replaced, and uses Java as a language.  You can sort of get the effect you are looking for -- i.e. modification of something that already e.http://staff.aist.go.jp/y-ichisugi/mj/

The book "Framing Software for Reuse" (http://www.amazon.com/Framing-Software-Reuse-Lessons-World/dp/013327859X) uses a text-based approach but does essentially what you mention.  The approach works in any language, but unfortunately it isn't AST based.

One of the problems with doing this via the AST is that you need very smart editors (or you have to express the changes in a different way to how you specify the original).  Subtextual gets around that by making everything based around its own editor (which allows refactorings and name changes without breaking code etc).  Another problem is that existing users of the things you change can suffer subtle (or not so subtle) bugs based on your change (possibly a breaking change).  The only answer for this is to "try it and see" (either breaks at compile or runtime) or go to a stronger model where most errors can be picked up by analysis.

I went down the "stronger model" route, but it has taken me well off the mainstream ;-) 

Cheers,
Andrew 


 

Slava Imeshev replied on Tue, 2008/07/22 - 5:19pm

Andrew,

I would not agree that the code reuse is a myth. Web UI and persistence frameworks are the proof. They are written once and used tens of thousand times.

What I would agree with is that forced or upfront re-use is a waste. The re-use is a process of learning about opportunities for re-use and acting on such opportunities by developing utility classes and frameworks. As such, it is always post-factum. Frameworks Up Front (FUF, pronounced as foof) on the contrary come to life without analysis of the prior opportunities for the re-use and do not provide the re-use as a result.

Myself I have seen these three or four times. All of that I attributed to the pure or luck of, basic engineering education. I have seen management or policy-imposed re-use only once but the result has been no different: complex, over-engineered, and hard-to-develop and maintain systems.

The systems that used frameworks that have grown by realizing known opportunities for re-use enjoyed all the benefits of the solved common problems.

Just my 2c.

Regards,

Slava Imeshev

Cacheonix: Distributed Java Cache

 

 

Andrew McVeigh replied on Tue, 2008/07/22 - 5:46pm in response to: Slava Imeshev

I would not agree that the code reuse is a myth. Web UI and persistence frameworks are the proof. They are written once and used tens of thousand times.

Agree with this example. Another example are the Java class libs. The best type of reuse occurs when 2 almost orthogonal areas are combined. E.g. a business app using a web framework. Or, a desktop app using widget class libraries. And how well it works also depends on how well the framework or library developers have foreseen the ways in which it will be extended. E.g. provision of hooks, methods to override, config options etc etc.

Frameworks Up Front (FUF, pronounced as foof) on the contrary come to life without analysis of the prior opportunities for the re-use and do not provide the re-use as a result.

Yes, it almost never works. The best results I've had were to engineer a framework quite close to the initial concrete app requirements, and then abstract away using interfaces. Worked well for me on a large, multi-year project and allowed us to customise and sell the same product to multiple companies.

But here's the problem even with this example. Assume you want to use someone's framework, and it almost, but not quite fits your needs. Perhaps you need to extend it in a way that simply isn't possible. What do you do? You work around it somehow (reflection, class shadowing etc), or you fork the code. However, forking may not be feasible -- the committers may not want to take your changes Even if they do, the next release will take time, and have lots of other changes too that you need to test against. What if the changes are major? If the committers won't accept your changes, you own your variant and have to manage bug fixes and upgrades. Lots of problems here.

Again, the problem may not be so apparent when looking at 2 more or less orthogonal areas: e.g. a business app (concerned, say with healthcare) and a web framework (concerned with webby things). The problem becomes more acute (in my experience) when for example, building a new healthcare app out of an old healthcare app. Even if the old one was built with extensibility in mind, the new customer requirements will soon show that you haven't foreseen all extension/customisation points. And how could you? Even growing a framework, you'll never be able to put in the right extension points and config options for all future needs. How far do you go? And even if you could, the app would be a generic mess, very difficult to understand.

The point I guess I'm making is that I feel we need languages that can express evolution or change instructions within the language, leading to much better reuse. E.g. I take a health care app, and i need to change it to fit my needs, i use the language primitives to "operate" on the app. I don't change the app per se, i simply take it and combine it with my "change instructions" and i have my new app. It's a little bit like how Aaron wanted to have a way to operate on the AST as a sort of copy/paste operation on steroids.

That way, i don't have to feed my changes back to a central place, and i can still accept changes from the original framework. Plus, my changes for my specific use cases don't pollute the original app -- they only live with me.

Cheers,
Andrew

Slava Imeshev replied on Tue, 2008/07/22 - 6:32pm in response to: Andrew McVeigh

[quote=andrewm]

The problem becomes more acute (in my experience) when for example, building a new healthcare app out of an old healthcare app.

[/quote]

I guess this should be considered on a case-by-case basis. A clear definition of "new" is needed here. Is it a full rewrite or some new features? What is the problem?

[quote=andrewm]

Even if the old one was built with extensibility in mind, the new customer requirements will soon show that you haven't foreseen all extension/customization points. And how could you?

[/quote]

I am not even sure if this is possible. What is possible though is to know that the application will evolve within some border and spend some time in the beginning on architecting the extensibility in. How is a separate story.

[quote=andrewm]

Even growing a framework, you'll never be able to put in the right extension points and config options for all future needs. How far do you go?

[/quote]

Personally I believe it does not make sense to develop in-house frameworks. The knowledge of the opportunities for re-use is limited, domain and time wise. Maximum what one could and should get is a set of utility classes.

[quote=andrewm]

And even if you could, the app would be a generic mess, very difficult to understand.

[/quote]

Exactly. So, what we should have here is a conscious effort in the beginning of the project to propose architecture that supports extension for a certain period of time. On top of this a periodical or on-demand reviewing of the architecture could be added to see if it begins showing signs of rigidness and correcting such signs or already existing issues soon. I should mention that this is a pretty tall order.

Slava

Andrew McVeigh replied on Tue, 2008/07/22 - 6:40pm in response to: Slava Imeshev

I guess this should be considered on a case-by-case basis. A clear definition of "new" is needed here. Is it a full rewrite or some new features? What is the problem?

Well, the problem is that sometimes making a new system by reusing an existing system is difficult, even if the systems are very, very similar. Reuse of the system is hampered by current technical techniques.

So, to make this concrete -- i was the architect for a smartcard provisioning system for a credit card company. We got the contract to make their system and delivered it (call it system X). It was pretty obvious fairly soon that other companies wanted similar systems. However, it was a medium sized system (500kloc) and it turned out that there were some deeply buried stuff in there about card lifecycles and security/data encryption that weren't correct for the 2nd and other customers.

At this point we had a choice. We could either distill a set of frameworks out of the 1st system, and add in new extension points for the 2nd customer. Or, we could fork the source. We chose the former option. It worked out well for us, and kept a common code base and were able to fix bugs / add new features in one place.

But, there was pain. We had to reengineer System X also to use the new frameworks. And some of the concepts/extension points in the frameworks were not applicable to System X, and so added complexity to it.

What I guess I'm saying is that reuse of systems, even if notionally quite similar, is difficult using existing techniques. You can look at the source and say "if only i could make this small change", but in practice there are forces preventing this, particularly if you are not the maintainer of the app/framework.

Personally I believe it does not make sense to develop in-house frameworks. The knowledge of the opportunities for re-use is two limited, domain and time wise. Maximum what one could and should get is a set of utility classes.

I've had some very positive experiences making in-house frameworks (above was one case of several), but only at the domain level. I'd be very reluctant to make my own infrastructure.

So, what we should have here is a conscious effort in the beginning of the project to propose architecture that supports extension for a certain period of time. On top of this a periodical or on-demand reviewing of the architecture to see if it begins showing signs of rigidness and correcting such signs or already existing issues soon.

Yes, this is exactly what I have found also with current techniques. The problem becomes magnified if you are not the primary owner of the architecture that you are wishing to extend. E.g. you build on a 3rd party trading system.

My research work is based (in part) on adding evolution primitives to a language to allow it to cope better with these scenarios. A user of a framework can customise it radically, without needing to feed the changes back to the owner, and without forking the source. Works quite well in that you can add your own extension points to an existing framework, freeing the original owner of the f'work to concentrate on conceptual clarity without needing to make it predictively generic.

Cheers,
Andrew

 

Slava Imeshev replied on Tue, 2008/07/22 - 7:09pm in response to: Andrew McVeigh

 

What would be interesting to see what both qualitative and quantitative characteristics of the design and the code that signify support for re-use are? 

Imagine that you are a 3rd party consultant hired to identify if the system supports re-use. What approach would you use to confirm or deny that support? Where would you start?

  

Slava

Andrew McVeigh replied on Wed, 2008/07/23 - 3:48am in response to: Slava Imeshev

What would be interesting to see what both qualitative and quantitative characteristics of the design and the code that signify support for re-use are?

In my experience, the Robert Martin metrics are excellent for identifying this sort of thing. http://www.tek271.com/articles/pood/PrinciplesOfOOD.java.html
The are old for sure, but still very relevant.

There are various tools that measure these things on a package by package basis (jdepend, stan4j etc). Of particular interest to me is "dependency inversion" --> i.e. has the system been designed to depend on interfacs rather than concrete classes.

Andrew

Steven Baker replied on Thu, 2008/07/24 - 1:21am

if your finding it hard or unfitting to reuse some code that was intended to be reuseable, than your prolly not doing it right inthe first place

Yannick Menager replied on Sun, 2008/07/27 - 8:11pm

The issues around code reuse have nothing to do with the technical aspects of the language and frameworks

Sure, some languages and frameworks can make it easier technically, but the real issues are not technical.

The first thing to keep in mind, is that code reuse is a double-edged sword.

The up side, is that you save time on your development, and *potentially* improve it's quality.

The downsides are the following:

- First, in order to be able to reuse something, you must know it exists. In large organizations, duplication of code is very common (go to any large corporation and you'll find logging and caching frameworks crawling all around the place). It is very rare for such organizations to have a central place where all reusable software can be listed and accessed.

- Second, when you reuse code, you've just created a dependency. And that dependency can have dependencies, which in turn can conflict with other dependencies, etc... It can get into a mess of biblical proportions, where an application you thought was 'modular' is in reality of a single monolithic monster because due to the dependency graph. Managing dependencies can be a real problem in those circonstances, especially when the library developer has been very dependency-happy and used every open source library then could a use for, even if it's just to save on a line of code.

- Third, when you reuse code, you're exposed to it's quality and how well (if at all) it is being maintained. Reusing a buggy library where such bugs take forever to be fixed can be more trouble than it's worth. Very often organizations will publish reusable components, but not have any resources assigned for it's maintenance.

Now, the impact of those issues can be reduced through several means.

The first (knowing what exists) can be through having a single place where all existing reusable code can be browsed (that's one of the reasons why i created DevPortal)

The second is to use a dependency management framework like Ivy or Maven (I highly recommend Ivy). Please note that even this approach has it's own double-edged factor, which is either to be exposed to having to maintain your own repository, or to use a public repository like ibiblio which is quite a bit messy, and generally not fully up-to-date. I'm actually adding some repository management to DevPortal which should help in that aspect.

The third is to take seriously and professionally the maintenance of reusable code. That means having skilled developers in charge of maintaining, resolving bugs quickly, designing the applications to reduce dependency hell (If you are working on  a reusable library/framework, don't bloody add a dependency on commons-lang just because you want to use StringUtils.hasText, just create your own static utility, or just type the whole txt != null && txt.trim().lenght() != 0).

Also proper care in release management must be taken. If you're releasing a new version 1.5.2.3 (from 1.5.2.2), don't just go around breaking all backward compability because you changed all the packages (like i've seen chaps from one of the largest consultancies doing while they were maintaining some common component libraries in one very large organization).

Steven Baker replied on Sun, 2008/07/27 - 8:20pm

good points Yannick

Andrew McVeigh replied on Mon, 2008/07/28 - 5:40am in response to: Yannick Menager

[quote=ym105613]

The issues around code reuse have nothing to do with the technical aspects of the language and frameworks

Sure, some languages and frameworks can make it easier technically, but the real issues are not technical.

[/quote]

Reuse is always possible, and i'm not saying that the biggest barriers are always technical.  I'm saying that we have a long way to go with creating better technical mechanisms that enable reuse.  Sure, finding libraries and components to reuse are often big issues, the culture of promoting reuse is a big issue.  But at the heart of the matter is the fact that we've never really solved the technical issues around reuse.

On the surface, what I'm saying isn't contentious.  Consider a historical example: Did adding inheritance and virtual functions help with reuse?  Sure, it helped a lot because people could introduce their own classes into a framework as per Robert Martin's "Open/Closed principle".  This is the basis of OO frameworks, which enable large scale reuse not possible in earlier systems written in languages like fortran.

My work involves looking at the technical barriers still preventing reuse.  This involves thinking beyond the current set of solutions and working out how to really address the issues.

Andrew

 

Andrew McVeigh replied on Mon, 2008/07/28 - 5:44am in response to: Andrew McVeigh

Ahh, i think i've found my problem ;-)

 http://mine.icanhascheezburger.com/view.aspx?ciid=1615025

Andrew

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.