The Code Reuse Myth

The main goal would be to allow to use the compiler as a cut'n'paste tool which checks the syntax and allows me to say "copy that method over there and replace xxx with yyy". Because that's why we think that code reuse could work: We see the same code over and over and over again and each time, the difference is just a tiny little bit of code which we can't "patch" because the language just doesn't allow it.

Interesting -- i completely agree with the last issue. I feel however that it isn't enough to just change methods or a bit of code. I wanted to remake the structure of things in a principled way against a full component model. AYour "copy that method over there and replace xxx with yyy" becomes "copy that component structure over there, but change these connections and these properties" in this model --> I call this "resemblance". A further interesting construct turned out to be "change that method/connection in its current place, affecting all existing instances" --> I call this "redefinition".

It's the basis of the phd research I'm finishing at the moment. Here's an early paper on the constructs I created to do this. An interesting part of the work turned out to be allowing multiple changes from independent developers to be combined automatically.

http://portal.acm.org/citation.cfm?id=1181195.1181206&coll=&dl=GUIDE

Andrew

 So right now, I'd be happy with something that would give me the AST and allow me to manipulate it to do thinks like copying code and replacing the type of certain variables or copy a piece of code and replace a few calls or lines.

Here are some things I've come across in my work that are related, you may (or may not!) find them interesting:

Subtext is a programming environment that starts with your premise, and designs a language around it.  The basic idea is that it offers principled support for copy/paste, without losing the link back to the original.
http://www.subtextual.org/
Quite an interesting editor.  Unfortunately, it uses its own language.  

MixJuice allows arbitrary methods in classes to be replaced, and uses Java as a language.  You can sort of get the effect you are looking for -- i.e. modification of something that already e.http://staff.aist.go.jp/y-ichisugi/mj/

The book "Framing Software for Reuse" (http://www.amazon.com/Framing-Software-Reuse-Lessons-World/dp/013327859X) uses a text-based approach but does essentially what you mention.  The approach works in any language, but unfortunately it isn't AST based.

One of the problems with doing this via the AST is that you need very smart editors (or you have to express the changes in a different way to how you specify the original).  Subtextual gets around that by making everything based around its own editor (which allows refactorings and name changes without breaking code etc).  Another problem is that existing users of the things you change can suffer subtle (or not so subtle) bugs based on your change (possibly a breaking change).  The only answer for this is to "try it and see" (either breaks at compile or runtime) or go to a stronger model where most errors can be picked up by analysis.

I went down the "stronger model" route, but it has taken me well off the mainstream ;-) 

Cheers,
Andrew 

I would not agree that the code reuse is a myth. Web UI and persistence frameworks are the proof. They are written once and used tens of thousand times.

Agree with this example. Another example are the Java class libs. The best type of reuse occurs when 2 almost orthogonal areas are combined. E.g. a business app using a web framework. Or, a desktop app using widget class libraries. And how well it works also depends on how well the framework or library developers have foreseen the ways in which it will be extended. E.g. provision of hooks, methods to override, config options etc etc.

Frameworks Up Front (FUF, pronounced as foof) on the contrary come to life without analysis of the prior opportunities for the re-use and do not provide the re-use as a result.

Yes, it almost never works. The best results I've had were to engineer a framework quite close to the initial concrete app requirements, and then abstract away using interfaces. Worked well for me on a large, multi-year project and allowed us to customise and sell the same product to multiple companies.

But here's the problem even with this example. Assume you want to use someone's framework, and it almost, but not quite fits your needs. Perhaps you need to extend it in a way that simply isn't possible. What do you do? You work around it somehow (reflection, class shadowing etc), or you fork the code. However, forking may not be feasible -- the committers may not want to take your changes Even if they do, the next release will take time, and have lots of other changes too that you need to test against. What if the changes are major? If the committers won't accept your changes, you own your variant and have to manage bug fixes and upgrades. Lots of problems here.

Again, the problem may not be so apparent when looking at 2 more or less orthogonal areas: e.g. a business app (concerned, say with healthcare) and a web framework (concerned with webby things). The problem becomes more acute (in my experience) when for example, building a new healthcare app out of an old healthcare app. Even if the old one was built with extensibility in mind, the new customer requirements will soon show that you haven't foreseen all extension/customisation points. And how could you? Even growing a framework, you'll never be able to put in the right extension points and config options for all future needs. How far do you go? And even if you could, the app would be a generic mess, very difficult to understand.

The point I guess I'm making is that I feel we need languages that can express evolution or change instructions within the language, leading to much better reuse. E.g. I take a health care app, and i need to change it to fit my needs, i use the language primitives to "operate" on the app. I don't change the app per se, i simply take it and combine it with my "change instructions" and i have my new app. It's a little bit like how Aaron wanted to have a way to operate on the AST as a sort of copy/paste operation on steroids.

That way, i don't have to feed my changes back to a central place, and i can still accept changes from the original framework. Plus, my changes for my specific use cases don't pollute the original app -- they only live with me.

Cheers,
Andrew

I guess this should be considered on a case-by-case basis. A clear definition of "new" is needed here. Is it a full rewrite or some new features? What is the problem?

Well, the problem is that sometimes making a new system by reusing an existing system is difficult, even if the systems are very, very similar. Reuse of the system is hampered by current technical techniques.

So, to make this concrete -- i was the architect for a smartcard provisioning system for a credit card company. We got the contract to make their system and delivered it (call it system X). It was pretty obvious fairly soon that other companies wanted similar systems. However, it was a medium sized system (500kloc) and it turned out that there were some deeply buried stuff in there about card lifecycles and security/data encryption that weren't correct for the 2nd and other customers.

At this point we had a choice. We could either distill a set of frameworks out of the 1st system, and add in new extension points for the 2nd customer. Or, we could fork the source. We chose the former option. It worked out well for us, and kept a common code base and were able to fix bugs / add new features in one place.

But, there was pain. We had to reengineer System X also to use the new frameworks. And some of the concepts/extension points in the frameworks were not applicable to System X, and so added complexity to it.

What I guess I'm saying is that reuse of systems, even if notionally quite similar, is difficult using existing techniques. You can look at the source and say "if only i could make this small change", but in practice there are forces preventing this, particularly if you are not the maintainer of the app/framework.

Personally I believe it does not make sense to develop in-house frameworks. The knowledge of the opportunities for re-use is two limited, domain and time wise. Maximum what one could and should get is a set of utility classes.

I've had some very positive experiences making in-house frameworks (above was one case of several), but only at the domain level. I'd be very reluctant to make my own infrastructure.

So, what we should have here is a conscious effort in the beginning of the project to propose architecture that supports extension for a certain period of time. On top of this a periodical or on-demand reviewing of the architecture to see if it begins showing signs of rigidness and correcting such signs or already existing issues soon.

Yes, this is exactly what I have found also with current techniques. The problem becomes magnified if you are not the primary owner of the architecture that you are wishing to extend. E.g. you build on a 3rd party trading system.

My research work is based (in part) on adding evolution primitives to a language to allow it to cope better with these scenarios. A user of a framework can customise it radically, without needing to feed the changes back to the owner, and without forking the source. Works quite well in that you can add your own extension points to an existing framework, freeing the original owner of the f'work to concentrate on conceptual clarity without needing to make it predictively generic.

Cheers,
Andrew

What would be interesting to see what both qualitative and quantitative characteristics of the design and the code that signify support for re-use are?

In my experience, the Robert Martin metrics are excellent for identifying this sort of thing. http://www.tek271.com/articles/pood/PrinciplesOfOOD.java.html
The are old for sure, but still very relevant.

There are various tools that measure these things on a package by package basis (jdepend, stan4j etc). Of particular interest to me is "dependency inversion" --> i.e. has the system been designed to depend on interfacs rather than concrete classes.

Andrew

The issues around code reuse have nothing to do with the technical aspects of the language and frameworks

Sure, some languages and frameworks can make it easier technically, but the real issues are not technical.

The first thing to keep in mind, is that code reuse is a double-edged sword.

The up side, is that you save time on your development, and *potentially* improve it's quality.

The downsides are the following:

- First, in order to be able to reuse something, you must know it exists. In large organizations, duplication of code is very common (go to any large corporation and you'll find logging and caching frameworks crawling all around the place). It is very rare for such organizations to have a central place where all reusable software can be listed and accessed.

- Second, when you reuse code, you've just created a dependency. And that dependency can have dependencies, which in turn can conflict with other dependencies, etc... It can get into a mess of biblical proportions, where an application you thought was 'modular' is in reality of a single monolithic monster because due to the dependency graph. Managing dependencies can be a real problem in those circonstances, especially when the library developer has been very dependency-happy and used every open source library then could a use for, even if it's just to save on a line of code.

- Third, when you reuse code, you're exposed to it's quality and how well (if at all) it is being maintained. Reusing a buggy library where such bugs take forever to be fixed can be more trouble than it's worth. Very often organizations will publish reusable components, but not have any resources assigned for it's maintenance.

Now, the impact of those issues can be reduced through several means.

The first (knowing what exists) can be through having a single place where all existing reusable code can be browsed (that's one of the reasons why i created DevPortal)

The second is to use a dependency management framework like Ivy or Maven (I highly recommend Ivy). Please note that even this approach has it's own double-edged factor, which is either to be exposed to having to maintain your own repository, or to use a public repository like ibiblio which is quite a bit messy, and generally not fully up-to-date. I'm actually adding some repository management to DevPortal which should help in that aspect.

The third is to take seriously and professionally the maintenance of reusable code. That means having skilled developers in charge of maintaining, resolving bugs quickly, designing the applications to reduce dependency hell (If you are working on  a reusable library/framework, don't bloody add a dependency on commons-lang just because you want to use StringUtils.hasText, just create your own static utility, or just type the whole txt != null && txt.trim().lenght() != 0).

Also proper care in release management must be taken. If you're releasing a new version 1.5.2.3 (from 1.5.2.2), don't just go around breaking all backward compability because you changed all the packages (like i've seen chaps from one of the largest consultancies doing while they were maintaining some common component libraries in one very large organization).

The issues around code reuse have nothing to do with the technical aspects of the language and frameworks

Sure, some languages and frameworks can make it easier technically, but the real issues are not technical.

[/quote]

Reuse is always possible, and i'm not saying that the biggest barriers are always technical.  I'm saying that we have a long way to go with creating better technical mechanisms that enable reuse.  Sure, finding libraries and components to reuse are often big issues, the culture of promoting reuse is a big issue.  But at the heart of the matter is the fact that we've never really solved the technical issues around reuse.

On the surface, what I'm saying isn't contentious.  Consider a historical example: Did adding inheritance and virtual functions help with reuse?  Sure, it helped a lot because people could introduce their own classes into a framework as per Robert Martin's "Open/Closed principle".  This is the basis of OO frameworks, which enable large scale reuse not possible in earlier systems written in languages like fortran.

My work involves looking at the technical barriers still preventing reuse.  This involves thinking beyond the current set of solutions and working out how to really address the issues.

Andrew