Mark is a graph advocate and field engineer for Neo Technology, the company behind the Neo4j graph database. As a field engineer, Mark helps customers embrace graph data and Neo4j building sophisticated solutions to challenging data problems. When he's not with customers Mark is a developer on Neo4j and writes his experiences of being a graphista on a popular blog at He tweets at @markhneedham. Mark is a DZone MVB and is not an employee of DZone and has posted 553 posts at DZone. You can read more from them at their website. View Full User Profile

Our Obsession With Efficiency – Dan North

  • submit to reddit

Oredev have put some of the videos from the conference on Vimeo and one of my favourites is 'Our obsession with efficiency' by my colleague Dan North.

The slides for the talk are available on SlideShare.

In this talk Dan leads from the following statement about efficiency:

So here's the thing, I don't believe in efficiency. It's our obsession with efficiency that has got us into the current technology mess, and which has led almost directly to heavy waterfall processes. Efficiency is how you let the big vendors sell their bloated technologies to the poor CIOs.

What did I learn?
  • Dan spends quite a bit of time explaining how what we should really care about is effectiveness and not efficiency. Efficiency is defined as:
    the accomplishment of or ability to accomplish a job with a minimum expenditure of time and effort

    which makes sense in a way but what tends to happen is that achieving that becomes our goal at the expense of everything else. For example adhering to the DRY principle is considered a good thing and not repeating code is efficient. However, if we take that to an extreme then it results in code that is difficult to understand and difficult to change which defeats the purpose of that efficiency.

    In lean terms we want to look to favour the big picture over local optimisations whereby we start to measure our effectiveness rather than efficiency. i.e. we focus on the results rather than the effort.

  • Later on he points out that effectiveness is often inefficient which makes a lot of sense to me.

    Pair programming is not an efficient way to get the most code produced – we can do that much more efficiently if we have people working individually. However it allows us to create a much greater shared understanding of the code, reduce the defects and increase the cohesion of that code which is a more important goal overall. The same can be said for something like set based concurrent engineering where we work on two solutions simultaneously and then throw one away. It's inefficient but we can delay potentially making the wrong decision so it makes sense to do it.

  • Dan also suggests that 'you get what you measure' and lists a series of examples where aiming for a certain target is actually not very effective and doesn't give us what we want anyway.

    For example if our goal is to have all the tests passing by the end of the month then we may be tempted to comment out failing tests to achieve this. One which I notice quite frequently is aiming for a story point total which tends to result in reduced communication between business and IT and we end up delivering features that may not have that much value. It might seem to be locally efficient but in the grand scheme of things it's not efficient at all.

  • There's also some discussion around the desire for people to be 'busy' all the time which is quite common from my experience. Dan points out that if we can get to a stage where we have time when we're not busy then we have more time to think about what we're doing and perhaps we can even spend this time innovating. I think the key here is to ensure that we're still doing something which is contributing to the overall system goal rather than not doing nothing at all!

There's loads more – it's a very good talk – but these are some of the bits that stand out for me.


Published at DZone with permission of Mark Needham, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Loren Kratzke replied on Mon, 2009/12/07 - 12:48pm

Interesting points, but I don't understand the recent fascination with extreem methodologies that I have seen emerge in the past few years. I just seems like a fad, but a well intentioned fad.

 I am reminded of my favorite quote from our previous CTO. "I hate throwing good hardware at bad software." And this quote: "Most good judgement comes from experience. Most experience comes from bad judgement." Between the two, I tend not to buy in too deeply or quickly to religious-like or shock-value laden methodologies. I try to keep it real as much as possible.

Nirav Assar replied on Mon, 2009/12/07 - 2:12pm


How do you know if something is a fad or not a fad if 1) you don't know the trends or have not researched them, and 2)ignore the fact that research and case studies exist that confirm Agile has become more mainstream than outdated methods.

Thoughworks is a well respected company that has implemented solutions using these "fads" and has been successful for years.  Many of their employees have authored books, given worldwide presentations and implemented many enterrpise solutions. 

I would suggest "keeping it real" is more researching and learning, and comparing old outdated methods to more recent trends, rather then just cubby holing youself into darkness and dismissing things just cause you haven't taken the time to learn something.

Loren Kratzke replied on Mon, 2009/12/07 - 4:08pm in response to: Nirav Assar

Actually I am emersed in an Agile workplace and have very few issues with Agile in general - with the exception of certain statements such as we see in the parent article and those which Agile coaches and consultants have been known to spout on a regular basis such as, "what we should really care about is effectiveness and not efficiency". That's not real, it is an abstract statement conviently stated out of the context of any real world situation purely for shock value. If you do not question that statement then you question nothing.

I say you need both. Call me crazy.

I have been programming for 20 years and before that I worked in electronics, manufacturing, and construction. In my opinion, if a product comes off the line and it is defective, or if the product doesn't make it off the line at all because the assembly line or crew is inefficient, then you are equally screwed in either case. So I call bullshit. You need a balance. It's an art, not a method or methodology. I know this to be true.

Particularly in manufacturing, if you slack then your competition will eat your lunch. It's that simple. In software development this fact is not immediately evident because nobody is in direct competition like Spacely Sprockets and Cogswell Cogs are. That vacuum tends to get filled with BS if one is not alert.

Agile does not solve every problem despite what many people wish to belive. It solves many problems and simply pushes the remainder into a different problem domain in an effort to come across as a silver bullet. What Agile brings to the table is visibility for business, scheduling benefits for everybody, and flexibility and feedback loops not inherant to waterfall methods. Most of these problems that are solved exist purely because of the skill gap (bi-directional) between business and developers - a gap that does not exist for instance on a construction crew and one that exists by design in a manufacturing situation. Yet software development still retains enough attributes of both to fall on the sword of idealistic thinking. It is this idealistic "extreme" thinking that I treat like the plague.

 I have forgotten more about trends than most people will ever know. All I am saying is keep it real and demand proof of concepts lest be led off a cliff like a lemming. Especially when it's YOUR money at stake.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.