Mark is a graph advocate and field engineer for Neo Technology, the company behind the Neo4j graph database. As a field engineer, Mark helps customers embrace graph data and Neo4j building sophisticated solutions to challenging data problems. When he's not with customers Mark is a developer on Neo4j and writes his experiences of being a graphista on a popular blog at He tweets at @markhneedham. Mark is a DZone MVB and is not an employee of DZone and has posted 534 posts at DZone. You can read more from them at their website. View Full User Profile

Pain Driven Development

  • submit to reddit

My colleague Pat Fornasier has been using an interesting spin on the idea of making decisions at the last responsible moment by encouraging our team to ‘feel the pain’ before introducing any constraint in our application.

These are some of the decisions which we’ve been delaying/are still delaying:

Dependency Injection

Everyone in our team comes from a Java/C# background and one of the first technical decisions that gets made on applications in those languages is which dependency injection container to use.

We decided to just create a trait where we wired up the dependencies ourself and then inject that trait into the entry point of our application. Effectively it acts as the ApplicationContext that a framework like Spring would provide.

I was fairly sure that we’d need to introduce a container fairly quickly but it’s been 10 weeks and we still haven’t felt the need to do that and our application is simpler as a result.

Data Ingestion

As I mentioned in an earlier post we have to import around 5 million documents into our database by the time the application goes live.

Our initial attempt at writing this code was single threaded and it was clear that there were many places where performance optimisations could be made.

Since we were only ingesting a few thousand documents at that stage it still ran pretty quickly so Pat encouraged us to wait until we felt the pain before making any changes.

That duly happened once the number of documents increased and it started taking 3/4 hours to run the job in our QA environment.

We then spent a couple of days working out how to make it possible to process the documents more quickly.

Complex markup in documents

As I mentioned a couple of months ago the application we’re working on is mainly about taking data from a database and applying some transformations on it before showing it to the user.

We decided to incrementally add different types of documents into the database.

This meant that initially all our transformations involved just getting a text representation of XML nodes even though we knew that eventually we’d need to do more processing on the data depending on which tags appeared.

These data transformations actually turned out to be more complicated than we’d imagined so we might have delayed the pain here a little bit too long.

On the other hand we were able to show early progress to our business stakeholders which probably wouldn’t have been the case if we’d tried to take on the complex markup all at once.


One thing to note with this approach is that we need to make sure there is a feedback mechanism to recognise when we are feeling pain otherwise we’ll end up going beyond the last responsible moment more frequently.

There will probably also be more complaints about things not being done ‘properly’ since we’re waiting for longer until we actually do that.

We have a code review that the whole team attends for an hour each week which acts as the feedback mechanism and we recently starting using Fabio’s effort/pain wall to work out which things were causing us most pain.



Published at DZone with permission of Mark Needham, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Mike Dzone replied on Tue, 2011/08/23 - 8:08am

I like your thoughts about dependency injection.  There is really no need to use Spring to do this kind of thing.  It is a software pattern and very easy to do yourself without having complexity of Spring. 

Aaron Digulla replied on Wed, 2011/08/24 - 6:41am

As for tackling complex cases early: Don't. Ever.

Why not? Several reasons:

- When you start, you know little about the problem domain. As you progress, you learn. So any later solution will become better because you simply know more

- Starting with the complex stuff will discourage/demotivate you, hampering progress

- If the code is clean, it should be simple to add complexity at any stage. Another reason to avoid it when you're least prepared

- If you grow with the complexity, your tools will grow as well. For example, you will have a framework to set up your tests. So when the first complex case comes along, you only have to worry about the small diff between what you have and what you need - instead of the diff between nothing and what you need.

- The complex cases will show you weaknesses in your framework but they will do so at a time when you already have some ideas to handle them. Those weaknesses don't appear out of the blue - you already know about them. You just didn't have the time/motivation to address them. So while it feels like "I should have fixed this in the first place," this is an illusion. In the first place, you didn't even knew what the problem would be.

Lund Wolfe replied on Sat, 2011/08/27 - 1:45am

"If your only tool is a hammer, everything looks like a nail"
"Keep It Simple Stupid"
"You Aren't Gonna Need It"
"Big Design Up Front"
"Premature optimization"

Not every application should be Spring/Hibernate. It's amazing how many applications out there are unintelligible and nearly unmaintainable, though they provide relatively simple functionality. This is usually caused by over-engineering and ill-conceived future proofing and too many levels of indirection that serve no purpose.

I'm maintaining an application that has the GUI components defined in a property file and methods that are executed through reflection which makes it very difficult to understand and learn using the IDE to trace through with "find usages" and "go to implementation".

Wait to implement a tool/technique until it will actually simplify your code. It's not Pain Driven Development. It's XP's "Do The Simplest Thing That Could Possibly Work".

Sebastian Beltramini replied on Thu, 2011/09/15 - 2:37pm in response to: Mike Dzone

What are those Spring complexities you talk about?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.