Mark is a graph advocate and field engineer for Neo Technology, the company behind the Neo4j graph database. As a field engineer, Mark helps customers embrace graph data and Neo4j building sophisticated solutions to challenging data problems. When he's not with customers Mark is a developer on Neo4j and writes his experiences of being a graphista on a popular blog at http://markhneedham.com/blog. He tweets at @markhneedham. Mark is a DZone MVB and is not an employee of DZone and has posted 492 posts at DZone. You can read more from them at their website. View Full User Profile

Trade-Offs: Choosing Where to Put the Complexity

03.07.2012
| 3920 views |
  • submit to reddit

On the current application I’m working on we need to make use of some data which comes from another system so we’ve created an import script which creates a copy of that data so that we can use it in our application.

In general we’ve been trying not to do too much manipulation of the data and keeping it close to the initial structure so that if something goes wrong with the import we can more easily trace the problem back to the original data source.

Complexity

While that approach has generally been fine we recently had a situation where the way the data was stored in the original database was quite de-normalised and recreating that structure made some of the code in the data access layer quite messy.

We therefore decided to change the import script to normalise the data, thereby simplifying our database access code.

For now the complexity trade off seems ok because we haven’t had to change the schema that much so it’ll still be reasonably easy to track the data back to the source.

We currently don’t have any tests specifically around the data import because there’s not very much logic going on but if the complexity increases and we start to see problems with the import process then we’ll need to change that.

While we were having this conversation about where we should place the complexity I found it interesting that you could argue for each approach equally convincingly and neither really seemed definitively better – it was back to the world of trade offs!

Published at DZone with permission of Mark Needham, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Lund Wolfe replied on Sat, 2012/03/10 - 4:30pm

It seems to me that your application is coded to your database based on your requirements.  I like the idea of extracting out the third party data/files, especially if the structure/requirements are in flux.  When their data changes structurally, your application and database don't change.  You just have to change your code or script that massages their new data into your database format.

If requirements are also changing on your end, then your code and your database format and your database layer will have to change accordingly.  Am I oversimplifying the problem ? 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.