Mark is a graph advocate and field engineer for Neo Technology, the company behind the Neo4j graph database. As a field engineer, Mark helps customers embrace graph data and Neo4j building sophisticated solutions to challenging data problems. When he's not with customers Mark is a developer on Neo4j and writes his experiences of being a graphista on a popular blog at http://markhneedham.com/blog. He tweets at @markhneedham. Mark is a DZone MVB and is not an employee of DZone and has posted 492 posts at DZone. You can read more from them at their website. View Full User Profile

Gaming the system: Some project examples

10.28.2011
| 2602 views |
  • submit to reddit

Earlier this year Liz Keogh gave a talk at QCon London titled ‘Learning and Perverse Incentives: The Evil Hat‘ where she eventually encouraged people to try and game the systems that they take part in.

Over the last month or so we’ve had two different metrics visibly on show and are therefore prime targets for being gamed.

The first metric is one we included on our build radiator which shows how many commits to the git repository each person has for that day.

We originally created the metric to try and see which people were embracing git and committing locally and which were still treating it like Subversion and only committing when they had something to push to the central repository.

The other advantage we wanted to try and encourage is that by creating lots of small commits it’s easier for someone browsing ‘git log’ to see what’s happened over time just from glancing at the commit messages.

Bigger commits tend to mean that changes have been made in multiple places and perhaps not all those changes are related to each other.

Since we made that metric visible the number of commits have visibly increased and it’s mostly been positive because people tend to push to the central repository quite frequently.

There have, however, been a couple of occasions where people have made 10/15 commits locally over the day and then pushed them all at the end of the day and gone straight to the top of the leader board.

IMG 20111026 175248 1

The disadvantage of this approach is that it means other people on the team aren’t integrating with your changes until right at the end of the day which can lead to merge hell for them.

There have also been some times when people’s count has artificially increased because they’ve checked in, broke the build and then checked in again to fix it.

We’re going to try and find a way to combine local commits with remote pushes in a combined metric as our next trick.

Another metric which we’ve recently made visible is the number of points that we’ve completed so far in the iteration.

Previously we’ve had this data available in our Project Manager’s head and in Mingle but since a big part of how the team is judged is based on the number of points ‘achieved’ the team asked for the score to be made visible.

Since that happened from my observation we’ve ‘achieved’ or got very close to the planned velocity every week whereas before that it was a bit hit and miss.

I think sub consciously the estimates made on stories have started to veer towards the cautious side whereas previously they were probably more optimistic.

Another change in behaviour I’ve noticed is that people tend to postpone any technical tasks they have to do when we’re near the end of an iteration and instead keep focus on the story to ensure it gets completed in time.

We’ve also seen a couple of occasions where people stayed 2/3 hours longer on the last day of the iteration to ensure that stories got signed off so the points could be counted.

It’s been quite interesting to observe how behaviour can change based on increasing the visibility of metrics even when in the first case it’s actually irrelevant to the perception of the team.

 

From http://www.markhneedham.com/blog/2011/10/26/gaming-the-system-some-project-examples/

Published at DZone with permission of Mark Needham, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Amara Amjad replied on Sun, 2012/03/25 - 2:39am

Don't forget about developers using squashed merges and interactive rebasing and decreasing the number of commits which eventually make it to the remote repository. I commit *very frequently* locally, but routinely squash them into fewer more logical commits before pushing. Depending on what the metrics end up being used for, it'd work out better for me to just push everything un-squashed, but that negatively affects the codebase in favor of gaming the results (unless people like "did the thing" commits!).

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.