DevOps Zone is brought to you in partnership with:

I work at Tangent Labs, a digital agency, writing applications in python and django. I spend most of my free time hacking. I run commandlinefu.com and am the author of the e-commerce framework django-oscar amongst other things. I used to be a mathematician; I have a PhD in Mathematics from the University of Nottingham and an associated interest in cryptic crosswords, chess and playing devil's advocate. David is a DZone MVB and is not an employee of DZone and has posted 27 posts at DZone. You can read more from them at their website. View Full User Profile

Continuously rebuild your project

06.19.2014
| 5573 views |
  • submit to reddit

New developers joining a project will often find that the project won't build cleanly on their machine, and hours of time will be sunk into setting up the project so work can start. This is sad and expensive for all concerned.

This is a particular menace in agencies (or anywhere with lots of small projects) where a large team of developers need to jump between projects. Tools like Vagrant and Docker can help but aren't the panacea they first seem to be [*].

Counter this by using continuous integration to build your project from scratch. Then any changes that break the build process (such as database schema changes not applying correctly) will be spotted early. New team members will be dishing out high-fives when their development environment is built and primed with sample data sixty seconds after cloning the repo.

Tips

It should be trivial to get a project working locally. At Tangent, projects use a makefile for common tasks. Setting up a working version of the project is as simple as:

$ make

It's helpful if you can template new projects to embed good practices like this. We frequently use Django and maintain a boilerplate Django project for this purpose. It includes a makefile along these lines:

# Build a working version of the project
build: clean virtualenv database

# Delete all temporary or untracked files
clean:
    -find . -type f -name "*.pyc" -delete
    -rm -rf www/public/media/*

# Update the virtualenv
virtualenv:
    pip install -r www/deploy/requirements.txt

# Create a database populated with data
database:
    python www/manage.py reset_db --router=default --noinput
    python www/manage.py syncdb --noinput
    python www/manage.py migrate
    # Load any project fixtures to pre-populate the initial database
    python www/manage.py loaddata fixtures/*.json

test:
    cd www && py.test

ci: test database

Witness the ci target which runs the test suite and builds the database, effectively smoke-testing that the migrations apply correctly and the fixtures load (which is where we've historically had pain).

We use Travis for CI and our template .travis.yml looks a little like this:

language: python

python:
  - '2.7'

install:
  - make virtualenv

# Use the same database as used in production
before_script:
  - psql -c 'CREATE ROLE test_role login createdb superuser;' -U postgres
  - psql -c 'CREATE DATABASE test_db OWNER test_role;' -U postgres

script:
  - make ci

which means that, by default, all new projects will be built from scratch as part of continuous integration. You should do this.

Django-specific issues

For the record, here's some of the build issues we've encountered in Django projects (both internal and external) Most stem from South migrations, which worked fine when applied piecemeal by the incumbent team but fail when run on a blank database. For instance:

  • Migrations fail to apply as there are dependencies between migrations which haven't been captured. This is easily solved by employing South's support for dependent migrations (eg adding depends_on to the relevant migration class).
  • Migrations fail as they import models directly rather than using the reconstituted models that South provides. This is a beginner mistake really but still quite common. Fortunately, it's trivial to fix.
  • Migrations import and call functions that are no longer defined (but did exist when the migration was originally written).
  • Migrations create instances of models from other apps where South's serialised version is out of sync with the database schema. This can be tricky to fix as you can get circular dependencies between migrations. Often you'll need to rewrite migrations to create models in the migrations of their own apps.

Andrew Ingram has written up an excellent summary of common South pitfalls.

[*] For instance, it's not trivial to share folders with a Docker container on OSX. See https://gist.github.com/codeinthehole/7ea69f8a21c67cc07293
Published at DZone with permission of David Winterbottom, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)