DevOps Zone is brought to you in partnership with:

Eric Berg is vice president of products at Okta. He has more than 18 years of experience across engineering, marketing and business development and has successfully driven product, business, and marketing for both early stage SaaS companies and high growth software businesses within larger organizations. Eric is a DZone MVB and is not an employee of DZone and has posted 35 posts at DZone. You can read more from them at their website. View Full User Profile

Our Simple Jenkins Configuration and Deployment

04.12.2012
| 17180 views |
  • submit to reddit

This post is originally authored by Denali Lumma on the Okta blog

At Okta, we’ve gone through many iterations of using Jenkins to build and test our software. We use a number of tools to make sure our code works properly, and we like to have Jenkins manage these. The list would be familiar to anyone using the Java environment; PMD, Cobertura, unit and functional tests with JUnit, Selenium tests with testNG, and also some more exotic tools like BURP Security scanner, MogoTest, and SLAMD.

We quickly found that manually installing and configuring Jenkins when we needed a new server for a given task or a special project, or rebuilding an existing server which had crashed or been eaten by Amazon was incredibly time-consuming and error-prone. We implemented a simple way of managing the Jenkins server configuration and job configurations for a given instance with code. I’m going to walk through the approach we took (simplified a bit) as an example of how to do this.

We happen to use Python and Jinja2, however this approach could be taken with any template language that has simple variable replacement and logical controls for looping and conditionals.

So let’s get started.

Library and Package Prerequisites

This assumes that we are running on CentOS, however, we can and do run these same things on Windows and Mac.

Python 2.6

We standardize on this version, but others will work, too. For tutorial reasons, though, I’ll reference this specific libraries we use. If you don’t have Python installed, or you have a different version, you can get 2.6 here: http://python.org/ftp/python/2.6.6/Python-2.6.6.tgz

 

wget http://python.org/ftp/python/2.6.6/Python-2.6.6.tgz
tar fxz Python-2.6.6.tgz
cd Python-2.6.6
./configure
make
sudo make altinstall
sudo ln -s /usr/local/bin/python2.6 /usr/bin/python2.6

 

setuptools

setuptools is needed for Jinja2 installation.

wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
sudo sh setuptools-0.6c11-py2.6.egg

 

Jinja2

This package can be retrieved from here: http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.5.5.tar.gz

wget http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.5.5.tar.gz
tar fxz Jinja2-2.5.5.tar.gz
cd Jinja2-2.5.5
sudo python2.6 setup.py install

Jenkins: The Heart of It

In order to dynamically create a Jenkins instance, it is necessary to understand which files are where and how they are used.

Jenkins is a great piece of software, and part of it’s greatness is the ease of use via the Jenkins web gui, especially for people just starting out. But at it’s heart, Jenkins uses configuration files defined with xml to control the application. Editing these xml configuration files has the same effect as editing Jenkins through the gui. It’s easier to generate xml files with code, but usually less complex than using the build in Jenkins APIs. So this is the approach we took for managing our Jenkins instances.

When Jenkins is fired up with java -jar jenkins.war or some more involved variant of that, it creates a directory called .jenkins which is referred to with the environment variable called JENKINS_HOME.  We will be working with the files created in this directory to control how the server and corresponding jobs are defined.

Go ahead and download the latest stable Jenkins war from the open source project here: http://jenkins-ci.org/

We’ll walk through the contents of the directory created by Jenkins.  This is key to understanding how to create a specific Jenkins instance and associated jobs with code.

Once downloaded, set the JENKINS_HOME variable to something called .jenkins in your current working directory:

export JENKINS_HOME=`pwd`/.jenkins

 

By default, it will be created in a file called .jenkins in the users home directory, but we don’t want that, we want it to be in the current working directory in a hidden directory called .jenkins, alongside the jenkins.war.

Start up the Jenkins web application:

java -jar jenkins.war

 

Let’s walk through the structure.

You should see something like this:

$ cd .jenkins
[.jenkins]$ ls -al
total 32
drwxr-xr-x 10 dlumma staff 340 Mar 19 15:22 .
drwxr-xr-x 4 dlumma staff 136 Mar 19 15:22 ..
-rw-r--r-- 1 dlumma staff 159 Mar 19 15:22 hudson.model.UpdateCenter.xml
-rw------- 1 dlumma staff 1675 Mar 19 15:22 identity.key
drwxr-xr-x 2 dlumma staff 68 Mar 19 15:22 jobs
-rw-r--r-- 1 dlumma staff 907 Mar 19 15:22 nodeMonitors.xml
drwxr-xr-x 16 dlumma staff 544 Mar 19 15:22 plugins
-rw-r--r-- 1 dlumma staff 64 Mar 19 15:22 secret.key
drwxr-xr-x 3 dlumma staff 102 Mar 19 15:22 userContent
drwxr-xr-x 30 dlumma staff 1020 Mar 19 15:22 war

 

This is the skeleton directory structure of what Jenkins uses to drive itself.  More files and directories are added as more functionality and configurations are specified. The directories we deal with and find most useful are 1) the jobs directory (this defines all the jobs shockingly) and the 2) plugins directories (where all the plugins are defined, also shocking, I know).  By default, there is nothing defined in the jobs directory, but there are already a few plugins installed like ant maven, ssh and subversion.

Aside from things in the jobs and plugins directories, the other file we manipulate a lot is called config.xml.  This file resides in the top level under .jenkins. This is the main configuration file for the Jenkins server. We can get Jenkins to generate this file by modifying the http://jenkinsinstance/configure page through the ui. Navigate the the configure page, and add a System Message “My Awesome Jenkins Server”, and click Save. The config.xml should now exist in the .jenkins directory.

$emacs .jenkins/config.xml shows:

 

<?xml version='1.0' encoding='UTF-8'?>

<hudson>
<disabledAdministrativeMonitors/>
<version>1.456</version>
<numExecutors>2</numExecutors>
<mode>NORMAL</mode>
<useSecurity>true</useSecurity>
<authorizationStrategy/>
<securityRealm/>
<projectNamingStrategy/>
<workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
<buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
<systemMessage>My Awesome Jenkins Server</systemMessage>
<jdks/>
<viewsTabBar/>
<myViewsTabBar/>
<clouds/>
<slaves/>
<quietPeriod>5</quietPeriod>
<scmCheckoutRetryCount>0</scmCheckoutRetryCount>
<views>
<hudson.model.AllView>
<owner reference="../../.."/>
<name>All</name>
<filterExecutors>false</filterExecutors>
<filterQueue>false</filterQueue>
<properties/>
</hudson.model.AllView>
</views>
<primaryView>All</primaryView>
<slaveAgentPort>0</slaveAgentPort>
<label></label>
<nodeProperties/>
<globalNodeProperties/>
</hudson>

 

After understanding the basic structure of the files which Jenkins uses to determine it’s configuration and functionality it is possible to go about defining it with source code.

Jenkinizer

Now feel free to blow away your Jenkins instance and all files associated with it and clone the repository setup for this blog post by Okta here:

git clone git@github.com:/okta/jenkinizer

Our jenkinizer repo has four kinds of files: python configuration files, xml template files, static resource files, and scripts.  The structure looks like this (it is a simplified version of our actual internal code and only contains a sub-set of code related to selenium testing for clarity):

jenkinizer/
– config/
—- servers/
—— selenium/
——– selenium_server.py
—- templates/
—— selenium/
——– selenium_server_config.xml
——– selenium_setup.xml
——– selenium_suite.xml
– install/
—- .jenkins/
—— plugins/
——– (a bunch of plugins we have installed)
—— hudson.plugins.emailext.ExtendedEmailPublisher.xml
—- jenkins-env.sh
—- jenkins.sh
—- jenkins.war
– mkconfig.py

Directories

 

config


This directory contains all the template and job configurations.

 

install


Static resources and initialization scripts for starting Jenkins are under install.

A master script called mkconfig.py resides in the main directory. Let’s walk through what this does.

class ServerOptionParser(OptionParser):
    def __init__(self):
        OptionParser.__init__(self)
        self.add_option("-t", "--jenkins-type", action="store", dest="jenkins_type", default="selenium",
                         help="Type of server to generate." )
        self.add_option("-j", "--jenkins-jobset", dest="jenkins_jobset", default="integration",
                          help="Specific host this Jenkins will be deployed to.")
        self.add_option("-o", "--install-dir", dest="install_dir", default="/ebs/ci-build/tools/jenkins",
                          help="Location to install the newly generated server.")
        self.add_option("-b", "--jenkins-branch", dest="jenkins_branch", default="release",
                          help="Branch to use in testing.")
        self.add_option("-e", "--recipient-email", dest="recipient_email", default=" ",
                          help="Email recipient")
        self.add_option("-d", "--discard-old-builds", action="store_true", dest="discard_old_builds", default=False,
                          help="If set, old builds will be discarded.")

This is a simple command line tool made from the lovely OptionParser library. We define a few parameters and arguments. The most important ones are the jenkins-type and jenkins-jobset. In this example I am showing a distilled version of our selenium servers. So the jenkins-type will be selenium and the jenkins-jobset will be either integration, pre-integration, or serial. We have a lot of other kinds of jenkins servers which I’m not including here for things like building and testing mobile, windows artifacts, unit and functional builds, master/ slave configurations and dashboards. If you want to make servers of different kinds, you could follow the pattern here by creating a new jenkins type and giving it a group of jobsets.

The next few methods are used for keeping test results when he upgrade the Jenkins instance itself. We sometimes don’t care, but sometimes do want to preserve the test results collected so far, and add a new job, or alter a setting slightly. These methods handle that.

def make_selected_archive(backup_file_name, base_dir, files, gzip = True):
    backup_file = tarfile.open(backup_file_name, 'w:gz' if gzip else "w")
    current_dir = os.getcwd()
    os.chdir(base_dir)
    for artifact in files:
        if os.path.exists(artifact):
            backup_file.add(artifact)
    backup_file.close()
    os.chdir(current_dir)

def restore_archive(backup_file_name, base_dir, gzip = True):
    backup_file = tarfile.open(backup_file_name, "r:gz" if gzip else "r")
    backup_file.extractall(base_dir)
    backup_file.close()

def backup_old_builds(jobs_dir, job_names):
    archives = {}
    for job in job_names:
        job_dir = jobs_dir + "/" + job
        if os.path.exists(job_dir):
            print "Backing up old builds in %s" % (job_dir)
            backup_file_name = tempfile.gettempdir() + "/" + job + ".tar.gz"
            make_selected_archive(backup_file_name, job_dir, ["./builds", "./lastStableBuild", "./lastSuccessfulBuild", "./nextBuildNumber"])
            archives[job] = backup_file_name
    return archives

Then comes the main functionality, which generated the configuration files for this Jenkins server and its jobs.
We get the type, jobset, installation_dir and email to be used in this instance.

def build_server(options, base_path = os.getcwd()):

    # We categorize Jenkins instances depending on the jobs they run.  We have Jenkins instances that run selenium,
    # run unit and functional tests, build artifacts and deploy artifacts. This defined the kind of Jenkins we are
    # deploying.
    jenkins_type = options.jenkins_type

    # The set of jobs to deploy.
    jenkins_jobset = options.jenkins_jobset

    # The location to install this Jenkins instance to on the filesystem.
    install_dir = options.install_dir

    # Email.
    recipient_email = options.recipient_email

We load up the template files defined for our type of server.

    # Make our python job and server configuration files reachable.
    sys.path.extend([base_path + "/config/servers"])

We import the specific dict defining the specified jobset and jenkins type.

    # Define the template environment.
    template_env = Environment(loader=FileSystemLoader(base_path + '/config/templates/' + jenkins_type + '/'))

We define the installation locations.

    # Import the specific server and job data for this jenkins type
    # and this jobset.
    server_config = __import__(jenkins_type).__dict__[jenkins_jobset]
    server_config["env"]["EMAIL"] = recipient_email
    server_config["env"]["BRANCH"] = options.jenkins_branch

If specified, we backup existing archives and then delete the current Jenkins installation.

    jenkins_dir = install_dir + "/.jenkins"
    jobs_dir = jenkins_dir + "/jobs"

    # Backup archived files and
    # remove the old instance if it exists.
    archives = {}
    if os.path.exists(install_dir):
        if not options.discard_old_builds:
            archives = backup_old_builds(jobs_dir, [job["job_name"] for job in server_config["jobs"]])
        shutil.rmtree(install_dir)

We copy all of the static resource files to the installation directory and print out an informational message.

    # Copy all of the static Jenkins files to the installation directory
    shutil.copytree(base_path + "/install", install_dir)

    print "Generating configuration for jenkins type %s (writing to %s)" % (jenkins_type, install_dir)

We generate the main Jenkins configuration file with the given configuration file and server configuration.

    # Create the jenkins main config file.
    with open(jenkins_dir + "/config.xml", "w") as f:
        t = template_env.get_template("server-config.xml")
        f.write(t.render(server_config))

And finally, we generate the jobs configuration files, write them to the installation directory and restore any archived files.

    # Create the jobs config files.
    for job in server_config["jobs"]:
        job_name = job[ "job_name"]
        job_template = job["template_file" ]
        job.update(server_config)

        print "Creating job %s from %s" % (job_name, job_template)
        job_dir = jobs_dir + "/" + job_name
        os.makedirs(job_dir)
        with open(job_dir + "/config.xml", "w") as f:
            t = template_env.get_template(job_template)
            f.write(t.render(job))

        if archives.has_key(job_name):
            print "Restoring old builds for %s" % (job_name)
            restore_archive(archives[job_name], job_dir)

Now let’s look at the files used. The template files under config/templates/selenium are xml files with simple variables defined in the selenium.py configuration file. These were originally copied from the Jenkins installation and edited with variables for dynamic rendering based on the data defined in the Python files.

The main job data is defined in config/servers/selenium.py. Each jobset is defined with a dict built from get_default() which contain the keys env, os, chained, views and jobs. The jobs key references another list which contains dicts which specific job data such as job_name, template_file, max_builds_to_keep, repo, and others.

Updating an existing Jenkins selenium server with the integration jobset would look like this:

$ python2.6 mkconfig.py -t selenium -j integration
Generating configuration for jenkins type selenium (writing to /ebs/ci-build/tools/jenkins)
Creating job okta.build from selenium-setup.xml
Creating job core.chrome from selenium-suite.xml
Creating job apps.chrome from selenium-suite.xml
Creating job plugin.apps.chrome from selenium-suite.xml
Creating job core.firefox.latest from selenium-suite.xml
Creating job apps.firefox.latest from selenium-suite.xml
Creating job plugin.apps.firefox.latest from selenium-suite.xml
Creating job core.ie.8 from selenium-suite.xml
Creating job apps.ie.8 from selenium-suite.xml
Creating job plugin.apps.ie.8 from selenium-suite.xml
Creating job core.firefox.3.6 from selenium-suite.xml
Creating job apps.firefox.3.6 from selenium-suite.xml
Creating job plugin.apps.firefox.3.6 from selenium-suite.xml
Creating job core.ie.9 from selenium-suite.xml
Creating job apps.ie.9 from selenium-suite.xml
Creating job plugin.apps.ie.9 from selenium-suite.xml

 

Now go to the install directory and start it up.

 

$ cd /ebs/ci-build/tools/jenkins
$ ./jenkins.sh
JENKINS_HOME=/ebs/ci-build/tools/jenkins/.jenkins

 

Tail the log to make sure everything is running smoothly.

 

$ tail -f jenkins.log
Running from: /ebs/ci-build/tools/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Apr 1, 2012 10:04:26 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Jenkins home directory: /ebs/ci-build/tools/jenkins/.jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Apr 1, 2012 10:04:29 PM winstone.Logger logInternal
INFO: HTTP Listener started: port=8080
Apr 1, 2012 10:04:29 PM winstone.Logger logInternal
INFO: AJP13 Listener started: port=8009
Apr 1, 2012 10:04:29 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled
Apr 1, 2012 10:04:30 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Apr 1, 2012 10:04:32 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Apr 1, 2012 10:04:36 PM org.apache.sshd.common.util.SecurityUtils$BouncyCastleRegistration run
INFO: Trying to register BouncyCastle as a JCE provider
Apr 1, 2012 10:04:36 PM org.apache.sshd.common.util.SecurityUtils$BouncyCastleRegistration run
INFO: Registration succeeded
Apr 1, 2012 10:04:36 PM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Apr 1, 2012 10:04:36 PM org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 52810
Apr 1, 2012 10:04:36 PM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@2be3d80c: display name [Root WebApplicationContext]; startup date [Sun Apr 01 22:04:37 PDT 2012]; root of context hierarchy
Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@2be3d80c]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1e94b0ca
Apr 1, 2012 10:04:37 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1e94b0ca: defining beans [authenticationManager]; root of factory hierarchy
Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@23a65a18: display name [Root WebApplicationContext]; startup date [Sun Apr 01 22:04:37 PDT 2012]; root of context hierarchy
Apr 1, 2012 10:04:37 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@23a65a18]: org.springframework.beans.factory.support.DefaultListableBeanFactory@29cc3436
Apr 1, 2012 10:04:37 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@29cc3436: defining beans [filter,legacy]; root of factory hierarchy
Apr 1, 2012 10:04:37 PM hudson.TcpSlaveAgentListener
INFO: JNLP slave agent listener started on TCP port 7070
Apr 1, 2012 10:04:42 PM hudson.triggers.SCMTrigger$Runner run
INFO: SCM changes detected in okta.build. Triggering #1
Apr 1, 2012 10:04:48 PM hudson.WebAppMain$2 run
INFO: Jenkins is fully up and running
Apr 1, 2012 10:04:58 PM hudson.model.DownloadService$Downloadable doPostBack
INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
Apr 1, 2012 10:04:58 PM hudson.model.DownloadService$Downloadable doPostBack
INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
Apr 1, 2012 10:04:58 PM hudson.model.DownloadService$Downloadable doPostBack
INFO: Obtained the updated data file for hudson.tools.JDKInstaller
Apr 1, 2012 10:05:01 PM hudson.model.UpdateSite doPostBack
INFO: Obtained the latest update center data file for UpdateSource default
Apr 1, 2012 10:05:42 PM hudson.triggers.SCMTrigger$Runner run
INFO: SCM changes detected in okta.build. Triggering #2
Apr 1, 2012 10:06:42 PM hudson.triggers.SCMTrigger$Runner run
INFO: SCM changes detected in okta.build. Triggering #3

 

Open your browser and see the newly created Jenkins instance at: http://localhost:8080

This is a simple and effective way to manage multiple Jenkins instances intended to do different kinds of things. It is more maintainable than configuring a new instance through the gui, which is very error prone and takes a lot of time. Yet it is straightforward and doesn’t involve any advanced programming. For managing a lot of Jenkins instances doing a lot of different things, this approach has worked well for us.

Published at DZone with permission of Eric Berg, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)