Continuous Delivery for Microservices with Jenkins and the Job DSL Plugin

In classical Monolith-based environments there are normally not so many separate release artifacts, so it’s relatively easy to maintain the Jenkins build jobs manually. But in a Microservice architecture the number of deployment units increases and an automated solution is needed here. Also in enterprise environments with a lot of build jobs it doesn’t really make sense to create every job manually. Furthermore, in a lot of jenkins installations we see often Continuous Delivery Pipelines with separate jobs for building, releasing, deploying and testing the applications.

A Continuous Delivery pipeline for a monolithic application looks often likes this:

1. build-application
2. release-application
3. deploy-application-to-dev
4. run-tests
5. deploy-application-to-qa
6. …

Goals

In this blog post I will show you a way how you can achieve these goals:

1. Automatically generate a jenkins job, when a user checks in a new repository/artifact into the SCM
2. Reduce the number of jenkins jobs to 1 job for each release artifact, which is able to build, release and deploy the application/microservice

Build and Release

Before we dive into the magic of the Job DSL Plugin I will show you my preferred way of releasing an artifact with Maven. Maybe some of you are also not really satisfied with the Maven Release Plugin. I see two problems here:

1. Two many redundant steps (e.g. 3 full clean/compile/test cycles!!)
2. Instability – the plugin modifies the pom.xml in the SCM, so there are often manual steps needed to revert the changes if something fails in the release build

For further informations I can recommend the blog posts from Axel Fontaine. He also shows an alternative to release your artifacts in a clean and simple way, which we are also using in this blog post.

The continuous build is very simple:

1. Jenkins triggers build process on SCM commit
2. Execute normal build process:

mvn clean package

The release build is also very simple:

1. User triggers release build
2. Replace SNAPSHOT-Version in pom.xml

mvn build-helper:parse-version versions:set -DnewVersion=${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}-${BUILD_NUMBER}

3. Execute normal build process

mvn clean package

4. Deploy artifact to artifact repository

mvn deploy

5. Tag version in SCM

mvn scm:tag

Generate a single job

To generate the jobs automatically we need the Job DSL Plugin in Jenkins. You can install it through the normal Jenkins plugin mechanism. I can definitely recommend this plugin. I’m using it in some projects for about a year to generate hundreds of jobs and we didn’t have any big problems yet. It’s also very simple to use. I recommend to start with this detailed tutorial here. In our setup the seed job id called “job-generator” and contains for the first step this DSL script:

job(type: Maven) {
    name("batch-boot-demo")
    triggers { scm("*/5 * * * *") }
    scm {
		git {
		    remote {
		        url("https://github.com/codecentric/spring-samples")
		    }
		    createTag(false)
		}
	}
	rootPOM("batch-boot-demo/pom.xml")
	goals("clean package")
	wrappers {
		preBuildCleanup()
		release {
			preBuildSteps {
				maven {
					mavenInstallation("Maven 3.0.4")
					rootPOM("${projectName}/pom.xml")
					goals("build-helper:parse-version")
					goals("versions:set")
					property("newVersion", "\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}-\${BUILD_NUMBER}")
				}
			}
			postSuccessfulBuildSteps {
				maven {
				    rootPOM("${projectName}/pom.xml")
					goals("deploy")
				}
				maven {
					rootPOM("${projectName}/pom.xml")
					goals("scm:tag")
				}
				downstreamParameterized {
					trigger("deploy-application") {
						predefinedProp("STAGE", "development")
					}
				}
			}
		}
	}
	publishers {
		groovyPostBuild("manager.addShortText(manager.build.getEnvironment(manager.listener)[\'POM_VERSION\'])")
	}
}

The above DSL script contains all the discussed build and release steps from the first section of the blog post. To integrate the release step into the existing build job, we are using the Jenkins Release Plugin, which adds a Release Button (see screenshot) to the build job UI. The “groovyPostBuild”-element adds the version number to the build history overview (see screenshot), so you can directly see it, when you open the job view. To play around with the plugin I prepared a Docker image which contains all the needed stuff here. Please follow the setup instructions on Github. Alternatively you can also use your own Jenkins and install the plugins by yourself (see a list here).

Deployment

Actually, the above job is able to build and release artifacts, but the deploy step is missing. Because we don’t want to add separate deploy jobs for each artifact, we use the Promoted Builds Plugin. This plugin introduces the notion of a “promotion”. A “promoted” build is a successful build that passes additional criteria. In our scenario we manually promote builds, when they are deployed to a specific stage. These promoted builds (released artifacts) get a nice star in a specific colour in the build history view (see screenshot below) for every deployment stage.

Add the following DSL snippet to the existing script (see also job-dsl-example.groovy):

promotions {
	promotion("Development") {
		icon("star-red")
		conditions {
			manual('')
		}
		actions {
			downstreamParameterized {
				trigger("deploy-application","SUCCESS",false,["buildStepFailure": "FAILURE","failure":"FAILURE","unstable":"UNSTABLE"]) {
					predefinedProp("ENVIRONMENT","dev.microservice.com")
					predefinedProp("APPLICATION_NAME", "\${PROMOTED_JOB_FULL_NAME}")
					predefinedProp("BUILD_ID","\${PROMOTED_NUMBER}")
				}
			}
		}
	}
	promotion("QA") {
		icon("star-yellow")
		conditions {
			manual('')
			upstream("Development")
		}
		actions {
			downstreamParameterized {
				trigger("deploy-application","SUCCESS",false,["buildStepFailure": "FAILURE","failure":"FAILURE","unstable":"UNSTABLE"]) {
					predefinedProp("ENVIRONMENT","qa.microservice.com")
					predefinedProp("APPLICATION_NAME", "\${PROMOTED_JOB_FULL_NAME}")
					predefinedProp("BUILD_ID","\${PROMOTED_NUMBER}")
				}
			}
		}
	}
	promotion("Production") {
		icon("star-green")
		conditions {
			manual('prod_admin')
			upstream("QA")
		}
		actions {
			downstreamParameterized {
				trigger("deploy-application","SUCCESS",false,["buildStepFailure": "FAILURE","failure":"FAILURE","unstable":"UNSTABLE"]) {
					predefinedProp("ENVIRONMENT","prod.microservice.com")
					predefinedProp("APPLICATION_NAME", "\${PROMOTED_JOB_FULL_NAME}")
					predefinedProp("BUILD_ID","\${PROMOTED_NUMBER}")
				}
			}
		}
	}
}

Through the element “manual(‘user’)” it’s possible to restrict the execution of a promotion to a specific user or group. Especially in production this does make sense When the promotion is manually approved the promotion actions get executed. In our scenario we only trigger the downstream deploy job. After the successful execution the build gets a coloured star. With the “upstream(‘promotionName’)”-element you can make promotions dependent on another promotion, e.g. a deployment to production is only allowed, when the artifact was already deployed to development and qa. This all works great, but there is also a bad message for you. The Promoted Builds Plugin is actually not officially supported by the Job DSL Plugin. My colleague Thomas has implemented some really great stuff, but the Pull Request is not merged until today. But there is an alpha release available, which can be used to generate the promotions. Hopefully we hear some better news from it soon. An alternative solution is to create a template job with the defined promotions which is then referenced with the using-element in the Job DSL definition.

Generate multiple jobs

In the step above the seed job generates only one jenkins job (“batch-boot-demo”). So let’s generate multiple jobs from the Job DSL template. In our example we are using an repository on Github with some Spring Boot projects. Thus it’s simple to get the contents of the repository over the Github API.

def repository = 'codecentric/spring-samples'
def contentApi = new URL("https://api.github.com/repos/${repository}/contents")
def projects = new groovy.json.JsonSlurper().parse(contentApi.newReader())
projects.each {
    def projectName = it.name
    job(type: Maven) {
        name("${projectName}")
        triggers { scm("*/5 * * * *") }
 
		[...]
	}
}

Alternatively you can detect your projects with a simple shell script (e.g. interact directly with the SCM-API) and write the contents into a file:

codecentric:spring-samples

And then read it inside the DSL script:

readFileFromWorkspace("projects.txt").eachLine {
 
	def repository = it.split(":")[0]
	def projectName = it.split(":")[1]
 
	[...]
 
}

Now it’s very important to create a SCM trigger for the seed job, because it should run at any time a project is checked in or deleted. The Job DSL Plugin is even so intelligent that it deletes a Jenkins job if the corresponding repository/project was removed.

What do you think about it? Are you already using the Job DSL Plugin?

The post Continuous Delivery for Microservices with Jenkins and the Job DSL Plugin appeared first on codecentric Blog.