An Introduction to CICD Pipelines

It can often be the case, even in 2021, that many teams still struggle to ship software customers hands at speed due to lack of consistency and excessive manual activities. Continuous integration (CI) and continuous delivery (CD) can deliver software to a production environment with speed, safety, and reliability in a wonderfully predictable and repeatable way. In this post, I will outline the basic concepts of a CICD pipeline and outline some of the common stages. If you are someone who already has already automated their way to application deployment greatness using pipelines, this article is probably not for you. But if you are thinking of joining the ranks of CICD pipeline heroes… please read on.

CICD is a phrase which has been has become standard fayre for any DevOps professional or developer in recent years. I am assuming you, dear reader, have joined me here because you are either seeking to learn more about CICD pipelines and how to implement them, or you simply want to validate your own thoughts on the subject. This article will provide a gentle introduction to CICD pipelines and how you might want to go about implementing them. This article is geared as an overview — I’ll talk more in future articles about the technical aspects of implementation.

“At an abstract level, a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users.”― David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Firstly, let’s define CICD in the context of this article. Sometimes, the “CD” part of this phrase can mean different things to different people, but for the purposes of this article I will call it “Continuous Delivery”. Whether that is delivery to a test environment, or all the way into production depends on the scale of your system and your level of process maturity. CD can also be referred to as “Continuous Deployment”. Continuous deployment extends continuous delivery in that every change that passes all stages of your production pipeline is released to your customers.

Continuous Integration and Continuous Delivery are software development disciplines that attempt to ‘shift left’ by detecting quality issues as early in the software development lifecycle as possible. This is under the premise that the further to the right in the development process that quality issues are found, the more costly it will be to your organisation in terms of time, resources, rework, customer satisfaction / confidence etc.

An Example Pipeline

For the purposes of this article, we will use the highly popular Jenkins to help orchestrate the execution of the various build and verification tasks involved in the develop, test and release phases of an automated CICD pipeline. Jenkins provides development teams with the ability to implement a project’s entire build/test/deploy pipeline in a “Jenkinsfile” and store that alongside their code. This allows your Jenkins pipeline config to be treated as another piece of code checked into source control, living happily alongside the very code of the application it builds. When teams introduce some form test automation (e.g. their application unit tests), those activities can easily be plugged into a Jenkins pipeline to introduce quality gateways into your build and produce faster feedback in the event of test failures.

When developing a pipeline, our focus should be on creating fast feedback loops through automation of tasks such as unit testing and static code analysis (CI) and automated deployment and testing (CD). The conclusion of these practices are fully automated ‘pipelines’ where new features and bug fixes can be deployed with no manual intervention or verification whatsoever. We must always operate with the concept of “Build once, deploy anywhere” foremost in our minds. Compiling code again each time you deploy is expensive from a pipeline latency perspective and a single build should be suitable for any intended deployment target. Remember, we are trying to create tight feedback loops so identifying and removing unnecessary operations is an important part of implementing CICD pipelines. Let’s take a look at an example pipeline below:

Code Commit
When code changes are committed to a repository such as Git, a post-commit hook can automatically start the Jenkins pipeline build process. Alternatively, Jenkins can “poll SCM” to build at scheduled times (i.e. if new changes are found). You also have the option of setting up a multibranch pipeline which will automatically detect a new branch and fire a build (as long as a jenkinsfile is present in the codebase you have branched from).

Build
This stage executes the appropriate build engine for your source code (e.g. maven for java). If required, this stage can also pull external references — for example, java projects may pull their maven repositories from an RPM utility such as Nexus, or download packages directly from the internet (though this is a slower option if you build regularly).

Code Analysis
Perform code analysis using a tool such as Sonarqube, PyCharm, Embold etc. This stage will execute a static scan analysis of source code using one or more sets of pre-defined rules. This is a vital pipeline step as it helps the development team identify code validation issues or vulnerabilities early in the lifecycle (i.e. immediately after commit or on pull request). Most static code analysis tools provide functionality for development teams to add their own rule profiles. You may also want to consider adding an additional pipeline stage for providing finer grained verification for your coding language (e.g. running “terraform fmt” for Terraform code).

Unit tests
Execute unit tests with the appropriate test framework (e.g. junit) and generate code coverage stats for unit test coverage (e.g. using Cobertura). Unit Test results are also available in the Jenkins job output and failure should result in the build being aborted.

Deploy
This stage can include infrastructure provisioning and application deployment.

We are in the era where demands for faster software shipping and packaging fast releases on the fly are expected. So being able to adopt a more robust project cycle that accelerates the delivery pipeline is crucial. Implementing DevOps services is the optimal way to get the effective collaboration between cross-functional teams that is needed, but… If you cannot deliver your code in a predictable and repeatable way through the CI/CD process, your team will never hit your full potential. We’ll talk more on this in future articles, but for now please contact me if you would like to learn more about any of the concepts here, or the way in which one may go about applying the more technical aspects of the pipeline.