INDUSTRY USE CASE OF JENKINS IN VOX- MEDIA PRODUCT

What is jenkins ?

→Jenkins is a continuous integration automation tool (open source) which is installed on the server where build takes place. Continuous Integration refers to a process where developers commit their code into a commonly created repository as soon as they complete the development.

→In the CI/CD process also results in increased overhead and hence we have pipelines in Jenkins to maintain the users and permissions tagged to them. The same is applied to increasing projects.

→Once the building increase, the corresponding need increases to create test agents along with their maintenance.

Vox Media, Inc. is an American mass media company based in Washington, D.C., and New York City. The company was established in November 2011 by Jim Bankoff and Trei Brundrett to encompass SB Nation, a sports blog network founded in 2005, and The Verge, a technology news website launched alongside Vox Media.

Problem :

For a while now, the Vox Media Development Experience team, formerly known as the Systems Engineering team, has relied on Jenkins to perform CI and CD tasks. It has proven to be a decent tool with customizable platform and low costs. However, they have run into bumps over the years:

→Regular breakage of agents requiring manual cleanup, including workspaces not getting cleaned up as expected in between jobs and resulting in, for example, duplicated docker networks that prevent jobs from running.

→While it is technically possible to have a Jenkins cluster with HA, it isn’t possible to ensure high availability on our Kubernetes cluster by increasing the number of replicas. A single master instance cannot be duplicated as there is no clear way to have multiple instances share states.

→With a logging view that is dense and confusing to read. The Blue Ocean view looks more modern but it is buggy (e.g. broken replay stage button, broken logs table), and it lacks some important features from the original UI including a way to add parameters to jobs.

→Jenkins uses its own version of a scripting language called Groovy which makes it difficult for teams to take ownership of their pipelines since they’re not necessarily familiar with it.

Since these issues were affecting engineers’ productivity, we’ve had to find some alternatives and explored three different tools: Circle CI, Semaphore, and GitHub Actions.

After some extensive research, their team chose GitHub Actions. It allows us to keep a level of customization similar to that provided by Jenkins, it is straightforward to set up and being included with GitHub means that it could empower engineers to work on their CI/CD pipelines. Finally, the docs and UX are great so far, and frankly, it is simply new and exciting!

Building the GitHub Actions Shared Library

Jenkins architecture uses a private shared library with scripts that let us deploy to Kubernetes, create database snapshots, and more. They first considered turning these scripts into individual Actions, but GitHub doesn’t support private shared actions yet. Instead, They opted to house all scripts under one private repo, appropriately named the GitHub Actions Shared Library (GASL).

They chose Bash as the initial official language because the whole team is familiar with it and it’s easier to get started as it doesn’t come with any boilerplate code. Ease of development mattered a lot since we were eager to get a functional release of GASL within a few months.

Given that the generic scripts provided by the Jenkins shared library, such as a helm linter, are available in the GitHub Actions marketplace, we only needed to port custom scripts from it. We also had to add some scripts to prepare GitHub runners’ environments for each workflow run; these included authenticating our GASL service account with ECR, setting up docker to work with our registry, and installing a docker container to allow the install of private dependencies inside containers by injecting the right SSH key.

Things looked simple enough but lacked a testing infrastructure and a general directory structure. More importantly, writing complex custom scripts in Bash was going to be hellish. For example, Docker images are environment specific and are tagged with a particular base64 encoding scheme that utilizes the git SHA. Writing this in Bash might’ve been possible, but at what cost? We were headed down a bad path and had to pivot.

Luckily, Github provides TypeScript Actions templates. It made more sense to get rid of the testing and structural gaps, and continue iterating on GASL with a framework specifically made for Actions. Getting to work with a new language was a welcomed bonus!

Using GASL

To use the shared library, repos must add the `actions/checkout` step to their workflow:

name: checkout shared library

uses: actions/checkout@v2

with:

repository: voxmedia/github-actions-shared-library token: ${{ secrets.GITHUB_ACCESS_TOKEN }}

ref: v3

→The `ref` parameter is particularly helpful because it lets us specify the desired GASL version. We use git tagging to create/update releases and pin versions to releases. It also makes updated releases automatically available to all the repos using it.

→After checking out the Bash shared library, a repo owner would’ve had to manually add a series of secrets in order for the authenticating scripts to work. This is no longer the case since adding a `push-secrets` action in the TypeScript shared library. This action pushes the necessary secrets to a list of repos within our organization and makes using the GASL even easier. A repo owner only needs to add their application to the list in order to get the secrets, et voila! They’re ready to use the shared library.

Result

GASL is still under development and it is looking better everyday. We’ve added several actions after the secrets pusher.

They include:

→a linter that ensures repos have a proper container setup. Amongst other things, it checks that Dockerfiles are using images from the right registry or that we’re always pointing to specific image tags versions instead of the umbrella `latest`.

→a tool that creates GASL releases upon merges to master. It avoids having to create tags manually.

→the helm setup needed to access our private charts.