Ramp Me Up, Scotty!

EN | DE

Scaling Jenkins Build Agents with Kubernetes Pods

2021-02-22

Based on the previous tutorial on running Jenkins inside a Kubernetes cluster, it is now time to leverage the Kubernetes infrastructure to scale build jobs across the cluster.

Sample Build Jobs

First, create two projects to test the setup. The jobs won’t do anything useful—they will just wait 10 seconds and then finish.

Create the jobs by clicking “New Item” in the dashboard and selecting “Freestyle Project”.

In the “Build” section, add an “Execute shell” build step.

The command the build job should perform is:

sleep 10

After saving, the new job appears in the dashboard. Create another job so there are two total.

When both jobs are triggered simultaneously, they appear under “Build Executor Status”.

For now, both jobs execute within the Jenkins pod. This won’t scale in the long run.

The goal is to run each build job instance in a newly created pod whose lifecycle is tied to the build job:

To achieve this, additional configuration is required.

Install Kubernetes Plugin

Install the Kubernetes plugin via:

Then:

The plugin will be installed:

Adding Kubernetes Secrets

Two Kubernetes secrets (service accounts + RBAC) are needed.

The first is jenkins-robot.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: jenkins-robot
 namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: jenkins-robot
 namespace: jenkins
 labels:
 "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
 resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims", "events"]
 verbs: ["get", "list", "watch"]
- apiGroups: [""]
 resources: ["pods", "pods/exec", "persistentvolumeclaims", "events"]
 verbs: ["create", "apply", "delete", "deletecollection", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: jenkins-robot-binding
 namespace: jenkins
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: jenkins-robot
subjects:
- kind: ServiceAccount
 name: jenkins-robot
 namespace: jenkins

The second is jenkins-robot-global.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: jenkins-robot-global
 namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: jenkins-robot-global
 namespace: jenkins
 labels:
 "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
 resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims"]
 verbs: ["get", "list", "watch"]
- apiGroups: [""]
 resources: ["pods", "pods/exec", "persistentvolumeclaims"]
 verbs: ["create", "apply", "delete", "deletecollection", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: jenkins-robot-global-binding
 namespace: jenkins
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: jenkins-robot-global
subjects:
- kind: ServiceAccount
 name: jenkins-robot-global
 namespace: jenkins

Apply both manifests:

kubectl apply -n jenkins -f jenkins-robot.yaml
kubectl apply -n jenkins -f jenkins-robot-global.yaml

These secrets must be added to Jenkins.

Retrieve the first secret with:

kubectl -n jenkins get serviceaccount jenkins-robot --template='{{range.secrets}}{{.name}}{{"\n"}}{{end}}' | xargs -n 1 kubectl -n jenkins get secret --template='{{ if.data.token }}{{.data.token }}{{end}}' | head -n 1 | base64 -d -

Copy the resulting token to the clipboard to paste in the next step.

In Jenkins, create the credential via:

Then click:

Navigate to:

Then:

Enter these values:

Repeat for the jenkins-robot-global secret, which can be retrieved with:

kubectl -n jenkins get serviceaccount jenkins-robot-global --template='{{range.secrets}}{{.name}}{{"\n"}}{{end}}' | xargs -n 1 kubectl -n jenkins get secret --template='{{ if.data.token }}{{.data.token }}{{end}}' | head -n 1 | base64 -d -

Both credentials should now appear in Jenkins:

Adding the Kubernetes Cloud to Jenkins

Jenkins must be configured with the Kubernetes cluster it will use. Add the Kubernetes cloud via:

Under “Kubernetes Cloud Details” provide:

The cluster master node IP can be found with:

kubectl get nodes -o wide | grep master

The IP for the Jenkins tunnel is the IP of the jenkins-jnlp service:

kubectl get service jenkins-jnlp -n jenkins

Leave the remaining values at their defaults:

Test the connection with the “Test Connection” button.

Adding Pod and Container Templates

Configure the Pod template as follows:

Pod Template:

For this tutorial, the agent containers use the jenkins/jnlp-agent-alpine image:

Add a HostPath volume:

Add an EmptyDir volume:

Save the configuration.

Finally, configure Jenkins so only agent nodes run build jobs:

Set:

Save.

Testing the Pod Build Agents

Trigger both build jobs simultaneously. You should see two pods launched under “Build Executor Status”.

When the pods are running, the jobs execute inside them:

Resources