{"id":1419,"date":"2019-03-07T19:39:55","date_gmt":"2019-03-07T19:39:55","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=1419"},"modified":"2019-03-07T21:06:35","modified_gmt":"2019-03-07T21:06:35","slug":"deploying-and-scaling-jenkins-on-kubernetes","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2019\/03\/07\/deploying-and-scaling-jenkins-on-kubernetes\/","title":{"rendered":"Deploying and Scaling Jenkins on Kubernetes"},"content":{"rendered":"<h2>Introduction<\/h2>\n<p><a href=\"https:\/\/jenkins.io\">Jenkins<\/a> is an open-source continuous integration and continuous delivery tool, which can be used to automate building, testing, and deploying software. It is widely considered the most popular automation server, being used by more than a million users worldwide. Some advantages of Jenkins include:<\/p>\n<ul>\n<li>Open-source software with extensive community support<\/li>\n<li>Java-based codebase, making it portable to all major platforms<\/li>\n<li>A rich ecosystem of more than 1000 plugins<\/li>\n<\/ul>\n<p>Jenkins works well with all popular Source Control Management systems (Git, SVN, Mercurial and CVS), popular build tools (Ant, Maven, Grunt), shell scripts and Windows batch commands, as well as testing frameworks and report generators. Jenkins plugins provide support for technologies like Docker and Kubernetes, which enable the creation and deployment of cloud-based microservice environments, both for testing as well as production deployments.<\/p>\n<p>Jenkins supports the master-agent architecture (many build agents completing work scheduled by a master server) making it highly scalable. The master\u2019s job is to schedule build jobs, distribute the jobs to agents for actual execution, monitor the agents, and get the build results. Master servers can also execute build job directly.<\/p>\n<p>The agents\u2019 task is to build the job sent by the master. A job can be configured to run on a particular type of agent, or if there are no special requirements, Jenkins can simply choose the next available agent.<\/p>\n<p>Jenkins scalability provides many benefits:<\/p>\n<ul>\n<li>Running many build plans in parallel<\/li>\n<li>Automatically spinning up and removing agents to save costs<\/li>\n<li>Distributing the load<\/li>\n<\/ul>\n<p>Even if Jenkins includes scalability features out-of-the-box, the process of configuring scaling is not always straightforward. There are many options available to scale Jenkins, but one of the powerful options is to use Kubernetes.<\/p>\n<h2>What is Kubernetes?<\/h2>\n<p>Kubernetes is an open-source container orchestration tool. Its main purpose is to manage containerized applications on clusters of nodes by helping operators deploy, scale, update, and maintain their services, and providing mechanisms for service discovery. You can learn more about what Kubernetes is and what it can do by checking out <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/overview\/what-is-kubernetes\">the official documentation<\/a>.<\/p>\n<p>Kubernetes is one of the best tools for managing scalable, container-based workloads. Most applications, including Jenkins, can be containerized, which makes Kubernetes a very good option.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/01-rancher-jenkins-master-slave-architecture.png\" alt=\"01\" \/><\/p>\n<h2>Project Goals<\/h2>\n<p>Before we begin, let\u2019s take a moment and describe the system we are attempting to build.<\/p>\n<p>We want to start by deploying a Jenkins master instance onto a Kubernetes cluster. We will use <a href=\"https:\/\/github.com\/jenkinsci\/kubernetes-plugin\">Jenkins\u2019 kubernetes plugin<\/a> to scale Jenkins on the cluster by provisioning dynamic agents to accommodate its current workloads. The plugin will create a Kubernetes Pod for each build by launching an agent based on a specific Docker image. When the build completes, Jenkins will remove the Pod to save resources. Agents will be launched using JNLP (Java Network Launch Protocol), so we the containers will be able to automatically connect to the Jenkins master once up and running.<\/p>\n<h2>Prerequisites and Setup<\/h2>\n<p>To complete this guide, you will need the following:<\/p>\n<ul>\n<li>A Linux box to run Rancher: We will also use this to build custom Jenkins images. Follow the <a href=\"https:\/\/rancher.com\/quick-start\/\">Rancher installation quick start guide<\/a> to install Docker and Rancher on an appropriate host.<\/li>\n<li>Docker Hub account: We will need an account with a container image repository to push the custom images for our Jenkins master and agents.<\/li>\n<li>GCP account: We will provision our Kubernetes cluster on GCP. The free-tier of Google\u2019s cloud platform should be enough to complete this guide.<\/li>\n<\/ul>\n<h2>Building Custom Images for Jenkins<\/h2>\n<p>Let\u2019s start by building custom images for our Jenkins components and pushing them to <a href=\"https:\/\/hub.docker.com\">Docker Hub<\/a>.<\/p>\n<p>Log in to the Linux server where you will be running Rancher and building images. If you haven\u2019t already done so, install Docker and Rancher on the host by following the <a href=\"https:\/\/rancher.com\/quick-start\/\">Rancher installation quick start guide<\/a>. Once the host is ready, we can prepare our <a href=\"https:\/\/docs.docker.com\/engine\/reference\/builder\/\">Dockerfiles<\/a>.<\/p>\n<h3>Writing the Jenkins Master Dockerfile<\/h3>\n<p>We can begin by creating a file called Dockerfile-jenkins-master in the current directory to define the Jenkins master image:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# vi Dockerfile-jenkins-master<\/p>\n<p>Inside, include the following Dockerfile build instructions. These instructions use the main Jenkins Docker image as a base and configure the plugins we will use to deploy onto a Kubernetes cluster:<\/p>\n<p>FROM jenkins\/jenkins:lts<\/p>\n<p># Plugins for better UX (not mandatory)<br \/>\nRUN \/usr\/local\/bin\/install-plugins.sh ansicolor<br \/>\nRUN \/usr\/local\/bin\/install-plugins.sh greenballs<\/p>\n<p># Plugin for scaling Jenkins agents<br \/>\nRUN \/usr\/local\/bin\/install-plugins.sh kubernetes<\/p>\n<p>USER jenkins<\/p>\n<p>Save and close the file when you are finished.<\/p>\n<h3>Writing the Jenkins Agent Dockerfiles<\/h3>\n<p>Next, we can create the Dockerfiles for our Jenkins agents. We will be creating two agent images to demonstrate how Jenkins can correctly identify the correct agent to provision for each job.<\/p>\n<p>Create an empty file in the current directory. We will copy this to the image as an identifier for each agent we are building:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# touch empty-test-file<\/p>\n<p>Now, create a new Dockerfile for the first agent image:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# vi Dockerfile-jenkins-slave-jnlp1<\/p>\n<p>This image will copy the empty file to a unique name to identify the agent being used.<\/p>\n<p>FROM jenkins\/jnlp-slave<\/p>\n<p># For testing purpose only<br \/>\nCOPY empty-test-file \/jenkins-slave1<\/p>\n<p>ENTRYPOINT [&#8220;jenkins-slave&#8221;]<\/p>\n<p>Save and close the file when you are finished.<\/p>\n<p>Finally, define a second agent. This is identical to the previous agent, but includes a different file identifier:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# vi Dockerfile-jenkins-slave-jnlp2FROM jenkins\/jnlp-slave<\/p>\n<p># For testing purpose only<br \/>\nCOPY empty-test-file \/jenkins-slave2<\/p>\n<p>ENTRYPOINT [&#8220;jenkins-slave&#8221;]<\/p>\n<p>Save the file when you are finished.<\/p>\n<p>You\u2019re working directory should now look like this:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# ls -l<br \/>\ntotal 16<br \/>\n-rw-r&#8211;r&#8211;. 1 root root 265 Oct 21 12:58 Dockerfile-jenkins-master<br \/>\n-rw-r&#8211;r&#8211;. 1 root root 322 Oct 21 13:16 Dockerfile-jenkins-slave-jnlp1<br \/>\n-rw-r&#8211;r&#8211;. 1 root root 315 Oct 21 13:05 Dockerfile-jenkins-slave-jnlp2<\/p>\n<h3>Building the Images and Pushing to Docker Hub<\/h3>\n<p>With the Dockerfiles written, we are now ready to build and push the images to Docker Hub.<\/p>\n<p>Let\u2019s start by building the image for the Jenkins master:<\/p>\n<p>Note: In the command below, replace &lt;dockerhub_user&gt; with your Docker Hub account name.<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# docker build -f Dockerfile-jenkins-master -t &lt;dockerhub_user&gt;\/jenkins-master .<\/p>\n<p>Click for full command output<\/p>\n<p>Sending build context to Docker daemon 12.29 kB<br \/>\nStep 1\/5 : FROM jenkins\/jenkins:lts<br \/>\nTrying to pull repository docker.io\/jenkins\/jenkins &#8230;<br \/>\nlts: Pulling from docker.io\/jenkins\/jenkins<br \/>\n05d1a5232b46: Pull complete<br \/>\n5cee356eda6b: Pull complete<br \/>\n89d3385f0fd3: Pull complete<br \/>\n80ae6b477848: Pull complete<br \/>\n40624ba8b77e: Pull complete<br \/>\n8081dc39373d: Pull complete<br \/>\n8a4b3841871b: Pull complete<br \/>\nb919b8fd1620: Pull complete<br \/>\n2760538fe600: Pull complete<br \/>\nbcb851da81db: Pull complete<br \/>\neacbf73f87b6: Pull complete<br \/>\n9a7e396a0cbd: Pull complete<br \/>\n8900cde5602e: Pull complete<br \/>\nc8f62fde3f4d: Pull complete<br \/>\neb91939ba069: Pull complete<br \/>\nb894a41fcbe2: Pull complete<br \/>\nb3c60e932390: Pull complete<br \/>\n18f663576636: Pull complete<br \/>\n4445e4b557b3: Pull complete<br \/>\nf09e9b4be8ed: Pull complete<br \/>\ne3abe5324295: Pull complete<br \/>\n432eff1ecbb4: Pull complete<br \/>\nDigest: sha256:d5c835407130a393becac222b979b120c675f8cd815fadd085adb76b216e4ce1<br \/>\nStatus: Downloaded newer image for docker.io\/jenkins\/jenkins:lts<br \/>\n&#8212;&gt; 9cff19ad8c8b<br \/>\nStep 2\/5 : RUN \/usr\/local\/bin\/install-plugins.sh ansicolor<br \/>\n&#8212;&gt; Running in ff752eeb107d<\/p>\n<p>Creating initial locks&#8230;<br \/>\nAnalyzing war&#8230;<br \/>\nRegistering preinstalled plugins&#8230;<br \/>\nUsing version-specific update center: https:\/\/updates.jenkins.io\/2.138&#8230;<br \/>\nDownloading plugins&#8230;<br \/>\nDownloading plugin: ansicolor from https:\/\/updates.jenkins.io\/2.138\/latest\/ansicolor.hpi<br \/>\n&gt; ansicolor depends on workflow-step-api:2.12;resolution:=optional<br \/>\nSkipping optional dependency workflow-step-api<\/p>\n<p>WAR bundled plugins:<\/p>\n<p>Installed plugins:<br \/>\nansicolor:0.5.2<br \/>\nCleaning up locks<br \/>\n&#8212;&gt; a018ec9e38e6<br \/>\nRemoving intermediate container ff752eeb107d<br \/>\nStep 3\/5 : RUN \/usr\/local\/bin\/install-plugins.sh greenballs<br \/>\n&#8212;&gt; Running in 3505e21268b2<\/p>\n<p>Creating initial locks&#8230;<br \/>\nAnalyzing war&#8230;<br \/>\nRegistering preinstalled plugins&#8230;<br \/>\nUsing version-specific update center: https:\/\/updates.jenkins.io\/2.138&#8230;<br \/>\nDownloading plugins&#8230;<br \/>\nDownloading plugin: greenballs from https:\/\/updates.jenkins.io\/2.138\/latest\/greenballs.hpi<\/p>\n<p>WAR bundled plugins:<\/p>\n<p>Installed plugins:<br \/>\nansicolor:0.5.2<br \/>\ngreenballs:1.15<br \/>\nCleaning up locks<br \/>\n&#8212;&gt; 0af36c7afa67<br \/>\nRemoving intermediate container 3505e21268b2<br \/>\nStep 4\/5 : RUN \/usr\/local\/bin\/install-plugins.sh kubernetes<br \/>\n&#8212;&gt; Running in ed0afae3ac94<\/p>\n<p>Creating initial locks&#8230;<br \/>\nAnalyzing war&#8230;<br \/>\nRegistering preinstalled plugins&#8230;<br \/>\nUsing version-specific update center: https:\/\/updates.jenkins.io\/2.138&#8230;<br \/>\nDownloading plugins&#8230;<br \/>\nDownloading plugin: kubernetes from https:\/\/updates.jenkins.io\/2.138\/latest\/kubernetes.hpi<br \/>\n&gt; kubernetes depends on workflow-step-api:2.14,apache-httpcomponents-client-4-api:4.5.3-2.0,cloudbees-folder:5.18,durable-task:1.16,jackson2-api:2.7.3,variant:1.0,kubernetes-credentials:0.3.0,pipeline-model-extensions:1.3.1;resolution:=optional<br \/>\nDownloading plugin: workflow-step-api from https:\/\/updates.jenkins.io\/2.138\/latest\/workflow-step-api.hpi<br \/>\nDownloading plugin: apache-httpcomponents-client-4-api from https:\/\/updates.jenkins.io\/2.138\/latest\/apache-httpcomponents-client-4-api.hpi<br \/>\nDownloading plugin: cloudbees-folder from https:\/\/updates.jenkins.io\/2.138\/latest\/cloudbees-folder.hpi<br \/>\nDownloading plugin: durable-task from https:\/\/updates.jenkins.io\/2.138\/latest\/durable-task.hpi<br \/>\nDownloading plugin: jackson2-api from https:\/\/updates.jenkins.io\/2.138\/latest\/jackson2-api.hpi<br \/>\nDownloading plugin: variant from https:\/\/updates.jenkins.io\/2.138\/latest\/variant.hpi<br \/>\nSkipping optional dependency pipeline-model-extensions<br \/>\nDownloading plugin: kubernetes-credentials from https:\/\/updates.jenkins.io\/2.138\/latest\/kubernetes-credentials.hpi<br \/>\n&gt; workflow-step-api depends on structs:1.5<br \/>\nDownloading plugin: structs from https:\/\/updates.jenkins.io\/2.138\/latest\/structs.hpi<br \/>\n&gt; kubernetes-credentials depends on apache-httpcomponents-client-4-api:4.5.5-3.0,credentials:2.1.7,plain-credentials:1.3<br \/>\nDownloading plugin: credentials from https:\/\/updates.jenkins.io\/2.138\/latest\/credentials.hpi<br \/>\nDownloading plugin: plain-credentials from https:\/\/updates.jenkins.io\/2.138\/latest\/plain-credentials.hpi<br \/>\n&gt; cloudbees-folder depends on credentials:2.1.11;resolution:=optional<br \/>\nSkipping optional dependency credentials<br \/>\n&gt; plain-credentials depends on credentials:2.1.5<br \/>\n&gt; credentials depends on structs:1.7<\/p>\n<p>WAR bundled plugins:<\/p>\n<p>Installed plugins:<br \/>\nansicolor:0.5.2<br \/>\napache-httpcomponents-client-4-api:4.5.5-3.0<br \/>\ncloudbees-folder:6.6<br \/>\ncredentials:2.1.18<br \/>\ndurable-task:1.26<br \/>\ngreenballs:1.15<br \/>\njackson2-api:2.8.11.3<br \/>\nkubernetes-credentials:0.4.0<br \/>\nkubernetes:1.13.0<br \/>\nplain-credentials:1.4<br \/>\nstructs:1.17<br \/>\nvariant:1.1<br \/>\nworkflow-step-api:2.16<br \/>\nCleaning up locks<br \/>\n&#8212;&gt; dd19890f3139<br \/>\nRemoving intermediate container ed0afae3ac94<br \/>\nStep 5\/5 : USER jenkins<br \/>\n&#8212;&gt; Running in c1066861d5a3<br \/>\n&#8212;&gt; 034e27e479c5<br \/>\nRemoving intermediate container c1066861d5a3<br \/>\nSuccessfully built 034e27e479c5<\/p>\n<p>When the command returns, check the newly created image:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# docker images<br \/>\nREPOSITORY TAG IMAGE ID CREATED SIZE<br \/>\n&lt;dockerhub_user&gt;\/jenkins-master latest 034e27e479c5 16 seconds ago 744 MB<br \/>\ndocker.io\/jenkins\/jenkins lts 9cff19ad8c8b 10 days ago 730 MB<\/p>\n<p>Log in to Docker Hub using the credentials of your account:<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# docker login<br \/>\nLogin with your Docker ID to push and pull images from Docker Hub. If you don&#8217;t have a Docker ID, head over to https:\/\/hub.docker.com to create one.<br \/>\nUsername:<br \/>\nPassword:<br \/>\nLogin Succeeded<\/p>\n<p>Now, push the image to your Docker Hub account:<\/p>\n<p>Note: In the command below, be sure to substitute your own Docker Hub account again.<\/p>\n<p>[root@rancher-instance jenkins-kubernetes]# docker push &lt;dockerhub_user&gt;\/jenkins-master<\/p>\n<p>Click for full command output<\/p>\n<p>The push refers to a repository [docker.io\/calinrus\/jenkins-master]<br \/>\nb267c63b5961: Pushed<br \/>\n2cd1dc56ef56: Pushed<br \/>\ne99d7d8d116f: Pushed<br \/>\n8d117101392a: Mounted from jenkins\/jenkins<br \/>\nc2607b4e8ae4: Mounted from jenkins\/jenkins<br \/>\n81e4bc7cb1f1: Mounted from jenkins\/jenkins<br \/>\n8bac294d4ee8: Mounted from jenkins\/jenkins<br \/>\n707f669f3d58: Mounted from jenkins\/jenkins<br \/>\nac2b51b56ac6: Mounted from jenkins\/jenkins<br \/>\n1b2b61bef21f: Mounted from jenkins\/jenkins<br \/>\nefe1c25100f5: Mounted from jenkins\/jenkins<br \/>\n8e656983ccf7: Mounted from jenkins\/jenkins<br \/>\nba000aef226d: Mounted from jenkins\/jenkins<br \/>\na046c3cdf994: Mounted from jenkins\/jenkins<br \/>\n67e27eb293e8: Mounted from jenkins\/jenkins<br \/>\nbdd1835d949d: Mounted from jenkins\/jenkins<br \/>\n84bbcb8ef932: Mounted from jenkins\/jenkins<br \/>\n0d67aa2185d5: Mounted from jenkins\/jenkins<br \/>\n3499b696191f: Pushed<br \/>\n3b2a1688b8f3: Pushed<br \/>\nb7c56a9790e6: Mounted from jenkins\/jenkins<br \/>\nab016c9ea8f8: Mounted from jenkins\/jenkins<br \/>\n2eb1c9bfc5ea: Mounted from jenkins\/jenkins<br \/>\n0b703c74a09c: Mounted from jenkins\/jenkins<br \/>\nb28ef0b6fef8: Mounted from jenkins\/jenkins<br \/>\nlatest: digest: sha256:6b2c8c63eccd795db5b633c70b03fe1b5fa9c4a3b68e3901b10dc3af7c3549f0 size: 5552<\/p>\n<p>You will need to repeat similar commands to build the two images for the Jenkins JNLP agents:<\/p>\n<p>Note: Substitute your Docker Hub account name for &lt;dockerhub_user&gt; in the commands below.<\/p>\n<p>docker build -f Dockerfile-jenkins-slave-jnlp1 -t &lt;dockerhub_user&gt;\/jenkins-slave-jnlp1 .<br \/>\ndocker push &lt;dockerhub_user&gt;\/jenkins-slave-jnlp1<\/p>\n<p>docker build -f Dockerfile-jenkins-slave-jnlp2 -t &lt;dockerhub_user&gt;\/jenkins-slave-jnlp2 .<br \/>\ndocker push &lt;dockerhub_user&gt;\/jenkins-slave-jnlp2<\/p>\n<p>If everything was successful, you should see something like this in your Docker Hub account:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/02-rancher-docker-hub.png\" alt=\"02\" \/><\/p>\n<h2>Using Rancher to Deploy a Cluster<\/h2>\n<p>Now that our images are published, we can use Rancher to help us deploy a GKE cluster. If you set up Rancher earlier, you should be able to log into your instance by visiting your server\u2019s IP address with a web browser.<\/p>\n<p>Next, create a new GKE cluster. You will need to log in to your Google Cloud account to create a service account with the appropriate access. Follow the <a href=\"https:\/\/rancher.com\/docs\/rancher\/v2.x\/en\/cluster-provisioning\/hosted-kubernetes-clusters\/gke\/\">Rancher documentation on creating a GKE cluster<\/a> to learn how to create a service account and the provision a cluster with Rancher.<\/p>\n<h2>Deploying Jenkins to the Cluster<\/h2>\n<p>As soon as the cluster is ready, we can deploy the Jenkins master and create some services. If you are familiar with kubectl, you can achieve this from command line, but you can easily deploy all of the components you need through Rancher\u2019s UI.<\/p>\n<p>Regardless of how you choose to submit workloads to your cluster, create the following files on your local computer to define the objects you need to create.<\/p>\n<p>Start by creating a file to define the Jenkins deployment:<\/p>\n<p>[root@rancher-instance k8s]# vi deployment.yml<\/p>\n<p>Inside, paste the following:<\/p>\n<p>Note: Make sure to change &lt;dockerhub_user&gt; to your Docker Hub account name in the file below.<\/p>\n<p>apiVersion: extensions\/v1beta1<br \/>\nkind: Deployment<br \/>\nmetadata:<br \/>\nname: jenkins<br \/>\nspec:<br \/>\nreplicas: 1<br \/>\ntemplate:<br \/>\nmetadata:<br \/>\nlabels:<br \/>\napp: jenkins<br \/>\nspec:<br \/>\ncontainers:<br \/>\n&#8211; name: jenkins<br \/>\nimage: &lt;dockerhub_user&gt;\/jenkins-master<br \/>\nenv:<br \/>\n&#8211; name: JAVA_OPTS<br \/>\nvalue: -Djenkins.install.runSetupWizard=false<br \/>\nports:<br \/>\n&#8211; name: http-port<br \/>\ncontainerPort: 8080<br \/>\n&#8211; name: jnlp-port<br \/>\ncontainerPort: 50000<br \/>\nvolumeMounts:<br \/>\n&#8211; name: jenkins-home<br \/>\nmountPath: \/var\/jenkins_home<br \/>\nvolumes:<br \/>\n&#8211; name: jenkins-home<br \/>\nemptyDir: {}<\/p>\n<p>Next, create a file to configure the two services we will create.<\/p>\n<p>One will be a LoadBalancer service which will provision a public IP allowing us to access Jenkins from Internet. The other one will be a ClusterIP service needed for internal communication between master and agents that will be provisioned in the future:<\/p>\n<p>[root@rancher-instance k8s]# vi service.yml<\/p>\n<p>Inside, paste the following YAML structure:<\/p>\n<p>apiVersion: v1<br \/>\nkind: Service<br \/>\nmetadata:<br \/>\nname: jenkins<br \/>\nspec:<br \/>\ntype: LoadBalancer<br \/>\nports:<br \/>\n&#8211; port: 80<br \/>\ntargetPort: 8080<br \/>\nselector:<br \/>\napp: jenkins<\/p>\n<p>&#8212;<\/p>\n<p>apiVersion: v1<br \/>\nkind: Service<br \/>\nmetadata:<br \/>\nname: jenkins-jnlp<br \/>\nspec:<br \/>\ntype: ClusterIP<br \/>\nports:<br \/>\n&#8211; port: 50000<br \/>\ntargetPort: 50000<br \/>\nselector:<br \/>\napp: jenkins<\/p>\n<p>From Rancher, click on your managed cluster (called jenkins in this demo). In the upper-left menu, select the Default project and then select the Workloads tab.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/03-rancher-deploy-workloads-from-yaml.png\" alt=\"03\" \/><\/p>\n<p>From here, click Import YAML. On the page that follows, click the Read from a file button in the upper-right corner. Choose the local deployment.yml file you created on your computer and click Import.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/04-rancher-import.png\" alt=\"04\" \/><\/p>\n<p>Rancher will deploy a pod based on your Jenkins master image to the cluster:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/06-rancher-workload-progress-done.png\" alt=\"06\" \/><\/p>\n<p>Next, we need to configure a way to access the UI on the Jenkins master.<\/p>\n<p>In Load Balancing tab, follow same process as you did to import the previous file. Click the Import YAML button, followed by the Read from a file button. Next, select the service.yml file from your computer and click the Import button:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/07-rancher-load-balancing.png\" alt=\"07\" \/><\/p>\n<p>Rancher will begin to create your services. Provisioning the load balancer may take a few minutes.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/08-rancher-load-balancing-status.png\" alt=\"08\" \/><\/p>\n<p>As soon as service is marked as Active, you can find its public IP address by clicking on the three vertical dots at the right end of the load balancer\u2019s row and select View\/Edit YAML. From here, scroll down to find the IP address under status &gt; loadBalancer &gt; ingress &gt; ip:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/09-rancher-get-public-IP.png\" alt=\"09\" \/><\/p>\n<p>We can access the Jenkins UI by typing this IP into a web browser:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/10-rancher-jenkins-first-access.png\" alt=\"10\" \/><\/p>\n<h2>Configuring Dynamic Build Agents<\/h2>\n<p>With the Jenkins master up and running, we can go ahead and configure dynamic build agents to automatically spin up Pods as necessary.<\/p>\n<h3>Disabling the Default Master Build Agents<\/h3>\n<p>In the Jenkins UI, under Build Executor Status on the left side, two executors are configured by default, waiting to pick up build jobs. These are provided by the Jenkins master.<\/p>\n<p>The master instance should only be in charge of scheduling build jobs, distributing the jobs to agents for execution, monitoring the agents, and getting the build results. Since we don\u2019t want our master instance to execute builds, we will disable these.<\/p>\n<p>Click on Manage Jenkins followed by Manage Nodes.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/13-rancher-jenkins-manage-nodes.png\" alt=\"13\" \/><\/p>\n<p>Click the gear icon associated with the master row.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/14_1-rancher-jenkins-master-configure.png\" alt=\"14_1\" \/><\/p>\n<p>On the following page, set # of executors to 0 and click Save.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/14_2-rancher-jenkins-master-remove-executors.png\" alt=\"14_2\" \/><\/p>\n<p>The two idle executors will be removed from the Build Executor Status on the left side of the UI.<\/p>\n<h3>Gathering Configuration Information<\/h3>\n<p>We need a few pieces of information to configure Jenkins to automatically provision build agents on our Kubernetes cluster. We need three pieces of information from our GCP account and one from our ClusterIP service.<\/p>\n<p>In your GCP account, select Kubernetes Engine, followed by Clusters and then click on the name of your cluster. In the Details column, copy the Endpoint IP address for later reference. This is the URL we need to give Jenkins to connect to the cluster:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/16-rancher-gke-show-credentials.png\" alt=\"16\" \/><\/p>\n<p>Next, click Show credentials to the right of the Endpoint. Copy the Username and Password for later reference.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/17-rancher-gke-get-credentials.png\" alt=\"17\" \/><\/p>\n<p>Now, switch over to the Rancher UI. In the upper-left menu, select the Default project on Jenkins cluster. Select the Workloads tab in the upper navigation pane and click the Service Discovery tab on the page:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/20-rancher-jenkins-cloud-3.png\" alt=\"20-3\" \/><\/p>\n<p>Click on the three vertical dots associated with the jenkins-jnlp row and click View\/Edit YAML. Copy values in the spec &gt; clusterIP and spec &gt; ports &gt; port for later reference.<\/p>\n<h3>Configuring the Jenkins Kubernetes Plugin<\/h3>\n<p>Back in the main Jenkins dashboard, click on Manage Jenkins, followed by Manage Plugins:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/11-rancher-jenkins-manage-plugins.png\" alt=\"11\" \/><\/p>\n<p>Click the Installed tab and check that the Kubernetes plugin is installed:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/12-rancher-jenkins-check-plugins.png\" alt=\"12\" \/><\/p>\n<p>We can now configure the plugin. Go to Manage Jenkins and select Configure System:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/18-rancher-jenkins-configure-system.png\" alt=\"18\" \/><\/p>\n<p>Scroll to the Cloud section at the bottom of the page. Click on Add a new cloud and select Kubernetes.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/19-rancher-jenkins-cloud.png\" alt=\"19\" \/><\/p>\n<p>On the form that follows, in the Kubernetes URL field, enter https:\/\/ followed by the cluster endpoint IP address you copied from your GCP account.<\/p>\n<p>Under Credentials, click the Add button and select Jenkins. On the form that appears, enter the username and password you copied from your GCP account and click the Add button at the bottom.<\/p>\n<p>When you return to the Kubernetes form, select the credentials you just added from the Credentials drop down menu and click the Test Connection button. If the configuration is correct, the test will show \u201cConnection test successful\u201d.<\/p>\n<p>Next, in the Jenkins tunnel field, enter the IP address and port that you retrieved from the jenkins-jnlp service in the Rancher UI, separated by a colon:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/20-rancher-jenkins-cloud-1.png\" alt=\"20-1\" \/><\/p>\n<p>Now, scroll down to the Images section at the bottom of the page, click the Add Pod Template button, and select Kubernetes Pod Template. Fill out the Name and Labels fields with unique values to identify your first agent. We will use the label to specify which agent image should be used to run each build.<\/p>\n<p>Next, in the Containers field, click the Add Container button and select Container Template. In the section that appears, fill out the following fields:<\/p>\n<ul>\n<li>Name: jnlp (this is required by the Jenkins agent)<\/li>\n<li>Docker image: &lt;dockerhub_user&gt;\/jenkins-slave-jnlp1 (make sure to change the Docker Hub username)<\/li>\n<li>Command to run: Delete the value here<\/li>\n<li>Arguments to pass to the command: Delete the value here<\/li>\n<\/ul>\n<p>The rest of the fields can be left as they are.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/21-rancher-jenkins-pod-template-1.png\" alt=\"21\" \/><\/p>\n<p>Next, click the Add Pod Template button and select Kubernetes Pod Template again. Repeat the process for the second agent image you created. Make sure to change the values to refer to your second image where applicable:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/22-rancher-jenkins-pod-template-2.png\" alt=\"22\" \/><\/p>\n<p>Click the Save button at the bottom to save your changes and continue.<\/p>\n<h2>Testing Dynamic Build Jobs<\/h2>\n<p>Now that our configuration is complete, we can create some build jobs to ensure that Jenkins can scale on top of Kubernetes. We will create five build jobs for each of our Jenkins agents.<\/p>\n<p>On the main Jenkins page, click New Item on the left side. Enter a name for the first build of your first agent. Select Freestyle project and click the OK button.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/23-racher-jenkins-job1.png\" alt=\"23\" \/><\/p>\n<p>On the next page, in the Label Expression field, type the label you set for your first Jenkins agent image. If you click out of the field, a message will appear indicating that the label is serviced by a cloud:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/24-rancher-jenkins-label-expression.png\" alt=\"24\" \/><\/p>\n<p>Scroll down to the Build Environment section and check Color ANSI Console Output.<\/p>\n<p>In the Build section, click Add build step and select Execute shell. Paste the following script in the text box that appears:<\/p>\n<p>#!\/bin\/bash<\/p>\n<p>RED=&#8217;33[0;31m&#8217;<br \/>\nNC=&#8217;33[0m&#8217;<\/p>\n<p>result=`ls \/ | grep -e jenkins-slave1 -e jenkins-slave2`<br \/>\necho -e &#8220;$Docker image is for $result $&#8221;<\/p>\n<p>Click the Save button when you are finished.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/25-rancher-jenkins-shell-1.png\" alt=\"25\" \/><\/p>\n<p>Create another four jobs for the first agent by clicking New Item, filling out a new name, and using the Copy from field to copy from your first build. You can save each build without changes to duplicate the first build exactly.<\/p>\n<p>Next, configure the first job for your second Jenkins agent. Click New Item, select a name for the first job for the second agent, and copy the job from your first agent again. This time, we will modify the fields on the configuration page before saving.<\/p>\n<p>First, change the Label Expression field to match the label for your second agent.<\/p>\n<p>Next, replace the script in the text box in the Build section with the following script:<\/p>\n<p>#!\/bin\/bash<\/p>\n<p>BLUE=&#8217;e[34m&#8217;<br \/>\nNC=&#8217;33[0m&#8217;<\/p>\n<p>result=`ls \/ | grep -e jenkins-slave1 -e jenkins-slave2`<br \/>\necho -e &#8220;$Docker image is for $result $&#8221;<\/p>\n<p>Click Save when you are finished.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/26-rancher-jenkins-shell-2.png\" alt=\"26\" \/><\/p>\n<p>Create four more builds for your second agent by copying from the job we just created.<\/p>\n<p>Now, go to the home screen and start all the ten jobs you just created by clicking on the icon on the far right side of each row. As soon as you start them, they will be queued for execution as indicated by the Build Queue section:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/28-rancher-jenkins-start-jobs.png\" alt=\"28\" \/><\/p>\n<p>After a few seconds, Pods will begin to be created to execute the builds (you can verify this in Rancher\u2019s Workload tab). Jenkins will create one Pod for each job. As each agent is started, it connects to the master and receives a job from the queue to execute.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/29-rancher-jenkins-executors.png\" alt=\"29\" \/><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/30-rancher-workloads-executors.png\" alt=\"30\" \/><\/p>\n<p>As soon as an agent finishes processing its job, it is automatically removed from the cluster:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/31-rancher-remove-pods.png\" alt=\"31\" \/><\/p>\n<p>To check the status of our jobs, we can click on one from each agent. Click the build from the Build History and then click Console Output. Jobs executed by the first agent should specify that the jenkins-slave1 Docker image was used, while builds executed by the second agent should indicate that the jenkins-slave2 image was used:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/32-rancher-test-results-1.png\" alt=\"32\" \/><img decoding=\"async\" src=\"https:\/\/rancher.com\/img\/blog\/2018\/scaling-jenkins\/33-rancher-test-results-2.png\" alt=\"33\" \/><\/p>\n<p>If you see the output above, Jenkins is configured correctly and functioning as intended. You can now begin to customize your Kubernetes-backed build system to help your team test and release software.<\/p>\n<h2>Conclusion<\/h2>\n<p>In this article, we configured Jenkins to automatically provision build agents on demand by connecting it with a Kubernetes cluster managed by Rancher. To achieve this, we completed the following steps:<\/p>\n<ul>\n<li>Created a cluster using Rancher<\/li>\n<li>Created custom Docker images for the Jenkins master and agents<\/li>\n<li>Deployed the Jenkins master and an L4 LoadBalancer service to the Kubernetes cluster<\/li>\n<li>Configured the Jenkins kubernetes plugin to automatically spawn dynamic agents on our cluster.<\/li>\n<li>Tested a scenario using multiple build jobs with dedicated agent images<\/li>\n<\/ul>\n<p>The main purpose of this article was to highlight the basic configuration necessary to set up a Jenkins master and agent architecture. We saw how Jenkins launched agents using JNLP and how the containers automatically connected to the Jenkins master to receive instructions. To achieve this, we used <a href=\"https:\/\/rancher.com\/\">Rancher<\/a> to create the cluster, deploy a workload, and monitor the resulting Pods. Afterwards, we relied on the <a href=\"https:\/\/github.com\/jenkinsci\/kubernetes-plugin\">Jenkins Kubernetes plugin<\/a> to glue together all of the different components.<\/p>\n<p><a href=\"https:\/\/rancher.com\/blog\/2018\/2018-11-27-scaling-jenkins\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Jenkins is an open-source continuous integration and continuous delivery tool, which can be used to automate building, testing, and deploying software. It is widely considered the most popular automation server, being used by more than a million users worldwide. Some advantages of Jenkins include: Open-source software with extensive community support Java-based codebase, making it &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.appservgrid.com\/paw93\/index.php\/2019\/03\/07\/deploying-and-scaling-jenkins-on-kubernetes\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Deploying and Scaling Jenkins on Kubernetes&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1419","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1419","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/comments?post=1419"}],"version-history":[{"count":2,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1419\/revisions"}],"predecessor-version":[{"id":1486,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1419\/revisions\/1486"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/media?parent=1419"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/categories?post=1419"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/tags?post=1419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}