Author: root

Introduction

Jenkins is an open-source continuous integration and continuous delivery tool, which can be used to automate building, testing, and deploying software. It is widely considered the most popular automation server, being used by more than a million users worldwide. Some advantages of Jenkins include:

• Open-source software with extensive community support
• Java-based codebase, making it portable to all major platforms
• A rich ecosystem of more than 1000 plugins

Jenkins works well with all popular Source Control Management systems (Git, SVN, Mercurial and CVS), popular build tools (Ant, Maven, Grunt), shell scripts and Windows batch commands, as well as testing frameworks and report generators. Jenkins plugins provide support for technologies like Docker and Kubernetes, which enable the creation and deployment of cloud-based microservice environments, both for testing as well as production deployments.

Jenkins supports the master-agent architecture (many build agents completing work scheduled by a master server) making it highly scalable. The master’s job is to schedule build jobs, distribute the jobs to agents for actual execution, monitor the agents, and get the build results. Master servers can also execute build job directly.

The agents’ task is to build the job sent by the master. A job can be configured to run on a particular type of agent, or if there are no special requirements, Jenkins can simply choose the next available agent.

Jenkins scalability provides many benefits:

• Running many build plans in parallel
• Automatically spinning up and removing agents to save costs
• Distributing the load

Even if Jenkins includes scalability features out-of-the-box, the process of configuring scaling is not always straightforward. There are many options available to scale Jenkins, but one of the powerful options is to use Kubernetes.

What is Kubernetes?

Kubernetes is an open-source container orchestration tool. Its main purpose is to manage containerized applications on clusters of nodes by helping operators deploy, scale, update, and maintain their services, and providing mechanisms for service discovery. You can learn more about what Kubernetes is and what it can do by checking out the official documentation.

Kubernetes is one of the best tools for managing scalable, container-based workloads. Most applications, including Jenkins, can be containerized, which makes Kubernetes a very good option.

Project Goals

Before we begin, let’s take a moment and describe the system we are attempting to build.

We want to start by deploying a Jenkins master instance onto a Kubernetes cluster. We will use Jenkins’ kubernetes plugin to scale Jenkins on the cluster by provisioning dynamic agents to accommodate its current workloads. The plugin will create a Kubernetes Pod for each build by launching an agent based on a specific Docker image. When the build completes, Jenkins will remove the Pod to save resources. Agents will be launched using JNLP (Java Network Launch Protocol), so we the containers will be able to automatically connect to the Jenkins master once up and running.

Prerequisites and Setup

To complete this guide, you will need the following:

• A Linux box to run Rancher: We will also use this to build custom Jenkins images. Follow the Rancher installation quick start guide to install Docker and Rancher on an appropriate host.
• Docker Hub account: We will need an account with a container image repository to push the custom images for our Jenkins master and agents.
• GCP account: We will provision our Kubernetes cluster on GCP. The free-tier of Google’s cloud platform should be enough to complete this guide.

Building Custom Images for Jenkins

Let’s start by building custom images for our Jenkins components and pushing them to Docker Hub.

Log in to the Linux server where you will be running Rancher and building images. If you haven’t already done so, install Docker and Rancher on the host by following the Rancher installation quick start guide. Once the host is ready, we can prepare our Dockerfiles.

Writing the Jenkins Master Dockerfile

We can begin by creating a file called Dockerfile-jenkins-master in the current directory to define the Jenkins master image:

[root@rancher-instance jenkins-kubernetes]# vi Dockerfile-jenkins-master

Inside, include the following Dockerfile build instructions. These instructions use the main Jenkins Docker image as a base and configure the plugins we will use to deploy onto a Kubernetes cluster:

FROM jenkins/jenkins:lts

# Plugins for better UX (not mandatory)
RUN /usr/local/bin/install-plugins.sh ansicolor
RUN /usr/local/bin/install-plugins.sh greenballs

# Plugin for scaling Jenkins agents
RUN /usr/local/bin/install-plugins.sh kubernetes

USER jenkins

Save and close the file when you are finished.

Writing the Jenkins Agent Dockerfiles

Next, we can create the Dockerfiles for our Jenkins agents. We will be creating two agent images to demonstrate how Jenkins can correctly identify the correct agent to provision for each job.

Create an empty file in the current directory. We will copy this to the image as an identifier for each agent we are building:

[root@rancher-instance jenkins-kubernetes]# touch empty-test-file

Now, create a new Dockerfile for the first agent image:

[root@rancher-instance jenkins-kubernetes]# vi Dockerfile-jenkins-slave-jnlp1

This image will copy the empty file to a unique name to identify the agent being used.

FROM jenkins/jnlp-slave

# For testing purpose only
COPY empty-test-file /jenkins-slave1

ENTRYPOINT [“jenkins-slave”]

Save and close the file when you are finished.

Finally, define a second agent. This is identical to the previous agent, but includes a different file identifier:

[root@rancher-instance jenkins-kubernetes]# vi Dockerfile-jenkins-slave-jnlp2FROM jenkins/jnlp-slave

# For testing purpose only
COPY empty-test-file /jenkins-slave2

ENTRYPOINT [“jenkins-slave”]

Save the file when you are finished.

You’re working directory should now look like this:

[root@rancher-instance jenkins-kubernetes]# ls -l
total 16
-rw-r–r–. 1 root root 265 Oct 21 12:58 Dockerfile-jenkins-master
-rw-r–r–. 1 root root 322 Oct 21 13:16 Dockerfile-jenkins-slave-jnlp1
-rw-r–r–. 1 root root 315 Oct 21 13:05 Dockerfile-jenkins-slave-jnlp2

Building the Images and Pushing to Docker Hub

With the Dockerfiles written, we are now ready to build and push the images to Docker Hub.

Let’s start by building the image for the Jenkins master:

Note: In the command below, replace <dockerhub_user> with your Docker Hub account name.

[root@rancher-instance jenkins-kubernetes]# docker build -f Dockerfile-jenkins-master -t <dockerhub_user>/jenkins-master .

Click for full command output

Sending build context to Docker daemon 12.29 kB
Step 1/5 : FROM jenkins/jenkins:lts
Trying to pull repository docker.io/jenkins/jenkins …
lts: Pulling from docker.io/jenkins/jenkins
05d1a5232b46: Pull complete
5cee356eda6b: Pull complete
89d3385f0fd3: Pull complete
80ae6b477848: Pull complete
40624ba8b77e: Pull complete
8081dc39373d: Pull complete
8a4b3841871b: Pull complete
b919b8fd1620: Pull complete
2760538fe600: Pull complete
bcb851da81db: Pull complete
eacbf73f87b6: Pull complete
9a7e396a0cbd: Pull complete
8900cde5602e: Pull complete
c8f62fde3f4d: Pull complete
eb91939ba069: Pull complete
b894a41fcbe2: Pull complete
b3c60e932390: Pull complete
18f663576636: Pull complete
4445e4b557b3: Pull complete
f09e9b4be8ed: Pull complete
e3abe5324295: Pull complete
432eff1ecbb4: Pull complete
Digest: sha256:d5c835407130a393becac222b979b120c675f8cd815fadd085adb76b216e4ce1
Status: Downloaded newer image for docker.io/jenkins/jenkins:lts
—> 9cff19ad8c8b
Step 2/5 : RUN /usr/local/bin/install-plugins.sh ansicolor
—> Running in ff752eeb107d

Creating initial locks…
Analyzing war…
Registering preinstalled plugins…
Using version-specific update center: https://updates.jenkins.io/2.138…
Downloading plugins…
Downloading plugin: ansicolor from https://updates.jenkins.io/2.138/latest/ansicolor.hpi
> ansicolor depends on workflow-step-api:2.12;resolution:=optional
Skipping optional dependency workflow-step-api

WAR bundled plugins:

Installed plugins:
ansicolor:0.5.2
Cleaning up locks
—> a018ec9e38e6
Removing intermediate container ff752eeb107d
Step 3/5 : RUN /usr/local/bin/install-plugins.sh greenballs
—> Running in 3505e21268b2

Creating initial locks…
Analyzing war…
Registering preinstalled plugins…
Using version-specific update center: https://updates.jenkins.io/2.138…
Downloading plugins…
Downloading plugin: greenballs from https://updates.jenkins.io/2.138/latest/greenballs.hpi

WAR bundled plugins:

Installed plugins:
ansicolor:0.5.2
greenballs:1.15
Cleaning up locks
—> 0af36c7afa67
Removing intermediate container 3505e21268b2
Step 4/5 : RUN /usr/local/bin/install-plugins.sh kubernetes
—> Running in ed0afae3ac94

Creating initial locks…
Analyzing war…
Registering preinstalled plugins…
Using version-specific update center: https://updates.jenkins.io/2.138…
Downloading plugins…
Downloading plugin: kubernetes from https://updates.jenkins.io/2.138/latest/kubernetes.hpi
> kubernetes depends on workflow-step-api:2.14,apache-httpcomponents-client-4-api:4.5.3-2.0,cloudbees-folder:5.18,durable-task:1.16,jackson2-api:2.7.3,variant:1.0,kubernetes-credentials:0.3.0,pipeline-model-extensions:1.3.1;resolution:=optional
Downloading plugin: workflow-step-api from https://updates.jenkins.io/2.138/latest/workflow-step-api.hpi
Downloading plugin: apache-httpcomponents-client-4-api from https://updates.jenkins.io/2.138/latest/apache-httpcomponents-client-4-api.hpi
Downloading plugin: cloudbees-folder from https://updates.jenkins.io/2.138/latest/cloudbees-folder.hpi
Downloading plugin: durable-task from https://updates.jenkins.io/2.138/latest/durable-task.hpi
Downloading plugin: jackson2-api from https://updates.jenkins.io/2.138/latest/jackson2-api.hpi
Downloading plugin: variant from https://updates.jenkins.io/2.138/latest/variant.hpi
Skipping optional dependency pipeline-model-extensions
Downloading plugin: kubernetes-credentials from https://updates.jenkins.io/2.138/latest/kubernetes-credentials.hpi
> workflow-step-api depends on structs:1.5
Downloading plugin: structs from https://updates.jenkins.io/2.138/latest/structs.hpi
> kubernetes-credentials depends on apache-httpcomponents-client-4-api:4.5.5-3.0,credentials:2.1.7,plain-credentials:1.3
Downloading plugin: credentials from https://updates.jenkins.io/2.138/latest/credentials.hpi
Downloading plugin: plain-credentials from https://updates.jenkins.io/2.138/latest/plain-credentials.hpi
> cloudbees-folder depends on credentials:2.1.11;resolution:=optional
Skipping optional dependency credentials
> plain-credentials depends on credentials:2.1.5
> credentials depends on structs:1.7

WAR bundled plugins:

Installed plugins:
ansicolor:0.5.2
apache-httpcomponents-client-4-api:4.5.5-3.0
cloudbees-folder:6.6
credentials:2.1.18
durable-task:1.26
greenballs:1.15
jackson2-api:2.8.11.3
kubernetes-credentials:0.4.0
kubernetes:1.13.0
plain-credentials:1.4
structs:1.17
variant:1.1
workflow-step-api:2.16
Cleaning up locks
—> dd19890f3139
Removing intermediate container ed0afae3ac94
Step 5/5 : USER jenkins
—> Running in c1066861d5a3
—> 034e27e479c5
Removing intermediate container c1066861d5a3
Successfully built 034e27e479c5

When the command returns, check the newly created image:

[root@rancher-instance jenkins-kubernetes]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<dockerhub_user>/jenkins-master latest 034e27e479c5 16 seconds ago 744 MB
docker.io/jenkins/jenkins lts 9cff19ad8c8b 10 days ago 730 MB

Log in to Docker Hub using the credentials of your account:

[root@rancher-instance jenkins-kubernetes]# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Password:
Login Succeeded

Now, push the image to your Docker Hub account:

Note: In the command below, be sure to substitute your own Docker Hub account again.

[root@rancher-instance jenkins-kubernetes]# docker push <dockerhub_user>/jenkins-master

Click for full command output

The push refers to a repository [docker.io/calinrus/jenkins-master]
b267c63b5961: Pushed
2cd1dc56ef56: Pushed
e99d7d8d116f: Pushed
8d117101392a: Mounted from jenkins/jenkins
c2607b4e8ae4: Mounted from jenkins/jenkins
81e4bc7cb1f1: Mounted from jenkins/jenkins
8bac294d4ee8: Mounted from jenkins/jenkins
707f669f3d58: Mounted from jenkins/jenkins
ac2b51b56ac6: Mounted from jenkins/jenkins
1b2b61bef21f: Mounted from jenkins/jenkins
efe1c25100f5: Mounted from jenkins/jenkins
8e656983ccf7: Mounted from jenkins/jenkins
ba000aef226d: Mounted from jenkins/jenkins
a046c3cdf994: Mounted from jenkins/jenkins
67e27eb293e8: Mounted from jenkins/jenkins
bdd1835d949d: Mounted from jenkins/jenkins
84bbcb8ef932: Mounted from jenkins/jenkins
0d67aa2185d5: Mounted from jenkins/jenkins
3499b696191f: Pushed
3b2a1688b8f3: Pushed
b7c56a9790e6: Mounted from jenkins/jenkins
ab016c9ea8f8: Mounted from jenkins/jenkins
2eb1c9bfc5ea: Mounted from jenkins/jenkins
0b703c74a09c: Mounted from jenkins/jenkins
b28ef0b6fef8: Mounted from jenkins/jenkins
latest: digest: sha256:6b2c8c63eccd795db5b633c70b03fe1b5fa9c4a3b68e3901b10dc3af7c3549f0 size: 5552

You will need to repeat similar commands to build the two images for the Jenkins JNLP agents:

Note: Substitute your Docker Hub account name for <dockerhub_user> in the commands below.

docker build -f Dockerfile-jenkins-slave-jnlp1 -t <dockerhub_user>/jenkins-slave-jnlp1 .
docker push <dockerhub_user>/jenkins-slave-jnlp1

docker build -f Dockerfile-jenkins-slave-jnlp2 -t <dockerhub_user>/jenkins-slave-jnlp2 .
docker push <dockerhub_user>/jenkins-slave-jnlp2

If everything was successful, you should see something like this in your Docker Hub account:

Using Rancher to Deploy a Cluster

Now that our images are published, we can use Rancher to help us deploy a GKE cluster. If you set up Rancher earlier, you should be able to log into your instance by visiting your server’s IP address with a web browser.

Next, create a new GKE cluster. You will need to log in to your Google Cloud account to create a service account with the appropriate access. Follow the Rancher documentation on creating a GKE cluster to learn how to create a service account and the provision a cluster with Rancher.

Deploying Jenkins to the Cluster

As soon as the cluster is ready, we can deploy the Jenkins master and create some services. If you are familiar with kubectl, you can achieve this from command line, but you can easily deploy all of the components you need through Rancher’s UI.

Regardless of how you choose to submit workloads to your cluster, create the following files on your local computer to define the objects you need to create.

Start by creating a file to define the Jenkins deployment:

[root@rancher-instance k8s]# vi deployment.yml

Inside, paste the following:

Note: Make sure to change <dockerhub_user> to your Docker Hub account name in the file below.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins
spec:
replicas: 1
template:
metadata:
labels:
app: jenkins
spec:
containers:
– name: jenkins
image: <dockerhub_user>/jenkins-master
env:
– name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
ports:
– name: http-port
containerPort: 8080
– name: jnlp-port
containerPort: 50000
volumeMounts:
– name: jenkins-home
mountPath: /var/jenkins_home
volumes:
– name: jenkins-home
emptyDir: {}

Next, create a file to configure the two services we will create.

One will be a LoadBalancer service which will provision a public IP allowing us to access Jenkins from Internet. The other one will be a ClusterIP service needed for internal communication between master and agents that will be provisioned in the future:

[root@rancher-instance k8s]# vi service.yml

Inside, paste the following YAML structure:

apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
type: LoadBalancer
ports:
– port: 80
targetPort: 8080
selector:
app: jenkins

—

apiVersion: v1
kind: Service
metadata:
name: jenkins-jnlp
spec:
type: ClusterIP
ports:
– port: 50000
targetPort: 50000
selector:
app: jenkins

From Rancher, click on your managed cluster (called jenkins in this demo). In the upper-left menu, select the Default project and then select the Workloads tab.

From here, click Import YAML. On the page that follows, click the Read from a file button in the upper-right corner. Choose the local deployment.yml file you created on your computer and click Import.

Rancher will deploy a pod based on your Jenkins master image to the cluster:

Next, we need to configure a way to access the UI on the Jenkins master.

In Load Balancing tab, follow same process as you did to import the previous file. Click the Import YAML button, followed by the Read from a file button. Next, select the service.yml file from your computer and click the Import button:

Rancher will begin to create your services. Provisioning the load balancer may take a few minutes.

As soon as service is marked as Active, you can find its public IP address by clicking on the three vertical dots at the right end of the load balancer’s row and select View/Edit YAML. From here, scroll down to find the IP address under status > loadBalancer > ingress > ip:

We can access the Jenkins UI by typing this IP into a web browser:

Configuring Dynamic Build Agents

With the Jenkins master up and running, we can go ahead and configure dynamic build agents to automatically spin up Pods as necessary.

Disabling the Default Master Build Agents

In the Jenkins UI, under Build Executor Status on the left side, two executors are configured by default, waiting to pick up build jobs. These are provided by the Jenkins master.

The master instance should only be in charge of scheduling build jobs, distributing the jobs to agents for execution, monitoring the agents, and getting the build results. Since we don’t want our master instance to execute builds, we will disable these.

Click on Manage Jenkins followed by Manage Nodes.

Click the gear icon associated with the master row.

On the following page, set # of executors to 0 and click Save.

The two idle executors will be removed from the Build Executor Status on the left side of the UI.

Gathering Configuration Information

We need a few pieces of information to configure Jenkins to automatically provision build agents on our Kubernetes cluster. We need three pieces of information from our GCP account and one from our ClusterIP service.

In your GCP account, select Kubernetes Engine, followed by Clusters and then click on the name of your cluster. In the Details column, copy the Endpoint IP address for later reference. This is the URL we need to give Jenkins to connect to the cluster:

Next, click Show credentials to the right of the Endpoint. Copy the Username and Password for later reference.

Now, switch over to the Rancher UI. In the upper-left menu, select the Default project on Jenkins cluster. Select the Workloads tab in the upper navigation pane and click the Service Discovery tab on the page:

Click on the three vertical dots associated with the jenkins-jnlp row and click View/Edit YAML. Copy values in the spec > clusterIP and spec > ports > port for later reference.

Configuring the Jenkins Kubernetes Plugin

Back in the main Jenkins dashboard, click on Manage Jenkins, followed by Manage Plugins:

Click the Installed tab and check that the Kubernetes plugin is installed:

We can now configure the plugin. Go to Manage Jenkins and select Configure System:

Scroll to the Cloud section at the bottom of the page. Click on Add a new cloud and select Kubernetes.

On the form that follows, in the Kubernetes URL field, enter https:// followed by the cluster endpoint IP address you copied from your GCP account.

Under Credentials, click the Add button and select Jenkins. On the form that appears, enter the username and password you copied from your GCP account and click the Add button at the bottom.

When you return to the Kubernetes form, select the credentials you just added from the Credentials drop down menu and click the Test Connection button. If the configuration is correct, the test will show “Connection test successful”.

Next, in the Jenkins tunnel field, enter the IP address and port that you retrieved from the jenkins-jnlp service in the Rancher UI, separated by a colon:

Now, scroll down to the Images section at the bottom of the page, click the Add Pod Template button, and select Kubernetes Pod Template. Fill out the Name and Labels fields with unique values to identify your first agent. We will use the label to specify which agent image should be used to run each build.

Next, in the Containers field, click the Add Container button and select Container Template. In the section that appears, fill out the following fields:

• Name: jnlp (this is required by the Jenkins agent)
• Docker image: <dockerhub_user>/jenkins-slave-jnlp1 (make sure to change the Docker Hub username)
• Command to run: Delete the value here
• Arguments to pass to the command: Delete the value here

The rest of the fields can be left as they are.

Next, click the Add Pod Template button and select Kubernetes Pod Template again. Repeat the process for the second agent image you created. Make sure to change the values to refer to your second image where applicable:

Click the Save button at the bottom to save your changes and continue.

Testing Dynamic Build Jobs

Now that our configuration is complete, we can create some build jobs to ensure that Jenkins can scale on top of Kubernetes. We will create five build jobs for each of our Jenkins agents.

On the main Jenkins page, click New Item on the left side. Enter a name for the first build of your first agent. Select Freestyle project and click the OK button.

On the next page, in the Label Expression field, type the label you set for your first Jenkins agent image. If you click out of the field, a message will appear indicating that the label is serviced by a cloud:

Scroll down to the Build Environment section and check Color ANSI Console Output.

In the Build section, click Add build step and select Execute shell. Paste the following script in the text box that appears:

#!/bin/bash

RED=’33[0;31m’
NC=’33[0m’

result=`ls / | grep -e jenkins-slave1 -e jenkins-slave2`
echo -e “$Docker image is for $result $”

Click the Save button when you are finished.

Create another four jobs for the first agent by clicking New Item, filling out a new name, and using the Copy from field to copy from your first build. You can save each build without changes to duplicate the first build exactly.

Next, configure the first job for your second Jenkins agent. Click New Item, select a name for the first job for the second agent, and copy the job from your first agent again. This time, we will modify the fields on the configuration page before saving.

First, change the Label Expression field to match the label for your second agent.

Next, replace the script in the text box in the Build section with the following script:

#!/bin/bash

BLUE=’e[34m’
NC=’33[0m’

result=`ls / | grep -e jenkins-slave1 -e jenkins-slave2`
echo -e “$Docker image is for $result $”

Click Save when you are finished.

Create four more builds for your second agent by copying from the job we just created.

Now, go to the home screen and start all the ten jobs you just created by clicking on the icon on the far right side of each row. As soon as you start them, they will be queued for execution as indicated by the Build Queue section:

After a few seconds, Pods will begin to be created to execute the builds (you can verify this in Rancher’s Workload tab). Jenkins will create one Pod for each job. As each agent is started, it connects to the master and receives a job from the queue to execute.

As soon as an agent finishes processing its job, it is automatically removed from the cluster:

To check the status of our jobs, we can click on one from each agent. Click the build from the Build History and then click Console Output. Jobs executed by the first agent should specify that the jenkins-slave1 Docker image was used, while builds executed by the second agent should indicate that the jenkins-slave2 image was used:

If you see the output above, Jenkins is configured correctly and functioning as intended. You can now begin to customize your Kubernetes-backed build system to help your team test and release software.

Conclusion

In this article, we configured Jenkins to automatically provision build agents on demand by connecting it with a Kubernetes cluster managed by Rancher. To achieve this, we completed the following steps:

• Created a cluster using Rancher
• Created custom Docker images for the Jenkins master and agents
• Deployed the Jenkins master and an L4 LoadBalancer service to the Kubernetes cluster
• Configured the Jenkins kubernetes plugin to automatically spawn dynamic agents on our cluster.
• Tested a scenario using multiple build jobs with dedicated agent images

The main purpose of this article was to highlight the basic configuration necessary to set up a Jenkins master and agent architecture. We saw how Jenkins launched agents using JNLP and how the containers automatically connected to the Jenkins master to receive instructions. To achieve this, we used Rancher to create the cluster, deploy a workload, and monitor the resulting Pods. Afterwards, we relied on the Jenkins Kubernetes plugin to glue together all of the different components.

Source

Introduction

With over 40,000 stars on Github, more than 70,000 commits, and with major contributors like Google and Redhat, Kubernetes has rapidly taken over the container ecosystem to become the true leader of container orchestration platforms.

Understanding Kubernetes and Its Abstractions

At the infrastructure level, a Kubernetes cluster is a set of physical or virtual machines acting in a specific role. The machines acting in the role of Master act as the brain of all operations and are charged with orchestrating containers that run on all of the Nodes.

• Master components manage the lifecycle of a pod, the base unit of deployment within a Kubernetes cluster. If a pod dies, the Controller creates a new one. If you scale the number of pod replicas up or down, the Controller creates or destroys pods to satisfy your request. The Master role includes the following components:
  • kube-apiserver – exposes APIs for the other master components.
  • etcd – a consistent and highly-available key/value store used for storing all internal cluster data.
  • kube-scheduler – uses information in the Pod spec to decide on which Node to run a Pod.
  • kube-controller-manager – responsible for Node management (detecting if a Node fails), pod replication, and endpoint creation.
  • cloud-controller-manager – runs controllers that interact with the underlying cloud providers.
• Node components are worker machines in Kubernetes and are managed by the Master. A node may be a virtual machine (VM) or physical machine, and Kubernetes runs equally well on both types of systems. Each node contains the necessary components to run pods:
  • kubelet: handles all communication between the Master and the node on which it is running. It interfaces with the container runtime to deploy and monitor containers.
  • kube-proxy: maintains the network rules on the host and handles transmission of packets between pods, the host, and the outside world.
  • container runtime: responsible for running containers on the host. The most popular engine is Docker, although Kubernetes supports container runtimes from rkt, runc and others.

From a logical perspective, a Kubernetes deployment is comprised of various components, each serving a specific purpose within the cluster.

• Pods are the basic unit of deployment within Kubernetes. A pod consists of one or more containers that share the same network namespace and IP address. Best practices recommend that you create one pod per application component so you can scale and control them separately.
• Services provide a consistent IP address in front of a set of pods and a policy that controls access to them. The set of pods targeted by a service is often determined by a label selector. This makes it easy to point the service to a different set of pods during upgrades or blue/green deployments.
• ReplicaSets are controlled by deployments and ensure that the desired number of pods for that deployment are running.
• Namespaces define a logical namespace for resources such as pods and services. They enable resources to use the same names, whereas resources in a single namespace must have unique names. Rancher uses namespaces with its role-based access control to provide a secure separation between namespaces and the resources running inside of them.
• Metadata marks containers based on their deployment characteristics.

Monitoring Kubernetes

Multiple services and namespaces can be spread across the infrastructure. As seen above, each of the services are made of pods, which can have one or more containers inside. With so many moving parts, monitoring even a small Kubernetes cluster can present a challenge. It requires a deep understanding of the application architecture and functionality in order to monitor it effectively.

Kubernetes ships with tools for monitoring the cluster:

• Probes actively monitor the health of a container. If the probe determines that a container is no longer healthy, the probe will restart it.
• cAdvisor is an open source agent that monitors resource usage and analyzes the performance of containers. Originally created by Google, cAdvisor is now integrated with the Kubelet. It collects, aggregates, processes and exports metrics such as CPU, memory, file and network usage for all containers running on a given node.
• The kubernetes dashboard is an add-on which gives an overview of the resources running on your cluster. It also gives a very basic means of deploying and interacting with those resources.

Kubernetes has tremendous capability for automatically recovering from failures. It can restart pods if a process crashes, and it will redistribute pods if a node fails. However, for all of its power, there are times when it cannot fix a problem. In order to detect those situations, we need additional monitoring.

Layers Of Monitoring

Infrastructure

All clusters should have monitoring of the underlying server components because problems at the server level will show up in the workloads.

What to monitor?

• CPU utilization. Monitoring the CPU will reveal both system and user consumption, and it will also show iowait. When running clusters in the cloud or with any network storage, iowait will indicate bottlenecks waiting for storage reads and writes (i/o processes). An oversubscribed storage framework can impact performance.
• Memory usage. Monitoring memory will show how much memory is in use and how much is available, either as free memory or as cache. Systems that run up against memory limits will begin to swap (if swap is available on the system), and swapping will rapidly degrade performance.
• Disk pressure. If a system is running write-intensive services like etcd or any datastore, running out of disk space can be catastrophic. The inability to write data will result in corruption, and that corruption can transfer to real-world losses. Technologies like LVM make it trivial to grow disk space as needed, but keeping an eye on it is imperative.
• Network bandwidth. In today’s era of gigabit interfaces, it might seem like you can never run out of bandwidth. However, it doesn’t take more than a few aberrant services, a data breach, system compromise, or DOS attack to eat up all of the bandwidth and cause an outage. Keeping awareness of your normal data consumption and the patterns of your application will help you keep costs down and also aid in capacity planning.
• Pod resources. The Kubernetes scheduler works best when it knows what resources a pod needs. It can then assure that it places pods on nodes where the resources are available. When designing your network, consider how many nodes can fail before the remaining nodes can no longer run all of the desired resources. Using a service such as a cloud autoscaling group will make recovery quick, but be sure that the remaining nodes can handle the increased load for the time that it takes to bring the failed node back online.

Kubernetes Services

All of the components that make up a Kubernetes Master or Worker, including etcd, are critical to the health of your applications. If any of these fail, the monitoring system needs to detect the failure and either fix it or send an alert.

Internal Services

The final layer is that of the Kubernetes resources themselves. Kubernetes exposes metrics about the resources, and we can also monitor the applications directly. Although we can trust that Kubernetes will work to maintain the desired state, if it’s unable to do so, we need a way for a human to intervene and fix the issue.

Monitoring with Rancher

In addition to managing Kubernetes clusters running anywhere, on any provider, Rancher will also monitor the resources running inside of those clusters and send alerts when they exceed defined thresholds.

There are already dozens of tutorials on how to deploy Rancher. If you don’t already have a cluster running, pause here and visit our quickstart guide to spin one up. When it’s running, return here to continue with monitoring.

The cluster overview gives you an idea of the resources in use and the state of the Kubernetes components. In our case, we’re using 78% of the CPU, 26% of the RAM and 11% of the maximum number of pods we can run within the cluster.

When you click on the Nodes tab, you’ll see additional information about each of the nodes running in the cluster, and when you click on a particular node, the view focuses on the health of that one member.

The Workloads tab shows the pods running in your cluster. If you don’t have anything running, launch a workload running the nginx image and scale it up to multiple replicas.

When you select the name of the workload, Rancher presents a page that shows information about it. At the top of this page it will show you each of the pods, which node they’re on, their IP address, and their state. Clicking on any individual pod takes us one level deeper, where now we see detailed information about only that pod. The hamburger menu icon in the top right corner lets us interact with the pod, and through this we can execute a shell, view the logs, or delete the pod.

Other tabs show information about different Kubernetes resources, including Load Balancing for ingress or services of type LoadBalancer, Service Discovery for other service types, and Volumes for information about any volumes configured in the cluster.

Use Prometheus for Monitoring

The information visible in the Rancher UI is useful for troubleshooting, but it’s not the best way to actively track the state of the cluster throughout every moment of its life. For that we’ll use Prometheus, a sibling project of Kubernetes under the care and guidance of the Cloud Native Computing Foundation. We’ll also use Grafana, a tool for converting time-series data into beautiful graphs and dashboards.

Prometheus is an open-source application for monitoring systems and generating alerts. It can monitor almost anything, from servers to applications, databases, or even a single process. In the Prometheus lexicon it monitors targets, and each unit of a target is called a metric. The act of retrieving information about a target is known as scraping. Prometheus will scrape targets at designated intervals and store the information in a time-series database. Prometheus has its own scripting language called PromQL.

Grafana is also open source and runs as a web application. Although frequently used with Prometheus, it also supports backend datastores such as InfluxDB, Graphite, Elasticsearch, and others. Grafana makes it easy to create graphs and assemble those graphs into dashboards. Those dashboards can be protected by a strong authentication and authorization layer, and they can also be shared with others without giving them access to the server itself. Grafana makes heavy use of JSON for its object definitions, which makes its graphs and dashboards extremely portable and easy to use with version control.

Rancher includes both Prometheus and Grafana in its application catalog, so we can deploy them with a few clicks.

Install Prometheus and Grafana

Visit the Catalog Apps page for your cluster and search for Prometheus. Installing it will also install Grafana and AlertManager. For this article it’s sufficient to leave everything at its defaults, but for a production deployment, read the information under Detailed Descriptions to see just how much configuration is available within the chart.

When you click Launch, Rancher will deploy the applications into your cluster, and after a few minutes, you’ll see all of the workloads in an Active state under the prometheus namespace.

The defaults set up a Layer7 ingress using xip.io, and we can see this on the Load Balancing tab. Clicking on the link will open the Grafana dashboard.

The Prometheus installation also deployed several dashboards into Grafana, so immediately we can see information about the cluster and start to view its performance over time.

Conclusion

Kubernetes works tirelessly to keep your applications running, but that doesn’t free you from the obligation of staying aware of how they’re doing. As soon as you begin to rely on Kubernetes to do work for you, the responsible thing to do is deploy a monitoring system that keeps you informed and empowers you to make decisions.

Prometheus and Grafana will do this for you, and when you use Rancher, the time it would normally take to deploy these two applications is reduced to mere minutes.

Calin Rus

Source

Today we announced a partnership with Arm to create a Kubernetes-based platform for IoT, edge, and data center nodes, all powered by Arm servers. The platform includes a port of Rancher Kubernetes Engine (RKE) and RancherOS to Arm Neoverse-based servers. We have also enhanced Rancher 2.1 so that a single Rancher server can manage mixed x86 and Arm 64 nodes. Customers can therefore have x86 clusters running in the data center and Arm clusters running on the edge.

Rancher and Arm are working jointly on a Smart City project in China. In that project, Arm servers are installed in buildings across large organizations. These ARM servers are used to collect data from various sensors including human presence, ambient temperature, and air quality. Sensor data is then processed in centralized data centers which is then used to coordinate power distribution and HVAC controls. Each edge node runs a standalone Arm Kubernetes cluster. Standard x86 Kubernetes clusters running in central data centers are used to process and analyze a large amount of data collected from edge nodes. A single Rancher server is used to manage all x86 and Arm Kubernetes clusters.

We learned a lot in the process of planning and executing this project. In the beginning, I did not understand why this customer wanted to run an independent Kubernetes cluster on each edge node. It seemed to be quite wasteful to run a separate instance of etcd, Kubernetes master, and Kubelet on each edge node. Why couldn’t we manage all the edge nodes as one large Kubernetes cluster? It turned out edge nodes had spotty network connectivity and therefore were not able to form a resilient multi-node Kubernetes cluster. Then why did the customer want to setup a Kubernetes cluster on the edge in the first place? Why couldn’t they just run standard Linux nodes and deploy rpm packages to the edge nodes? It turned out the software stack they were trying to deploy was quite sophisticated. It involved multiple services and required service discovery. The components had to be updated from time to time. Kubernetes was a great platform to manage application deployment.

The main challenge this customer had was that they needed a platform that can manage multiple Kubernetes clusters. Rancher suited their needs very well. Even better, we enhanced Rancher 2.1 so that it can manage mixed x86 and Arm clusters. Today, we simply rebuild edge clusters when they fail. In Rancher 2.2, we will be able to backup and restore RKE clusters, so that users no longer need to worry about losing etcd databases on edge node failures.

Kubernetes was designed to be a scalable platform. Our experience shows that it also works well as a single-node edge computing platform. We do wish, however, that Kubernetes itself consumed less resources. The single-node Kubernetes platform worked well on nodes with more than 8GB of memory, but it struggled on 4GB nodes. We are exploring ways to create a special-purpose Kubernetes distro that consumes less resources. Darren Shepherd, our Chief Architect, has been working on a minimal Kubernetes distro. It is still a work in progress. I hope, when he’s done, we will be able to fit the entire Kubernetes platform (etcd, master, kubelet) in less than 1GB of memory, so that our edge-solution can work on 4GB nodes.

If you are interested in exploring Kubernetes on the edge or reduce the footprint of your Kubernetes distros, please reach out to us at info@rancher.com. We’d love to work with you. We are super excited about the potential for applying Kubernetes technology on the edge. Kubernetes holds the potential to be the common platform for the cloud and for the edge.

Source

Survey after survey is showing that 70% of organizations are using cloud infrastructure. The last bits of the late majority will be joining this year and the laggards will come within a few years. Regardless of the stage of a company’s cloud native journey, security still remains an issue. The same survey states that security ranks as first priority with enterprises.

At Giant Swarm we pay close attention to the need for security. Last year we shared a 5 part series about security on our blog (I, II, III, IV, V). It presented some of the basics and our philosophy when it comes to security.

So…we talk the talk, great! What is important to our customers is that we also walk the walk.

In December 2018, a severe vulnerability was discovered in the Kubernetes API server. It allowed an unauthenticated user to perform privilege escalation and gain full admin privileges on a cluster.

The details of the vulnerability were discussed at length in the Kubernetes community. The chain of events is well documented across GitHub and Google Groups. Other contributors to the Kubernetes ecosystem provided analyses of the problem. One could easily find information about the problem, its identification and suggested mitigation.

The recommendation was to upgrade Kubernetes and new releases that included the fix were created for all active versions (v1.10.11, v1.11.5, v1.12.3). Earlier versions, did not receive an upgrade, so their upgrade deficit grew to include a security vulnerability.

At Giant Swarm, we were ready to upgrade all our customers to the secure version the next day. Regardless of the Kubernetes version, or the cloud provider. Customers on AWS, Azure, and on-premises, were all proactively notified of the vulnerability and its solution. Most of our customers don’t have Kubernetes APIs exposed to the public internet. Still, all benefited from a quick and transparent upgrade that allowed them to keep running their businesses – threat free.

This incident highlights how important it is to have several layers of security. But also that only with an automated update system, as well as the ability to quickly test and release upgrades, you and your business can really be safe.

Want to find out how the Giant Swarm Infrastructure deploys and scales Kubernetes Clusters? Request your free trial here by taking our survey and find out if you’re eligible.

Source

Introduction

Elasticsearch is an open-source search engine based on Apache Lucene and developed by Elastic. It focuses on features like scalability, resilience, and performance, and companies all around the world, including Mozilla, Facebook, Github, Netflix, eBay, the New York Times, and others, use it every day. Elasticsearch is one of the most popular analytics platforms for large datasets and is present almost everywhere that you find a search engine. It uses a document-oriented approach when manipulating data, and it can parse it in almost real-time while a user is performing a search. It stores data in JSON and organizes data by index and type.

If we draw analogs between the components of a traditional relational database and those of Elasticsearch, they look like this:

• Database or Table -> Index
• Row/Column -> Document with properties

Elasticsearch Advantages

• It originates from Apache Lucene, which provides the most robust full-text search capabilities of any open source product.
• It uses a document-oriented architecture to store complex real-world entities as structured JSON documents. By default, it indexes all fields, which provides tremendous performance when searching.
• It doesn’t use a schema with its indices. Documents add new fields by including them, which gives the freedom to add, remove, or change relevant fields without the downtime associated with a traditional database schema upgrade.
• It performs linguistic searches against documents, returning those that match the search condition. It scores the results using the TFIDF algorithm, bringing more relevant documents higher up in the list of results.
• It allows fuzzy searching, which helps find results even with misspelled search terms.
• It supports real-time search autocompletion, returning results while the user types their search query.
• It uses a RESTful API, exposing its power via a simple, lightweight interface.
• Elasticsearch executes complex queries with tremendous speed. It also caches queries, returning cached results for other requests that match a cached filter.
• It scales horizontally, making it possible to extend resources and balance the load between cluster nodes.
• It breaks indices into shards, and each shard has any number of replicas. Each node knows the location of every document in the cluster and routes requests internally as necessary to retrieve the data.

Terminology

Elasticsearch uses specific terms to define its components.

• Cluster: A collection of nodes that work together.
• Node: A single server that acts as part of the cluster, stores the data, and participates in the cluster’s indexing and search capabilities.
• Index: A collection of documents with similar characteristics.
• Document: The basic unit of information that can be indexed.
• Shards: Indexes are divided into multiple pieces called shards, which allows the index to scale horizontally.
• Replicas: Copies of index shards

Prerequisites

To perform this demo, you need one of the following:

• An existing Rancher deployment and Kubernetes cluster, or
• Two nodes in which to deploy Rancher and Kubernetes, or
• A node in which to deploy Rancher and a Kubernetes cluster running in a hosted provider such as GKE.

This article uses the Google Cloud Platform, but you may use any other provider or infrastructure.

Launch Rancher

If you don’t already have a Rancher deployment, begin by launching one. The quick start guide covers the steps for doing so.

Launch a Cluster

Use Rancher to set up and configure your cluster according to the guide most suited to your environment.

Deploy Elasticsearch

If you are already comfortable with kubectl, you can apply the manifests directly. If you prefer to use the Rancher user interface, scroll down for those instructions.

We will deploy Elasticsearch as a StatefulSet with two Services: a headless service for communicating with the pods and another for interacting with Elasticsearch from outside of the Kubernetes cluster.

svc-cluster.yaml

apiVersion: v1
kind: Service
metadata:
name: elasticsearch-cluster
spec:
clusterIP: None
selector:
app: es-cluster
ports:
– name: transport
port: 9300$ kubectl apply -f svc-cluster.yaml
service/elasticsearch-cluster created

svc-loadbalancer.yaml

apiVersion: v1
kind: Service
metadata:
name: elasticsearch-loadbalancer
spec:
selector:
app: es-cluster
ports:
– name: http
port: 80
targetPort: 9200
type: LoadBalancer$ kubectl apply -f svc-loadbalancer.yaml
service/elasticsearch-loadbalancer created

es-sts-deployment.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: es-config
data:
elasticsearch.yml: |
cluster.name: my-elastic-cluster
network.host: “0.0.0.0”
bootstrap.memory_lock: false
discovery.zen.ping.unicast.hosts: elasticsearch-cluster
discovery.zen.minimum_master_nodes: 1
xpack.security.enabled: false
xpack.monitoring.enabled: false
ES_JAVA_OPTS: -Xms512m -Xmx512m
—
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: esnode
spec:
serviceName: elasticsearch
replicas: 2
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: es-cluster
spec:
securityContext:
fsGroup: 1000
initContainers:
– name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
command: [“sysctl”, “-w”, “vm.max_map_count=262144”]
containers:
– name: elasticsearch
resources:
requests:
memory: 1Gi
securityContext:
privileged: true
runAsUser: 1000
capabilities:
add:
– IPC_LOCK
– SYS_RESOURCE
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.0
env:
– name: ES_JAVA_OPTS
valueFrom:
configMapKeyRef:
name: es-config
key: ES_JAVA_OPTS
readinessProbe:
httpGet:
scheme: HTTP
path: /_cluster/health?local=true
port: 9200
initialDelaySeconds: 5
ports:
– containerPort: 9200
name: es-http
– containerPort: 9300
name: es-transport
volumeMounts:
– name: es-data
mountPath: /usr/share/elasticsearch/data
– name: elasticsearch-config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
volumes:
– name: elasticsearch-config
configMap:
name: es-config
items:
– key: elasticsearch.yml
path: elasticsearch.yml
volumeClaimTemplates:
– metadata:
name: es-data
spec:
accessModes: [ “ReadWriteOnce” ]
resources:
requests:
storage: 5Gi$ kubectl apply -f es-sts-deployment.yaml
configmap/es-config created
statefulset.apps/esnode created

Deploy Elasticsearch via the Rancher UI

If you prefer, import each of the manifests above into your cluster via the Rancher UI. The screenshots below shows the process for each of them.

Import svc-cluster.yaml

Import svc-loadbalancer.yaml

Import es-sts-deployment.yaml

Retrieve the Load Balancer IP

You’ll need the address of the load balancer that we deployed. You can retrieve this via kubectl or the UI.

Use the CLI

$ kubectl get svc elasticsearch-loadbalancer
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-loadbalancer LoadBalancer 10.59.246.186 35.204.239.246 80:30604/TCP 33m

Use the UI

Test the Cluster

Use the address we retrieved in the previous step to query the cluster for basic information.

$ curl 35.204.239.246
{
“name” : “d7bDQcH”,
“cluster_name” : “my-elastic-cluster”,
“cluster_uuid” : “e3JVAkPQTCWxg2vA3Xywgg”,
“version” : {
“number” : “6.5.0”,
“build_flavor” : “default”,
“build_type” : “tar”,
“build_hash” : “816e6f6”,
“build_date” : “2018-11-09T18:58:36.352602Z”,
“build_snapshot” : false,
“lucene_version” : “7.5.0”,
“minimum_wire_compatibility_version” : “5.6.0”,
“minimum_index_compatibility_version” : “5.0.0”
},
“tagline” : “You Know, for Search”
}

Query the cluster for information about its nodes. The asterisk in the master column highlights the current master node.

$ curl 35.204.239.246/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.56.2.8 24 97 5 0.05 0.12 0.13 mdi – d7bDQcH
10.56.0.6 28 96 4 0.01 0.05 0.04 mdi * WEOeEqC

Check the available indices:

$ curl 35.204.239.246/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

Because this is a fresh install, it doesn’t have any indices or data. To continue this tutorial, we’ll inject some sample data that we can use later. The files that we’ll use are available from the Elastic website. Download them and then load them with the following commands:

$ curl -H ‘Content-Type: application/x-ndjson’ -XPOST
‘http://35.204.239.246/shakespeare/doc/_bulk?pretty’ –data-binary @shakespeare_6.0.json
$ curl -H ‘Content-Type: application/x-ndjson’ -XPOST
‘http://35.204.239.246/bank/account/_bulk?pretty’ –data-binary @accounts.json
$ curl -H ‘Content-Type: application/x-ndjson’ -XPOST
‘http://35.204.239.246/_bulk?pretty’ –data-binary @logs.json

When we recheck the indices, we see that we have five new indices with data.

$ curl 35.204.239.246/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2015.05.20 MFdWJxnsTISH0Z9Vr0aT3g 5 1 4750 0 49.9mb 25.2mb
green open logstash-2015.05.18 lLHV2nzvTOG9mzlpKaG9sg 5 1 4631 0 46.5mb 23.5mb
green open logstash-2015.05.19 PqNnVUgXTyaDSfmCQZwbLQ 5 1 4624 0 48.2mb 24.2mb
green open shakespeare rwl3xBgmQtm8B3V7GFeTZQ 5 1 111396 0 46mb 23.1mb
green open bank z0wVGsbrSiG2cQwRXwaCOg 5 1 1000 0 949.2kb 474.6kb

Each of these contains a different type of document. For the shakespeare index, we can search for the name of a play. For the logstash-2015.05.19 index we can query and filter data based on an IP address, and for the bank index we can search for information about a particular account.

Conclusion

Elasticsearch is extremely powerful. It is both simple and complex – simple to deploy and use, and complex in the way that it interacts with its data.

This article has shown you the basics of how to deploy it with Rancher and Kubernetes and how to query it via the RESTful API.

If you wish to explore ways to use Elasticsearch in everyday situations, we encourage you to explore the other parts of the ELK stack: Kibana, Logstash, and Beats. These tools round out an Elasticsearch deployment and make it useful for storing, retrieving, and visualizing a broad range of data from systems and applications.

Calin Rus

Source

DevOps engineers have grown so reliant on the power and scalability of Kubernetes (K8s) clusters that one server platform can seldom accommodate them all. More and more enterprises now run their containerized applications in clusters across multiple platforms at once, in public clouds and on-prem servers.

That can fuel a chaotic stampede in an enterprise-class system – who has control, and which builds do you trust?

Rancher offers a solution for managing multiple K8s clusters, and an enhanced Kubernetes distribution with additional features for central control of those clusters. Rancher’s multi-cluster operations management features provide a unified experience across public and private providers, VMware clusters, and bare metal servers that run in production across your organization, with common policies for provisioning and upgrades.

Kubernetes registry enables trust

While containerized applications help provide great stability through features like immutability and declarative configuration, they don’t guarantee that the software they contain is trusted. Without full control of and visibility into the source and dependencies that go into your containers, elements you don’t want or need can sneak into your builds.

JFrog Artifactory can provide the hybrid Kubernetes registry you need that gives you full visibility into your containers. Artifactory enables trust by giving you insight into your code-to-cluster process while providing visibility into each layer of each application. Moreover, a hybrid K8s registry will help you run applications effectively and safely across all clusters in all of the infrastructure environments you use.

Installing Artifactory with Rancher

Rancher makes it easy for you to install a high-availability instance of Artifactory through its catalog of applications directly into a Kubernetes cluster that you create for Artifactory. In this way, Artifactory instances can run in any of the infrastructure types you use, either on a public cloud or an on-prem server.

To start, install the Rancher Kubernetes Engine (RKE) onto a server and set up an admin account.

Step 1: Add a Cluster

From Rancher’s UI, add a new K8s cluster in the platform where your Artifactory instance will run.

• You can use a node template for nodes hosted by an infrastructure provider such as Google Cloud Platform (GCP), Amazon Web Services (AWS) or Azure, or set up a custom node for a local on-prem server.
• For a cluster on a hosted service like GKE, you may need to have a service account created by your support team that provides the privileges that you need.
• When you create the cluster, select a machine type powerful enough to support Artifactory (recommended minimum is 2 vCPUs, 7.5 Gb RAM)
• When you have completed your settings, provision the cluster. This may take several minutes to complete.

Step 2: Create a Project and Namespace

You can install Artifactory into the Default Rancher project that is automatically created when adding a cluster. However, it’s a good practice to create a Rancher project and namespace for Artifactory to run in,

For example, a project my-project and a namespace my-project-artifactory:

![Rancher Namespaces](https://rancher.com/img/blog/2018/Jfrog-Rancher-Namespaces.jpg

Step 3: Create a Certificate

The NGINX server used by Artifactory requires a certificate to run.
From the main menu, select Resources > Certificates. In the resulting page, supply the Private Key and Certificate, and assign the Name as artifactory-ha-tls.

![Rancher Certificate](https://rancher.com/img/blog/2018/Jfrog-Rancher-Certificate.png

When complete, click Save.

Step 4: Add a ConfigMap

Artifactory will require a ConfigMap for general configuration information needed by its load balancer.

The following example ConfigMap should be used for a standard setup:

## add HA entries when ha is configure.
upstream artifactory {
server artifactory-ha-artifactory-ha-primary:8081;
server artifactory-ha:8081;
}
## add ssl entries when https has been set in config
ssl_certificate /var/opt/jfrog/nginx/ssl/tls.crt;
ssl_certificate_key /var/opt/jfrog/nginx/ssl/tls.key;
ssl_session_cache shared:SSL:1m;
ssl_prefer_server_ciphers on;
## server configuration
server {
listen 443 ssl;
listen 80 ;
server_name ~(?<repo>.+).jfrog.team jfrog.team;

if ($http_x_forwarded_proto = ”) {
set $http_x_forwarded_proto $scheme;
}
## Application specific logs
## access_log /var/log/nginx/jfrog.team-access.log timing;
## error_log /var/log/nginx/jfrog.team-error.log;
rewrite ^/$ /artifactory/webapp/ redirect;
rewrite ^/artifactory/?(/webapp)?$ /artifactory/webapp/ redirect;
rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2;
chunked_transfer_encoding on;
client_max_body_size 0;
location /artifactory/ {
proxy_read_timeout 2400s;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
if ( $request_uri ~ ^/artifactory/(.*)$ ) {
proxy_pass http://artifactory/artifactory/$1;
}
proxy_pass http://artifactory/artifactory/;
proxy_next_upstream http_503 non_idempotent;
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port/artifactory;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

From the main menu, select Resources > Config Maps, then click Add Config Map.

![Rancher ConfigMap](https://rancher.com/img/blog/2018/Jfrog-Rancher-ConfigMap.jpg

1. In the Name field, enter art-nginx-conf
2. In the Namespace field, enter the name of the created namespace.
3. In the Key field, enter artifactory-ha.conf
4. Copy the example ConfigMap and paste it into the Value field.
5. Click Save

The ConfigMap will be used when Artifactory is installed.

Step 5: Install Artifactory

Once you have a cluster, project, and namespace that Artifactory can run in, you can install it easily through Rancher’s catalog of applications.

1. In the Rancher UI, click on Catalog Apps, then click the Launch button.

![Launch Rancher Catalog](https://rancher.com/img/blog/2018/Jfrog-Rancher-Catalog_Launch.jpg

1. In the catalog, find the JFrog artifactory-ha template marked “Partner”

![Rancher JFrog Catalog Item](https://rancher.com/img/blog/2018/Jfrog-Rancher-Catalog_JFrog.jpg

1. Click View Details

![Rancher JFrog Catalog Details](https://rancher.com/img/blog/2018/Jfrog-Rancher-Catalog_Install.jpg

Scroll down to set the Configuration Options. Set the name, enable persistent storage, and set the persistent volume size to a value large enough to accommodate your expected needs.

Set the Container Images to use the Default Image, and the Services and Load Balancing settings to use the NGINX server, assign the artifactory-ha-tls secret and the art-nginx-conf ConfigMap that were created in the prior steps.

![Rancher JFrog Catalog Details](https://rancher.com/img/blog/2018/Jfrog-Rancher-Settings.jpg

Set the Database Settings to enable and configure PostgreSQL.

![Rancher JFrog Database Settings](https://rancher.com/img/blog/2018/Jfrog-Rancher-Settings-Database.jpg

![Rancher JFrog Storage Settings](https://rancher.com/img/blog/2018/Jfrog-artifactory-storage.png

Click Launch to perform the installation.

![Rancher JFrog Install Launched](https://rancher.com/img/blog/2018/Jfrog-Rancher-Install_Launch.png

1. The installation will likely take several minutes to complete. When finished, it will present the JFrog artifactory-ha app as Active. The URL for the Artifactory HA installation is presented as a hotlink (for example, 443/tcp, 80/tcp). Click on the link to access the Artifactory HA application.

![Rancher JFrog Install Completed](https://rancher.com/img/blog/2018/Jfrog-Rancher-Install_Complete.png

Give it a try

Rancher and Artifactory both bring many pieces that would be challenging to manage independently into a single system, bringing control and visibility to the process. Together, they help enforce uniform policies, promotion flow, and more under a set of universal managers, quelling the risk of disorder.

Rancher’s integration of Artifactory through its catalog makes it especially easy to deploy and manage a hybrid Kubernetes Registry across all of the clusters you need across your organization.

If you’re already a Rancher user, you can install Artifactory immediately through the catalog of applications.

![Rancher JFrog Activate](https://rancher.com/img/blog/2018/Jfrog-Rancher-Activate.png

If you are new to Artifactory, you can request a set of three Artifactory Enterprise licenses for a free trial by emailing rancher-jfrog-licenses@jfrog.com.

Jainish Shah

JFrog Software Engineer

Source

The CNCF recently released 9 Kubernetes Security Best Practices Everyone Must Follow, in which they outline nine basic actions that they recommend people take with their Kubernetes clusters.

Although their recommendations are a good start, the article leans heavily on GKE. For those of you who are committed to using Google’s services, GKE is a good solution. However, others want to run in Amazon, Azure, DigitalOcean, on their own infrastructure, or anywhere else they can think of, and having solutions that point to GKE doesn’t help them.

For these people, Rancher is a great open source solution.

Rancher Labs takes security seriously. Darren Shepherd, who is one of the founders, discovered the bug that resulted in CVE-2018-1002105 in December, 2018. Security isn’t an afterthought or something you remember to do after you deploy an insecure cluster. You don’t, for example, build a house, move all of your belongings into it, and then put locks on the door.

In this article, I’ll respond to each of the points raised by the CNCF and walk you through how Rancher and RKE satisfy these security recommendations by default.

Upgrade to the Latest Version

This is sound advice that doesn’t only apply to Kubernetes. Unpatched software is the most common entry point for attackers when they breach systems. When a CVE is released and proof of concept code is made publicly available, tool suites such as Metasploit quickly include the exploits in their standard kit. Anyone with the skill to copy and paste commands from the Internet can find themselves in control of your systems.

When using Rancher Kubernetes Engine (RKE), either standalone or when installed by Rancher, you can choose the version of Kubernetes to install. Rancher Labs uses native upstream Kubernetes, and this enables the company to quickly respond to security alerts, releasing patched versions of the software. Because RKE runs the Kubernetes components within Docker containers. operations teams can perform zero-downtime upgrades of critical infrastructure.

I recommend that you follow Rancher Labs on Twitter to receive announcements about new releases. I also strongly recommend that you test new versions in a staging environment before upgrading, but in the event that an upgrade goes awry, Rancher makes it just as easy to roll back to a previous version.

Enable Role-Based Access Control (RBAC)

RKE installs with RBAC enabled by default. If you’re only using RKE, or any other standalone Kubernetes deployment, you’re responsible for configuring the accounts, roles, and bindings to secure your cluster.

If you’re using Rancher, it not only installs secure clusters, but it proxies all communication to those clusters through the Rancher server. Rancher plugs into a number of backend authentication providers, such as Active Directory, LDAP, SAML, Github, and more. When connected in this way, Rancher enables you to extend your existing corporate authentication out to all of the Kubernetes clusters under Rancher’s umbrella, no matter where they’re running.

Rancher enables roles at the global, cluster, and project level, and it makes it possible for administrators to define roles in a single place and apply them to all clusters.

This combination of RBAC-by-default and strong controls for authentication and authorization means that from the moment you deploy a cluster with Rancher or RKE, that cluster is secure.

Use Namespaces to Establish Security Boundaries

Because of the special way that Kubernetes treats the default namespace, I don’t recommend that you use it. Instead, create a namespace for each of your applications, defining them as logical groups.

Rancher defines an additional layer of abstraction called a Project. A Project is a collection of namespaces, onto which roles can be mapped. Users with access to one Project cannot see any or interact with any workload running in another Project to which they do not have access. This effectively creates single-cluster multi-tenancy.

Using Projects makes it easier for administrators to grant access to multiple namespaces within a single cluster. It minimizes duplicated configuration and reduces human error.

Separate Sensitive Workloads

This is a good suggestion, in that it presumes the question, “what happens if a workload is compromised?” Acting in advance to reduce the blast radius of a breach makes it harder for an attacker to escalate privileges, but it doesn’t make it impossible. If anything, this might buy you additional time.

Kubernetes allows you to set taints and tolerations, which control where a Pod might be deployed.

Rancher also lets you control scheduling of workloads through Kubernetes labels. In addition to taints and tolerations, when deploying a workload you can set the labels that a host must have, should have, or can have for a Pod to land there. You can also schedule workloads to a specific node if your environment is that static.

This suggestion states that it sensitive metadata “can sometimes be stolen or misused,” but it fails to outline the conditions of when or how. The article references a disclosure from Shopify, presented at Kubecon NA on December 13, 2018. Although this piece of the article points out a GKE feature for “metadata concealment,” it’s worth noting that the service which leaked the credentials in the first place was the Google Cloud metadata API.

There is nothing that shows the same vulnerability exists with any other cloud provider.

The only place this vulnerability might exist would be in a hosted Kubernetes service such as GKE. If you deploy RKE onto bare metal or cloud compute instances, either directly or via Rancher, you’ll end up with a cluster that cannot have credentials leaked via the cloud provider’s metadata API.

If you’re using GKE, I recommend that you activate this feature to prevent any credentials from leaking via the metadata service.

I would also argue that cloud providers should never embed credentials into metadata accessible via an API. Even if this exists for convenience, it’s an unnecessary risk with unimaginable consequences.

Create and Define Cluster Network Policies

RKE clusters, deployed directly or by Rancher, use Canal by default, although you can also choose Calico or Flannel. Both Canal and Calico include support for NetworkPolicies. Rancher-deployed clusters, when using Canal as a network provider, also support ProjectNetworkPolicies. When activated, workloads can speak to other workloads within their Project, and the System project, which includes cluster-wide components such as ingress controllers, can communicate with all projects.

Earlier versions of Rancher enabled ProjectNetworkPolicies by default, but this created confusion for some users who weren’t aware of the extra security. To provide the best experience across the entire user base, this feature is now off by default but can be easily activated at launch time or later if you change your mind.

Run a Cluster-wide Pod Security Policy

A Pod Security Policy (PSP) controls what capabilities and configuration Pods must have in order to run within your cluster. For example, you can block privileged mode, host networking, or containers running as root. When installing a cluster via Rancher or RKE, you choose if you want a restricted PSP enabled by default. If you choose to enable it, your cluster will immediately enforce strong limitations on the workload permissions.

The restricted and unrestricted PSPs are the same within RKE and Rancher, so what they activate at install is identical. Rancher allows an unlimited number of additional PSP templates, all handled at the global level. Administrators define PSPs and then apply them to every cluster that Rancher manages. This, like the RBAC configuration discussed earlier, keeps security configuration in a single place and dramatically simplifies the configuration and application of the policies.

When something is easy to do, more people will do it.

Harden Node Security

This isn’t a Kubernetes-specific suggestion, but it’s a good general policy. Anything that interacts with traffic that you don’t control, such as user traffic hitting an application running within Kubernetes, should be running on nodes with a small attack surface. Disable and uninstall unneeded services. Restrict root access via SSH and require a password for sudo. Use passphrases on SSH keys, or use 2FA, U2F keys, or a service like Krypton to bind keys to devices that your users have. These are examples of basic, standard configurations for secure systems.

Rancher requires nothing on the host beyond a supported version of Docker. RKE requires nothing but SSH access, and it will install the latest version of Docker supported by Kubernetes before continuing to install Kubernetes itself.

If you want to reduce the attack surface even more, take a look at RancherOS, a lightweight Linux operating system that runs all processes as Docker containers. The System Docker runs only the smallest number of processes necessary to provide access and run an instance of Docker in userspace for the actual workloads. Both lightweight and secure, RancherOS is what an operating system should be: secure by default.

Turn on Audit Logging

The Rancher Server runs inside of an RKE cluster, so in addition to the Kubernetes audit logging, it’s important to activate audit logging for API calls to the server itself. This log will show all activities that users execute to any cluster, including what happened, who did it, when they did it, and what cluster they did it to.

It’s also important to ship these logs off of the servers in question. Rancher connects to Splunk, Elasticsearch, Fluentd, Kafka, or any syslog endpoint, and from these you can generate dashboards and alerts for suspicious activity.

Information on enabling audit logging for the Rancher Server is available in our documentation.

For information on enabling audit logging for RKE clusters, please see the next section.

Ongoing Security

It takes more than nine changes to truly secure a Kubernetes cluster. Rancher has a hardening guide and a self assessment guide that cover more than 100 controls from the CIS Benchmark for Securing Kubernetes.

If you’re serious about security, Rancher, RKE, and RancherOS will help you stay that way.

ChangeLog:

• 2019-01-24: added clarification around ProjectNetworkPolicies and additional images
• 2019-01-23: added main image at top of article

Adrian Goins

Senior Solutions Architect

Adrian has been online since 1986, when he first got his hands on a 300 baud modem for his C64. He fell in love with computers and started writing software in 1988, moving into Unix and Linux and launching a career building Internet infrastructure in 1996. Fluent in languages spoken by humans and computers alike, Adrian is a champion for Rancher and Kubernetes. He is passionate about automation and efficiency, and he loves to teach anyone who wants to learn about technology. When not pushing Kubernetes to its limits, you’ll find him flying drones or working on his farm in the Chilean central valley.

Source

Introduction

Kubernetes clusters can manage large numbers of unrelated workloads concurrently and organizations often choose to deploy projects created by separate teams to shared clusters. Even with relatively light use, the number of deployed objects can quickly become unmanageable, slowing down operational responsiveness and increasing the chance of dangerous mistakes.

Kubernetes uses a concept called namespaces to help address the complexity of organizing objects within a cluster. Namespaces allow you to group objects together so you can filter and control them as a unit. Whether applying customized access control policies or separating all of the components for a test environment, namespaces are a powerful and flexible concept for handling objects as a group.

In this article, we’ll discuss how namespaces work, introduce a few common use cases, and cover how to use namespaces to manage your Kubernetes objects. Towards the end, we’ll also take a look at a Rancher feature called projects that builds on and extends the namespaces concept.

What are Namespaces and Why Are They Important?

Namespaces are the organizational mechanism that Kubernetes provides to categorize, filter by, and manage arbitrary groups of objects within a cluster. Each workload object added to a Kubernetes cluster must be placed within exactly one namespace.

Namespaces impart a scope for object names within a cluster. While names must be unique within a namespace, the same name can be used in different namespaces. This can have some important practical benefits for certain scenarios. For example, if you use namespaces to segment application life cycle environments — like development, staging, and production — you can maintain copies of the same objects, with the same names, in each environment.

Namespaces also allow you to easily apply policies to specific slices of your cluster. You can control resource usage by defining ResourceQuota objects, which set limits on consumption on a per-namespace basis. Similarly, when using a CNI (container network interface) that supports network policies on your cluster, like Calico or Canal (Calico for policy with flannel for networking), you can apply a NetworkPolicy to the namespace with rules that dictate how pods can be communicate with one another. Different namespaces can be given different policies.

One of the greatest benefits of using namespaces is being able to take advantage of Kubernetes RBAC (role-based access control). RBAC allows you to develop roles, which group a list of permissions or abilities, under a single name. ClusterRole objects exist to define cluster-wide usage patterns, while the Role object type is applied to a specific namespace, giving greater control and granularity. Once a Role is created, a RoleBinding can grant the defined capabilities to a specific user or group of users within the context of a single namespace. In this way, namespaces let cluster operators map the same policies to organized sets of resources.

Common Namespace Usage Patterns

Namespaces are an incredibly flexible feature that doesn’t impose a specific structure or organizational pattern. That being said, there are some common patterns that many teams find useful.

Mapping Namespaces to Teams or Projects

One convention to use when setting up namespaces is to create one for each discrete project or team. This melds well with many of the namespace characteristics we mentioned earlier.

By giving a team a dedicated namespace, you can allow self-management and autonomy by delegating certain responsibilities with RBAC policies. Adding and removing members from the namespace’s RoleBinding objects is a simple way to control access to the team’s resources. It is also often useful to set resource quotas for teams and projects. This way, you can ensure equitable access to resources based the organization’s business requirements and priorities.

Using Namespaces to Partition Life Cycle Environments

Namespaces are well suited for carving out development, staging, and production environments within cluster. While it recommended to deploy production workloads to an entirely separate cluster to ensure maximum isolation, for smaller teams and projects, namespaces can be a workable solution.

As with the previous use case, network policies, RBAC policies, and quotas are big factors in why this can be successful. The ability to isolate the network to control communication to your components is a fundamental requirement when managing environments. Likewise, namespace-scoped RBAC policies allow operators to set strict permissions for production environments. Quotas help you guarantee access to important resources for your most sensitive environments.

The ability to reuse object names is also helpful here. Objects can be rolled up to new environments as they they are tested and released while retaining their original name. This helps avoid confusion around which objects are analogous across environments and reduces cognitive overhead.

Using Namespaces to Isolate Different Consumers

Another use case that namespaces can help with is segmenting workloads by their intended consumers. For instance, if your cluster provides infrastructure for multiple customers, segmenting by namespace allows you to manage each independently while keeping track of usage for billing purposes.

Once again, namespace features allow you to control network and access policies and define quotas for your consumers. In cases where the offering is fairly generic, namespaces allow you to develop and deploy a different instance of the same templated environment for each of your users. This consistency can make management and troubleshooting significantly easier.

Understanding the Preconfigured Kubernetes Namespaces

Before we take a look at how to create your own namespaces, let’s discuss what Kubernetes sets up automatically. By default, three namespaces are available on new clusters:

• default: Adding an object to a cluster without providing a namespace will place it within the default namespace. This namespace acts as the main target for new user-added resources until alternative namespaces are established. It cannot be deleted.
• kube-public: The kube-public namespace is intended to be globally readable to all users with or without authentication. This is useful for exposing any cluster information necessary to bootstrap components. It is primarily managed by Kubernetes itself.
• kube-system: The kube-system namespace is used for Kubernetes components managed by Kubernetes. As a general rule, avoid adding normal workloads to this namespace. It is intended to be managed directly by the system and as such, it has fairly permissive policies.

While these namespaces effectively segregate user workloads the system-managed workloads, they do not impose any additional structure to help categorize and manage applications. Thankfully, creating and using additional namespaces is very straightforward.

Working with Namespaces

Managing namespaces and the resources they contain is fairly straightforward with kubectl. In this section we will demonstrate some of the most common namespace operations so you can start effectively segmenting your resources.

Viewing Existing Namespaces

To display all namespaces available on a cluster, use use the kubectl get namespaces command:

NAME STATUS AGE
default Active 41d
kube-public Active 41d
kube-system Active 41d

The command will show all available namespaces, whether they are currently active, and the resource’s age.

To get more information about a specific namespace, use the kubectl describe command:

kubectl describe namespace defaultName: default
Labels: field.cattle.io/projectId=p-cmn9g
Annotations: cattle.io/status={“Conditions”:[{“Type”:”ResourceQuotaInit”,”Status”:”True”,”Message”:””,”LastUpdateTime”:”2018-12-17T23:17:48Z”},{“Type”:”InitialRolesPopulated”,”Status”:”True”,”Message”:””,”LastUpda…
field.cattle.io/projectId=c-7tf7d:p-cmn9g
lifecycle.cattle.io/create.namespace-auth=true
Status: Active

No resource quota.

No resource limits.

This command can be used to display the labels and annotations associated with the namespace, as well as any quotas or resource limits that have been applied.

Creating a Namespace

To create a new namespace from the command line, use the kubectl create namespace command. Include the name of the new namespace as the argument for the command:

kubectl create namespace demo-namespacenamespace “demo-namespace” created

You can also create namespaces by applying a manifest from a file. For instance, here is a file that defines the same namespace that we created above:

# demo-namespace.yml
apiVersion: v1
kind: Namespace
metadata:
name: demo-namespace

Assuming the spec above is saved to a file called demo-namespace.yml, you can apply it by typing:

kubectl apply -f demo-namespace.yml

Regardless of how we created the namespace, if we check our available namespaces again, the new namespace should be listed (we use ns, a shorthand for namespaces, the second time around):

NAME STATUS AGE
default Active 41d
demo-namespace Active 2m
kube-public Active 41d
kube-system Active 41d

Our namespace is available and ready to use.

Filtering and Performing Actions by Namespace

If we deploy a workload object to the cluster without specifying a namespace, it will be added to the default namespace:

kubectl create deployment –image nginx demo-nginxdeployment.extensions “demo-nginx” created

We can verify the deployment was created in the default namespace with kubectl describe:

kubectl describe deployment demo-nginx | grep Namespace

If we try to create a deployment with the same name again, we will get an error because of the namespace collision:

kubectl create deployment –image nginx demo-nginxError from server (AlreadyExists): deployments.extensions “demo-nginx” already exists

To apply an action to a different namespace, we must include the –namespace= option in the command. Let’s create a deployment with the same name in the demo-namespace namespace:

kubectl create deployment –image nginx demo-nginx –namespace=demo-namespacedeployment.extensions “demo-nginx” created

This newest deployment was successful even though we’re still using the same deployment name. The namespace provided a different scope for the resource name, avoiding the naming collision we experienced earlier.

To see details about the new deployment, we need to specify the namespace with the –namespace= option again:

kubectl describe deployment demo-nginx –namespace=demo-namespace | grep NamespaceNamespace: demo-namespace

This confirms that we have created another deployment called demo-nginx within our demo-namespace namespace.

Selecting Namespace by Setting the Context

If you want to avoid providing the same namespace for each of your commands, you can change the default namespace that commands will apply to by configuring your kubectl context. This will modify the namespace that actions will apply to when that context is active.

To list your context configuration details, type:

kubectl config get-contextsCURRENT NAME CLUSTER AUTHINFO NAMESPACE
* Default Default Default

The above indicates that we have a single context called Default that is being used. No namespace is specified by the context, so the default namespace applies.

To change the namespace used by that context to our demo-context, we can type:

kubectl config set-context $(kubectl config current-context) –namespace=demo-namespaceContext “Default” modified.

We can verify that the demo-namespace is currently selected by viewing the context configuration again:

kubectl config get-contextsCURRENT NAME CLUSTER AUTHINFO NAMESPACE
* Default Default Default demo-namespace

Validate that our kubectl describe command now uses demo-namespace by default by asking for our demo-nginx deployment without specifying a namespace:

kubectl describe deployment demo-nginx | grep NamespaceNamespace: demo-namespace

Deleting a Namespace and Cleaning Up

If you no longer require a namespace, you can delete it.

Deleting a namespace is very powerful because it not only removes the namespaces, but it also cleans up any resources deployed within it. This can be very convenient, but also incredibly dangerous if you are not careful.

It is always a good idea to list the resources associated with a namespace before deleting to verify the objects that will be removed:

kubectl get all –namespace=demo-namespaceNAME READY STATUS RESTARTS AGE
pod/demo-nginx-676fc7d85d-gkdz2 1/1 Running 0 56m

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/demo-nginx 1 1 1 1 56m

NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-nginx-676fc7d85d 1 1 1 56m

Once we are comfortable with the scope of the action, we can delete the demo-namespace namespace and all of the resources within it by typing:

kubectl delete namespace demo-namespace

The namespace and its resources will be removed from the cluster:

NAME STATUS AGE
default Active 41d
kube-public Active 41d
kube-system Active 41d

If you previously changed the selected namespace in your kubectl context, you can clear the namespace selection by typing:

kubectl config set-context $(kubectl config current-context) –namespace=Context “Default” modified.

While cleaning up demo resources, remember to remove the original demo-nginx deployment we initially provisioned to the default namespace:

kubectl delete deployment demo-nginx

Your cluster should now be in the state you began with.

Extending Namespaces with Rancher Projects

If you are using Rancher to manage your Kubernetes clusters, you have access to the extended functionality provided by the projects feature. Rancher projects are an additional organizational layer used to bundle multiple namespaces together.

Rancher projects overlay a control structure on top of namespaces that allow you to group namespaces into logical units and apply policy to them. Projects mirror namespaces in most ways, but act as a container for namespaces instead of for individual workload resources. Each namespace in Rancher exists in exactly one project and namespaces inherit all of the policies applied to the project.

By default, Rancher clusters define two projects:

• Default: This project contains the default namespace.
• System: This project contains all of the other preconfigured namespaces, including kube-public, kube-system, and any namespaces provisioned by the system.

You can see the projects available within your cluster by visiting the Projects/Namespaces tab after selecting your cluster:

Fig. 1: Rancher projects view

From here, you can add projects by clicking on the Add Project button. When creating a project, you can configure the project members and their access rights and can configure security policies and resource quotas.

You can add a namespace to an existing project by clicking the project’s Add Namespace button. To move a namespace to a different project, select the namespace and then click the Move button. Moving a namespace to a new project switches immediately modifies the permissions and policies applied to the namespace.

Rather than introducing new organizational models, Rancher projects simply apply the same abstractions to namespaces that namespaces apply to workload objects. They fill in some usability gaps if you appreciate namespaces functionality but need an additional layer of control.

Conclusion

In this article, we introduced the concept of Kubernetes namespaces and how they can help organize cluster resources. We discussed how namespaces segment and scope resource names within a cluster and how policies applied at the namespace level can influence user permissions and resource allotment.

Afterwards, we covered some common patterns that teams employ to segment their clusters into logical pieces and we described Kubernetes’ preconfigured namespaces and their purpose. Then we took a look at how to create and work with namespaces within a cluster. We ended by taking a look at Rancher projects and how they extend the namespaces concept by grouping namespaces themselves.

Namespaces are an incredibly straightforward concept that help teams organize cluster resources and compartmentalize complexity. Taking a few minutes to get familiar with their benefits and characteristics can help you configure your clusters effectively and avoid trouble down the road.

Justin Ellingwood

Rancher Content Manager

Source

Introduction

Etcd is an open-source distributed key-value store created by the CoreOS team, now managed by the Cloud Native Computing Foundation. It is pronounced “et-cee-dee”, making reference to distributing the Unix “/etc” directory, where most global configuration files live, across multiple machines. It serves as the backbone of many distributed systems, providing a reliable way for storing data across a cluster of servers. It works on a variety of operating systems including here Linux, BSD and OS X.

Etcd has the following properties:

• Fully Replicated: The entire store is available on every node in the cluster
• Highly Available: Etcd is designed to avoid single points of failure in case of hardware or network issues
• Consistent: Every read returns the most recent write across multiple hosts
• Simple: Includes a well-defined, user-facing API (gRPC)
• Secure: Implements automatic TLS with optional client certificate authentication
• Fast: Benchmarked at 10,000 writes per second
• Reliable: The store is properly distributed using the Raft algorithm

How Does Etcd Work?

To understand how Etcd works, it is important to define three key concepts: leaders, elections, and terms. In a Raft-based system, the cluster holds an election to choose a leader for a given term.

Leaders handle all client requests which need cluster consensus. Requests not requiring consensus, like reads, can be processed by any cluster member. Leaders are responsible for accepting new changes, replicating the information to follower nodes, and then committing the changes once the followers verify receipt. Each cluster can only have one leader at any given time.

If a leader dies, or is no longer responsive, the rest of the nodes will begin a new election after a predetermined timeout to select a new leader. Each node maintains a randomized election timer that represents the amount of time the node will wait before calling for a new election and selecting itself as a candidate.

If the node does not hear from the leader before a timeout occurs, the node begins a new election by starting a new term, marking itself as a candidate, and asking for votes from the other nodes. Each node votes for the first candidate that requests its vote. If a candidate receives a vote from the majority of the nodes in the cluster, it becomes the new leader. Since the election timeout differs on each node, the first candidate often becomes the new leader. However, if multiple candidates exist and receive the same number of votes, the existing election term will end without a leader and a new term will begin with new randomized election timers.

As mentioned above, any changes must be directed to the leader node. Rather than accepting and committing the change immediately, Etcd uses the Raft algorithm to ensure that the majority of nodes all agree on the change. The leader sends the proposed new value to each node in the cluster. The nodes then send a message confirming receipt of the new value. If the majority of nodes confirm receipt, the leader commits the new value and messages each node that the value is committed to the log. This means that each change requires a quorum from the cluster nodes in order to be committed.

Etcd in Kubernetes

Since its adoption as part of Kubernetes) in 2014, the Etcd community has grown exponentially. There are many contributing members including CoreOS, Google, Redhat, IBM, Cisco, Huawei and more. Etcd is used successfully in production environments by large cloud providers such as AWS, Google Cloud Platform, and Azure.

Etcd’s job within Kubernetes is to safely store critical data for distributed systems. It’s best known as Kubernetes’ primary datastore used to store its configuration data, state, and metadata. Since Kubernetes usually runs on a cluster of several machines, it is a distributed system that requires a distributed datastore like Etcd.

Etcd makes it easy to store data across a cluster and watch for changes, allowing any node from Kubernetes cluster to read and write data. Etcd’s watch functionality is used by Kubernetes to monitor changes to either the actual or the desired state of its system. If they are different, Kubernetes makes changes to reconcile the two states. Every read by the kubectl command is retrieved from data stored in Etcd, any change made (kubectl apply) will create or update entries in Etcd, and every crash will trigger value changes in etcd.

Deployment and Hardware Recommendations

For testing or development purposes, Etcd can run on a laptop or a light cloud setup. However, when running Etcd clusters in production, we should take in consideration the guidelines offered by Etcd’s official documentation. The page offers a good starting point for a robust production deployment. Things to keep in mind:

• Since Etcd writes data to disk, SSD is highly recommended
• Always use an odd number of cluster members as quorum is needed to agree on updates to the cluster state
• For performance reasons, clusters should usually not have more than seven nodes

Let’s go over the steps required to deploy an Etcd cluster in Kubernetes. Afterward, we will demonstrate some basic CLI commands or API calls. We will use a combination of Kubernetes’ concepts like StatefulSets and PersistentVolumes for our deployment.

Prerequisites

To follow along with this demo, you will need the following:

• a Google Cloud Platform account: The free tier should be more than enough. You should be able to use most other cloud providers with little modification.
• A server to run Rancher

Starting a Rancher Instance

To begin, start a Rancher instance on your control server. There is a very intuitive getting started guide for this purpose here.

Using Rancher to Deploy a GKE Cluster

Use Rancher to set up and configure a Kubernetes cluster in your GCP account using this guide.

Install the Google Cloud SDK and kubelet command on the same server hosting our Rancher instance. Install the SDK by following the link provided above, and install kubelet through the Rancher UI.

Make sure that the gcloud command has access to your GCP account by authenticating with gcloud init and gcloud auth login.

As soon as cluster is deployed, check basic kubectl functionality by typing:

NAME STATUS ROLES AGE VERSION
gke-c-ggchf-default-pool-df0bc935-31mv Ready <none> 48s v1.11.6-gke.2
gke-c-ggchf-default-pool-df0bc935-ddl5 Ready <none> 48s v1.11.6-gke.2
gke-c-ggchf-default-pool-df0bc935-qqhx Ready <none> 48s v1.11.6-gke.2

Before deploying the Etcd cluster (through kubectl or by importing YAML files in Rancher’s UI), we need to configure a few items. In GCE, the default persistent disk is pd-standard. We will configure pd-ssd for our Etcd deployment. This is not mandatory, but as per Etcd recommendations, SSD is very good option. Please check this page to learn about other cloud providers’ storage classes.

Let’s check the available storage class that GCE offers. As expected, we see the default one, called standard:

NAME PROVISIONER AGE
standard (default) kubernetes.io/gce-pd 2m

Let’s apply this YAML file, updating the value of zone to match your preferences, so we can benefit of SSD storage:

# storage-class.yaml
—
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ssd
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
zone: europe-west4-c # Change this value

If we check again, we can see that, along the default standard class, ssd is now available:

kubectl apply -f storage-class.yaml
kubectl get storageclassstorageclass.storage.k8s.io/ssd created

NAME PROVISIONER AGE
ssd kubernetes.io/gce-pd 7s
standard (default) kubernetes.io/gce-pd 4m

We can now proceed with deploying the Etcd cluster. We will create a StatefulSet with three replicas, each of which have a dedicated volume with the ssd storageClass. We will also need to deploy two services, one for internal cluster communication and the other to access the cluster externally via the API.

When forming the cluster, we need to pass a few parameters to the Etcd binary to the datastore. The listen-client-urls and listen-peer-urls options specify the local addresses the Etcd server uses to accepting incoming connections. Specifying 0.0.0.0 as the IP address means that Etcd will listen for connections on all available interfaces. The advertise-client-urls and initial-advertise-peer-urls parameters specify the addresses Etcd clients or other etcd members should use to contact the etcd server.

The following YAML file defines our two services and the Etcd StatefulSet:

# etcd-sts.yaml
—
apiVersion: v1
kind: Service
metadata:
name: etcd-client
spec:
type: LoadBalancer
ports:
– name: etcd-client
port: 2379
protocol: TCP
targetPort: 2379
selector:
app: etcd
—
apiVersion: v1
kind: Service
metadata:
name: etcd
spec:
clusterIP: None
ports:
– port: 2379
name: client
– port: 2380
name: peer
selector:
app: etcd
—
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: etcd
labels:
app: etcd
spec:
serviceName: etcd
replicas: 3
template:
metadata:
name: etcd
labels:
app: etcd
spec:
containers:
– name: etcd
image: quay.io/coreos/etcd:latest
ports:
– containerPort: 2379
name: client
– containerPort: 2380
name: peer
volumeMounts:
– name: data
mountPath: /var/run/etcd
command:
– /bin/sh
– -c
– |
PEERS=”etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380″
exec etcd –name $
–listen-peer-urls http://0.0.0.0:2380
–listen-client-urls http://0.0.0.0:2379
–advertise-client-urls http://$.etcd:2379
–initial-advertise-peer-urls http://$:2380
–initial-cluster-token etcd-cluster-1
–initial-cluster $
–initial-cluster-state new
–data-dir /var/run/etcd/default.etcd
volumeClaimTemplates:
– metadata:
name: data
spec:
storageClassName: ssd
accessModes: [ “ReadWriteOnce” ]
resources:
requests:
storage: 1Gi

Apply the YAML file by typing:

kubectl apply -f etcd-sts.yaml service/etcd-client created
service/etcd created
statefulset.apps/etcd created

After applying the YAML file, we can the resources it defines within the different tabs Rancher offers:

Fig. 1: Etcd StatefulSet as seen in the Rancher Workloads tab

Fig. 2: Etcd Service as seen in the Rancher Service Discovery tab

Fig. 3: Etcd volume as seen in the Rancher Volumes tab

Interacting with Etcd

There are two primary ways to interact with Etcd: either using etcdctl command or directly through the RESTful API. We will cover both of these briefly, but you can find more in depth information and additional examples by visiting the full documentation here and here.

etcdctl is a command-line interface for interacting with an Etcd server. It can be used to perform a variety of actions such as setting, updating, or removing keys, verifying the cluster health, adding or removing Etcd nodes, and generating database snapshots. By default, etcdctl talks to the Etcd server with the v2 API for backward compatibility. If you want etcdctl to speak to Etcd using the v3 API, you must set the version to “3” via the ETCDCTL_API environment variable.

As for the API, every request sent to an Etcd server is a gRPC remote procedure call. This gRPC gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Let’s find the external IP we need to use for API calls:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd ClusterIP None <none> 2379/TCP,2380/TCP 1m
etcd-client LoadBalancer 10.15.247.17 35.204.136.231 2379:30525/TCP 1m
kubernetes ClusterIP 10.15.240.1 <none> 443/TCP 3m

We should also find the names of our three Pods so that we can use the etcdctl command:

NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 6m
etcd-1 1/1 Running 0 6m
etcd-2 1/1 Running 0 6m

Let’s check the Etcd version. For this, we can use the API or CLI (both v2 and v3). The output will be slightly different depending on your chosen method.

Use this command to contact the API directly:

curl http://35.204.136.231:2379/version{“etcdserver”:”3.3.8″,”etcdcluster”:”3.3.0″}

To check for the version with v2 of the etcdctl client, type:

kubectl exec -it etcd-0 — etcdctl –versionetcdctl version: 3.3.8
API version: 2

To use the etcdctl with v3 of the API, type:

kubectl exec -it etcd-0 — /bin/sh -c ‘ETCDCTL_API=3 etcdctl version’etcdctl version: 3.3.8
API version: 3.3

Next, let’s list the cluster members, just as we did above.

We can query the API with:

curl 35.204.136.231:2379/v2/members{“members”:[{“id”:”2e80f96756a54ca9″,”name”:”etcd-0″,”peerURLs”:[“http://etcd-0.etcd:2380″],”clientURLs”:[“http://etcd-0.etcd:2379”]},{“id”:”7fd61f3f79d97779″,”name”:”etcd-1″,”peerURLs”:[“http://etcd-1.etcd:2380″],”clientURLs”:[“http://etcd-1.etcd:2379”]},{“id”:”b429c86e3cd4e077″,”name”:”etcd-2″,”peerURLs”:[“http://etcd-2.etcd:2380″],”clientURLs”:[“http://etcd-2.etcd:2379”]}]}

With etcdctl using v2 of the API:

kubectl exec -it etcd-0 — etcdctl member list2e80f96756a54ca9: name=etcd-0 peerURLs=http://etcd-0.etcd:2380 clientURLs=http://etcd-0.etcd:2379 isLeader=true
7fd61f3f79d97779: name=etcd-1 peerURLs=http://etcd-1.etcd:2380 clientURLs=http://etcd-1.etcd:2379 isLeader=false
b429c86e3cd4e077: name=etcd-2 peerURLs=http://etcd-2.etcd:2380 clientURLs=http://etcd-2.etcd:2379 isLeader=false

With etcdctl using v3 of the API:

kubectl exec -it etcd-0 — /bin/sh -c ‘ETCDCTL_API=3 etcdctl member list –write-out=table’+——————+———+——–+————————-+————————-+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+——————+———+——–+————————-+————————-+
| 2e80f96756a54ca9 | started | etcd-0 | http://etcd-0.etcd:2380 | http://etcd-0.etcd:2379 |
| 7fd61f3f79d97779 | started | etcd-1 | http://etcd-1.etcd:2380 | http://etcd-1.etcd:2379 |
| b429c86e3cd4e077 | started | etcd-2 | http://etcd-2.etcd:2380 | http://etcd-2.etcd:2379 |
+——————+———+——–+————————-+————————-+

Setting and Retrieving Values in Etcd

The last example we will cover is creating a key and checking it’s value on all the 3 Pods in the Etcd cluster. Then we will kill the leader, etcd-0 in our scenario, and see how a new leader is elected. Finally, once the cluster has recovered, we will verify the value of our previously created key on all members. We will see that there is no data loss, and cluster simply goes on with a different leader.

We can verify that the cluster is initially healthy by typing:

kubectl exec -it etcd-0 — etcdctl cluster-healthmember 2e80f96756a54ca9 is healthy: got healthy result from http://etcd-0.etcd:2379
member 7fd61f3f79d97779 is healthy: got healthy result from http://etcd-1.etcd:2379
member b429c86e3cd4e077 is healthy: got healthy result from http://etcd-2.etcd:2379
cluster is healthy

Next, verify the current leader by typing the following. The last field indicates that etcd-0 is the leader in our cluster:

kubectl exec -it etcd-0 — etcdctl member list2e80f96756a54ca9: name=etcd-0 peerURLs=http://etcd-0.etcd:2380 clientURLs=http://etcd-0.etcd:2379 isLeader=true
7fd61f3f79d97779: name=etcd-1 peerURLs=http://etcd-1.etcd:2380 clientURLs=http://etcd-1.etcd:2379 isLeader=false
b429c86e3cd4e077: name=etcd-2 peerURLs=http://etcd-2.etcd:2380 clientURLs=http://etcd-2.etcd:2379 isLeader=false

Using the API, we will create a key called message and assign it a value. Remember to substitute the IP address you retrieved for your cluster in the command below:

curl http://35.204.136.231:2379/v2/keys/message -XPUT -d value=”Hello world”{“action”:”set”,”node”:{“key”:”/message”,”value”:”Hello world”,”modifiedIndex”:9,”createdIndex”:9}}

The key will have same value regardless the member we query. This helps us validate that the value has been replicated to the other nodes and committed to the log:

kubectl exec -it etcd-0 — etcdctl get message
kubectl exec -it etcd-1 — etcdctl get message
kubectl exec -it etcd-2 — etcdctl get messageHello world
Hello world
Hello world

Demonstrating High Availability and Recovery

Next, we can kill the Etcd cluster leader. This will let us see how a new leader is elected and how the cluster recovers from it’s degraded state. Delete the pod associated with the Etcd leader you discovered above:

kubectl delete pod etcd-0

Now, let’s check the cluster health:

kubectl exec -it etcd-2 — etcdctl cluster-healthfailed to check the health of member 2e80f96756a54ca9 on http://etcd-0.etcd:2379: Get http://etcd-0.etcd:2379/health: dial tcp: lookup etcd-0.etcd on 10.15.240.10:53: no such host
member 2e80f96756a54ca9 is unreachable: [http://etcd-0.etcd:2379] are all unreachable
member 7fd61f3f79d97779 is healthy: got healthy result from http://etcd-1.etcd:2379
member b429c86e3cd4e077 is healthy: got healthy result from http://etcd-2.etcd:2379
cluster is degraded
command terminated with exit code 5

The above message indicates that the cluster is in a degraded state due to the loss of its leader node.

Once Kubernetes responds to the deleted pod by spinning up a new instance, the Etcd cluster should recover:

kubectl exec -it etcd-2 — etcdctl cluster-healthmember 2e80f96756a54ca9 is healthy: got healthy result from http://etcd-0.etcd:2379
member 7fd61f3f79d97779 is healthy: got healthy result from http://etcd-1.etcd:2379
member b429c86e3cd4e077 is healthy: got healthy result from http://etcd-2.etcd:2379
cluster is healthy

We can see that a new leader has been elected by typing:

kubectl exec -it etcd-2 — etcdctl member list2e80f96756a54ca9: name=etcd-0 peerURLs=http://etcd-0.etcd:2380 clientURLs=http://etcd-0.etcd:2379 isLeader=false
7fd61f3f79d97779: name=etcd-1 peerURLs=http://etcd-1.etcd:2380 clientURLs=http://etcd-1.etcd:2379 isLeader=true
b429c86e3cd4e077: name=etcd-2 peerURLs=http://etcd-2.etcd:2380 clientURLs=http://etcd-2.etcd:2379 isLeader=false

In our case, the etcd-1 node was elected as leader.

If we will check the value for the message key again, we can verify that there was no data loss:

kubectl exec -it etcd-0 — etcdctl get message
kubectl exec -it etcd-1 — etcdctl get message
kubectl exec -it etcd-2 — etcdctl get messageHello world
Hello world
Hello world

Conclusion

Etcd is a very powerful, highly available, and reliable distributed key-value store designed for specific use cases. Common examples are storing database connection details, cache settings, feature flags, and more. It was designed to be sequentially consistent, so that every event is stored in the same order throughout the cluster.

We saw how to get an etcd cluster up and running in Kubernetes with the help of Rancher. Afterwards, we were able to play with few basic Etcd commands. In order to learn more about the project, how keys can be organized, how to set TTLs for keys, or how to back up all the data, the official Etcd repo is a great starting point.

Calin Rus

Source