Art2Dec SoftLab

May 15, 2018

by Adrian Mouat

Kubernetes is a fantastic tool for building large containerised software systems in a manner that is both resilient and scalable. But the architecture and design of Kubernetes has evolved over time, and there are some areas that could do with tweaking or rethinking. This post digs into some issues related to how image tags are handled in Kubernetes and how they are treated differently in plain Docker.

First, let’s take a look at one of the first issues that people can face. I have the following demo video that shows a developer trying to deploy a new version of a Rust webapp to a Kubernetes cluster:

The video starts by building and pushing a version of the pizza webapp that serves quattro formaggi pizza. The developer then tries to deploy the webapp to Kubernetes and ends up in a confusing situation where it’s running, yet not serving the kind of pizza we expect. We can see what’s going on by doing some more inspection:

It turns out 3 different versions of our webapp are running inside a single Kubernetes Replica Set, as evidenced by the 3 different digests.

The reason this can happen comes down to the Kubernetes imagePullPolicy. The default is IfNotPresent, which means nodes will use an existing image rather than pull a new one. In our demo, each node happened to have a different version of the image left over from previous runs. Personally, I’m disappointed that this is the default behaviour, as it’s unexpected and confusing for new users. I understand that it evolved over-time and in some cases it is the wanted behaviour, but we should be able to change this default for the sake of usability.

The simplest mitigation for the problem is to set the pull policy to AlwaysPull:

This can even be made the default for all deployments by using the AlwaysPullImages Admission Controller.

However, there is still a rather large hole in this solution. Imagine a new deployment occurs concurrently with the image being updated in the registry. It’s quite likely that different nodes will pull different versions of the image even with AlwaysPull set. We can see a better solution in the way Docker Swarm Mode works – the Swarm Mode control plane will resolve images to a digest prior to asking nodes to run the image, that way all containers are guaranteed to run the same version of the image. There’s no reason we can’t do something similar in Kubernetes using an Admission Controller, and my understanding is that Docker EE does exactly this when running Kubernetes pods. I haven’t been able to find an existing open source Admission Controller that does this, but we’re working on one at CS and I’ll update this post when I have something.

Going a little deeper, the real reason behind this trouble is a difference between the way image tags are viewed in Kubernetes and Docker. Kubernetes assumes image tags are immutable. That is to say, if I call my image amouat/pizza:today, Kubernetes assumes it will only ever refer to that unique image; the tag won’t get reused for new versions in the future. This may sound like a pain at first, but immutable images solve a lot of problems; any potential confusion about which version of an image a tag refers to simply evaporates. It does require using an appropriate naming convention; in the case of amouat/pizza:today a better version would be to use the date e.g. amouat/pizza:2018-05-12, in other cases SemVer or git hashes can work well.

In contrast, Docker treats tags as mutable and even trains us to think this way. For example, when building an application that runs in a container, I will repeatedly run docker build -t test . or similar, constantly reusing the tag so that the rest of my workflow doesn’t need to change. Also, the official images on the Docker Hub typically have tags for major and minor versions of images that get updated over time e.g. redis:3 is the same image as redis:3.2.11 at the time of writing, but in the past would have pointed at redis:3.2.10 etc.

This split is a real practical problem faced by new users. Solving it seems reasonably straightforward; can’t we have both immutable and mutable tags? This would require support from registries and (preferably) the Docker client, but the advantages seem worth it. I am hopeful that the new OCI distribution specification will tackle this issue.

To sum up; be careful when deploying images to Kubernetes and make sure you understand how images actually get deployed to your cluster. And if you happen
to have any influence on the direction of Kubernetes or the Distribution spec; can we please try to make the world a bit nicer?

Because of these and some other issues, Container Solutions have started work on Trow; an image management solution for Kubernetes that includes a registry component that runs inside the cluster. Trow will support immutable tags and include admission controllers that pin images to digests. If this sounds useful to you, please head over to trow.io and let us know!

Further Viewing

This blog was based on my talk Establishing Image Provenance and Security in Kubernetes given at KubeCon EU 2018, which goes deeper into some of the issues surrounding images.

Looking for a new challenge? We’re hiring!

Source

A Detailed Overview of Rancher’s Architecture

This newly-updated, in-depth guidebook provides a detailed overview of the features and functionality of the new Rancher: an open-source enterprise Kubernetes platform.

Get the eBook

Kubernetes 1.12 will be released this week on Thursday, September 27, 2018. Version 1.12 ships just three months after Kubernetes 1.11 and marks the third major release of this year. The short cycle is inline with the quarterly release cycle the project has followed since it’s GA in 2015.

Kubernetes releases 2018

| Kubernetes Release | Date |
|——————–|——————–|
| 1.10 | March 26, 2018 |
| 1.11 | June 27, 2018 |
| 1.12 | September 27, 2018 |

Whether you are a developer using Kubernetes or an admin operating clusters, it’s worth getting an idea about the new features and fixes that you can expect in Kubernetes 1.12.

A total of 38 features are included in this milestone. Let’s have a look at some of the highlights.

Kubelet certificate rotation

Kubelet certificate rotation was promoted to beta status. This functionality allows for automated renewal of key and a certificate for the kubelet API server as the current certificate approaches expiration. Until the official 1.12 docs have been published, you can read the beta documentation on this feature here.

Network Policies: CIDR selector and egress rules

Two formerly beta features have now reached stable status: One of them is the ipBlock selector, which allows specifying ingress/egress rules based on network addresses in CIDR notation. The second one adds support for filtering the traffic that is leaving the pods by specifying egress rules. The below example illustrates the use of both features:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: app
policyTypes:
– Egress
egress:
– to:
– ipBlock:
cidr: 10.0.0.0/24
(…)

As previoulsy beta features, both egress and ipBlock are already described in the official network policies documentation.

Mount namespace propagation

Mount namespace propagation, i.e. the ability to mount a volume rshared so that any mounts from inside the container are reflected in the root (= host) mount namespace, has been promoted to stable. You can read more about this feature in the Kubernetes volumes docs.

Taint nodes by condition

This feature introduced in 1.8 as early alpha has been promoted to beta. Enabling it’s featureflag causes the node controller to create taints based on node conditions and the scheduler to filter nodes based on taints instead of conditions. The official documentation is available here.

Horizontal pod autoscaler with custom metrics

While support for custom metrics in HPA continuous to be in beta status, version 1.12 adds various enhancements like the the ability to select metrics based on the labels available in your monitoring pipeline. If you are interested in autoscaling pods based on application-level metrics provided by monitoring systems such as Prometheus, Sysdig or Datadog, I recommend to checkout the design proposal for external metrics in HPA.

RuntimeClass

RuntimeClass is a new cluster-scoped resource “that surfaces container runtime properties to the control plane”. In other words: This early alpha feature will enable users to select and configure (per pod) a specific container runtime (such as Docker, Rkt or Virtlet) by providing the runtimeClass field in the PodSpec. You can read more about it in these docs.

Resource Quota by priority

Resource quotas allow administrators to limit the resource consumption in namespaces. This is especially practical in scenarios where the available compute and storage resources in a cluster are shared by several tenants (users, teams). The beta feature Resource quota by priority allows admins to fine-tune resource allocation within the namespace by scoping quotas based on the PriorityClass of pods. You can find more details here.

Volume Snapshots

One of the most exciting new 1.12 features for storage is the early alpha implementation of persistent volume snapshots. This feature allows users to create and restore snapshots at a particular point in time backed by any CSI storage provider. As part of this implementation three new API resources have been added:
VolumeSnapshotClass defines how snapshots for existing volumes are provisioned. VolumeSnapshotContent represents existing snapshots and VolumeSnapshot allows users to request a new snapshot of a persistent volume like so:

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
name: new-snapshot-test
spec:
snapshotClassName: csi-hostpath-snapclass
source:
name: pvc-test
kind: PersistentVolumeClaim

For the nitty gritty details take a look at the 1.12 documentation branch on Github.

Topology aware dynamic provisioning

Another storage related feature, topology aware dynamic provisioning, was introduced in v1.11 and has been promoted to beta in 1.12. It addresses some limitations with dynamic provisioning of volumes in clusters spread across multiple zones where single-zone storage backends are not globally accessible from all nodes.

Enhancements for Azure Cloud provider

These two improvements regarding running Kubernetes in Azure are shipping in 1.12:

Cluster autoscaler support

The cluster autoscaler support for Azure was promoted to stable. This will allow for automatic scaling of the number of Azure nodes in Kubernetes clusters based on global resource usage.

Azure availability zone support

Kubernetes v1.12 adds alpha support for Azure availability zones (AZ). Nodes in an availability zone will be added with label failure-domain.beta.kubernetes.io/zone=<region>-<AZ> , and topology-aware provisioning is added for Azure managed disks storage class.

Anything else?

Kubernetes 1.12 contains many bug fixes and improvements of internal components, clearly focusing on stabilising the core, maturing existing beta features and improving the release velocity by adding more automated tests to the projects CI pipeline. A noteworthy example for the latter is the addition of CI e2e conformance tests for arm, arm64, ppc64, s390x and windows platforms to the projects test harness.

For a full list of changes in 1.12 see the release notes.

Rancher will support Kubernetes 1.12 on hosted clusters as soon as it becomes available on the particular provider. For RKE provisioned clusters it will be supported starting with Rancher 2.2.

Source

We recently open sourced the Operator Framework and today we’re happy to share the next milestone: Operator Metering. Operator Metering is designed to help you gain more knowledge about the usage and costs to run and manage Kubernetes native applications (Operators). This joins the other Operator Framework components – SDK and Lifecycle Management – that are a part of the Operator Framework family, an open source toolkit designed to manage Operators in a more effective, automated, and scalable way.

Now available as an open source version, Operator Metering enables usage reporting for Operators that provide specialized lifecycle services for applications running on Kubernetes. The project is designed to tie into the cluster’s CPU and memory reporting, as well as calculate Infrastructure-as-a-Service (IaaS) costs and customized metrics such as licensing. Examples of such services could be metering products running in Kubernetes for use in on-demand billing or to derive DevOps insights, such as tracking heal operations across Gluster storage clusters.

We believe Operator Metering will enable Kubernetes operations teams to be able to associate the cost of their underlying infrastructure to the applications running on their Kubernetes clusters in a consistent way, across any environment, be it on public cloud infrastructure or on premises.

Metering is an important tool for organizations that use capacity from a Kubernetes cluster to run their own services, as well as for IT departments that manage these clusters. In the past, many departments tended to overestimate their resource needs. This could easily result in wasted capacity and wasted capital.

Today, management teams want to understand more concretely where budget is spent and by whom, and for which service. Metering provides that information, providing an understanding of how much it costs to run specific services, while also providing usage information that can lead to improved budgeting and capacity planning. With this information, IT can also internally bill departments to reflect the costs directly associated with their actual infrastructure usage, driving accountability for service costs. This helps to eliminate some of the more manual IT “plumbing” work in tallying costs and usage by hand or managing spreadsheets – instead, by using metering, IT teams can free up their time to tackle bigger problems and even drive business-wide innovation.

Here are some examples of how metering could be applied in the real world:

• Cloud budgeting: Teams can gain insight into how cloud resources are being used, especially in autoscaled clusters or hybrid cloud deployments.
• Cloud billing: Resource usage can be tracked by billing codes or labels that reflect your internal hierarchy.
• Telemetry/aggregation: Service usage and metrics can be viewed across many namespaces or teams, such as a Postgres Operator running hundreds of databases.

We are extremely excited to share Operator Metering with the community as part of our commitment to making Kubernetes more extensible and widely usable. Operator Metering is currently in alpha, so we are looking forward to your feedback and comments on bugs, tweaks, and future updates. Our current plan is to incorporate feedback, stabilize the code base, and fix any critical problems before moving on to adding more features.

Learn more about the Operator Metering project at https://github.com/operator-framework/operator-metering.

Source

In this tutorial, we will use Azure Kubernetes Service (AKS) to deploy an application in the form of a Helm package. We will use Codefresh for CI/CD and also get a brief look at Draft for creating Helm charts.

What you will learn

• How to create a Kubernetes cluster in Azure
• How to install Helm -the Kubernetes package manager – on the cluster
• How to add/view/edit the cluster in Codefresh
• How to use Draft to create a starting Helm chart for the application
• How to push Helm charts with Codefresh
• How to deploy Helm charts with Codefresh

Prerequisites

If you want to follow along as you read the tutorial, you will need:

• A free Codefresh Account
• An Azure subscription (trial should work fine)
• A Github repository with your application (Codefresh also supports Gitlab and Bitbucket)

The programming language of your application does not really matter, as long as it comes with a Dockerfile. Even if you don’t have a Dockerfile at hand we will see how you can create one easily with Draft.

Creating a Kubernetes cluster in Azure

The documentation for AKS explains how you can create a cluster using the command line or the GUI of Azure Portal. Both ways are valid but since we are also going to use Helm from the command line, it makes sense to dive into the terminal right away. For the sake of convenience, however, we will use the terminal offered in the Azure portal (Azure cloud shell) which offers some nice features such as automatic authentication as well as preinstalled kubectl and helm executables.

Login into Azure portal with your account and launch the cloud shell from the top right of the GUI.

Azure Cloud shell

After a few moments, you should have access to the command line right from your browser!

Before you create your first Kubernetes cluster in Azure you need to understand the concept of Resource Groups. Everything in Azure is created under a Resource Group. If you don’t have a Resource Group yet you need to create one first following the documentation.

A Resource Group was already available for the purposes of this tutorial and therefore creating an Azure cluster was a single line in the terminal:

az aks create –resource-group kostis-rg –name myDemoAKSCluster –node-count 1 –generate-ssh-keys

Adjust the node count according to your preference but be aware that this affects billing. In this tutorial we’ll use one, but adding more won’t change how you interact with Kubernetes.

After a few minutes, the cluster will be created and you can even visit it from the GUI and see its details.

Azure Kubernetes cluster

The next step is to setup kubectl access to the cluster. Kubectl is already available in the command line but you still need to match its config with the cluster we just created. The command that does this uses the Azure CLI:

az aks get-credentials –resource-group kostis-rg –name myDemoAKSCluster

Once that is done, we have full access to the cluster with kubectl. We can now enter any of the familiar commands and see the resources of the cluster (pods, deployments, services etc).

Connecting the Azure Kubernetes cluster to Codefresh

Codefresh has native support for Azure Kubernetes clusters which includes a nice dashboard that can be used both for viewing existing resources as well as modifying them and built-in deploy steps through pipelines.

To connect the Azure Kubernetes cluster, log in to your Codefresh account and click Kubernetes on the left sidebar. Then click “Add Cluster”

Codefresh integrations

From the drop-down menu select Microsoft Azure and enter your cluster credentials (url, certificate, and token). To obtain these values, Codefresh documentation explains the kubectl commands that you need to execute. You can execute them in the Azure cloud shell directly, as we have already seen in the previous section that kubectl is configured to point at the newly created cluster.

Once all the details are filled in, test your settings and save the cluster configuration. If everything goes well, you will be able to click on Kubernetes on the left sidebar and view all the cluster details within Codefresh.

Kubernetes Dashboard

The last step is to create a registry pull secret so that the cluster can pull Docker images from the Codefresh Registry. Each Codefresh account comes with a built-in free Docker registry that can be used for image storage. Of course, you can also use your own external registry instead. (Azure offers a Docker Registry as well.)
First, you need to create a token for the Codefresh registry, then execute the following kubectl command in the Azure cloud shell:

kubectl create secret docker-registry my-codefresh-registry –docker-server=r.cfcf.io –docker-username=<codefresh_username> –docker-password=<codefresh_token> –docker-email=<email>

Of course, you need to enter your own values here. Notice also that this secret is for the default namespace, so if you want to use another namespace add it as a parameter on the command above.

Tip – If you don’t want to run the command line to create the secret, you can goto the Kubernetes dashboard and click “Add Service” then click on “Image Pull Secret” to add a secret for any of your connected repositories.

That’s it! The Kubernetes cluster is now ready to deploy Docker applications. At this point, we could use plain Kubernetes manifests to deploy our application. In this tutorial, however, we will see how we can use Helm.

Installing Helm on the Azure Kubernetes cluster

Helm is a package manager for Kubernetes. It offers several extra features on top of vanilla Kubernetes deployments, some of which are:

• The ability to group multiple Kubernetes manifests together and treat them as a single entity (for deployments, rollbacks, and storage).
• Built-in templating for Kubernetes manifests, putting an end to custom template systems that you might use for replacing things such as the Docker tag inside a manifest.
• The ability to package the collection of manifests in Charts which contain the templates as well as default values.
• The ability to create catalogs of applications (Helm repositories) that function similar to traditional package repositories (think deb, rpm, nuget,brew, npm etc).

Helm comes in two parts, the client (Helm) and the server component called Tiller. The Azure cloud shell comes with the client already preinstalled. The client can be used to install the server part as well. Therefore, just execute in the Azure cloud shell the following:

The helm executable uses the same configuration as kubectl. Because kubectl was already configured to point to our Kubernetes cluster in the beginning of this tutorial, the helm init command works out of the box without any extra configuration.

Codefresh also comes with a Helm dashboard that you can now visit. It should be empty because we haven’t deployed anything yet.

Empty Helm Dashboard

Creating a starter Helm chart using Draft

Creating a Helm template and associated default values is a well-documented process. Maybe you are already familiar with Helm templates and your application already has its own chart. In that case, skip this section.

In this tutorial, however, we will select the easy way out and auto-generate a chart that will serve as a starting point for the Helm package. We will use Draft for this purpose.

Draft is a tool geared towards local Kubernetes deployments (i.e. before you actually commit your code) such as Minikube. It is developed by the same team that develops Helm so obviously, they play along very well together.

We will explore the deployment capabilities of Draft in a future article. For the present situation, we will exploit its capability to create a Helm chart for any of the supported languages.

Download Draft for your OS and then execute it at the root directory of your project.

This will create a default Helm chart under the charts folder. Notice that Draft will even create a Dockerfile for your application if it doesn’t contain one already.

Edit the values.yaml file to your preference. At the very least you should change:

• The ports for the Kubernetes service
• The type of Service from ClusterIP to LoadBalancer

You can also add the Docker repository/tag values, although this is not strictly necessary as Codefresh will do this for us during the actual deployment.

Once you are finished, commit the chart folder to your Git repository.

Preparing a Helm Chart with Codefresh

Apart from a built-in Docker registry, all Codefresh accounts also include an integrated Helm Repository. A Helm Repository can be used to catalog your own Helm Charts in addition to the public charts already included with Codefresh from KubeApps.

To push the Helm Chart, create a new Codefresh Pipeline with the following yml build file:

version: ‘1.0’ steps: BuildingDockerImage: title: Building Docker Image type: build image_name: kostis-codefresh/python-flask-sampleapp working_directory: ./ tag: ${} dockerfile: Dockerfile StoreChart: title: Storing Helm Chart image: ‘codefresh/cfstep-helm:2.9.1’ environment: – ACTION=push – CHART_REF=charts/python

This yml file contains two steps. The first step, BuildingDockerImage creates a Docker image from a Dockerfile that exists at the root of the project.

The second step uses the premade Codefresh Helm Docker image and pushes the Helm chart located at charts/python in the root folder. The names of steps are arbitrary.

In order to make this pipeline have access to the internal Helm Repository you also need to import its configuration. Select “import from shared configuration” in your Codefresh pipeline and choose the CF_HELM_DEFAULT config.

Codefresh Helm configuration

After you execute the pipeline you will see the chart getting pushed in the Codefresh Helm Repository. To browse this repository select Kubernetes->Helm Releases from the left sidebar and expand the CF_HELM_DEFAULT Repo.

Codefresh Helm repository

You can also manually deploy this chart to your Azure cluster by clicking the Install icon as shown on the right-hand side. Even though this is a convenient way to deploy applications via the Codefresh GUI, it is best if we fully automate this process as you will see in the next section.

Automatically deploying a Helm Chart with Codefresh

In the previous section, we saw how you can store the Helm chart in the integrated Codefresh Helm repository. The final step is to actually deploy the application and get a full CI/CD pipeline. Here are the respective pipeline steps:

DeployChart: image: ‘codefresh/cfstep-helm:2.9.1’ environment: – CHART_REF=charts/python – RELEASE_NAME=mypython-chart-prod – KUBE_CONTEXT=myDemoAKSCluster – VALUE_image_pullPolicy=Always – VALUE_image_tag=${}

This is a Helm step as before, but with the following parameters:

• We define again which chart we will deploy
• We select a release name. (This is optional and if not provided, Helm with autogenerate one for us using a funky naming pattern.)
• The Kube context selects the Azure Cluster as the target of the deployment (Codefresh supports linking multiple Kubernetes clusters from multiple cloud providers).

The last two lines show the power of Helm. Instead of having to use custom replacement scripts or external templating system that are needed for plain Kubernetes manifests, Helm has built-in templating. The two lines override the default values for the chart.

We change the pull policy to “always” (by default it was IfNotPresent) and also we make sure that we use the Docker image from the branch that we are building from. This is just a suggestion and you can use any other combination of tags on your Docker images. You can find all built-in Codefresh variables in the documentation.

Once the application is deployed you should be able to access it from its external endpoint. If you visit again the Codefresh Helm dashboard you should now see the release along with its templates and values.

Helm Release

Congratulations! You have now deployed a Helm chart in the Azure Kubernetes cluster and the application is ready to be used.

You can also see the release from the Azure Cloud shell if you execute

This will print out the release details in the terminal.

Rolling back deployments without re-running the pipelines

Another big advantage of Helm is the way it gives you easy rollbacks for free. If you make some commits in your project, Helm will keep the same deployment and add new revisions on it.

You can easily rollback to any previous version without actually re-running the pipeline.

Helm Rollback

The server part of Helm keeps a history of all releases and knows the exact contents of each respective Helm package.

Codefresh allows you to do this right from the GUI. Select the History tab in the Helm release and from the list of revisions you can select any of them as the rollback target. Notice that rolling back will actually create a new revision. So you can go backward and forward in time to any revision possible.

Conclusion

We have reached the end of this tutorial. You have now seen:

• How to create an Azure Kubernetes cluster
• How to install Helm
• How to give access of the Codefresh Registry to the Azure cluster
• How to connect the Azure cluster in Codefresh
• How to inspect Kubernetes resources and Helm chart/releases from the Codefresh GUI
• How to create a Helm chart using Draft and how to edit default values
• How to use Codefresh to store the chart in the integrated Helm repository
• How to use Codefresh to deploy applications to the Azure cluster from a Helm chart
• How to easily rollback Helm releases

Now you know how to create a complete CI/CD pipeline that deploys applications to Kubernetes using Helm packaging.

New to Codefresh? Create Your Free Account Today!

Source

 

Support for Azure VMSS, Cluster-Autoscaler and User Assigned Identity

Author: Krishnakumar R (KK) (Microsoft), Pengfei Ni (Microsoft)

Introduction

With Kubernetes v1.12, Azure virtual machine scale sets (VMSS) and cluster-autoscaler have reached their General Availability (GA) and User Assigned Identity is available as a preview feature.

Azure VMSS allow you to create and manage identical, load balanced VMs that automatically increase or decrease based on demand or a set schedule. This enables you to easily manage and scale multiple VMs to provide high availability and application resiliency, ideal for large-scale applications like container workloads [1].

Cluster autoscaler allows you to adjust the size of the Kubernetes clusters based on the load conditions automatically.

Another exciting feature which v1.12 brings to the table is the ability to use User Assigned Identities with Kubernetes clusters [12].

In this article, we will do a brief overview of VMSS, cluster autoscaler and user assigned identity features on Azure.

VMSS

Azure’s Virtual Machine Scale sets (VMSS) feature offers users an ability to automatically create VMs from a single central configuration, provide load balancing via L4 and L7 load balancing, provide a path to use availability zones for high availability, provides large-scale VM instances et. al.

VMSS consists of a group of virtual machines, which are identical and can be managed and configured at a group level. More details of this feature in Azure itself can be found at the following link [1].

With Kubernetes v1.12 customers can create k8s cluster out of VMSS instances and utilize VMSS features.

Cluster components on Azure

Generally, standalone Kubernetes cluster in Azure consists of the following parts

• Compute – the VM itself and its properties.
• Networking – this includes the IPs and load balancers.
• Storage – the disks which are associated with the VMs.

Compute

Compute in cloud k8s cluster consists of the VMs. These VMs are created by provisioning tools such as acs-engine or AKS (in case of managed service). Eventually, they run various system daemons such as kubelet, kube-api server etc. either as a process (in some versions) or as a docker container.

Networking

In Azure Kubernetes cluster various networking components are brought together to provide features required for users. Typically they consist of the network interfaces, network security groups, public IP resource, VNET (virtual networks), load balancers etc.

Storage

Kubernetes clusters are built on top of disks created in Azure. In a typical configuration, we have managed disks which are used to hold the regular OS images and a separate disk is used for etcd.

Cloud provider components

Kubernetes cloud provider interface provides interactions with clouds for managing cloud-specific resources, e.g. public IPs and routes. A good overview of these components is given in [2]. In case of Azure Kubernetes cluster, the Kubernetes interactions go through the Azure cloud provider layer and contact the various services running in the cloud.

The cloud provider implementation of K8s can be largely divided into the following component interfaces which we need to implement:

1. Load Balancer
2. Instances
3. Zones
4. Routes

In addition to the above interfaces, the storage services from the cloud provider is linked via the volume plugin layer.

Azure cloud provider implementation and VMSS

In the Azure cloud provider, for every type of cluster we implement, there is a VMType option which we specify. In case of VMSS, the VM type is “vmss”. The provisioning software (acs-engine, in future AKS etc.) would setup these values in /etc/kubernetes/azure.json file. Based on this type, various implementations would get instantiated [3]

The load balancer interface provides access to the underlying cloud provider load balancer service. The information about the load balancers and the control operations on them are required for Kubernetes to handle the services which gets hosted on the Kubernetes cluster. For VMSS support the changes ensure that the VMSS instances are part of the load balancer pool as required.

The instances interfaces help the cloud controller to get various details about a node from the cloud provider layer. For example, the details of a node like the IP address, the instance id etc, is obtained by the controller by means of the instances interfaces which the cloud provider layer registers with it. In case of VMSS support, we talk to VMSS service to gather information regarding the instances.

The zones interfaces help the cloud controller to get zone information for each node. Scheduler could spread pods to different availability zones with such information. It is also required for supporting topology aware dynamic provisioning features, e.g. AzureDisk. Each VMSS instances will be labeled with its current zone and region.

The routes interfaces help the cloud controller to setup advanced routes for Pod network. For example, a route with prefix node’s podCIDR and next hop node’s internal IP will be set for each node. In case of VMSS support, the next hops are VMSS virtual machines’ internal IP address.

The Azure volume plugin interfaces have been modified for VMSS to work properly. For example, the attach/detach to the AzureDisk have been modified to perform these operations at VMSS instance level.

Setting up a VMSS cluster on Azure

The following link [4] provides an example of acs-engine to create a Kubernetes cluster.

acs-engine deploy –subscription-id <subscription id>
–dns-prefix <dns> –location <location>
–api-model examples/kubernetes.json

API model file provides various configurations which acs-engine uses to create a cluster. The API model here [5] gives a good starting configuration to setup the VMSS cluster.

Once a VMSS cluster is created, here are some of the steps you can run to understand more about the cluster setup. Here is the output of kubectl get nodes from a cluster created using the above command:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-agentpool1-92998111-vmss000000 Ready agent 1h v1.12.0-rc.2
k8s-agentpool1-92998111-vmss000001 Ready agent 1h v1.12.0-rc.2
k8s-master-92998111-0 Ready master 1h v1.12.0-rc.2

This cluster consists of two worker nodes and one master. Now how do we check which node is which in Azure parlance? In VMSS listing, we can see a single VMSS:

$ az vmss list -o table -g k8sblogkk1
Name ResourceGroup Location Zones Capacity Overprovision UpgradePolicy
—————————- ————— ———- ——- ———- ————— —————
k8s-agentpool1-92998111-vmss k8sblogkk1 westus2 2 False Manual

The nodes which we see as agents (in the kubectl get nodes command) are part of this vmss. We can use the following command to list the instances which are part of the VM scale set:

$ az vmss list-instances -g k8sblogkk1 -n k8s-agentpool1-92998111-vmss -o table
InstanceId LatestModelApplied Location Name ProvisioningState ResourceGroup VmId
———— ——————– ———- —————————— ——————- ————— ————————————
0 True westus2 k8s-agentpool1-92998111-vmss_0 Succeeded K8SBLOGKK1 21c57d6c-9c8f-4a62-970f-63ed0fcba53f
1 True westus2 k8s-agentpool1-92998111-vmss_1 Succeeded K8SBLOGKK1 840743b9-0076-4a2e-920e-5ba9da296665

The node name does not match the name in the vm scale set, but if we run the following command to list the providerID we can find the matching node which resembles the instance name:

$ kubectl describe nodes k8s-agentpool1-92998111-vmss000000| grep ProviderID
ProviderID: azure:///subscriptions/<subscription id>/resourceGroups/k8sblogkk1/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-92998111-vmss/virtualMachines/0

Current Status and Future

Currently the following is supported:

1. VMSS master nodes and worker nodes
2. VMSS on worker nodes and Availability set on master nodes combination.
3. Per vm disk attach
4. Azure Disk & Azure File support
5. Availability zones (Alpha)

In future there will be support for the following:

1. AKS with VMSS support
2. Per VM instance public IP

Cluster Autoscaler

A Kubernetes cluster consists of nodes. These nodes can be virtual machines, bare metal servers or could be even virtual node (virtual kubelet). To avoid getting lost in permutations and combinations of Kubernetes ecosystem ;-), let’s consider that the cluster we are discussing consists of virtual machines, which are hosted in a cloud (eg: Azure, Google or AWS). What this effectively means is that you have access to virtual machines which run Kubernetes agents and a master node which runs k8s services like API server. A detailed version of k8s architecture can be found here [11].

The number of nodes which are required on a cluster depends on the workload on the cluster. When the load goes up there is a need to increase the nodes and when it subsides, there is a need to reduce the nodes and clean up the resources which are no longer in use. One way this can be taken care of is to manually scale up the nodes which are part of the Kubernetes cluster and manually scale down when the demand reduces. But shouldn’t this be done automatically ? Answer to this question is the Cluster Autoscaler (CA).

The cluster autoscaler itself runs as a pod within the kubernetes cluster. The following figure illustrates the high level view of the setup with respect to the k8s cluster:

Since Cluster Autoscaler is a pod within the k8s cluster, it can use the in-cluster config and the Kubernetes go client [10] to contact the API server.

Internals

The API server is the central service which manages the state of the k8s cluster utilizing a backing store (an etcd database), runs on the management node or runs within the cloud (in case of managed service such as AKS). For any component within the Kubernetes cluster to figure out the state of the cluster, like for example the nodes registered in the cluster, contacting the API server is the way to go.

In order to simplify our discussion let’s divide the CA functionality into 3 parts as given below:

The main portion of the CA is a control loop which keeps running at every scan interval. This loop is responsible for updating the autoscaler metrics and health probes. Before this loop is entered auto scaler performs various operations such as claiming the leader state after performing a Kubernetes leader election. The main loop initializes static autoscaler component. This component initializes the underlying cloud provider based on the parameters passed onto the CA.

Various operations performed by the CA to manage the state of the cluster is passed onto the cloud provider component. Some examples like – increase target size, decrease target size etc, results in the cloud provider component talking to the cloud services internally and performing operations such as adding a node or deleting a node. These operations are performed on group of nodes in the cluster. The static autoscaler also keeps tab on the state of the system by querying the API server – operations such as list pods and list nodes are used to get hold of such information.

The decision to make a scale up is based on pods which remain unscheduled and a variety of checks and balances. The nodes which are free to be scaled down are deleted from the cluster and deleted from the cloud itself. The cluster autoscaler applies checks and balances before scaling up and scaling down – for example the nodes which have been recently added are given special consideration. During the deletion the nodes are drained to ensure that no disruption happens to the running pods.

Setting up CA on Azure:

Cluster Autoscaler is available as an add-on with acs-engine. The following link [15] has an example configuration file used to deploy autoscaler with acs-engine. The following link [8] provides details on manual step by step way to do the same.

In acs-engine case we use the the regular command line to deploy:

acs-engine deploy –subscription-id <subscription id>
–dns-prefix <dns> –location <location>
–api-model examples/kubernetes.json

The main difference are the following lines in the config file at [15] makes sure that CA is deployed as an addon:

“addons”: [
{
“name”: “cluster-autoscaler”,
“enabled”: true,
“config”: {
“minNodes”: “1”,
“maxNodes”: “5”
}
}
]

The config section in the json above can be used to provide the configuration to the cluster autoscaler pod, eg: min and max nodes as above.

Once the setup completes we can see that the cluster-autoscaler pod is deployed in the system namespace:

$kubectl get pods -n kube-system | grep autoscaler
cluster-autoscaler-7bdc74d54c-qvbjs 1/1 Running 1 6m

Here is the output from the CA configmap and events from a sample cluster:

$kubectl -n kube-system describe configmap cluster-autoscaler-status
Name: cluster-autoscaler-status
Namespace: kube-system
Labels: <none>
Annotations: cluster-autoscaler.kubernetes.io/last-updated=2018-10-02 01:21:17.850010508 +0000 UTC

Data
====
status:
—-
Cluster-autoscaler status at 2018-10-02 01:21:17.850010508 +0000 UTC:
Cluster-wide:
Health: Healthy (ready=3 unready=0 notStarted=0 longNotStarted=0 registered=3 longUnregistered=0)
LastProbeTime: 2018-10-02 01:21:17.772229859 +0000 UTC m=+3161.412682204
LastTransitionTime: 2018-10-02 00:28:49.944222739 +0000 UTC m=+13.584675084
ScaleUp: NoActivity (ready=3 registered=3)
LastProbeTime: 2018-10-02 01:21:17.772229859 +0000 UTC m=+3161.412682204
LastTransitionTime: 2018-10-02 00:28:49.944222739 +0000 UTC m=+13.584675084
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2018-10-02 01:21:17.772229859 +0000 UTC m=+3161.412682204
LastTransitionTime: 2018-10-02 00:39:50.493307405 +0000 UTC m=+674.133759650

NodeGroups:
Name: k8s-agentpool1-92998111-vmss
Health: Healthy (ready=2 unready=0 notStarted=0 longNotStarted=0 registered=2 longUnregistered=0 cloudProviderTarget=2 (minSize=1, maxSize=5))
LastProbeTime: 2018-10-02 01:21:17.772229859 +0000 UTC m=+3161.412682204
LastTransitionTime: 2018-10-02 00:28:49.944222739 +0000 UTC m=+13.584675084
ScaleUp: NoActivity (ready=2 cloudProviderTarget=2)
LastProbeTime: 2018-10-02 01:21:17.772229859 +0000 UTC m=+3161.412682204
LastTransitionTime: 2018-10-02 00:28:49.944222739 +0000 UTC m=+13.584675084
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2018-10-02 01:21:17.772229859 +0000 UTC m=+3161.412682204
LastTransitionTime: 2018-10-02 00:39:50.493307405 +0000 UTC m=+674.133759650

Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal ScaleDownEmpty 42m cluster-autoscaler Scale-down: removing empty node k8s-agentpool1-92998111-vmss000002

As can be seen the events, the cluster autoscaler scaled down and deleted a node as there was no load on this cluster. The rest of the configmap in this case indicates that there are no further actions which the autoscaler is taking at this moment.

Current status and future:

Cluster Autoscaler currently supports four VM types: standard (VMAS), VMSS, ACS and AKS. In the future, Cluster Autoscaler will be integrated within AKS product, so that users can enable it by one-click.

User Assigned Identity

Inorder for the Kubernetes cluster components to securely talk to the cloud services, it needs to authenticate with the cloud provider. In Azure Kubernetes clusters, up until now this was done using two ways – Service Principals or Managed Identities. In case of service principal the credentials are stored within the cluster and there are password rotation and other challenges which user needs to incur to accommodate this model. Managed service identities takes out this burden from the user and manages the service instances directly [12].

There are two kinds of managed identities possible – one is system assigned and another is user assigned. In case of system assigned identity each vm in the Kubernetes cluster is assigned a managed identity during creation. This identity is used by various Kubernetes components needing access to Azure resources. Examples to these operations are getting/updating load balancer configuration, getting/updating vm information etc. With the system assigned managed identity, user has no control over the identity which is assigned to the underlying vm. The system automatically assigns it and this reduces the flexibility for the user.

With v1.12 we bring user assigned managed identity support for Kubernetes. With this support user does not have to manage any passwords but at the same time has the flexibility to manage the identity which is used by the cluster. For example if the user needs to allow access to a cluster for a specific storage account or a Azure key vault, the user assigned identity can be created in advance and key vault access provided.

Internals

To understand the internals, we will focus on a cluster created using acs-engine. This can be configured in other ways, but the basic interactions are of the same pattern.

The acs-engine sets up the cluster with the required configuration. The /etc/kubernetes/azure.json file provides a way for the cluster components (eg: kube-apiserver) to gather configuration on how to access the cloud resources. In a user managed identity cluster there is a value filled with the key as UserAssignedIdentityID. This value is filled with the client id of the user assigned identity created by acs-engine or provided by the user, however the case may be. The code which does the authentication for Kubernetes on azure can be found here [14]. This code uses Azure adal packages to get authenticated to access various resources in the cloud. In case of user assigned identity the following API call is made to get new token:

adal.NewServicePrincipalTokenFromMSIWithUserAssignedID(msiEndpoint,
env.ServiceManagementEndpoint,
config.UserAssignedIdentityID)

This calls hits either the instance metadata service or the vm extension [12] to gather the token which is then used to access various resources.

Setting up a cluster with user assigned identity

With the upstream support for user assigned identity in v1.12, it is now supported in the acs-engine to create a cluster with the user assigned identity. The json config files present here [13] can be used to create a cluster with user assigned identity. The same step used to create a vmss cluster can be used to create a cluster which has user assigned identity assigned.

acs-engine deploy –subscription-id <subscription id>
–dns-prefix <dns> –location <location>
–api-model examples/kubernetes-msi-userassigned/kube-vmss.json

The main config values here are the following:

“useManagedIdentity”: true
“userAssignedID”: “acsenginetestid”

The first one useManagedIdentity indicates to acs-engine that we are going to use the managed identity extension. This sets up the necessary packages and extensions required for the managed identities to work. The next one userAssignedID provides the information on the user identity which is to be used with the cluster.

Current status and future

Currently we support the user assigned identity creation with the cluster using deploy of the acs-engine. In future this will become part of AKS.

Get involved

For azure specific discussions – please checkout the Azure SIG page at [6] and come and join the #sig-azure slack channel for more.

For CA, please checkout the Autoscaler project here [7] and join the #sig-autoscaling Slack for more discussions.

For the acs-engine (the unmanaged variety) on Azure docs can be found here: [9]. More details about the managed service from Azure Kubernetes Service (AKS) here [5].

References

1) https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview

2) https://kubernetes.io/docs/concepts/architecture/cloud-controller/

3) https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_vmss.go

4) https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/deploy.md

5) https://docs.microsoft.com/en-us/azure/aks/

6) https://github.com/kubernetes/community/tree/master/sig-azure

7) https://github.com/kubernetes/autoscaler

8) https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/README.md

9) https://github.com/Azure/acs-engine

10) https://github.com/kubernetes/client-go

11) https://kubernetes.io/docs/concepts/architecture/

12) https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

13) https://github.com/Azure/acs-engine/tree/master/examples/kubernetes-msi-userassigned

14) https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/auth/azure_auth.go

15) https://github.com/Azure/acs-engine/tree/master/examples/addons/cluster-autoscaler

Source

31/Aug 2018

By Hannah Morris

Simon, our Head of Growth, details his experience as part of the growing commercial team at Jetstack.

What are your main duties as Head of Growth at Jetstack?

‘Head of Growth’ is a relatively new title that has been more recently adopted by fast growing tech companies. It can mean different things to different people but the role is usually focused on scaling a business, product or customers. In my case, at Jetstack, I lead the business development side of the organisation which includes our Sales & Go-to-Market, Marketing and PR functions. We’ve already achieved such a fantastic engineering reputation since being founded in 2015, and my main goal is to continue revenue growth by maturing the commercial arm of the business.

Team drinks at the Founder’s Arms

What’s your career history and what made you want to work for Jetstack?

I’ve always been a technologist. I loved building overclocked watercooled PCs as a teen (sinking hours into Call of Duty and FIFA), which led me to start my career in a DevOps team at IBM. As I progressed, I moved into more customer-facing technical sales roles at companies like Cloud Sherpas and Accenture, as I enjoyed solving complex customer business challenges with technology.

I’ve been very fortunate to meet with a number of CIO/CTOs in recent years, nearly all of whom were facing challenges in turning overused buzz-terms like ‘digital transformation’ into tangible projects. They were trying to identify work streams that struck a balance between delivering value to their organisation and helping them to compete against disruptive startups.

I was introduced to Jetstack via an ex-colleague and was immediately impressed by their reputation among the cloud vendors, customers and developers in the open source community. Although I’ve previously worked for some very big companies, I was witnessing a change in how customers (both startups and enterprises alike) are becoming less likely to work with uber-broad systems integrators. In fact, I found that they had a preference to work with smaller technology specialists, who focus on really deep subject matter expertise. For Jetstack, this is Kubernetes.

When I joined Jetstack, sales operations was almost greenfield – we were using Gitlab as our CRM! It’s been one of my most enjoyable roles yet to work with the founders and talented wider team to build a Go-to-Market strategy, sales process and value proposition, whilst looking after some of our strategic new business.

Video Editing at Kube Con 2018

What is day-to-day life like in your role?

My role is incredibly varied and that’s what I love the most. Yesterday, for example, I spent the morning on a company-wide call reporting our booked business, and forecasting what we expect to close in Q4. I also discussed the partner news from Google NEXT 2018 in San Francisco, which I’d attended a few weeks prior. That afternoon, I met with a customer to discuss the challenges around upskilling their team on Kubernetes, Day 2 operations and a future multi-cloud strategy. I finished up the day working with some colleagues on a revamped careers section on the Jetstack website!

I also occasionally have time to to mess about with some filming/editing…

In my opinion, Jetstack hits the mark on flexibility, autonomy and accountability. We still have a startup culture with a very friendly atmosphere, and there is a large amount of expertise on hand for anyone new to learn and grow personally, and professionally.

What is the most exciting part of your work at Jetstack?

Playing table tennis 5 times a day….?!

In all seriousness, the most exciting part is definitely the growth potential of the company in an industry that is exploding. In only six months at Jetstack, I’ve worked with both ‘born-in-the-cloud’ and enterprise companies alike. It’s been a very interesting process to work on often complex deals and see the fantastic achievements of both the engineers and the commercial team.

The Jetstack Team at Kube Con 2018

I tweet about work related topics @simonjohnparker on twitter, but make sure to follow Jestack’s accounts on instagram and twitter also. I’m always on the lookout for talented sales people as we grow – if you are interested, reach out to me at simon@jetstack.io.

Source

David Cheney, Steve Sloka, Alex Brand, and Ross Kukulinski

After several months of hard work behind the scenes, Heptio is proud to take the wraps off a brand new version of Heptio Contour, our Envoy based Ingress controller for Kubernetes.

An iceberg is 90% below the waterline

At its heart, Heptio Contour is a translator between Kubernetes’ API server and Envoy. This is its raison d’être. The big news for Heptio Contour 0.6 is that this translation layer has been entirely rewritten.

Previously, Heptio Contour would translate Kubernetes objects directly into their Envoy counterparts. In Heptio Contour 0.6, the Kubernetes objects that describe a web application deployment— ​hostnames, TLS certificates, routes, and backend services — ​are used to build a Directed Acyclic Graph (DAG).

(You can generate graphs like these from your Heptio Contour installation. See the troubleshooting documentation.)

The DAG abstracts away the specifics of the Kubernetes Ingress object and allows Heptio Contour to model the semantics of a web application deployment without being tied closely to either Kubernetes’ or Envoy’s data models.

For example, routes are treated as first class objects in the DAG, rather than fragments of YAML in the Ingress spec. Another example: because the canonical representation of your web service is defined by the contents of the DAG, adding support for multiple services connected to a single route becomes trivial. Finally, using the DAG allows Heptio Contour to support alternative ways of describing a web application deployment, like our headline 0.6 release feature, IngressRoute.

IngressRoute what?

Since it was added in Kubernetes 1.1, Ingress hasn’t seen much attention — ​it’s still in beta — ​but boy is it popular! There are close to a dozen Ingress controllers in use today trying to do their best with the unfinished Ingress object. Mostly this involves a cornucopia of annotations on the Ingress object to clarify, restrict, or augment the structure imposed by the Ingress object. In this respect, Heptio Contour is much the same as any other Ingress controller.

At the same time a number of web application deployment patterns like blue/green deployments, explicit load balancing strategies, and presenting more than one Kubernetes Service behind a single route, are difficult to achieve with Ingress as it stands today.

In collaboration with Actapio, a subsidiary of Yahoo Japan Corporation, we’ve added a new kind of layer seven ingress object to Heptio Contour. We call it IngressRoute.

We designed the IngressRoute CRD spec to do two things. The first is to provide a sensible home for configuration parameters that were previously crammed into annotations. The second is a mechanism to make it safer to share an ingress controller across multiple namespaces and teams in the same Kubernetes cluster. We do this using a process we call delegation.

Much like the way a subdomain is delegated from one domain name server to another, an IngressRoute may delegate control of some, or all, of the HTTP routing prefixes it controls to another IngressRoute.

# root.ingressroute.yaml
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: bar-com-root
namespace: default
spec:
virtualhost:
fqdn: root.bar.com
routes:
– match: /
services:
– name: s1
port: 80
# delegate the path, `/service2` to the IngressRoute object in this namespace with the name `service2`
– match: /service2
delegate:
name: bar-com-service
—
# service2.ingressroute.yaml
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: bar-com-service
namespace: default
spec:
routes:
– match: /service2
services:
– name: s2
port: 80
– match: /service2/blog
services:
– name: blog
port: 80

In this example the routes below http://root.bar.com/service2 are managed by the author of the bar-com-service IngressRoute object. The IngressRoute object bar-com-root is functioning as the “root” and the bar-com-service is the “delegation” IngressRoute object. This delegation also works across namespaces, allowing the ownership of the root of an IngressRoute, its hostname and TLS secret, to be handled in a namespace that is separate from the IngressRoute routes and Kubernetes Services that provide the web application itself.

That all means that Heptio Contour 0.6 has an ability to restrict the namespaces it uses to respond to IngressRoute roots. We’re excited about the possibilities this offers for deploying Heptio Contour in multi-team Kubernetes clusters.

You can read more about IngressRoute in the Heptio Contour docs/ directory

What next?

Over the next few weeks we’ll be writing about how you can use the new features of IngressRoute to implement patterns like blue/green deployment, load balancing strategies, and cross namespace delegation.

If you’re interested in hearing more about IngressRoute developments, you can find us on the Kubernetes #contour Slack channel or follow us on Twitter.

Last updated 2018–09–24 17:20:50 AEST

Source