Art2Dec SoftLab

At Heptio, our HKS offering is redefining the technical support experience.

When you hear the term “tech support” what comes to mind? Perhaps a group of early-career individuals who are pretty good with computers and software answering calls and responding to tickets. You may picture someone who is friendly, empathetic and measured on how quickly they will respond to you, when you make contact. While all of these attributes are fantastic, the key facet here is that you must initiate contact. Therefore, “tech support” is extremely reactive.

While not everyone loathes contacting technical support, too many people are often disappointed in the lack of scope, depth, technical expertise and ownership provided by the tech support individuals on the other end of the line. If Customer Service has evolved to “Customer Experience” what is the alternative to “Technical Support?”

Enter Customer Reliability Engineering (CRE).

At Heptio, we strive to treat our customers with the same level of proactive diligence and investment as if their environments were our own. To us, CRE means shifting the focus away from Support and into the emerging Site Reliability mindset, as inspired by the work Google pioneered. The guiding principles we use to advance our Heptio Kubernetes Subscription (HKS) include:

• Be proactive, not reactive. Connect with customers continuously to prevent production issues from occurring in the first place. CRE is based on a close partnership between the customer and Heptio.
• Drive customer insights into product innovation. Deeply understand the key issues our customers face and apply these key insights to our products and to the open source community.
• Synthesize key learnings. Our aspiration is to systematize a solution to any issue a customer might encounter, so the first time anyone encounters a production issue it is the last time anyone encounters it.
• Act as the gateway to the upstream community. We don’t want to solve a problem for one customer when we can solve it for the entire Kubernetes community.
• Work ourselves out of a job. Relentlessly automate and improve our technology with a focus on knowledge transfer. Our goal is to drive down the complexities of deploying and running Kubernetes so we can focus on adding value in higher impact technical areas.
• Invest in tooling. Use tools, such as Sonobuoy, to ensure customers avoid issues in the first place and to make it easier to determine causation when issues do occur.

These principles have guided Heptio into a space where we can advise our customers on best practices, offer proactive outreach and act as the key gateway to the Kubernetes community. We deeply understand the opportunity costs our customers are facing and we want to enable them to save time, reduce risk and install confidence so they can add real value; while we have their backs ensuring their environments have enterprise-grade reliability.

So what does this mean exactly?

While we are inspired by Google’s practices, our implementation has been customized to meet the needs of our customer base. Though our CRE team does handle inbound queries from customers in the form of tickets (traditional technical support) it is not core to their work. We allocate the lion’s share of each CRE’s time to:

• Engaging in proactive outreach, such as working with customers to build upgrade plans that include a full review of their Kubernetes environment, core dependencies and rollback plan.
• Analyzing the key drivers of operational cost and complexity, and advocating for product improvements to further solve enterprise use cases.
• Developing thought leadership and tactical content.
• Interacting with the Kubernetes community.

We believe this approach evolves the concept of Technical Support into a function that continuously resolves customer issues through product innovation. The CRE team has the latitude to deeply understand the customer’s environment and responsibility to ensure reliability.

What now? You can learn more about HKS by going here and in case you are interested in joining our CRE team, we are hiring!

Source

A lot of us at Giant Swarm were at KubeCon in Copenhagen back in May. As well as being 3 times the size of the previous edition in Berlin the atmosphere felt very different. Far more enterprises were present and it felt like Kubernetes has now gone mainstream.

As strong supporters of Kubernetes, it being the most widely deployed container orchestrator makes us happy. However, this poses the same question that James Governor from RedMonk wrote about Kubernetes won – so now what?

Part of this growth is that there are now a wide range of Cloud Native tools that provide rich functionality to help users develop and operate their applications. We already run many of these CNCF projects such as Prometheus, Helm, and CoreDNS as part of our stack. Rather than install and manage these tools themselves our customers want us to provide them, too.

At Giant Swarm our vision is to offer a wide range of managed services running on top of Kubernetes, as well as managed Kubernetes. These services will help our customers manage their applications, which is what they actually care about. We call this the Managed Cloud Native Stack and we’ll be launching it at the end of summer. This post is to give you a preview of what’s coming.

Managed Kubernetes is still important

We’re expanding our focus to provide a managed Cloud Native Stack but managed Kubernetes will remain an essential part of our product. Not least because all our managed services will be running on Kubernetes. We continue to extend our Kubernetes offering and have recently linked up with Microsoft as Azure Partners. So we now have Azure support to add to AWS and KVM for running on-premise.

Our customers each have their own Giant Swarm Installation. This is an independent control plane that lets them create as many tenant clusters as they require. This gives flexibility as they can have development, staging, and production clusters. Teams new to Kubernetes can have their own clusters and later these workloads can be consolidated onto larger shared clusters to improve resource utilisation and reduce costs. A big benefit is, there are no pet clusters. All these clusters are vanilla Kubernetes running fixed versions. They are also all upgradeable.

This ability to easily create clusters also means we manage 1 to 2 orders of magnitude more clusters than a typical in-house team. When we find a problem it is identified and fixed for all our customers. This means in many cases that our customers never even see a problem because it was fixed for another customer already.

Each tenant cluster is managed 24/7 by our operations team. To do this a key part of our stack is Prometheus which we use for monitoring and alerting. Prometheus will also be in the first set of managed services we provide. It will be easy to install Prometheus and it will also be managed by us 24/7. Using our operational knowledge of running Prometheus in production at scale.

Helm and chart-operator

Helm is a key part of our stack. As its templating support makes it easy to manage the large number of YAML files needed to deploy complex applications on Kubernetes. Helm charts are also the most popular packaging format for Kubernetes. There are alternatives like ksonnet, but Helm is the tool we use at Giant Swarm.

We use automation wherever possible in our stack. A lot of this automation uses the Operator pattern originally proposed by CoreOS. This consists of a CRD (Custom Resource Definition) that extends the Kubernetes API and a custom controller which we develop in Go using our OperatorKit library.

To enable the Managed Cloud Native Stack we’ve developed chart-operator. This automates the deployment of Helm charts in our tenant clusters. We use Quay.io as our registry for container images and also for charts using their Application Registry. This approach lets us do continuous deployment of cluster components including managed services across our entire estate of tenant clusters.

chart-operator comes with support for release channels. As well as providing stable channels for production there can also be alpha and beta channels. This lets users try out new features on development clusters. It also lets us have incubation charts for new tools. This is important because of the pace of development of cloud native tools under the umbrella of the CNCF.

We’re already using chart-operator to manage cluster components like node-exporter and kube-state-metrics that support our Prometheus monitoring. We also use it for the Nginx Ingress Controller which we pre-install in all tenant clusters. With managed services, this will become an optional component but also configurable. For example, customers will be able to install separate public and private Ingress Controllers.

The charts we use are production grade based on our experience of operating these tools. This means they often have features not present in the community Helm charts. The charts are also designed to work with our monitoring stack. So for example, if an Ingress Controller breaks in the middle of the night our operations team will be alerted and resolve the problem.

Service Catalogue

As part of the Managed Cloud Native Stack we’re adding a Service Catalogue to our web UI, API, and gsctl our command line tool. This easily shows which components are running and lets users select which tools from the Cloud Native Stack they wish to install.

In the screenshot above you can see a typical cluster. The management and ingress components are all being managed by chart-operator. Additional tools can be selected from the service catalogue shown at the beginning of the post.

In the screenshot above you can see how components can be configured. In this case for the Istio service mesh, you can decide whether to inject the Istio sidecar pod into your applications.

Conclusion

We think as the Kubernetes and Cloud Native ecosystems continue to evolve, providing a wider set of services is essential. These services will be pre-installed or available later in the clusters we manage and also be managed by us. This helps our customers focus on running their applications and still take advantage of the rich functionality these tools provide. If this matches what you think a Managed Cloud Native Stack should provide, we’d like to hear about your use case. Request your free trial of the Giant Swarm Infrastructure here.

Source

Feb 7, 2018

by Ian Crosby

I recently had an interesting project building a proof of concept for a cloud based platform. The PoC is composed of a few services, a message queue and a couple simple UIs. All the pieces are containerized and deployed on Kubernetes (GKE). As a final task before sending this over to the client, I needed to expose the front end publicly. What seemed like a straightforward task turned out to be a bit more time consuming, partially due to the specific requirements and partially due to an intricate technical complexity (typo in a config file). I learned quite a bit in the process so I thought I would share the steps I went through for anyone else who travels down this road.

Inter service communication (within the Kubernetes cluster) comes out of the box. By leveraging the internal DNS we simply reference another service by name and our requests will be routed appropriately. This is an over simplification of course, but in most cases this just works. However, if we want to expose any of our applications to the outside world we require another solution. There are three main ways we can do this with Kubernetes:

• Service Type NodePort – will expose our service on a port from a pre-configured range (default: 30000-32767) across each worker node.
• Service Type LoadBalancer – will spin up a load balancer to front the service. This works only on supported cloud platforms (e.g. AWS, GCP, Azure).
• Ingress – a collection of routing rules which are fulfilled by an Ingress Controller.

Ingress is the most flexible and configurable of the three, so this is the solution we chose. It is comprised mainly of two pieces, the Ingress resource, which is a list of routing rules, and the Ingress controller, which is deployed in a pod (or pods) on the cluster and fulfills these rules. Ingress primarily deals with HTTP traffic, the rules are a combination of host and paths which map to an associated backend. In most cases you will run multiple instances of the ingress controller, and front them with a load balancer.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

+———————-+ | | | Load Balancer | | | +———-+———–+ | | | +—————————————-+ | +—————————————-+ | Node 1 | | | Node 2 | |—————————————-| | |—————————————-| | +—————-+ | | | +—————-+ | | | | | | | | | | | | Ingress | | | | | Ingress | | | | Controller |——+ | | Controller | | | | | | | | | | | | | | | | | | | +————-+ +——-+——–+ | | +————-+ +—————-+ | | | | | | | | | | | | | | | | | | | | | App 1 | | | | | App 2 | | | | |———–+ | | | | | | | | | | | | | | +————-+ | | +————-+ | +—————————————-+ +—————————————-+

 

*Not all Ingress Controllers are set up in the above manner.

Kubernetes Ingress has become quite mature and is relatively painless to setup at this point, however we had a few additional requirements which made the setup a bit more complex:

• We had to support TLS
• We wanted to protect the UI with Basic Authentication.
• We needed websocket support.

As with many of the Kubernetes ‘add-ons’ there is no built-in Ingress controller. This allows us the freedom to choose among the various available implementations.

Since the cluster was on Google’s Container Engine (GKE) the default controller is Google’s L7, however we quickly found out that this does not support Basic Auth, so we then moved on to Nginx. Interestingly there are two (actually three) different nginx ingress controllers, one maintained by the Kubernetes community (kubernetes/ingress-nginx) and another by nginx themselves (nginxinc/kubernetes-ingress). There are some subtle differences between the controllers, here are a few of them I ran into:

• Websockets requires adding an annotation when using the nginx (nginxinc) controller.
• TCP/UDP connections can be handled by the nginx (kubernetes) controller by using a ConfigMap, not yet supported by others.
• Authentication (Basic Auth, Digest) is not supported by the GCE controller.
• Replacing the ingress path (rewrite-target) is only supported by the nginx (kubernetes) controller. (Note this is also supported by Traefik but with a different annotation)

A more complete view of the features supported by different controllers can be found here: https://github.com/kubernetes/ingress-nginx/blob/master/docs/annotations.md

Another requirement we had was to setup DNS to be able to expose our application on a subdomain of ours, and TLS so we could support https. For DNS, there is an interesting project currently in incubator (https://github.com/kubernetes-incubator/external-dns) which we looked into, but it was a bit overkill for our simple demo. Instead we simply manually created the managed zone within Google’s Cloud DNS and pointed it to the IP of our ingress load balancer.

As for TLS, this is quite easy to setup thanks to the kube-lego project (https://github.com/jetstack/kube-lego). Once deployed to your Kubernetes cluster, kube-lego creates a user account with LetsEncrypt, and will then create certificates for each Ingress resource marked with the proper annotation (kubernetes.io/tls-acme: “true”). Kube-lego is actually now in maintenance mode, and the recommended replacement is cert-manager (https://github.com/jetstack/cert-manager/).

The last requirement we needed to account for was that our front end we are exposing uses websockets. In general, websockets have become quite well supported by ingress controllers. Although I ran into a couple tricky pieces:

• When using the nginx ingress controller (from Nginx) an annotation is required for services which require websocket support, while for the others (including the Kubernetes backed nginx ingress controller) no annotation is needed.
• If the ingress controller is fronted with a Load Balancer we need to increase the response timeout for the backend (which defaults to 30 seconds).

One complication which added a bit of frustration in going through this process is that switching between ingress controllers is not transparent. In addition to the specific features which may or may not be supported by different controllers as mentioned above, there are also discrepancies within features.

One example is how URL paths are handled. For example if I want to route everything under /foo/ path to service-a, (e.g. /foo/bar and /foo/baz). This is achieved differently in the GCE ingress controller (/foo/*) vs the Nginx ingress controller (/foo/). There is an open issue (https://github.com/kubernetes/ingress-nginx/issues/555) to standardise or document the different implementations.

These differences can be a bit frustrating, it’s best to determine your ingress requirements up front and find the controller which satisfies those. Ideally there will be some standardization around ingress controllers so that swapping out one for another would be seamless.

Exposing applications outside of a cluster is a very common use case. There are several ways to go about this, with ingress being the most flexible. The implementation (the ingress controller) is not standardized, leaving us with several options. On the one hand this is great because it allows us the freedom to choose the tool/technology that fits best, but the lack of consistency can make setting things up more challenging than it needs to be.

That being said, all the necessary pieces to properly expose applications from your cluster to the outside world are available, thanks to projects like External-DNS and kube-lego/cert-manager. This is a testament to the Kubernetes community which is the most active open source community at the moment. When a need arises it is soon filled (possibly by multiple projects). It will be interesting to watch this space to see how it evolves, with new tools such as Istio (https://istio.io/docs/tasks/traffic-management/ingress.html) offering more complete solutions to Kubernetes traffic management.

We have a course about Production Grade Kubernetes. Click on the image to find out more or email us at info@container-solutions.com.

Source

How to Migrate from Rancher 1.6 to Rancher 2.1 Online Meetup

Key terminology differences, implementing key elements, and transforming Compose to YAML

Watch the video

If you are running a user-facing application drawing a lot of traffic, the goal is always to serve user requests efficiently without having any of your users get a server busy! sign. The typical solution is to horizontally scale your deployment so that there are multiple application containers ready to serve the user requests. This technique, however, needs a solid routing capability that efficiently distributes the traffic across your multiple servers. This use case is where the need for load balancing solutions arise.

Rancher 1.6, which is a container orchestration platform for Docker and Kubernetes, provides feature rich support for load balancing. As outlined in the Rancher 1.6 documentation, you can provide HTTP/HTTPS/TCP hostname/path-based routing using the out-of-the-box HAProxy load balancer provider.

In this article, we will explore how these popular load balancing techniques can be implemented with the Rancher 2.0 platform that uses Kubernetes for orchestration.

Rancher 2.0 Load Balancer Options

Out-of-the-box, Rancher 2.0 uses the native Kubernetes Ingress functionality backed by NGINX Ingress Controller for Layer 7 load balancing. Kubernetes Ingress has support for only HTTP and HTTPS protocols. So currently load balancing is limited to these two protocols if you are using Ingress support.

For the TCP protocol, Rancher 2.0 supports configuring a Layer 4 TCP load balancer in the cloud provider where your Kubernetes cluster is deployed. We will also go over a method of configuring the NGINX Ingress Controller for TCP balancing via ConfigMaps later in this article.

HTTP/HTTPS Load Balancing Options

With Rancher 1.6, you added the port/service rules to configure the HAProxy load balancer for balancing target services. You could also configure the hostname/path-based routing rules.

For example, let’s take a service that has two containers launched on Rancher 1.6. The containers launched are listening on private port 80.

To balance the external traffic between the two containers, we can create a load balancer for the application as shown below. Here we configured the load balancer to forward all traffic coming in to port 80 to the target service’s container port and Rancher 1.6 then placed a convenient link to the public endpoint on the load balancer service.

Rancher 2.0 provides a very similar load balancer functionality using Kubernetes Ingress backed by the NGINX Ingress Controller. Let us see how we can do that in the sections below.

Rancher 2.0 Ingress Controller Deployment

An Ingress is just a specification of the rules that a controller component applies to your actual load balancer. The actual load balancer can be running outside of your cluster or can also be deployed within the cluster.

Rancher 2.0 out-of-the-box deploys NGINX Ingress Controller and load balancer on clusters provisioned via RKE [Rancher’s Kubernetes installer] to process the Kubernetes Ingress rules. Please note that the NGINX Ingress Controller gets installed by default on RKE provisioned clusters only. Clusters provisioned via cloud providers like GKE have their own Ingress Controllers that configure the load balancer. This article’s scope is the RKE-installed NGINX Ingress Controller only.

RKE deploys NGINX Ingress Controller as a Kubernetes DaemonSet – so an NGINX instance is deployed on every node in the cluster. NGINX acts like an Ingress Controller listening to Ingress creation within your entire cluster, and it also configures itself as the load balancer to satisfy the Ingress rules. The DaemonSet is configured with hostNetwork to expose two ports- port 80 and port 443. For a detail look at how NGINX Ingress Controller DaemonSet is deployed and deployment configuration options, refer here.

If you are a Rancher 1.6 user, the deployment of the Rancher 2.0 Ingress Controller as a DaemonSet brings forward an important change that you should know of.

In Rancher 1.6 you could deploy a scalable load balancer service within your stack. Thus if you had say four hosts in your Cattle environment, you could deploy one load balancer service with scale two and point to your application via port 80 on those two host IP Addresses. Then, you can also launch another load balancer on the remaining two hosts to balance a different service again via port 80 (since load balancer is using different host IP Addresses).

The Rancher 2.0 Ingress Controller is a DaemonSet – so it is globally deployed on all schedulable nodes to serve your entire Kubernetes Cluster. Therefore, when you program the Ingress rules you need to use unique hostname and path to point to your workloads, since the load balancer Node IP addresses and ports 80/443 are common access points for all workloads.

Now let’s see how the above 1.6 example can be deployed to Rancher 2.0 using Ingress. On Rancher UI, we can navigate to the Kubernetes Cluster and Project and choose the Deploy Workloads functionality to deploy a workload under a namespace for the desired image. Lets set the scale of our workload to two replicas, as depicted below.

Here is how the workload gets deployed and listed on the Workloads tab:

For balancing between these two pods, you must create a Kubernetes Ingress rule. To create this rule, navigate to your cluster and project, and then select the Load Balancing tab.

Similar to a service/port rules in Rancher 1.6, here you can specify rules targeting your workload’s container port.

Host- and Path-Based Routing

Rancher 2.0 lets you add Ingress rules that are based on host names or URL path. Based on your rules, the NGINX Ingress Controller routes traffic to multiple target workloads. Let’s see how we can route traffic to multiple services in your namespace using the same Ingress spec. Consider the following two workloads deployed in the namespace:

We can add an Ingress to balance traffic to these two workloads using the same hostname but different paths.

Rancher 2.0 also places a convenient link to the workloads on the Ingress record. If you configure an external DNS to program the DNS records, this hostname can be mapped to the Kubernetes Ingress address.

The Ingress address is the IP address in your cluster that the Ingress Controller allocates for your workload. You can reach your workload by browsing to this IP address. Use kubectl to see the Ingress address assigned by the controller.

You can use Curl to test if the hostname/path-based routing rules work correctly, as depicted below.

Here is the Rancher 1.6 configuration spec using hostname/path-based rules in comparison to the 2.0 Kubernetes Ingress YAML Specs.

HTTPS/Certificates Option

Rancher 2.0 Ingress functionality also supports the HTTPS protocol. You can upload certificates and use them while configuring the Ingress rules as shown below.

Select the certificate while adding Ingress rules:

Ingress Limitations

• Even though Rancher 2.0 supports HTTP-/HTTPS- hostname/path-based load balancing, one important difference to highlight is the need to use unique hostname/paths while configuring Ingress for your workloads. Reasons being that the Ingress functionality only allows ports 80/443 to be used for routing and the load balancer and the Ingress Controller is launched globally for the cluster as a DaemonSet.
• There is no support for the TCP protocol via Kubernetes Ingress as of the latest Rancher 2.x release, but we will discuss a workaround using NGINX Ingress Controller in the following section.

TCP Load Balancing Options

Layer-4 Load Balancer

For the TCP protocol, Rancher 2.0 supports configuring a Layer 4 load balancer in the cloud provider where your Kubernetes cluster is deployed. Once this load balancer appliance is configured for your cluster, when you choose the option of a Layer-4 Load Balancer for port-mapping during workload deployment, Rancher creates a LoadBalancer service. This service will make the corresponding cloud provider from Kubernetes configure the load balancer appliance. This appliance will then route the external traffic to your application pods. Please note that this needs a Kubernetes cloud provider to be configured as documented here to fulfill the LoadBalancer services created.

Once configuration of the load balancer is successful, Rancher will provide a link in the Rancher UI to your workload’s public endpoint.

NGINX Ingress Controller TCP Support via ConfigMaps

As noted above, Kubernetes Ingress itself does not support the TCP protocol. Therefore, it is not possible to configure the NGINX Ingress Controller for TCP balancing via Ingress creation, even if TCP is not a limitation of NGINX.

However, there is a way to use NGINX’s TCP balancing capability through creation of a Kubernetes ConfigMap, as noted here. The Kuberenetes ConfigMap object can be created to store pod configuration parameters as key-value pairs, separate from the pod image. Details can be found here.

To configure NGINX to expose your services via TCP, you can add/update the ConfigMap tcp-services that should exist in the ingress-nginx namespace. This namespace also contains the NGINX Ingress Controller pods.

The key in the ConfigMap entry should be the TCP port you want to expose for public access and the value should be of the format <namespace/service name>:<service port>. As shown above I have exposed two workloads present in the Default namespace. For example, the first entry in the ConfigMap above tells NGINX that I want to expose the workload myapp that is running in the namespace default and listening on private port 80, on the external port 6790.

Adding these entries to the Configmap will auto-update the NGINX pods to configure these workloads for TCP balancing. You can exec into these pods deployed in the ingress-nginx namespace and see how these TCP ports get configured in the /etc/nginx/nginx.conf file. The workloads exposed should be available on the <NodeIP>:<TCP Port> after the NGINX config /etc/nginx/nginx.conf is updated. If they are not accessible, you might have to expose the TCP port explicitly using a NodePort service.

Rancher 2.0 Load Balancing Limitations

Cattle provided feature-rich load balancer support that is well documented here. Some of these features do not have equivalents in Rancher 2.0. This is the list of such features:

• No support for SNI in current NGINX Ingress Controller.
• TCP load balancing requires a load balancer appliance enabled by cloud provider within the cluster. There is no Ingress support for TCP on Kubernetes.
• Only ports 80/443 can be configured for HTTP/HTTPS routing via Ingress. Also Ingress Controller is deployed globally as a Daemonset and not launched as a scalable service. Also, users cannot assign random external ports to be used for balancing. Therefore, users need to ensure that they configure unique hostname/path combinations to avoid routing conflicts using the same two ports.
• There is no way to specify port rule priority and ordering.
• Rancher 1.6 added support for draining backend connections and specifying a drain timeout. This is not supported in Rancher 2.0.
• There is no support for specifying a custom stickiness policy and a custom load balancer config to be appended to the default config as of now in Rancher 2.0. There is some support, however, available in native Kubernetes for customizing the NGINX configuration as noted here.

Migrate Load Balancer Config via Docker Compose to Kubernetes YAML?

Rancher 1.6 provided load balancer support by launching its own microservice that launched and configured HAProxy. The load balancer configuration that users add is specified in rancher-compose.yml file and not the standard docker-compose.yml. The Kompose tool we used earlier in this blog series works on standard docker-compose parameters and therefore cannot parse the Rancher load balancer config constructs. So as of now, we cannot use this tool for converting the load balancer configs from Compose to Kubernetes YAML.

Conclusion

Since Rancher 2.0 is based on Kubernetes and uses NGINX Ingress Controller (as compared to Cattle’s use of HAProxy), some of the load balancer features supported by Cattle do not have direct equivalents currently. However Rancher 2.0 does support the popular HTTP/HTTPS hostname/path-based routing, which is most often used in real deployments. There also is Layer 4 (TCP) support using the cloud providers via the Kubernetes Load Balancer service. The load balancing support in 2.0 also has a similar intuitive UI experience.

The Kubernetes ecosystem is constantly evolving, and I am sure it’s possible to find equivalent solutions to all the nuances in load balancing going forward!

Prachi Damle

Principal Software Engineer

Source

 

Building a Network Bootable Server Farm for Kubernetes with LTSP

Author: Andrei Kvapil (WEDOS)

In this post, I’m going to introduce you to a cool technology for Kubernetes, LTSP. It is useful for large baremetal Kubernetes deployments.

You don’t need to think about installing an OS and binaries on each node anymore. Why? You can do that automatically through Dockerfile!

You can buy and put 100 new servers into a production environment and get them working immediately – it’s really amazing!

Intrigued? Let me walk you through how it works.

First, we need to understand how exactly it works.

In short, for all nodes we have prepared the image with the OS, Docker, Kubelet and everything else that you need there. This image with the kernel is building automatically by CI using Dockerfile. End nodes are booting the kernel and OS from this image via the network.

Nodes are using overlays as the root filesystem and after reboot any changes will be lost (like in Docker containers). You have a config-file where you can describe mounts and some initial commands which should be executed during node boot (Example: set root user ssh-key and kubeadm join commands)

Image Preparation Process

We will use LTSP project because it’s gives us everything we need to organize the network booting environment. Basically, LTSP is a pack of shell-scripts which makes our life much easier.

LTSP provides a initramfs module, a few helper-scripts, and some configuration systems which prepare the system during the early state of boot, before the main init process call.

This is what the image preparation procedure looks like:

• You’re deploying the basesystem in the chroot environment.
• Make any needed changes there, install software.
• Run the ltsp-build-image command

After that, you will get the squashed image from the chroot with all the software inside. Each node will download this image during the boot and use it as the rootfs. For the update node, you can just reboot it. The new squashed image will be downloaded and mounted into the rootfs.

Server Components

The server part of LTSP includes two components in our case:

• TFTP-server – TFTP is the initial protocol, it is used the download the kernel, initramfs and main config.
• NBD-server – NBD protocol is used to distribute the squashed rootfs image to the clients. It is the fastest way, but if you want, it can be replaced by the NFS or AoE protocol.

You should also have:

• DHCP-server – it will distribute the IP-settings and a few specific options to the clients to make it possible for them to boot from our LTSP-server.

Node Booting Process

This is how the node is booting up

• The first time, the node will ask DHCP for IP-settings and next-server, filename options.
• Next, the node will apply settings and download bootloader (pxelinux or grub)
• Bootloader will download and read config with the kernel and initramfs image.
• Then bootloader will download the kernel and initramfs and execute it with specific cmdline options.
• During the boot, initramfs modules will handle options from cmdline and do some actions like connect NBD-device, prepare overlay rootfs, etc.
• Afterwards it will call the ltsp-init system instead of the normal init.
• ltsp-init scripts will prepare the system on the earlier stage, before the main init will be called. Basically it applies the setting from lts.conf (main config): write fstab and rc.local entries etc.
• Call the main init (systemd) which is booting configured system as usual, mounts shares from fstab, start targets and services, executes commands from rc.local file.
• In the end you have a fully configured and booted system ready for further operations.

As I said before, I’m preparing the LTSP-server with the squashed image automatically using Dockerfile. This method is quite good because you have all steps described in your git repository.
You have versioning, branches, CI and everything that you used to use for preparing your usual Docker projects.

Otherwise, you can deploy the LTSP server manually by executing all steps by hand. This is a good practice for learning and understanding the basic principles.

Just repeat all the steps listed here by hand, just to try to install LTSP without Dockerfile.

Used Patches List

LTSP still has some issues which authors don’t want to apply, yet. However LTSP is easy customizable so I prepared a few patches for myself and will share them here.

I’ll create a fork if the community will warmly accept my solution.

• feature-grub.diff
  LTSP does not support EFI by default, so I’ve prepared a patch which adds GRUB2 with EFI support.
• feature_preinit.diff
  This patch adds a PREINIT option to lts.conf, which allows you to run custom commands before the main init call. It may be useful to modify the systemd units and configure the network. It’s remarkable that all environment variables from the boot environment are saved and you can use them in your scripts.
• feature_initramfs_params_from_lts_conf.diff
  Solves s problem with NBD_TO_RAM option, after this patch you can specify it on lts.conf inside chroot. (not in tftp directory)
• nbd-server-wrapper.sh
  This is not a patch but a special wrapper script which allows you to run NBD-server in the foreground. It is useful if you want to run it inside a Docker container.

Dockerfile Stages

We will use stage building in our Dockerfile to leave only the needed parts in our Docker image. The unused parts will be removed from the final image.

ltsp-base
(install basic LTSP server software)
|
|—basesystem
| (prepare chroot with main software and kernel)
| |
| |—builder
| | (build additional software from sources, if needed)
| |
| ‘—ltsp-image
| (install additional software, docker, kubelet and build squashed image)
|
‘—final-stage
(copy squashed image, kernel and initramfs into first stage)

Stage 1: ltsp-base

Let’s start writing our Dockerfile. This is the first part:

FROM ubuntu:16.04 as ltsp-base

ADD nbd-server-wrapper.sh /bin/
ADD /patches/feature-grub.diff /patches/feature-grub.diff
RUN apt-get -y update
&& apt-get -y install
ltsp-server
tftpd-hpa
nbd-server
grub-common
grub-pc-bin
grub-efi-amd64-bin
curl
patch
&& sed -i ‘s|in_target mount|in_target_nofail mount|’
/usr/share/debootstrap/functions
# Add EFI support and Grub bootloader (#1745251)
&& patch -p2 -d /usr/sbin < /patches/feature-grub.diff
&& rm -rf /var/lib/apt/lists
&& apt-get clean

At this stage our Docker image has already been installed:

• NBD-server
• TFTP-server
• LTSP-scripts with grub bootloader support (for EFI)

Stage 2: basesystem

In this stage we will prepare a chroot environment with basesystem, and install basic software with the kernel.

We will use the classic debootstrap instead of ltsp-build-client to prepare the base image, because ltsp-build-client will install GUI and few other things which we don’t need for the server deployment.

FROM ltsp-base as basesystem

ARG DEBIAN_FRONTEND=noninteractive

# Prepare base system
RUN debootstrap –arch amd64 xenial /opt/ltsp/amd64

# Install updates
RUN echo ”
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiversen
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiversen
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse”
> /opt/ltsp/amd64/etc/apt/sources.list
&& ltsp-chroot apt-get -y update
&& ltsp-chroot apt-get -y upgrade

# Installing LTSP-packages
RUN ltsp-chroot apt-get -y install ltsp-client-core

# Apply initramfs patches
# 1: Read params from /etc/lts.conf during the boot (#1680490)
# 2: Add support for PREINIT variables in lts.conf
ADD /patches /patches
RUN patch -p4 -d /opt/ltsp/amd64/usr/share < /patches/feature_initramfs_params_from_lts_conf.diff
&& patch -p3 -d /opt/ltsp/amd64/usr/share < /patches/feature_preinit.diff

# Write new local client config for boot NBD image to ram:
RUN echo “[Default]nLTSP_NBD_TO_RAM = true”
> /opt/ltsp/amd64/etc/lts.conf

# Install packages
RUN echo ‘APT::Install-Recommends “0”;nAPT::Install-Suggests “0”;’
>> /opt/ltsp/amd64/etc/apt/apt.conf.d/01norecommend
&& ltsp-chroot apt-get -y install
software-properties-common
apt-transport-https
ca-certificates
ssh
bridge-utils
pv
jq
vlan
bash-completion
screen
vim
mc
lm-sensors
htop
jnettop
rsync
curl
wget
tcpdump
arping
apparmor-utils
nfs-common
telnet
sysstat
ipvsadm
ipset
make

# Install kernel
RUN ltsp-chroot apt-get -y install linux-generic-hwe-16.04

Note that you may encounter problems with some packages, such as lvm2.
They have not fully optimized for installing in an unprivileged chroot.
Their postinstall scripts try to call some privileged commands which can fail with errors and block the package installation.

Solution:

• Some of them can be installed before the kernel without any problems (like lvm2)
• But for some of them you will need to use this workaround to install without the postinstall script.

Stage 3: builder

Now we can build all the necessary software and kernel modules. It’s really cool that you can do that automatically in this stage.
You can skip this stage if you have nothing to do here.

Here is example for install latest MLNX_EN driver:

FROM basesystem as builder

# Set cpuinfo (for building from sources)
RUN cp /proc/cpuinfo /opt/ltsp/amd64/proc/cpuinfo

# Compile Mellanox driver
RUN ltsp-chroot sh -cx
‘ VERSION=4.3-1.0.1.0-ubuntu16.04-x86_64
&& curl -L http://www.mellanox.com/downloads/ofed/MLNX_EN-$/mlnx-en-$.tgz
| tar xzf –
&& export
DRIVER_DIR=”$(ls -1 | grep “MLNX_OFED_LINUX-|mlnx-en-“)”
KERNEL=”$(ls -1t /lib/modules/ | head -n1)”
&& cd “$DRIVER_DIR”
&& ./*install –kernel “$KERNEL” –without-dkms –add-kernel-support
&& cd –
&& rm -rf “$DRIVER_DIR” /tmp/mlnx-en* /tmp/ofed*’

# Save kernel modules
RUN ltsp-chroot sh -c
‘ export KERNEL=”$(ls -1t /usr/src/ | grep -m1 “^linux-headers” | sed “s/^linux-headers-//g”)”
&& tar cpzf /modules.tar.gz /lib/modules/$/updates’

Stage 4: ltsp-image

In this stage we will install what we built in the previous step:

FROM basesystem as ltsp-image

# Retrieve kernel modules
COPY –from=builder /opt/ltsp/amd64/modules.tar.gz /opt/ltsp/amd64/modules.tar.gz

# Install kernel modules
RUN ltsp-chroot sh -c
‘ export KERNEL=”$(ls -1t /usr/src/ | grep -m1 “^linux-headers” | sed “s/^linux-headers-//g”)”
&& tar xpzf /modules.tar.gz
&& depmod -a “$”
&& rm -f /modules.tar.gz’

Then do some additional changes to finalize our ltsp-image:

# Install docker
RUN ltsp-chroot sh -c
‘ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add –
&& echo “deb https://download.docker.com/linux/ubuntu xenial stable”
> /etc/apt/sources.list.d/docker.list
&& apt-get -y update
&& apt-get -y install
docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk “”)’

# Configure docker options
RUN DOCKER_OPTS=”$(echo
–storage-driver=overlay2
–iptables=false
–ip-masq=false
–log-driver=json-file
–log-opt=max-size=10m
–log-opt=max-file=5
)”
&& sed “/^ExecStart=/ s|$| $DOCKER_OPTS|g”
/opt/ltsp/amd64/lib/systemd/system/docker.service
> /opt/ltsp/amd64/etc/systemd/system/docker.service

# Install kubeadm, kubelet and kubectl
RUN ltsp-chroot sh -c
‘ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add –
&& echo “deb http://apt.kubernetes.io/ kubernetes-xenial main”
> /etc/apt/sources.list.d/kubernetes.list
&& apt-get -y update
&& apt-get -y install kubelet kubeadm kubectl cri-tools’

# Disable automatic updates
RUN rm -f /opt/ltsp/amd64/etc/apt/apt.conf.d/20auto-upgrades

# Disable apparmor profiles
RUN ltsp-chroot find /etc/apparmor.d
-maxdepth 1
-type f
-name “sbin.*”
-o -name “usr.*”
-exec ln -sf “{}” /etc/apparmor.d/disable/ ;

# Write kernel cmdline options
RUN KERNEL_OPTIONS=”$(echo
init=/sbin/init-ltsp
forcepae
console=tty1
console=ttyS0,9600n8
nvme_core.default_ps_max_latency_us=0
)”
&& sed -i “/^CMDLINE_LINUX_DEFAULT=/ s|=.*|=”$”|”
“/opt/ltsp/amd64/etc/ltsp/update-kernels.conf”

Then we will make the squashed image from our chroot:

# Cleanup caches
RUN rm -rf /opt/ltsp/amd64/var/lib/apt/lists
&& ltsp-chroot apt-get clean

# Build squashed image
RUN ltsp-update-image

Stage 5: Final Stage

In the final stage we will save only our squashed image and kernels with initramfs.

FROM ltsp-base
COPY –from=ltsp-image /opt/ltsp/images /opt/ltsp/images
COPY –from=ltsp-image /etc/nbd-server/conf.d /etc/nbd-server/conf.d
COPY –from=ltsp-image /var/lib/tftpboot /var/lib/tftpboot

Ok, now we have docker image which includes:

• TFTP-server
• NBD-server
• configured bootloader
• kernel with initramfs
• squashed rootfs image

OK, now when our docker-image with LTSP-server, kernel, initramfs and squashed rootfs fully prepared we can run the deployment with it.

We can do that as usual, but one more thing is networking.
Unfortunately, we can’t use the standard Kubernetes service for our deployment, because during the boot, our nodes are not part of Kubernetes cluster and they requires ExternalIP, but Kubernetes always enables NAT for ExternalIPs, and there is no way to disable this behavior.

For now I have two ways for avoid this: use hostNetwork: true or use pipework. The second option will also provide you redundancy because, in case of failure, the IP will be moved with the Pod to another node. Unfortunately, pipework is not native and a less secure method.
If you have some better option for that please let me know.

Here is example for deployment with hostNetwork:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ltsp-server
labels:
app: ltsp-server
spec:
selector:
matchLabels:
name: ltsp-server
replicas: 1
template:
metadata:
labels:
name: ltsp-server
spec:
hostNetwork: true
containers:
– name: tftpd
image: registry.example.org/example/ltsp:latest
command: [ “/usr/sbin/in.tftpd”, “-L”, “-u”, “tftp”, “-a”, “:69”, “-s”, “/var/lib/tftpboot” ]
lifecycle:
postStart:
exec:
command: [“/bin/sh”, “-c”, “cd /var/lib/tftpboot/ltsp/amd64; ln -sf config/lts.conf .” ]
volumeMounts:
– name: config
mountPath: “/var/lib/tftpboot/ltsp/amd64/config”

– name: nbd-server
image: registry.example.org/example/ltsp:latest
command: [ “/bin/nbd-server-wrapper.sh” ]

volumes:
– name: config
configMap:
name: ltsp-config

As you can see it also requires configmap with lts.conf file.
Here is example part from mine:

apiVersion: v1
kind: ConfigMap
metadata:
name: ltsp-config
data:
lts.conf: |
[default]
KEEP_SYSTEM_SERVICES = “ssh ureadahead dbus-org.freedesktop.login1 systemd-logind polkitd cgmanager ufw rpcbind nfs-kernel-server”

PREINIT_00_TIME = “ln -sf /usr/share/zoneinfo/Europe/Prague /etc/localtime”
PREINIT_01_FIX_HOSTNAME = “sed -i ‘/^127.0.0.2/d’ /etc/hosts”
PREINIT_02_DOCKER_OPTIONS = “sed -i ‘s|^ExecStart=.*|ExecStart=/usr/bin/dockerd -H fd:// –storage-driver overlay2 –iptables=false –ip-masq=false –log-driver=json-file –log-opt=max-size=10m –log-opt=max-file=5|’ /etc/systemd/system/docker.service”

FSTAB_01_SSH = “/dev/data/ssh /etc/ssh ext4 nofail,noatime,nodiratime 0 0”
FSTAB_02_JOURNALD = “/dev/data/journal /var/log/journal ext4 nofail,noatime,nodiratime 0 0”
FSTAB_03_DOCKER = “/dev/data/docker /var/lib/docker ext4 nofail,noatime,nodiratime 0 0”

# Each command will stop script execution when fail
RCFILE_01_SSH_SERVER = “cp /rofs/etc/ssh/*_config /etc/ssh; ssh-keygen -A”
RCFILE_02_SSH_CLIENT = “mkdir -p /root/.ssh/; echo ‘ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBSLYRaORL2znr1V4a3rjDn3HDHn2CsvUNK1nv8+CctoICtJOPXl6zQycI9KXNhANfJpc6iQG1ZPZUR74IiNhNIKvOpnNRPyLZ5opm01MVIDIZgi9g0DUks1g5gLV5LKzED8xYKMBmAfXMxh/nsP9KEvxGvTJB3OD+/bBxpliTl5xY3Eu41+VmZqVOz3Yl98+X8cZTgqx2dmsHUk7VKN9OZuCjIZL9MtJCZyOSRbjuo4HFEssotR1mvANyz+BUXkjqv2pEa0I2vGQPk1VDul5TpzGaN3nOfu83URZLJgCrX+8whS1fzMepUYrbEuIWq95esjn0gR6G4J7qlxyguAb9 admin@kubernetes’ >> /root/.ssh/authorized_keys”
RCFILE_03_KERNEL_DEBUG = “sysctl -w kernel.unknown_nmi_panic=1 kernel.softlockup_panic=1; modprobe netconsole netconsole=@/vmbr0,@10.9.0.15/”
RCFILE_04_SYSCTL = “sysctl -w fs.file-max=20000000 fs.nr_open=20000000 net.ipv4.neigh.default.gc_thresh1=80000 net.ipv4.neigh.default.gc_thresh2=90000 net.ipv4.neigh.default.gc_thresh3=100000”
RCFILE_05_FORWARD = “echo 1 > /proc/sys/net/ipv4/ip_forward”
RCFILE_06_MODULES = “modprobe br_netfilter”
RCFILE_07_JOIN_K8S = “kubeadm join –token 2a4576.504356e45fa3d365 10.9.0.20:6443 –discovery-token-ca-cert-hash sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855”

• KEEP_SYSTEM_SERVICES – during the boot, LTSP automatically removes some services, this variable is needed to prevent this behavior.
• PREINIT_* – commands listed here will be executed before systemd runs (this function was added by the feature_preinit.diff patch)
• FSTAB_* – entries written here will be added to the /etc/fstab file.
  As you can see, I use the nofail option, that means that if a partition doesn’t exist, it will continue to boot without error.
  If you have fully diskless nodes you can remove the FSTAB settings or configure the remote filesystem there.
• RCFILE_* – those commands will be written to rc.local file, which will be called by systemd during the boot.
  Here I load the kernel modules and add some sysctl tunes, then call the kubeadm join command, which adds my node to the Kubernetes cluster.

You can get more details on all the variables used from lts.conf manpage.

Now you can configure your DHCP. Basically you should set the next-server and filename options.

I use ISC-DHCP server, and here is an example dhcpd.conf:

shared-network ltsp-netowrk {
subnet 10.9.0.0 netmask 255.255.0.0 {
authoritative;
default-lease-time -1;
max-lease-time -1;

option domain-name “example.org”;
option domain-name-servers 10.9.0.1;
option routers 10.9.0.1;
next-server ltsp-1; # write LTSP-server hostname here

if option architecture = 00:07 {
filename “/ltsp/amd64/grub/x86_64-efi/core.efi”;
} else {
filename “/ltsp/amd64/grub/i386-pc/core.0”;
}

range 10.9.200.0 10.9.250.254;
}

You can start from this, but what about me, I have multiple LTSP-servers and I configure leases statically for each node via the Ansible playbook.

Try to run your first node. If everything was right, you will have a running system there.
The node also will be added to your Kubernetes cluster.

Now you can try to make your own changes.

If you need something more, note that LTSP can be easily changed to meet your needs.
Feel free to look into the source code and you can find many answers there.

Source

1/May 2018

By Matt Barker

We are delighted to announce Jetstack Subscription, comprising tried-and-tested Kubernetes Reference Architecture, the highest quality training, and continuous support for organisations adopting Kubernetes.

As a leading Kubernetes company in Europe, Jetstack Subscription has been designed and refined to give organisations the confidence to take Kubernetes to production environments.

If you’re keen to upskill, need expert assistance with your clusters, or want to deploy Kubernetes to best practice, find out how the features of Jetstack Subscription can facilitate your Kubernetes experience:

Subscription provides Reference Architecture for Kubernetes. The ready-made blueprints help accelerate adoption and integration of Kubernetes into environments with enterprise requirements, shortcutting many months of engineering effort building and designing these systems from scratch.

Built in association with a number of Jetstack customers, our reference architecture is an open source toolkit that combines best-of-breed open source tools from the Cloud Native Landscape, for Kubernetes cluster lifecycle management. It focuses on best-practice cluster security, management and operation (the ‘day 2’ operations that are often overlooked, in our experience).

Developed from the ground-up to be cloud provider-agnostic, it provides a means for consistent and reliable cluster deployment and management, across clouds and on-premises environments.

Jetstack Subscription entitles you to Reference Architecture implementation, integration and assistance from the creators and maintainers of the open source toolkit, with close visibility and contribution to the roadmap.

With a tried-and-tested Reference Architecture, you will have the confidence to deploy and operate your clusters to best practice.

Jetstack Subscription gives you access to up-to-date and on-demand Kubernetes modules and operational playbooks in order to prepare you for running Kubernetes in production and keep your skills sharp.

Maybe you’re upskilling for a new role, working towards a certification or planning to use Kubernetes in production. Perhaps you’re approaching the software for the first time, or have been using it for a while and want to test your existing knowledge.

Jetstack Subscription gives you on-demand, real-world Kubernetes training, whatever your experience.

Jetstack training courses have been built around the knowledge gained in delivering production-ready Kubernetes to our customers.

We have run dozens of Kubernetes training workshops, over the course of several years, educating more than 1000 engineers. The training materials are regularly refined and updated, and align with the CNCF curricula (CKA and CKAD). We know exactly what you need to up-skill quickly and effectively.

What do you get?

• Training modules covering beginner, intermediate and advanced material for cluster and app ops users.
• On-demand wargaming scenarios to test how you would respond to Kubernetes production issues, and prepare you for the worst.
• Operational playbooks offering step-by-step fixes to common production issues.
• Regularly updated materials to reflect new Kubernetes features, and the latest ecosystem tools.

With Jetstack Subscription, you can feel safe in the knowledge that our experienced team of Operations Engineers are on hand to help. It’s always good to know that someone has your back.

Jetstack Subscription provides you and your production team with ongoing support and assistance. Our engineers can help, regardless of whether you’re gearing up to run Kubernetes in production, you’ve just built your first Kubernetes cluster, or you’ve been running Kubernetes in production for some time.

Our engineers have seen Kubernetes used at every scale, from bootstrapped startups on a budget to the largest of enterprises replatforming across multiple environments. Our skilled team know where the sharp edges are, and through experience with our customers, we have seen all sorts of production issues and cluster breaks.

You will have access to invaluable advice on how to build Kubernetes to best practice when you are starting out, and ongoing assistance if you run into trouble.

With Jetstack Subscription you know your Kubernetes clusters are in good hands.

Jetstack Subscription launches at KubeCon EU 2018 in Copenhagen. If you’re there, come and find us at stand C07 to talk more about this new offering and how it can help you and your organisation.

Otherwise, visit the Subscription page at the website, and feel free to share your details so one of our team can get back to you.

Source