Category: Kubernetes

The ability to autoscale workloads based on metrics such as CPU and memory usage is one of the most powerful features of Kubernetes. Of course, to enable this feature we first need a method of gathering and storing these metrics. Today this is most often accomplished using Heapster, but this method can be cumbersome and support from the various contributors to the project has been inconsistent – and in fact it may soon be phased out.

Fortunately, the new Kubernetes metrics APIs are paving the way for a more consistent and efficient way to supply metrics data for the purpose of autoscaling based on Prometheus. It’s no secret that we at CoreOS are big fans of Prometheus, so in this post we will explain the metrics APIs, what’s new, and our recommended method of scaling Kubernetes workloads, going forward.

This post assumes you have a basic understanding of Kubernetes and monitoring.

The Heapster problem

Heapster provides metric collection and basic monitoring capabilities and it supports multiple data sinks to store the collected metrics. The code for each sink resides within the Heapster repository. Heapster also enables the use of the Horizontal Pod Autoscaler (HPA) to automatically scale workloads based on metrics.

There are two problems with the architecture Heapster has chosen to implement. First, it assumes the data store is a bare time-series database for which there is a direct write path. This makes it fundamentally incompatible with Prometheus, as Prometheus uses a pull-based model. Because the rest of the Kubernetes ecosystem has first class Prometheus support, however, it’s not uncommon to run Prometheus, Heapster, an an additional non-Prometheus data store exclusively for Heapster (which typically is InfluxDB) – a less-than-ideal scenario.

Second, because the code for each sink is considered part of the core Heapster code base, the result is a “vendor dump,” where vendors implement support for their systems but often swiftly abandon the code. This is a common cause of frustration when maintaining Heapster. At the time of this writing, many of the 15 available sinks have been unsupported for a long time.

What’s more, even though Heapster doesn’t implement Prometheus as a data sink, it exposes metrics in Prometheus format. This often causes additional confusion.

A bit over a year ago, SIG-Instrumentation was founded and this problem was one of the first we tackled. Contributors and maintainers of the Heapster, Prometheus, and Kubernetes projects came together to design the Kubernetes resource and custom metrics APIs, which point the way forward to a better approach to autoscaling.

Resource and custom metrics APIs

To avoid repeating Heapster’s mistakes, the resource and custom metrics APIs were intentionally created as mere API definitions and not implementations. They are installed into a Kubernetes cluster as aggregated APIs, which allows implementations to be switched out while the APIs stay the same. Both APIs are defined to respond with the current value of the requested metric/query and are both available in beta starting with Kubernetes 1.8.0. Historical metrics APIs may be defined and implemented in the future.

The canonical implementation of the resource metrics API is the Metrics Server, which simply gathers what is referred to as the resource metrics: CPU and memory (and possibly more in the future). It gathers these from all the kubelets in a cluster through the kubelet’s stats AP and simply keeps all values on Pods and Nodes in memory.

The custom metrics API, as the name says, allows requesting arbitrary metrics. Custom metrics API implementations are specific to the respective backing monitoring system. Prometheus was the first monitoring system that an adapter was developed for, simply due to it being a very popular choice to monitor Kubernetes. This Kubernetes Custom Metrics Adapter for Prometheus can be found in the k8s-prometheus-adapter repository on GitHub. Requests to the adapter (aka the Prometheus implementation of the custom metrics API) are converted to a Prometheus query and executed against the respective Prometheus server. The result Prometheus returns is then returned by the custom metrics API adapter.

This architecture solves all the problems we intended to solve:

• Resource metrics can be used more reliably and consistently.
• There is no “vendor dump” for data sinks. Whoever implements an adapter must maintain it.
• Pull-based as well as push-based monitoring systems can be supported.
• Running Heapster with a datastore like InfluxDB in addition to Prometheus will not be necessary anymore.
• Prometheus can consistently be used to monitor, alert and autoscale.

Better yet, because the Kubernetes metrics APIs are standardized, we can now also consistently autoscale on custom metrics, such as worker queue size, in addition to plain CPU and memory.

What to do going forward

Using the Custom Metrics Adapter for Prometheus means we can autoscale on arbitrary metrics that we already collect with Prometheus, without the need to run Heapster at all. In fact, one of the areas SIG-Instrumentation is currently working on is phasing out Heapster – meaning it will eventually be unsupported. Thus, I recommend switching to using the resource and custom metrics APIs sooner rather than later. To enable using the resource and custom metrics APIs with the HPA one must pass the following flag to the kube-controller-manager:

–horizontal-pod-autoscaler-use-rest-clients

If you have any questions, feel free to follow up with me on Twitter (@FredBrancz) or Kubernetes Slack (@brancz). I also want to give Solly Ross (@directXMan12) a huge shout-out as he worked on all of this from the HPA to defining the resource and custom metrics APIs as well as implementing the Custom Metrics Adapter for Prometheus.

Finally, if you are interested in this area and would like to contribute, please join us on the SIG-Instrumentation biweekly call on Thursdays at 17:30 UTC. See you there!

Source

Hands On With Linkerd 2.0

Author: Thomas Rampelberg (Buoyant)

Linkerd 2.0 was recently announced as generally available (GA), signaling its readiness for production use. In this tutorial, we’ll walk you through how to get Linkerd 2.0 up and running on your Kubernetes cluster in a matter seconds.

But first, what is Linkerd and why should you care? Linkerd is a service sidecar that augments a Kubernetes service, providing zero-config dashboards and UNIX-style CLI tools for runtime debugging, diagnostics, and reliability. Linkerd is also a service mesh, applied to multiple (or all) services in a cluster to provide a uniform layer of telemetry, security, and control across them.

Linkerd works by installing ultralight proxies into each pod of a service. These proxies report telemetry data to, and receive signals from, a control plane. This means that using Linkerd doesn’t require any code changes, and can even be installed live on a running service. Linkerd is fully open source, Apache v2 licensed, and is hosted by the Cloud Native Computing Foundation (just like Kubernetes itself!)

Without further ado, let’s see just how quickly you can get Linkerd running on your Kubernetes cluster. In this tutorial, we’ll walk you through how to deploy Linkerd on any Kubernetes 1.9+ cluster and how to use it to debug failures in a sample gRPC application.

Step 1: Install the demo app 🚀

Before we install Linkerd, let’s start by installing a basic gRPC demo application called Emojivoto onto your Kubernetes cluster. To install Emojivoto, run:

curl https://run.linkerd.io/emojivoto.yml | kubectl apply -f –

This command downloads the Kubernetes manifest for Emojivoto, and uses kubectl to apply it to your Kubernetes cluster. Emojivoto is comprised of several services that run in the “emojivoto” namespace. You can see the services by running:

kubectl get -n emojivoto deployments

You can also see the app live by running

minikube -n emojivoto service web-svc –url # if you’re on minikube

… or:

kubectl get svc web-svc -n emojivoto -o jsonpath=”{.status.loadBalancer.ingress[0].*}” #

… if you’re somewhere else

Click around. You might notice that some parts of the application are broken! If you were to inspect your handly local Kubernetes dashboard, you wouldn’t see very much interesting—as far as Kubernetes is concerned, the app is running just fine. This is a very common situation! Kubernetes understands whether your pods are running, but not whether they are responding properly.

In the next few steps, we’ll walk you through how to use Linkerd to diagnose the problem.

Step 2: Install Linkerd’s CLI

We’ll start by installing Linkerd’s command-line interface (CLI) onto your local machine. Visit the Linkerd releases page, or simply run:

curl -sL https://run.linkerd.io/install | sh

Once installed, add the linkerd command to your path with:

export PATH=$PATH:$HOME/.linkerd2/bin

You should now be able to run the command linkerd version, which should display:

Client version: v2.0
Server version: unavailable

“Server version: unavailable” means that we need to add Linkerd’s control plane to the cluster, which we’ll do next. But first, let’s validate that your cluster is prepared for Linkerd by running:

linkerd check –pre

This handy command will report any problems that will interfere with your ability to install Linkerd. Hopefully everything looks OK and you’re ready to move on to the next step.

Step 3: Install Linkerd’s control plane onto the cluster

In this step, we’ll install Linkerd’s lightweight control plane into its own namespace (“linkerd”) on your cluster. To do this, run:

linkerd install | kubectl apply -f –

This command generates a Kubernetes manifest and uses kubectl command to apply it to your Kubernetes cluster. (Feel free to inspect the manifest before you apply it.)

(Note: if your Kubernetes cluster is on GKE with RBAC enabled, you’ll need an extra step: you must grant a ClusterRole of cluster-admin to your Google Cloud account first, in order to install certain telemetry features in the control plane. To do that, run: kubectl create clusterrolebinding cluster-admin-binding-$USER –clusterrole=cluster-admin –user=$(gcloud config get-value account).)

Depending on the speed of your internet connection, it may take a minute or two for your Kubernetes cluster to pull the Linkerd images. While that’s happening, we can validate that everything’s happening correctly by running:

linkerd check

This command will patiently wait until Linkerd has been installed and is running.

Finally, we’re ready to view Linkerd’s dashboard! Just run:

linkerd dashboard

If you see something like below, Linkerd is now running on your cluster. 🎉

Step 4: Add Linkerd to the web service

At this point we have the Linkerd control plane installed in the “linkerd” namespace, and we have our emojivoto demo app installed in the “emojivoto” namespace. But we haven’t actually added Linkerd to our service yet. So let’s do that.

In this example, let’s pretend we are the owners of the “web” service. Other services, like “emoji” and “voting”, are owned by other teams–so we don’t want to touch them.

There are a couple ways to add Linkerd to our service. For demo purposes, the easiest is to do something like this:

kubectl get -n emojivoto deploy/web -o yaml | linkerd inject – | kubectl apply -f –

This command retrieves the manifest of the “web” service from Kubernetes, runs this manifest through linkerd inject, and finally reapplies it to the Kubernetes cluster. The linkerd inject command augments the manifest to include Linkerd’s data plane proxies. As with linkerd install, linkerd inject is a pure text operation, meaning that you can inspect the input and output before you use it. Since “web” is a Deployment, Kubernetes is kind enough to slowly roll the service one pod at a time–meaning that “web” can be serving traffic live while we add Linkerd to it!

We now have a service sidecar running on the “web” service!

Step 5: Debugging for Fun and for Profit

Congratulations! You now have a full gRPC application running on your Kubernetes cluster with Linkerd installed on the “web” service. Of course, that application is failing when you use it–so now let’s use Linkerd to track down those errors.

If you glance at the Linkerd dashboard (the linkerd dashboard command), you should see all services in the “emojivoto” namespace show up. Since “web” has the Linkerd service sidecar installed on it, you’ll also see success rate, requests per second, and latency percentiles show up.

That’s pretty neat, but the first thing you might notice is that success rate is well below 100%! Click on “web” and let’s dig in.

You should now be looking at the Deployment page for the web service. The first thing you’ll see here is that web is taking traffic from vote-bot (a service included in the Emojivoto manifest to continually generate a low level of live traffic), and has two outgoing dependencies, emoji and voting.

The emoji service is operating at 100%, but the voting service is failing! A failure in a dependent service may be exactly what’s causing the errors that web is returning.

Let’s scroll a little further down the page, we’ll see a live list of all traffic endpoints that “web” is receiving. This is interesting:

There are two calls that are not at 100%: the first is vote-bot’s call the “/api/vote” endpoint. The second is the “VotePoop” call from the web service to the voting service. Very interesting! Since /api/vote is an incoming call, and “/VotePoop” is an outgoing call, this is a good clue that the failure of the vote service’s VotePoop endpoint is what’s causing the problem!

Finally, if we click on the “tap” icon for that row in the far right column, we’ll be taken to live list of requests that match this endpoint. This allows us to confirm that the requests are failing (they all have gRPC status code 2, indicating an error).

At this point we have the ammunition we need to talk to the owners of the vote “voting” service. We’ve identified an endpoint on their service that consistently returns an error, and have found no other obvious sources of failures in the system.

We hope you’ve enjoyed this journey through Linkerd 2.0. There is much more for you to explore. For example, everything we did above using the web UI can also be accomplished via pure CLI commands, e.g. linkerd top, linkerd stat, and linkerd tap.

Also, did you notice the little Grafana icon on the very first page we looked at? Linkerd ships with automatic Grafana dashboards for all those metrics, allowing you to view everything you’re seeing in the Linkerd dashboard in a time series format. Check it out!

Want more?

In this tutorial, we’ve shown you how to install Linkerd on a cluster, add it as a service sidecar to just one service–while the service is receiving live traffic!—and use it to debug a runtime issue. But this is just the tip of the iceberg. We haven’t even touched any of Linkerd’s reliability or security features!

Linkerd has a thriving community of adopters and contributors, and we’d love for YOU to be a part of it. For more, check out the docs and GitHub repo, join the Linkerd Slack and mailing lists (users, developers, announce), and, of course, follow @linkerd on Twitter! We can’t wait to have you aboard!

Source

12/Feb 2018

By Hannah Morris

This blog post provides an insight into how we run our Kubernetes workshops as we prepare for even more from Jetstack training in 2018.

In 2017, Jetstack ran more than 25 Kubernetes in Practice workshops: We trained engineers from over 80 different companies in London and across Europe, and had a great time doing so!

2018 promises to be an even busier year for Jetstack training, with several dates already in the diary for our first and second series of Beginner and Intermediate workshops. In addition, we will be running a number of workshops in our Kubernetes for Startups series. If you want to participate, you can find dates for Q1 at the end of this post.

In the run up to our 2018 workshops, we have been developing new material for our Advanced courses and working hard on the Jetstack Subscription, which will soon provide users with on-demand Kubernetes training modules and playbooks.

“Lots of lightbulb moments!” ~ Workshop Attendee

We run a number of our workshops in association with Google Cloud at the Google Academy in London Victoria. A typical day commences at 9.30am and finishes around 5pm, with a break for lunch. This gives us enough time to cover course content, answer questions, and pause to pick the brains of fellow Kubernetes enthusiasts over a cup of coffee!

Jetstack training courses are developed around the knowledge and experience gained by our engineers when deploying Kubernetes for clients. Our training modules are continuously updated and refined in order to ensure that they are consistent with the constantly changing Kubernetes ecosystem.

We focus on making our courses interactive, with a mixture of presentations, demos and hands-on labs. They are designed to prepare you to deploy, use and operate Kubernetes efficiently.

“The trainers were enthusiastic and knowledgeable.” ~ Workshop Attendee

We have a range of courses tailored for Kubernetes users of all levels. We run a two-day Kubernetes in Practice course (Beginner and Intermediate) and we now offer our Advanced workshop as part of a two-day Kubernetes Operator course.

Kubernetes in Practice

Day 1 – Beginner

• Introduction to core concepts of Kubernetes.
• Hands-on labs covering: persistent volume types, application logs, and using Helm to package and deploy applications.
• Demo: continuous integration and deployment (CI/CD) on Kubernetes. An application is deployed into multiple environments before being rolled out to production.

“The workshop helped me understand what Kubernetes is made for, what the building blocks are and how we are supposed to use them.” ~ Beginner Workshop Attendee

Day 2 – Intermediate

• More advanced Kubernetes features and how to use them to lessen operational burden.
• Hands-on labs covering: Autoscaling, the Control Plane, Ingress and StatefulSet.
• Set up a CI/CD pipeline running on Kubernetes, deploying to multiple environments with Helm.
• Cluster Federation demo.

“I now know the concepts and building blocks and I have a sense of what’s possible.” ~ Intermediate Workshop Attendee

Kubernetes Operator

Our Advanced Wargaming workshop is now part of a two-day Advanced Operations course.

Advanced Operations – Day 1

• Provision a Kubernetes cluster manually from the ground up.
• Learn how to configure cluster components (apiserver, controller-manager, scheduler, etcd, kubelet, kube-proxy, container runtime) and how the configuration of these components affects cluster behaviour.
• Comparison of Kubernetes deployment tools such as Kubeadm, Kops, Tarmak.

Advanced Operations – Day 2 (Wargaming)

• Deep dive into Kubernetes Internals.
• Team Wargaming: work together to overcome various production issues and common cluster failures.

“Had a really insightful day…just proves how important it is to have the operational tasks rehearsed!” ~ Advanced Workshop Attendee

“Yes yes please please more training at an advanced level!” ~ Workshop Attendee

We make workshop slides available to participants following the session, and warmly welcome feedback. We develop new material with our participants’ comments in mind; we need to know which approaches work best from an educational perspective, as well as any further topics and issues that could be tackled in future workshops.

We were grateful to receive some very positive feedback from our 2017 series, with an average score of 9⁄10 from participants. We aim to build on this by constantly refining our course content and adding new material, and we are looking forward to delivering in 2018.

“Boom!”

Kubernetes for Startups 2018

Dates for Q1 training:

• Thursday 8th March – Beginner (London)
• Friday 9th March – Intermediate (London)
• Monday 26th March – Beginner (Paris)
• Tuesday 27th March – Intermediate (Paris)

If you work in a Startup and would like to attend any of these workshops, please email hannah@jetstack.io

Source

In May I had the honor of sitting on a diversity panel at Kubecon EU with an awesome group of folks from Heptio, Google, Redhat and StorageOS.

When I first started telling people that I was going to be on a diversity panel, I got a few different reactions. The most surprising were people telling me I shouldn’t do it, that they themselves wouldn’t want to be considered the “token” diversity advocate, that it is too hard to be the person speaking publicly about diversity. People don’t even want to talk about diversity among their families or co-workers, much less speak publicly about the issue. This got me thinking about my own experiences.

My father is a retired military veteran so I moved about every three years my entire life. I have lived in several countries and several different states. Living on a military base was always a multi-cultural experience full of families of all shapes and sizes and made up of people from all over the world. At some point, everyone was the new kid, so being supportive and open to meeting new people was a natural part of surviving. To me, diversity and inclusion were part of my life and seemed like the way things worked everywhere. As I grew up and lived on my own I started to pull my head up, look around and realize how far from the truth my assumptions had been.

I started to notice the problem once I moved into more senior roles and experienced first-hand many examples of sexist, bigoted, and otherwise non-inclusive behavior in companies. I suddenly had a broader view of the organization and could see these issues more clearly. I was running up against real problems affecting not just me but others on my teams. I realized I really did need to be part of the change and part of the conversation. I realized I needed to be a person that advocated for diversity and for a workplace that allowed everyone to succeed.

The problem is, a lot of people are afraid to talk about diversity. Not just publicly but even privately. This point makes it really hard to make change and to keep educating people. It also makes it really difficult to feel like we can have truly open dialogue.

We are never going to learn from each other if we don’t start talking to each other; and when I say talking to each other, I really mean listening to each other. We have to start having more open dialogue about issues in a way that ensures we are educating. We have to do that in a way that is open to listening and void of our tendencies to lecture or shame. A true human connection can set the stage for real change.

I want to encourage everyone to get involved, educate, listen, and work hard not to make assumptions about others. We don’t always know how someone identifies in relation to race, ethnicity, gender, religion, sexual orientation, geographical representation, political beliefs and more, and we need to take the time to learn about others and understand who they are so we can really start to understand how to create change.

I want to see real companies making real change and I want to be involved in the conversation and I want to make real change. I want to continue to evolve my ideas and my thinking and I want to ensure I share those ideas and this mission with my children in hopes things are better when they start their careers. Taking the time to truly understand others will make this much more possible.

The goal of the Kubecon diversity panel was to encourage action. Here are some actions you can take to encourage more conversation.

The next time someone asks you to be in a panel or write a blog post about diversity, consider it an opportunity to share your personal journey. I can guarantee at least one other person will relate to your journey and feel inspired and supported by you sharing it.

The next time you see someone sharing their personal story about diversity in a panel or blog, let them know you appreciate it. It can be lonely to put your words out there so hearing that someone enjoyed it will encourage that person to speak out more.

Thanks to everyone who attended the panel and reached out after. If you missed it you can check out the panel here.

We would love to have more people joining the discussion so please join the diversity slack channel in the Kubernetes slack.

Source

The Learning Curve

The first round of dust settled pretty quick. The daily routine taking hold, legal to work in Germany, got the equipment, met the team. So, what’s this Kubernetes thing and how do we use it?

Positions and Titles of my Dev. Colleagues

Some of the titles of my colleagues are easy. We have the Co-Founders, CTO, UX/UI, Working Students, etc. But better understanding what my colleagues do, besides run Kubernetes infrastructure has been a real learning curve. We have Solutions Engineers, Platform Reliability Engineers, Support Engineers, a Developer Advocate only to mention a few. Each developer does their thing and I do mine – and from what I’ve found, they know as much about what I do as I now about what they do, all the way from the daily routine to the terms and acronyms we use.

Terms and Acronyms

Admittedly, this is about the most fun part of the learning curve. Every acronym that you nail, gets you closer to actually (feeling like) you know what you’re talking about and even more importantly, what (the Hell) they’re talking about. Here are a few when using #Slack, GitHub, VS Code, Kubernetes.

K8s – It’s what we do day in and day out. Kubernetes.

Repository – A repository is usually used to organize a single project. Repositories can contain folders and files, images, videos, spreadsheets, and data sets – anything your project needs.

Bare Metal – A computer system running on physical hardware compared to using a virtual server provided by a cloud provider like AWS or Azure.

Commit – These are saved changes on Github. Each commit has an associated message, which is a description explaining why a particular change was made. These messages capture the history of your changes, so other contributors can understand what you’ve done and why.

AFK – Away from Keyboard. It’s to let your colleagues know that you’re not available for a period of time.

Retro – Look back at the month (time dependant on organization).

Branch – Branching is the way to work on different versions of a repository at one time.

SIG – Special Interest Group. These are groups that are used across teams to both help in a specific category (channel) or communicate to the entire team.

Platform Reliability Engineer – That’s Joe.

Pull Request (PR) – To implement a change using team-sourcing. Requesting approvals and comments from chosen team members. Once the PR is approved, it gets merged and goes live.

mühsam ernährt sich das Eichhörnchen (it’s a German startup) – Idiom: used in the English context of Slowly but Surely.

The Giant Swarm Infrastructure

Not going to go too deep into our tool. If you are interested in learning what it does, we do have several tutorials online that explain how we launch and scale Kubernetes clusters. Just know that there is a learning curve with this one that is ever-changing alongside the product – big one.

Competitors

As you may know, competitive research in online marketing is essential. I spent a long time researching competitors and running analysis based on standard practices. Things like what marketing automation system are they using, how many pieces of content do they post weekly/monthly, who picks up their stories in the media, how many followers, etc. Competition however in the Open Source space is very different. There are your standard competitors, which we all agree are not as good as us but there are also so many available tools for Open Source that it is tough to say who adds value to your product compared to who is your actual competitor.

Even with Publicly Traded companies who are obvious competitors, our team may know a dozen people who work there and we may even invite them to speak at a meetup. As long as they offer good insight to Kubernetes as a whole. That makes it easy to be a part of the ecosystem but very difficult to make sure you don’t plug a product that is the one gunning after you.

Settled in as the Marketing Guy: Days 61 – 99 »

Source

How to Migrate from Rancher 1.6 to Rancher 2.1 Online Meetup

Key terminology differences, implementing key elements, and transforming Compose to YAML

Watch the video

Service discovery is one of the core functionalities of any container-based environment. Once you have packaged your application and launched it using containers, the next step is making it discoverable to other application containers in your environment or the external world.

In this article we will go over the service discovery support provided by Rancher 2.0 and see how the Rancher 1.6 feature set maps to the latest version.

Service Discovery in Rancher 1.6

Rancher 1.6 provided service discovery within Cattle environments. Rancher’s own DNS microservice provided the internal DNS functionality.

Rancher’s internal DNS provides the following key features:

• Service discovery within stack and across stack

  All services in the stack are resolvable by <service_name> and by <service_name>.<stack_name> across stacks.

• Container discovery

  All containers are resolvable globally by their name.

• Creating a service alias name

  Adding an alias name to services and linking to other services using aliases.

• Discovery of external services

  Pointing to services deployed outside of Rancher using the external IP(s) OR a domain name.

Service Discovery in Rancher 2.0

Rancher 2.0 uses the native Kubernetes DNS support to provide equivalent service discovery for Kubernetes workloads and pods. A Cattle user will be able to replicate all the service discovery features in Rancher 2.0 without loss of any functionality.

Similar to the Rancher 1.6 DNS microservice, Kubernetes schedules a DNS pod and service in the cluster and configures the kubelets to route all DNS lookups to this DNS service. Rancher 2.0’s Kubernetes cluster deploys skyDNS as the Kubernetes DNS service, which is a flavor of the default Kube-DNS implementation.

Service Resolution

As noted in this previous article, a Rancher 1.6 service maps to a Kubernetes workload of a certain type. A short summary on the popular types of workloads can be found here.

Kubernetes workloads are objects that specify the deployment rules for pods that are launched for the workload. Workload objects by themselves are not resolvable via DNS to other objects in the Kubernetes cluster. To lookup and access a workload, a Kubernetes Service needs to be created for the workload. Here are some details about a Kubernetes Service.

Any service created within Kubernetes gets a DNS name. The DNS A record created for the service is of the form <service_name>.<namespace_name>.svc.cluster.local. The DNS name for the service resolves to the cluster IP of the service. The cluster IP is an internal IP assigned to the service which is resolvable within the cluster.

Within the Kubernetes namespace, the service is resolvable directly by the <service_name> and outside of the namespace using <service_name>.<namespace_name>. This convention is similar to the service discovery within stack and across stack for Rancher 1.6.

Thus to lookup and access your application workload, a service needs to be created that gets a DNS record assigned.

Rancher simplifies this process by automatically creating a service along with the workload, using the service port and service type you select in the UI while deploying the workload and service name identical to the workload’s name. If no port is exposed, port 42 is used. This practice makes the workload discoverable within and across namespaces by its name.

For example, as seen below, I deploy a few workloads of type Deployment in two namespaces using Rancher 2.0 UI.

I can see the corresponding DNS records auto-created by Rancher for the workloads under Cluster > Project > Service Discovery tab.

The workloads become accessible to any other workload within and across the namespaces as demonstrated below.

Pod Resolution

Individual pods running in the Kubernetes cluster also get a DNS record assigned, which is in the form <pod_ip_address>.<namespace_name>.pod.cluster.local. For example, a pod with an IP of 10.42.2.7 in the namespace default with a DNS name of cluster.local would have an entry of 10-42-2-7.default.pod.cluster.local.

Pods can also be resolved using the hostname and subdomain fields if set in the pod spec. Details about this resolution is covered in the Kubernetes docs here.

Creating Alias Names for Workloads and External Services

Just as you can create an alias for Rancher 1.6 services, you can do the same for Kubernetes workloads using Rancher 2.0. Similarly you can also create DNS records pointing to externally running services using their hostname or IP address in Rancher 2.0. These DNS records are Kubernetes service objects.

Using the 2.0 UI, navigate to the Cluster > Project view and choose the Service Discovery tab. Here, all the existing DNS records created for your workloads will be listed under each namespace.

Click on Add Record to create new DNS records and view the various options supported to link to external services or to create aliases for another workload/DNS record/set of pods.

One thing to note is that out of these options for creating DNS records, the following options are supported natively by Kubernetes:

• Point to an external hostname
• Point to a set of pods which match a selector

The remaining options are implemented by Rancher leveraging Kubernetes:

• Point to external IP address
• Create alias for another DNS record
• Point to another workload

Docker Compose to Kubernetes YAML

Now let’s see what is needed if we want to migrate an application from 1.6 to 2.0 using Compose files instead of deploying it over the 2.0 UI.

As noted above, when we deploy workloads using the Rancher 2.0 UI, Rancher internally takes care of creating the necessary Kubernetes ClusterIP service for service discovery. However, if you deploy the workload via Rancher CLI or Kubectl client, what should you do to ensure that the same service discovery behavior is accomplished?

Service Discovery Within and Across Namespaces via Compose

Lets start with the following docker-compose.yml file, which shows two services (foo and bar) within a stack. Within a Cattle stack, these two services can reach each other by using their service names.

version: ‘2’
services:
bar:
image: user/testnewhostrouting
stdin_open: true
tty: true
labels:
io.rancher.container.pull_image: always
foo:
image: user/testnewhostrouting
stdin_open: true
tty: true
labels:
io.rancher.container.pull_image: always

What happens to service discovery if we migrate these two services to a namespace in Rancher 2.0?

We can convert this docker-compose.yml file from Rancher 1.6 to Kubernetes YAML using the Kompose tool, and then deploy the application in the Kubernetes cluster using Rancher CLI.

Now this conversion generates the *-deployment.yaml files, and deploying them using Rancher CLI creates the corresponding workloads within a namespace.

Can these workloads reach each other within the namespace? We can exec into the shell of workload foo using Rancher 2.0 UI and see if pinging the other workload bar works.

No! The reason is because we only created the workload objects of type Deployment. To make these workloads discoverable, they each need a service of type ClusterIP pointing to them that will be assigned a DNS record. The Kubernetes YAML for such a service should look like the sample below.

Note that ports is a required field. Therefore, we need to provide it using some port number, such as 42 as shown here.

apiVersion: v1
kind: Service
metadata:
annotations:
io.rancher.container.pull_image: always
creationTimestamp: null
labels:
io.kompose.service: bar
name: bar
spec:
clusterIP: None
ports:
– name: default
port: 42
protocol: TCP
targetPort: 42
selector:
io.kompose.service: bar

After deploying this service via CLI, service foo can successfully ping service bar!

Thus if you take the Compose-to-Kubernetes-YAML route to migrate your 1.6 services to Rancher 2.0, make sure you also deploy corresponding ClusterIP services for the workloads. The same solution also applies to cross-namespace referencing of workloads.

Links/External_Links via Compose

If you are a Cattle user, you know that in Rancher 1.6 you can create a service-link/alias pointing to another service, and use that alias name in your application to discover that linked target service.

For example, consider the application below, where the web service links to the database service using the alias name mongo.

Using Kompose, converting this Compose file to Kubernetes YAML generated the corresponding deployment and service YAML specs. If your services in docker-compose.yml expose ports, Kompose generates a Kubernetes ClusterIP service YAML spec by default.

Deploying these using Rancher CLI generated the necessary workloads.

However the service link mongo is missing, as the Kompose conversion does not support links in the docker-compose.yml file. As a result, the workload web encounters an error, and its pods keep restarting, failing to resolve the mongo link to database service.

How do we fix the broken DNS link? The solution is to create another ClusterIP service spec and set its name to the alias name of the link in docker-compose.

Deploying this service creates the necessary DNS record, and the link mongo is created, making the web workload available!

The following image shows that the pods launched for the web workload entered a Running state.

Transitioning from SkyDNS to CoreDNS in the Future

As of v2.0.7, Rancher deploys skyDNS as supported by Kubernetes version 1.10.x. In Kubernetes version 1.11 and later, CoreDNS can be installed as a DNS provider. We are evaluating CoreDNS as well, and it will be presentable as an alternative to skyDNS in the future versions of Rancher.

Conclusion

This article looked at how equivalent service discovery can be supported in Rancher 2.0 via Kubernetes DNS functionality. In the upcoming article, I plan to look at load balancing options supported by Rancher 2.0 and any limitations present in comparison to Rancher 1.6.

Prachi Damle

Principal Software Engineer

Source

Today we are issuing patches for two newly disclosed security vulnerabilities affecting all versions of Tectonic and Kubernetes versions 1.3 through 1.10. The vulnerabilities have been assigned CVE-2017-1002101 and CVE-2017-1002102, respectively.

Both bugs affect all versions of Tectonic and versions of Kubernetes from 1.3 to 1.10 that use Pod Security Policies (PSPs). The bugs can be used to bypass a PSP. If you aren’t using PSPs, you don’t have anything to worry about.

The first vulnerability involves the subPath parameter, which is commonly used to reference a volume multiple times within a Pod. A successful exploit can allow an attacker to access unauthorized files on a Pod with any kind of volume mount, including files on the host.

The second bug relates to mounting ConfigMaps and Secrets as volumes within a Pod. Maliciously crafted Pods can trigger deletion of any file or directory on the host.

To address these vulnerabilities, today we’re releasing two new versions of Tectonic:

• Tectonic 1.7.14-tectonic.1 to our 1.7 production and preproduction channels
• Tectonic 1.8.9-tectonic.1 to our 1.8 production and preproduction channels

Apply the update to your clusters by clicking “Check for Update” in your cluster settings.

In addition, this bug impacts the kubelet, which is managed at the infrastructure level. New clusters will install the patched kubelet version. If you have enabled PSPs on an existing cluster (which are not on by default), you will need to update your autoscaling group user-data or provisioning tool to install the 1.7.14 or 1.8.9 version of the kubelet, or update it manually.

All current Tectonic customers will receive an email alert about the bugs and the need to update. More information on the Tectonic update process and how to use the two channels can be found in our documentation.

Source

Kubernetes 1.12: Kubelet TLS Bootstrap and Azure Virtual Machine Scale Sets (VMSS) Move to General Availability

Author: The 1.12 Release Team

We’re pleased to announce the delivery of Kubernetes 1.12, our third release of 2018!

Today’s release continues to focus on internal improvements and graduating features to stable in Kubernetes. This newest version graduates key features such as security and Azure. Notable additions in this release include two highly-anticipated features graduating to general availability: Kubelet TLS Bootstrap and Support for Azure Virtual Machine Scale Sets (VMSS).

These new features mean increased security, availability, resiliency, and ease of use to get production applications to market faster. The release also signifies the increasing maturation and sophistication of Kubernetes on the developer side.

Let’s dive into the key features of this release:

Introducing General Availability of Kubelet TLS Bootstrap

We’re excited to announce General Availability (GA) of Kubelet TLS Bootstrap. In Kubernetes 1.4, we introduced an API for requesting certificates from a cluster-level Certificate Authority (CA). The original intent of this API is to enable provisioning of TLS client certificates for kubelets. This feature allows for a kubelet to bootstrap itself into a TLS-secured cluster. Most importantly, it automates the provision and distribution of signed certificates.

Before, when a kubelet ran for the first time, it had to be given client credentials in an out-of-band process during cluster startup. The burden was on the operator to provision these credentials. Because this task was so onerous to manually execute and complex to automate, many operators deployed clusters with a single credential and single identity for all kubelets. These setups prevented deployment of node lockdown features like the Node authorizer and the NodeRestriction admission controller.

To alleviate this, SIG Auth introduced a way for kubelet to generate a private key and a CSR for submission to a cluster-level certificate signing process. The v1 (GA) designation indicates production hardening and readiness, and comes with the guarantee of long-term backwards compatibility.

Alongside this, Kubelet server certificate bootstrap and rotation is moving to beta. Currently, when a kubelet first starts, it generates a self-signed certificate/key pair that is used for accepting incoming TLS connections. This feature introduces a process for generating a key locally and then issuing a Certificate Signing Request to the cluster API server to get an associated certificate signed by the cluster’s root certificate authority. Also, as certificates approach expiration, the same mechanism will be used to request an updated certificate.

Support for Azure Virtual Machine Scale Sets (VMSS) and Cluster-Autoscaler is Now Stable

Azure Virtual Machine Scale Sets (VMSS) allow you to create and manage a homogenous VM pool that can automatically increase or decrease based on demand or a set schedule. This enables you to easily manage, scale, and load balance multiple VMs to provide high availability and application resiliency, ideal for large-scale applications that can run as Kubernetes workloads.

With this new stable feature, Kubernetes supports the scaling of containerized applications with Azure VMSS, including the ability to integrate it with cluster-autoscaler to automatically adjust the size of the Kubernetes clusters based on the same conditions.

Additional Notable Feature Updates

RuntimeClass is a new cluster-scoped resource that surfaces container runtime properties to the control plane being released as an alpha feature.

Snapshot / restore functionality for Kubernetes and CSI is being introduced as an alpha feature. This provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers.

Topology aware dynamic provisioning is now in beta, meaning storage resources can now understand where they live. This also includes beta support to AWS EBS and GCE PD.

Configurable pod process namespace sharing is moving to beta, meaning users can configure containers within a pod to share a common PID namespace by setting an option in the PodSpec.

Taint node by condition is now in beta, meaning users have the ability to represent node conditions that block scheduling by using taints.

Arbitrary / Custom Metrics in the Horizontal Pod Autoscaler is moving to a second beta to test some additional feature enhancements. This reworked Horizontal Pod Autoscaler functionality includes support for custom metrics and status conditions.

Improvements that will allow the Horizontal Pod Autoscaler to reach proper size faster are moving to beta.

Vertical Scaling of Pods is now in beta, which makes it possible to vary the resource limits on a pod over its lifetime. In particular, this is valuable for pets (i.e., pods that are very costly to destroy and re-create).

Encryption at rest via KMS is now in beta. This adds multiple encryption providers, including Google Cloud KMS, Azure Key Vault, AWS KMS, and Hashicorp Vault, that will encrypt data as it is stored to etcd.

Availability

Kubernetes 1.12 is available for download on GitHub. To get started with Kubernetes, check out these interactive tutorials. You can also install 1.12 using Kubeadm.

5 Day Features Blog Series

If you’re interested in exploring these features more in depth, check back next week for our 5 Days of Kubernetes series where we’ll highlight detailed walkthroughs of the following features:

• Day 1 – Kubelet TLS Bootstrap
• Day 2 – Support for Azure Virtual Machine Scale Sets (VMSS) and Cluster-Autoscaler
• Day 3 – Snapshots Functionality
• Day 4 – RuntimeClass
• Day 5 – Topology Resources

Release team

This release is made possible through the effort of hundreds of individuals who contributed both technical and non-technical content. Special thanks to the release team led by Tim Pepper, Orchestration & Containers Lead, at VMware Open Source Technology Center. The 36 individuals on the release team coordinate many aspects of the release, from documentation to testing, validation, and feature completeness.

As the Kubernetes community has grown, our release process represents an amazing demonstration of collaboration in open source software development. Kubernetes continues to gain new users at a rapid clip. This growth creates a positive feedback cycle where more contributors commit code creating a more vibrant ecosystem. Kubernetes has over 22,000 individual contributors to date and an active community of more than 45,000 people.

Project Velocity

The CNCF has continued refining DevStats, an ambitious project to visualize the myriad contributions that go into the project. K8s DevStats illustrates the breakdown of contributions from major company contributors, as well as an impressive set of preconfigured reports on everything from individual contributors to pull request lifecycle times. On average, 259 different companies and over 1,400 individuals contribute to Kubernetes each month. Check out DevStats to learn more about the overall velocity of the Kubernetes project and community.

User Highlights

Established, global organizations are using Kubernetes in production at massive scale. Recently published user stories from the community include:

• Ygrene, a PACE (Property Assessed Clean Energy) financing company, is using cloud native to bring security and scalability to the finance industry, cutting deployment times down to five minutes with Kubernetes.
• Sling TV, a live TV streaming service, uses Kubernetes to enable their hybrid cloud strategy and deliver a high-quality service for their customers.
• ING, a Dutch multinational banking and financial services corporation, moved to Kubernetes with the intent to eventually be able to go from idea to production within 48 hours.
• Pinterest, a web and mobile application company that is running on 1,000 microservices and hundreds of thousands of data jobs, moved to Kubernetes to build on-demand scaling and simply the deployment process.
• Pearson, a global education company serving 75 million learners, is using Kubernetes to transform the way that educational content is delivered online and has saved 15-20% in developer productivity.

Is Kubernetes helping your team? Share your story with the community.

Ecosystem Updates

• CNCF recently released the findings of their bi-annual CNCF survey, finding that the use of cloud native technologies in production has grown over 200% within the last six months.
• CNCF expanded its certification offerings to include a Certified Kubernetes Application Developer exam. The CKAD exam certifies an individual’s ability to design, build, configure, and expose cloud native applications for Kubernetes. More information can be found here.
• CNCF added a new partner category, Kubernetes Training Partners (KTP). KTPs are a tier of vetted training providers who have deep experience in cloud native technology training. View partners and learn more here.
• CNCF also offers online training that teaches the skills needed to create and configure a real-world Kubernetes cluster.
• Kubernetes documentation now features user journeys: specific pathways for learning based on who readers are and what readers want to do. Learning Kubernetes is easier than ever for beginners, and more experienced users can find task journeys specific to cluster admins and application developers.

KubeCon

The world’s largest Kubernetes gathering, KubeCon + CloudNativeCon is coming to Shanghai from November 13-15, 2018 and Seattle from December 10-13, 2018. This conference will feature technical sessions, case studies, developer deep dives, salons and more! Register today!

Webinar

Join members of the Kubernetes 1.12 release team on November 6th at 10am PDT to learn about the major features in this release. Register here.

Get Involved

The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly community meeting, and through the channels below.

Thank you for your continued feedback and support.

• Post questions (or answer questions) on Stack Overflow
• Join the community portal for advocates on K8sPort
• Follow us on Twitter @Kubernetesio for latest updates
• Chat with the community on Slack
• Share your Kubernetes story

Source

8/Mar 2018

By Hannah Morris

Introduction

As ever, the Jetstack team are incredibly busy. Recent months have seen back-to-back Kubernetes consulting, training and open source development, as more and more companies adopt Kubernetes in order to meet the demands of their business.

It has to be said that at Jetstack we are scaling to meet the demands of our business: Just 3 months into 2018, and we have already grown by 3 members! We are delighted to welcome to our team Matt (yes, another!), Charlie and Simon, who all recently joined our London office.

Matt Turner

Solutions Engineer, London

Matt has been a software engineer for 10 years, following a degree in Computer Science. He started with embedded systems, moved to development tools, then orchestrated VMs for Cisco. Most recently at Skyscanner, he introduced Kubernetes and cloud-native practices. His idea of “full-stack” is POSIX, Kubernetes, and now Istio.

It would be true to say that Matt enjoys travelling: He has been on holiday to Chernobyl, caving in Cuba, and he went to Comicon San Diego last year (outfitless, apparently). He once helped a friend move to Sweden by driving there in an £800 van. When asked about his music taste, Matt expresses his regret at not having seen S Club 7 live, but he has seen Steps and the Vengaboys. And Slayer. Oh, and he once tuned an engine so much it melted.

Charlie Egan

Solutions Engineer, London

Charlie comes from an application development background, but he’s been tinkering with containers on the side for a few years. He’s interested in how container tech can improve the developer experience and reduce anxiety around deploying software, and is looking forward to working with the team at Jetstack.

Away from the keyboard, Charlie enjoys ‘Parkrunning’ and posting pretty pictures on Instagram. He also makes a good homemade soup, which has helped him through his first weeks at Jetstack HQ during the snowy spells. Charlie is from Scotland and makes fairly regular trips home on the Caledonian Sleeper, which he promotes at almost every opportunity.

Simon Parker

Head of Growth, London

Simon has over 11 years experience in the industry working for companies that include IBM, Accenture and Cloud Sherpas. He has been helping new startups disrupt and large enterprise organisations adapt to remain relevant through digital transformation. Although primarily focused on building strategic partnerships with customers, marketing and business growth, Simon is a technologist at heart and can often be found spinning up containers to test open source solutions to his problems!

In previous roles Simon travelled extensively around Europe, but he also managed to escape from working life long enough to spend some time with an indigenous tribe in Peru. He is a self-proclaimed Adrenaline Junkie, and is currently the proud owner of a Suzuki motorbike. He also sits on the board for two charities, where he shares his passion for technology in helping them leverage building tools with AI/Machine learning and releasing a mobile app.

Join us!

hello@jetstack.io

The Jetstack team is growing as we take on a variety of challenging customer projects, training programmes and open source development. We have an exciting and busy road ahead, and if you’d like to be a part of it please contact us.

We are currently looking for Solutions and Operations Engineers in UK/EU. Read about what it’s like to be a Solutions Engineer at Jetstack.

Source

Congrats team! Seattle Business’ Washington’s 100 Best Companies to Work For has recognized Heptio as #5 in its 2018 list of great workplaces. This vote came directly from our employees and I am so incredibly proud to be a part of this amazing team.

At Heptio, we’re working hard to build a culture with intention. In hand with our technical expertise, our team and values have been a growth accelerant. Even as we quadrupled headcount to support our customers’ journeys, we’ve been careful to cultivate a culture that we believe is helping us grow the right way — starting from our values. (1) Carry the fire, (2) Honest technology and (3) We before me

These values permeate everything we do, including a couple of areas where we believe Heptio stands apart:

Attract diverse perspectives [We before me]. We place a huge emphasis on fostering diversity inside and outside our company. Our founders built diversity into the foundation of this company and it has been a focus from the beginning. We take equal pay seriously and built our compensation bands early in the company lifecycle to ensure we enforce equal pay for equal work. We think fostering a diverse community is also important so have sponsored diversity panels at conferences and had Heptonians talk about what it means to them. For KubeCon Austin 2017, our own Kris Nova fundraised trips for 100 diversity attendees. And for each download of our Gartner Cool Vendor report, we make a donation to Black Girls Code.

Obsess over communication [Honest technology]. Our CEO, Craig McLuckie, sends an email to all Heptionians every Sunday. We hold an all-hands every second Monday where we talk about everything going well and things we need to work on. We also make sure to talk about how we are spending our money, everyone is an owner so it is important we all understand how we are doing. Each Wednesday, the team, including remote folks, gathers in-person and via conferencing for an hour-long water cooler break where we shoot the breeze. When we do a breakfast celebration at HQ, we send coffee gift cards to remote Heptionians so they can toast with us. We’re not perfect, but we obsess over communication to create a strong connection with all employees.

Have a purpose [Carry the fire]. Heptionians have a purpose and are motivated to drive change. We donate ideas and projects to keep Kubernetes open source because we believe this model best serves the user community. We draw inspiration from our rapid growth ‒ facilitated by partners and customers who value our approach. Our collaborations with Amazon and Microsoft underscore the appeal of open source Kubernetes even to tech titans who have deep resources to build proprietary alternatives. And our customers share incredible feedback on the tools, training, services and support we bring to unlock the full potential of Kubernetes.

A key to shaping our culture is working with others who can be themselves. Given our values we have attracted some amazing, outspoken people, and so we work together to create an environment where we can engage others and advocate for the causes that matter in our lives. We’re growing fast and we need more people who care deeply about their work and life. If you are interested in joining us on our mission, please take a look at our openings.

Ok, in true Heptio fashion, that’s enough time celebrating … we’ve got more work to do.

Source