Art2Dec SoftLab

9/Oct 2018

By Charlie Egan

Introduction

Our Kubernetes training programme forms a considerable part of our services at Jetstack. In 2017 alone we trained more than 1,000 engineers from over 50 different companies, and so far in 2018 we have already delivered over 60 courses. We are constantly making an effort to ensure that our course content is refined and up-to-date, and that it reflects both the real-world experience of our engineers and also the evolving Kubernetes ecosystem. 2018 has seen us dedicate a lot of time to our training programme: As well as maintaining our current courses, we have developed online resources available through Flight Deck, the Jetstack Subscription customer portal, for self-paced learning.

However, we had a recurring theme in feedback from attendees and customers. As more and more of them deployed Kubernetes, they wanted to learn how to make best use of the features it has to offer. For this reason, we decided to build the Application Developer’s course, to be announced at Google Cloud Next London ‘18. This blog post details our motivation for building the course, as well as some of the main topics.

Why Application Development?

Jetstack’s current course offering is largely aimed at those in an operational role – deploying and managing clusters. After the beginner level course, we immediately get stuck into more ‘cluster admin’ type tasks as well as the internals of Kubernetes itself.

With this new course, we’re introducing a new ‘track’ for a developer audience. This course is for developers building and architecting applications to be deployed on a Kubernetes-based platform.

Agenda

The course is very hands-on and entirely based around workshops where attendees extend a simple application to explore Kubernetes from an application developer’s perspective.

We start the day with a number of workshops around better integrating your applications with Kubernetes, covering topics such as probes and managing pod environment. The course then progresses to cover a number of features that are specifically useful to application developers, including CronJobs, Lifecycle Hooks and Init Containers.

Further workshops are designed to be selected based on the interests of attendees with more in-depth workshops available on the following topics:

• Wrapping an application in a Helm chart
• Developing in-cluster with skaffold
• Debugging with kubectl
• Connecting to an database deployed with Helm
• Logging and custom metrics

Kubernetes features for Application Developers

One highlight is the workshops covering features that can directly reduce complexity versus a more traditional deployment.

As an example, CronJobs greatly simplify the common problem of running cron for recurring jobs on a fleet of autoscaling application instances. Solutions to handle duplicate jobs and the issues that arise from this architecture are often handmade and a bit of a hack. In this workshop we see how this is much simpler leveraging Kubernetes to run new containers for jobs on a schedule.

Recurring jobs are a feature that a huge number of applications need. Kubernetes offers a language-agnostic standard for running this type of work – it’s really valuable.

Conclusion

If you think that you and your team would benefit from this course, contact us to take part in the pilot scheme. We will be running one-day courses from November 2018.

Source

It has been a truly amazing quarter for Heptio. Kubernetes has emerged as the de facto standard for container orchestration and Heptio has stepped up as a partner to many Fortune 500 companies on their cloud native journey. It is hard to believe that we started the company less than two years ago given the size and strength of the team — we recently passed our 100 employee threshold.

Today we were listed as a winner of the CNBC Upstart 100 award, following on the heels of being recognized with the Seattle Tech Impact Gold Award (in the cloud category). These accolades are greatly appreciated and are a credit to the amazing folks that joined our company. Our collective mission is to make enterprise developers more intrinsically productive and to empower operators. And we believe that the best way to deliver that impact is through a highly consistent multi-cloud and on-premises computing environment without fear of vendor lock-in.

Since announcing HKS (Heptio Kubernetes Subscription) earlier this year, Heptio has been selected as a partner to organizations large and small to help them understand the technology, align it with their internal operating practices and provide the mission critical support needed to bring it into production. We are humbled that a company of our age and size has a customer base that includes 3 of the 4 largest retailers in the world, 4 of the 5 largest telecommunication companies in the United States, and several global financial services institutions. Working alongside these customers has been immensely rewarding and their experiences fuel our future direction.

Looking forward we will continue to drive innovation in the open source ecosystem, and will continue to invest heavily in the OSS projects we have already created (collectively they have garnered more than 4700 stars on Github). We have established a strong release cadence with our existing technologies (with recent community excitement around the release of Ark 0.9, and Contour 0.6). You can expect to see some interesting new capabilities being shipped in the not too distant future.

A big thank you to the whole team at Heptio for another successful quarter. I could not be more excited about the coming months and years!

Source

Image attribution below

So you’ve finally succeeded in convincing your organization to use Kubernetes and you’ve even gotten first services in production. Congratulations!

You know uptime of your production workloads is of utmost importance so you set up your production cluster(s) to be as reliable as possible. You add all kinds of monitoring and alerting, so that if something breaks your SREs get notified and can fix it with the highest priority.

But this is expensive and you want to have staging and development clusters, too – maybe even some playgrounds. And as budgets are always tight, you start thinking…

What’s with DEV? Certainly can’t be as important as PROD, right? Wrong!

The main goal with all of these nice new buzzwordy technologies and processes was Developer Productivity. We want to empower developers and enable them to ship better software faster.

But if you put less importance on the reliability of your DEV clusters, you are basically saying “It’s ok to block my developers”, which indirectly translates to “It’s ok to pay good money for developers (internal and external) and let them sit around half a day without being able to work productively”.

Ah yes, the SAP DEV Cluster is also sooo important because of that many external and expensive consultants. Fix DEV first, than PROD which is earning all the money.

— Andreas Lehr (@shakalandy) September 13, 2018

Furthermore, no developer likes to hear that they are less important than your customers.

We consider our dev cluster a production environment, just for a different set of users (internal vs external).

— Greg Taylor (@gctaylor) September 18, 2018

What could go wrong?

Let’s look at some of the issues you could run into, when putting less importance on DEV, and the impact they might have.

I did not come up with these. We’ve witnessed these all happen before over the last 2+ years.

Scenario 1: K8s API of the DEV cluster is down

Your nicely built CI/CD pipeline is now spitting a mountain of errors. Almost all your developers are now blocked, as they can’t deploy and test anything they are building.

This is actually much more impactful in DEV than in production clusters as in PROD your most important assets are your workloads, and those should still be running when the Kubernetes API is down. That is, if you did not build any strong dependencies on the API. You might not be able to deploy a new version, but your workloads are fine.

Scenario 2: Cluster is full / Resource pressure

Some developers are now blocked from deploying their apps. And if they try (or the pipeline just pushes new versions), they might increase the resource pressure.

Pods start to get killed. Now your priority and QoS classes kick in – you did remember to set those, right? Or was that something that was not important in DEV? Hopefully, you have at least protected your Kubernetes components and critical addons. If not, you’ll see nodes going down, which again increases resource pressure. Thought DEV clusters could do with less buffer? Think again.

This sadly happens much more in DEV because of two things:

1. Heavy CI running in DEV
2. Less emphasis on clean definition of resources, priorities, and QoS classes.

Scenario 3: Critical addons failing

In most clusters, CNI and DNS are critical to your workloads. If you use an Ingress Controller to access them, then that counts also as critical. You’re really cutting edge and are already running a service mesh? Congratulations, you added another critical component (or rather a whole bunch of them – looking at you Istio).

Now if any of the above starts having issues (and they do partly depend on each other), you’ll start seeing workloads breaking left and right, or, in the case of the Ingress Controller, them not being reachable outside the cluster anymore. This might sound small on the impact scale, but just looking at our past postmortems, I must say that the Ingress Controller (we run the community NGINX variant) has the biggest share of them.

What happened?

A multitude of thinkable and unthinkable things can happen and lead to one of the scenarios above.

Most often we’ve seen issues arising because of misconfiguration of workloads. Maybe you’ve seen one of the below (the list is not conclusive).

• CI is running wild and filling up your cluster with Pods without any limits set
• CI “DoSing” your API
• Faulty TLS certs messing up your Ingress Controller
• Java containers taking over whole nodes and killing them
• …

Sharing DEV with a lot of teams? Gave each team cluster-admin rights? You’re in for some fun. We’ve seen pretty much anything, from “small” edits to the Ingress Controller template file, to someone accidentally deleting the whole cluster.

Conclusion

If it wasn’t clear from the above: DEV clusters are important!

Just consider this: If you use a cluster to work productively then it should be considered similarly important in terms of reliability as PROD.

DEV clusters usually need to be reliable at all times. Having them reliable only at business hours is tricky. First, you might have distributed teams and externals working at odd hours. Second, an issue that happens at off-hours might just get bigger and then take longer to fix once business hours start. The latter is one of the reasons why we always do 24/7 support, even if we could offer only business hours support for a cheaper price.

Some things you should consider (not only for DEV):

• Be aware of issues with resource pressure when sizing your clusters. Include buffers.
• Separate teams with namespaces (with access controls!) or even different clusters to decrease the blast radius of misuse.
• Configure your workloads with the right requests and limits (especially for CI jobs!).
• Harden your Kubernetes and Addon components against resource pressure.
• Restrict access to critical components and do not give out cluster-admin credentials.
• Have your SREs on standby. That means people will get paged for DEV.
• If possible enable your developers to easily rebuild DEV or spin up clusters for development by themselves.

Why don’t devs have the capability to rebuild dev 🤷‍♂️

— Chris Love (@chrislovecnm) September 14, 2018

If you really need to save money, you can experiment with downscaling in off-hours. If you’re really good at spinning up or rebuilding DEV, i.e. have it all automated from cluster creation to app deployments, then you could experiment with “throw-away-clusters”, i.e. clusters that get thrown away at the end of the day and start a new shortly before business hours.

Whatever you decide, please, please, please, do not block your developers, they will be much happier, and you will get better software, believe me.

P.S. Thanks to everyone responding and giving feedback on Twitter!

Image attribution:
Image created using https://xkcd-excuse.com/ by Mislav Cimperšak.
Original image created by Randall Munroe from XKCD. Released under Creative Commons Attribution-NonCommercial 2.5 License.

Source

Aug 17, 2018

by Catalin Jora

Cloud computing changed the required skills for software engineers and system administrators. IT departments that didn’t continuously researched and adopted new technologies are usually caught off-guard when they decide to “move to the cloud”. A cloud native IT department requires new skills as the software they have will need refactoring and repackaging to run and take advantage of the new environment, the cloud. About those skills and responsibilities I’ll be speaking in this article. Internally, we call this role the cloud native engineer*.

A cloud native engineer needs to take an application from a repository through a continuous integration and delivery pipeline. During this journey, she would make sure that the code builds, passes the tests, gets deployed safely and secure up to the to the production environment.

The list of relevant technologies in the cloud native context is overwhelming:

But don’t worry, you don’t need to know them all. You’ can’t either have “3+ years of production experience” as some naive recruiters would expect, because it’s a very new field. By the way, there’s another map for serverless.

So let’s go one by one over the required building blocks of every cloud project, the bread and butter of the cloud native engineer skillset**.

Software Architecture (from monoliths to microservices and serverless)

You’ll need to figure out if you’re dealing with a micro-service architecture or with a monolith. If you’re not assigned to a greenfield project, most of the time you’ll have to move a project from a monolith to microservices. Missing the knowledge about microservices? Check a microservice-demo app here: https://microservices-demo.github.io
To understand what microservices are all about, read Martin Fawler’s article on the subject. Sam Newman’s book is the “bible” in this domain.

Building the software (development and build scripts)

Once you know with what type of architecture you’re dealing, you’ll need some basic understanding on how to run and build that project. If you need to adjust/debug the application, experience or familiarity with the programming language is required.

Packaging (containers: Docker | Serverless)

In the cloud native context, applications get delivered as containers (serverless is gaining traction as well). So you’ll need to be able to understand and package applications into these new standards. My colleague’s (Adrian Mouat) Using Docker book is probably your best option for getting up to speed.

Continuous integration/delivery (CI/CD tools like Jenkins, GitLab, Travis and friends)

Once the code (both applications and infrastructure) is written, you will need to deploy it somewhere. Most of the “old” continuous integration tools are still alive and kicking and they’ve learned how to do continuous delivery. New tools are emerging, but it looks like Jenkins is still the standard, mainly due to poor alternatives. The CI tool will drive the entire delivery process, so designing a good pipeline will result in a better developer experience and faster response to failures/bugs. All types of tests and code quality tools will have to get integrated here as well.

Container Schedulers and Orchestrators (Kubernetes, Docker Swarm, DC/OS)

If you package your application in containers you’ll need to understand their lifecycle. Kubernetes is becoming the standard in this space, with managed solutions from all major cloud providers. Go over the basic tutorials or look for a training if you intend to go deeper into the subject, as this will be your new playground once the apps will get deployed.

Building the infrastructure (the operations part)

OK, you have some application code that works. Your next step (for greenfield projects it’s the first step) is to build the infrastructure where that code will run. This involves a mix of managed services and writing code for the missing parts. Tools like Terraform, Cloud Formation, Ansible are the usual suspects in this area.

Monitoring and observability

Monitoring plays a central role in a distributed system. The old tools didn’t manage to catch up with the new landscape, so you’ll need to look into SaaS solutions, the default services offered by your cloud, or rely on open source projects (like Prometheus/Grafana).

Cloud computing building blocks

For most companies, the infrastructure will get replaced by an API, some credentials and a bill that will track the usage. Understanding of cloud computing primitives like compute units, databases, storage, network for whatever cloud/infrastructure provider you are using is mandatory. Understanding pricing for the new services will help you avoid nasty surprises at the end of the month.

Once your IT department implements those block, there are other areas that usually gain attention:

• Deployment patterns for your application
  • Canary deployments
  • Blue/Green
  • GitOps
  • Networking for microservice architecture
• Service mesh (istio, envoy)
• Security for cloud architecture
  • container vulnerability scans
• Secrets management

The shift to cloud computing reflects as well in the technical skills required to develop and operate the applications on this new platforms. Continuous learning is the only way you can keep up. If you can’t do it at your current workplace, probably it’s time to look for a new challenge somewhere else.

Notes

*The skillset and responsibility of this role are different than the Google’s Site Reliability Engineer role: “SRE is what happens when you ask a software engineer to design an operations team” – Benjamin Treynor Sloss – SRE Founder. The DevOps Engineer is a weird name/role that is too vague and always in flux to make comparisons, but it does overlap on some of the skills.

**While shorter than the cloud native landscape, this list is overwhelming as well. I initially assembled it for a colleague (hi Riccardo!) but it is useful for everybody who wants to make a career step in this direction. Most of the people will know the basics and have deep knowledge in a few areas that interest them most.

Looking for a new challenge? We’re hiring!

Source

Sep 19, 2018

by José Moreira

The approach for deploying a containerised application typically involves building a Docker container image from a Dockerfile once and deploying that same image across several deployment environments (development, staging, testing, production).

If following security best practices, each deployment environment will require a different configuration (data storage authentication credentials, external API URLs) and the configuration is injected into the application inside the container through environment variables or configuration files. Our Hamish Hutchings takes a deeper look at 12-factor app in this blog post. Also, the possible configuration profiles might not be predetermined, if for example, the web application should be ready to be deployed both on public or private cloud (client premises), as it is also common for several configuration profiles to be added to source code and the required profile to be loaded at build time.

The structure of a web application project typically contains a ‘src’ directory with source code and executing npm run-script build triggers the Webpack asset build pipeline. Final asset bundles (HTML, JS, CSS, graphics, and fonts) are written to a dist directory and contents are either uploaded to a CDN or served with a web server (NGINX, Apache, Caddy, etc).

For context in this article, let’s assume the web application is a single-page frontend application which connects to a backend REST API to fetch data, for which the API endpoint will change across deployment environments. The backend API endpoint should, therefore, be fully configurable and configuration approach should support both server deployment and local development and assets are served by NGINX.

Deploying client-side web application containers requires a different configuration strategy compared to server-side applications containers. Given the nature of client-side web applications, there is no native executable that can read environment variables or configuration files in runtime, the runtime is the client-side web browser and configuration has to be hard-coded in the Javascript source code either by hard-coding values during the asset build phase or hard-coding rules (a rule would be to deduce current environment based on the domain name, ex: ‘staging.app.com’).

There is one OS process which is relevant to configuration in which reading values from environment values is useful, which is the asset build Node JS process and this is helpful for configuring the app for local development with local development auto reload.

For the configuration of the webapp across several different environments, there are a few solutions:

1. Rebuild the Webpack assets on container start during each deployment with the proper configuration on the destination server node(s):
   • Adds to deployment time. Depending on deployment rate and size of the project, the deployment time overhead might be considerable.
   • Is prone to build failures at end of deployment pipeline even if the image build has been tested before.
   • Build phase can fail for example due to network conditions although this can probably be minimised by building on top of a docker image that already has all the dependencies installed.
   • Might affect rollback speed
2. Build one image per environment (again with hardcoded configuration):
   • Similar solutions (and downsides) of solution #1 except that it adds clutter to the docker registry/daemon.
3. Build image once and rewrite configuration bits only during each deployment to target environment:
   • Image is built once and ran everywhere. Aligns with the configuration pattern of other types of applications which is good for normalisation
   • Scripts that rewrite configuration inside the container can be prone to failure too but they are testable.

I believe solution #1 is viable and, in some cases, simpler and probably required if the root path where the web application is hosted needs to change dynamically, ex.: from ‘/’ to ‘/app’, as build pipelines can hardcode the base path of fonts and other graphics in CSS files with the root path, which is a lot harder to change post build.

Solution #3 is the approach I have been implementing for the projects where I have been responsible for containerising web applications (both at my current and previous roles), which is the solution also implemented by my friend Israel and for which he helped me out implementing the first time around and the approach that will be described in this article.

Application-level configuration

Although it has a few moving parts, the plan for solution #3 is rather straightforward:

For code samples, I will utilise my fork of Adam Sandor micro-service Doom web client, which I have been refactoring to follow this technique, which is a Vue.js application. The web client communicates with two micro-services through HTTP APIs, the state and the engine, endpoints which I would like to be configurable without rebuilding the assets.

Single Page Applications (SPA) have a single “index.html” as a entry point to the app and during deployment. meta tags with optional configuration defaults are added to the markup from which application can read configuration values. script tags would also work but I found meta tags simple enough for key value pairs.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

<!DOCTYPE html> <html lang=”en”> <head> <meta charset=”utf-8″> <meta http-equiv=”X-UA-Compatible” content=”IE=edge”> <meta name=”viewport” content=”width=device-width,initial-scale=1.0″> <meta property=”DOOM_STATE_SERVICE_URL” content=”http://localhost:8081/” /> <meta property=”DOOM_ENGINE_SERVICE_URL” content=”http://localhost:8082/” /> <link rel=”icon” href=”./favicon.ico”> <title>frontend</title> </head> <body> <noscript> <strong>We’re sorry but frontend doesn’t work properly without JavaScript enabled. Please enable it to continue.</strong> </noscript> <div id=”app”></div> <!– built files will be auto injected –> </body> </html>

For reading configuration values from meta tags (and other sources), I wrote a simple Javascript module (“/src/config.loader.js”):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

/** * Get config value with precedence: * – check `process.env` * – check current web page meta tags * @param key Configuration key name */ function getConfigValue (key) { let value = null if (process.env && process.env[`$`] !== undefined) { // get env var value value = process.env[`$`] } else { // get value from meta tag return getMetaValue(key) } return value } /** * Get value from HTML meta tag */ function getMetaValue (key) { let value = null const node = document.querySelector(`meta[property=$]`) if (node !== null) { value = node.content } return value } export default { getConfigValue, getMetaValue }

This module will read configuration “keys” by looking them up in the available environment variables (“process.env”) first, so that configuration can be overridden with environment variables when developing locally (webpack dev server) and then the current document meta tags.

I also abstracted the configuration layer by adding a “src/config/index.js” that exports an object with the proper values:

import loader from ‘./loader’ export default { DOOM_STATE_SERVICE_URL: loader.getConfigValue(‘DOOM_STATE_SERVICE_URL’), DOOM_ENGINE_SERVICE_URL: loader.getConfigValue(‘DOOM_ENGINE_SERVICE_URL’) }

which can then be utilised in the main application by importing the “src/config” module and accessing the configuration keys transparently:

import config from ‘./config’ console.log(config.DOOM_ENGINE_SERVICE_URL)

There is some room for improvement in the current code as it not DRY (list of required configuration variables is duplicated in several places in the project) and I’ve considered writing a simple Javascript package to simplify this approach as I’m not aware if something already exists. Writing the Docker & Docker Compose files The Dockerfile for the SPA adds source-code to the container to the ‘/app’ directory, installs dependencies and runs a production webpack build (“NODE_ENV=production”). Assets bundles are written to the “/app/dist” directory of the image:

FROM node:8.11.4-jessie RUN mkdir /app WORKDIR /app COPY package.json . RUN npm install COPY . . ENV NODE_ENV production RUN npm run build CMD npm run dev

The docker image contains a Node.js script (“/app/bin/rewrite-config.js”) which copies “/app/dist” assets to another target directory before rewriting the configuration. Assets will be served by NGINX and therefore copied to a directory that NGINX can serve, in this case, a shared (persistent) volume. Source and destination directories can be defined through container environment variables:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

#!/usr/bin/env node const cheerio = require(‘cheerio’) const copy = require(‘recursive-copy’) const fs = require(‘fs’) const rimraf = require(‘rimraf’) const DIST_DIR = process.env.DIST_DIR const WWW_DIR = process.env.WWW_DIR const DOOM_STATE_SERVICE_URL = process.env.DOOM_STATE_SERVICE_URL const DOOM_ENGINE_SERVICE_URL = process.env.DOOM_ENGINE_SERVICE_URL // – Delete existing files from public directory // – Copy `dist` assets to public directory // – Rewrite config meta tags on public directory `index.html` rimraf(WWW_DIR + ‘/*’, {}, function() { copy(`$`, `$`, , function(error, results) { if (error) { console.error(‘Copy failed: ‘ + error); } else { console.info(‘Copied ‘ + results.length + ‘ files’); rewriteIndexHTML(`$/index.html`, { DOOM_STATE_SERVICE_URL: DOOM_STATE_SERVICE_URL, DOOM_ENGINE_SERVICE_URL: DOOM_ENGINE_SERVICE_URL }) } }); }) /** * Rewrite meta tag config values in “index.html”. * @param file * @param values */ function rewriteIndexHTML(file, values) { console.info(`Reading ‘$’`) fs.readFile(file, ‘utf8’, function (error, data) { if (!error) { const $ = cheerio.load(data) console.info(`Rewriting values ‘$’`) for (let [key, value] of Object.entries(values)) { console.log(key, value); $(`[property=$]`).attr(“content”, value); } fs.writeFile(file, $.html(), function (error) { if (!error) { console.info(`Wrote ‘$’`) } else { console.error(error) } }); } else { console.error(error) } }); }

The script utilises CheerioJS to read the “index.html” into memory, replaces values of meta tags according to environment variables and overwrites “index.html”. Although “sed” would have been sufficient for search & replace, I chose CherioJS as a more reliable solution that also allows expanding into more complex solutions like script injections.

Deployment with Kubernetes

Let’s jump into the Kubernetes Deployment manifest:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

— # Source: doom-client/templates/deployment.yaml apiVersion: apps/v1beta2 kind: Deployment metadata: name: doom-client labels: name: doom-client spec: replicas: 1 selector: matchLabels: name: doom-client template: metadata: labels: name: doom-client spec: initContainers: – name: doom-client image: “doom-client:latest” command: [“/app/bin/rewrite-config.js”] imagePullPolicy: IfNotPresent env: – name: DIST_DIR value: “/app/dist” – name: WWW_DIR value: “/tmp/www” – name: DOOM_ENGINE_SERVICE_URL value: “http://localhost:8081/” – name: DOOM_STATE_SERVICE_URL value: “http://localhost:8082/” volumeMounts: – name: www-data mountPath: /tmp/www containers: – name: nginx image: nginx:1.14 imagePullPolicy: IfNotPresent volumeMounts: – name: www-data mountPath: /usr/share/nginx/html – name: doom-client-nginx-vol mountPath: /etc/nginx/conf.d volumes: – name: www-data emptyDir: {} – name: doom-client-nginx-vol configMap: name: doom-client-nginx

The Deployment manifest defines an “initContainer” which executes the “rewrite-config.js” Node.js script to prepare and update the shared storage volume with the asset bundles. It also defines an NGINX container for serving our static assets. Finally, it also creates a shared Persistent Volume which is mounted on both of the above containers. In the NGINX container the mount point is “/var/www/share/html” but on the frontend container “/tmp/www” for avoiding creating extra directories. “/tmp/www” will be the directory where the Node.js script will copy asset bundles to and rewrite the “index.html”. Local development with Docker Compose

The final piece of our puzzle is the local Docker Compose development environment. I’ve included several services that allow both developing the web application with the development server and testing the application when serving production static assets through NGINX. It is perfectly possible to separate these services into several YAML files (“docker-compose.yaml”, “docker-compose.dev.yaml” and “docker-compose.prod.yaml”) and do some composition but I’ve added a single file for the sake of simplicity.

Apart from the “doom-state” and “doom-engine” services which are our backend APIs, the “ui” service starts the webpack development server with “npm run dev” and the “ui-deployment” service, which runs a container based on the same Dockerfile, runs the configuration deployment script. The “nginx” service serves static assets from a persistent volume (“www-data”) which is also mounted on the “ui-deployment” script.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

# docker-compose.yaml version: ‘3’ services: ui: build: . command: [“npm”, “run”, “dev”, ] ports: – “8080:8080” environment: – HOST=0.0.0.0 – PORT=8080 – NODE_ENV=development – DOOM_ENGINE_SERVICE_URL=http://localhost:8081/ – DOOM_STATE_SERVICE_URL=http://localhost:8082/ volumes: – .:/app # bind volume inside container for source mount not shadow image dirs – /app/node_modules – /app/dist doom-engine: image: microservice-doom/doom-engine:latest environment: – DOOM_STATE_SERVICE_URL=http://doom-state:8080/ – DOOM_STATE_SERVICE_PASSWORD=enginepwd ports: – “8081:8080” doom-state: image: microservice-doom/doom-state:latest ports: – “8082:8080” # run production deployment script ui-deployment: build: . command: [“/app/bin/rewrite-config.js”] environment: – NODE_ENV=production – DIST_DIR=/app/dist – WWW_DIR=/tmp/www – DOOM_ENGINE_SERVICE_URL=http://localhost:8081/ – DOOM_STATE_SERVICE_URL=http://localhost:8082/ volumes: – .:/app # bind volume inside container for source mount not shadow image dirs – /app/node_modules – /app/dist # shared NGINX static files dir – www-data:/tmp/www depends_on: – nginx # serve docker image production build with nginx nginx: image: nginx:1.14 ports: – “8090:80” volumes: – www-data:/usr/share/nginx/html volumes: www-data:

Since the webpack dev server is a long running process which also hot-reloads the app on source code changes, the Node.js config module will yield configuration from environment variables, based on the precedence I created. Also, although source code changes can trigger client-side updates without restarts (hot reload), it will not update the production build, which has to be manual but straightforward with a $ docker-compose build && docker-compose up.

Summarizing, although there are a few improvements points, including on the source code I wrote for this implementation, this setup has been working pretty well for the last few projects and is flexible enough to also support deployments to CDNs, which is as simple as adding a step for pushing assets to the cloud instead of a shared volume with NGINX.

If you have any comments feel free to get in touch on Twitter or comment under the article.

Download our free whitepaper, Kubernetes: Crossing the Chasm below.

Source

 

Expert Training in Kubernetes and Rancher

Join our free online training sessions to learn how to manage Kubernetes workloads with Rancher.

Sign up here

Docker vs Kubernetes: The Journey from Docker to Kubernetes

The need to deploy applications from one computing environment to another quickly, easily, and reliably has become a critical part of enterprise’s business requirements and DevOps team’s daily workflow.

It’s unsurprising then, that container technologies, which make application deployment and management easier for teams of all sizes, have risen dramatically in recent years. At the same time, however, virtual machines (VM) as computing resources have reached their peak use in virtualized data centers. Since VMs existed long before containers, you may wonder what the need isfor containers and why they have become so popular.

The Benefits and Limitations of Virtual Machines

Virtual machines allow you to run a full copy of an operating system on top of virtualized hardware as if it is a separate machine. In cloud computing, physical hardware on a bare metal server are virtualized and shared between virtual machines running on a host machine in a data center by the help of hypervisor (i.e. virtual machine manager).

Even though virtual machines bring us great deal of advantages such as running different operating systems or versions, VMs can consume a lot of system resources and also take longer boot time. On the other hand, containers share the same operating system kernel with collocated containers each one running as isolated processes. Containers are lightweight alternative by taking up less space (MBs) and can be provisioning rapidly (milliseconds) as opposed to VM’s slow boot time (minutes) and more storage space requirements (GBs). This allows containers to operate at an unprecedented scale and maximize the number of applications running on minimum number of servers. Therefore, containerization shined drastically in the recent years because of all these advantages for many software projects of enterprises.

Since its initial release in 2013, Docker has become the most popular container technology worldwide, despite a host of other options, including RKT from CoreOS, LXC, LXD from Canonical, OpenVZ, and Windows Containers.

However, Docker technology alone is not enough to reduce the complexity of managing containerized applications, as software projects get more and more complex and require the use tens of thousands of Docker containers. To address these larger container challenges, substantial number of container orchestration systems, such as Kubernetes and Docker Swarm, have exploded onto the scene shortly after the release of Docker.

There has been some confusion surrounding Docker and Kubernetes for awhile: “what they are?”, “what they are not?”, “where are they used?”, and “why are both needed?”

This post aims to explain the role of each technology and how each technology helps companies ease their software development tasks. Let’s use a made-up company, NetPly (sounds familiar?), as a case study to highlight the issues we are addressing.

NetPly is an online and on-demand entertainment movie streaming company with 30 million members in over 100 countries. NetPly delivers video streams to your favorite devices and provides personalized movie recommendations to their customers based on their previous activities, such as sharing or rating a movie. To run their application globally, at scale, and provide quality of service to their customers, NetPly runs 15,000 production servers worldwide and follow agile methodology to deploy new features and bug fixes to the production environment at a fast clip.

However, NetPly has been struggling with two fundamental issues in their software development lifecycle:

Issue 1- Code that runs perfectly in a development box, sometimes fails on test and/or production environments. Therefore, NetPly would like to keep code and configuration consistent across their development, test, and production environments to reduce the issues arising from application hosting environments.

Issue 2- Viewers experience a lot of lags as well as poor quality and degraded performance for video streams during weekends, nights, and holidays, when incoming requests spike. To resolve this potentially-devastating issue, NetPly would like to use load-balancing and auto scaling techniques and automatically adjust the resource capacity (e.g. increase or decrease number of computing resources) to maintain application availability, provide stable application performance, and optimize operational costs as computing demand increases or decreases. These requests also require NetPly to manage the complexity of computing resources and the connections between the flood of these resources in production.

Docker can be used to resolve Issue 1 by following a container-based approach; in other words, packaging application code along with all of its dependencies, such as libraries, files, and necessary configurations, together in a Docker image.

Docker is an open-source operating system level virtualized containerization platform with a light-weight application engine to run, build and distribute applications in Docker containers that run nearly anywhere. Docker containers, as part of Docker, are portable and light-weight alternative to virtual machines, and eliminate the waste of esources and longer boot times of the virtual-machine approach. Docker containers are created using Docker images, which consist of a prebuilt application stack required to launch the applications inside the container.

With that explanation of a Docker container in mind, let’s go back our successful company that is under duress: NetPly. As more users simultaneously request movies to watch on the site, NetPly needs to scale up more Docker containers at a reasonably fast rate and scale down when the traffic lowers. However, Docker alone is not capable of taking care of this job, and writing simple shell scripts to scale the number of Docker containers up or down by monitoring the network traffic or number of requests that hit to the server would not be a viable and practicable solution.

As the number of containers increases to tens of hundreds to thousands, and the NetPly IT team starts managing fleets of containers across multiple heterogeneous host machines, it becomes a nightmare to execute Docker commands like “docker run”, “docker kill”, and “docker network” manually.

Right at the point where the team starts launching containers, wiring them together, ensuring high availability even when a host goes down, and distributing the incoming traffic to the appropriate containers, the team wishes they had something that handled all these manual tasks with no or minimal intervention. Exit human, enter program.

To sum up: Docker by itself is not enough to handle these resources demands at scale. Simple shell commands alone are not sufficient to handle tasks for a tremendous number of containers on a cluster of bare metal or virtual servers. Therefore, another solution is needed to handle all these hurdles for the NetPly team.

This is where the magic starts with Kubernetes. Kubernetes is as container orchestration engine (COE), originally developed by Google and used to resolve NetPly’s Issue 2. Kubernetes allows you to handle fleets of containers. Kubernetes automatically manages the deployment, scaling and networking of containers, as well as container failovers by launching a new one with ease.

The following are some of the fundamental features of Kubernetes.

• Load balancing
• Configuration management
• Automatic IP assignment
• Container scheduling
• Health checks and self healing
• Storage management
• Auto rollback and rollout
• Auto scaling

Container Orchestration Alternatives

Although Kubernetes seems to solve the challenges our NetPly team faces, there are a good deal of container management tool alternatives for Kubernetes out there.

Docker Swarm, Marathon on Apache Mesos, and Nomad are all container orchestration engines that can also be used for managing your fleet of containers.

Why choose anything other than Kubernetes? Although Kubernetes has a lot of great qualities, it has challenges too. The most arresting issues people face with Kubernetes are:

1) the steep learning curve to its commands;

2) setting Kubernetes up for different operating systems.

As opposed to Kubernetes, Docker Swarm uses the Docker CLI to manage all container services. Docker Swarm is easy to set up, has less commands to learn to get started rapidly, and is cheaper to train employees. A drawback of Docker Swarm bounds you to the limitations of the Docker API.

Another option is the Marathon framework on Apache Mesos. It’s extremely fault-tolerant and scalable for thousands of servers. However, it may be too complicated to set up and manage small clusters with Marathon, making it impractical for many teams.

Each container management tool comes with its own set of advantages and disadvantages. However, Kubernetes with its heritage based in Google’s Borg system, has been greatly adopted and supported by the large community as well as industry for many years and become the most popular container management solution among other players. With the power of both Docker and Kubernetes, it seems like journey of the power and popularity of these technologies will continue to rise and being adopted by even larger communities.

In our next article in this series, we will compare in more depth Kubernetes and Docker Swarm.

Faruk Caglar, PhD

Cloud Computing Researcher and Solution Architect

Faruk Caglar received his PhD from the Electrical Engineering and Computer Science Department at Vanderbilt University. He is a researcher in the fields of Cloud Computing, Big Data, Internet of Things (IoT) as well as Machine Learning and solution architect for cloud-based applications. He has published several scientific papers and has been serving as reviewer at peer-reviewed journals and conferences. He also has been providing professional consultancy in his research field.

Source

Recap

Last time we talked about PV, PVC, Storage Class and Provisioner.

To quickly recap:

1. Originally PV was designed to be a piece of storage pre-allocated by administrator. Though after the introduction of Storage Class and Provisioner, users are able to dynamically provision PVs now.

2. PVC is a request for a PV. When used with Storage Class, it will trigger the dynamic provisioning of a matching PV.

3. PV and PVC are always one to one mapping.

4. Provisioner is a plugin used to provision PV for users. It helps to remove the administrator from the critical path of creating a workload that needs persistent storage.

5. Storage Class is a classification of PVs. The PV in the same Storage Class can share some properties. In most cases, while being used with a Provisioner, it can be seen as the Provisioner with predefined properties. So when users request it, it can dynamically provision PVs with those predefined properties.

But those are not the only ways to use persistent storage in Kubernetes.

Take a deep dive into Best Practices in Kubernetes Networking

From overlay networking and SSL to ingress controllers and network security policies, we’ve seen many users get hung up on Kubernetes networking challenges. In this video recording, we dive into Kubernetes networking, and discuss best practices for a wide variety of deployment options.

Watch the video

Volume

In the previous article, I mentioned that there is also a concept of Volume in Kubernetes. In order to differentiate Volume from Persistent Volume, people sometimes call it In-line Volume, or Ephemeral Volume.

Let me quote the definition of Volume here:

A Kubernetes volume … has an explicit lifetime – the same as the Pod that encloses it. Consequently, a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts. Of course, when a Pod ceases to exist, the volume will cease to exist, too. Perhaps more importantly than this, Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

At its core, a volume is just a directory, possibly with some data in it, which is accessible to the Containers in a Pod. How that directory comes to be, the medium that backs it, and the contents of it are determined by the particular volume type used.

One important property of Volume is that it has the same lifecycle as the Pod it belongs to. It will be gone if the Pod is gone. That’s different from Persistent Volume, which will continue to exist in the system until users delete it. Volume can also be used to share data between containers inside the same Pod, but this isn’t the primary use case, since users normally only have one container per Pod.

So it’s easier to treat Volume as a property of Pod, instead of as a standalone object. As the definition said, it represents a directory inside the pod, and Volume type defines what’s in the directory. For example, Config Map Volume type will create configuration files from the API server in the Volume directory; PVC Volume type will mount the filesystem from the corresponding PV in the directory, etc. In fact, Volume is almost the only way to use storage natively inside Pod.

It’s easy to get confused between Volume, Persistent Volume and Persistent Volume Claim. So if you can imagine that there is a data flow, it will look like this: PV -> PVC -> Volume. PV contains the real data, bound to PVC, which used as Volume in Pod in the end.

However, Volume is also confusing in the sense that besides PVC, it can be backed by pretty much any type of storage supported by Kubernetes directly.

Remember we already have Persistent Volume, which supports different kinds of storage solutions. We also have Provisioner, which supports the similar (but not exactly the same) set of solutions. And we have different types of Volume as well.

So, how are they different? And how to choose between them?

Many ways of persisting data

Take AWS EBS for example. Let’s start counting the ways of persisting data in Kubernetes.

Volume Way

awsElasticBlockStore is a Volume type.

You can create a Pod, specify a volume as awsElasticBlockStore, specify the volumeID, then use your existing EBS volume in the Pod.

The EBS volume must exist before you use it with Volume directly.

PV way

AWSElasticBlockStore is also a PV type.

So you can create a PV that represents an EBS volume (assuming you have the privilege to do that), then create a PVC bound to it. Finally, use it in your Pod by specifying the PVC as a volume.

Similar to Volume Way, EBS volume must exist before you create the PV.

Provisioner way

kubernetes.io/aws-ebs is also a Kubernetes built-in Provisioner for EBS.

You can create a Storage Class with Provisioner kubernetes.io/aws-ebs, then create a PVC using the Storage Class. Kubernetes will automatically create the matching PV for you. Then you can use it in your Pod by specifying the PVC as a volume.

In this case, you don’t need to create EBS volume before you use it. The EBS Provisioner will create it for you.

Third-Party Way

All the options listed above are the built-in options of Kubernetes. There are also some third-party implementations of EBS in the format of Flexvolume driver, to help you hook it up to Kubernetes if you’re not yet satisfied by any options above.

And there are CSI drivers for the same purpose if Flexvolume doesn’t work for you. (Why? More on this later.)

VolumeClaimTemplate Way

If you’re using StatefulSet, congratulations! You now have one more way to use EBS volume with your workload – VolumeClaimTemplate.

VolumeClaimTemplate is a StatefulSet spec property. It provides a way to create matching PVs and PVCs for the Pod that Statefulset created. Those PVCs will be created using Storage Class so they can be created automatically when StatefulSet is scaling up. When a StatefulSet has been scaled down, the extra PVs/PVCs will be kept in the system. So when the StatefulSet scales up again, they will be used again for the new Pods created by Kubernetes. We will talk more on StatefulSet later.

As an example, let’s say you created a StatefulSet named www with replica 3, and a VolumeClaimTemplate named data with it. Kubernetes will create 3 Pods, named www-0, www-1, www-2 accordingly. Kubernetes will also create PVC www-data-0 for Pod www-0, www-data-1 for www-1, and www-data-2 for www-2. If you scale the StatefulSet to 5, Kubernetes will create www-3, www-data-3, www-4 and www-data-4 accordingly. Then you scale the StatefulSet down to 1, all www-1 to www-4 will be deleted, but www-data-1 to www-data-4 will remain in the system. So when you decide to scale up to 5 again, Pod www-1 to www-4 will be created, and PVC www-data-1 will still serve Pod www-1, www-data-2 for www-2, etc. That’s because the identity of Pod are stable in StatefulSet. The name and relationship are predictable when using StatefulSet.

VolumeClaimTemplate is important for the block storage solutions like EBS and Longhorn. Because those solutions are inherently ReadWriteOnce, you cannot share it between the Pods. Deployment won’t work well with them if you have more than one Pod running with persistent data. So VolumeClaimTemplate provides a way for the block storage solution to scale horizontally for a Kubernetes workload.

How to choose between Volume, Persistent Volume and Provisioner

As you see, there are built-in Volume types, PV types, Provisioner types, plus external plugins using Flexvolume and/or CSI. The most confusing part is that they just provide largely the same but also slightly different functionality.

I thought, at least, there should be a guideline somewhere on how to choose between them.

But I cannot find it anywhere.

So I’ve plowed through codes and documents, to bring you the comparison matrix, and the guideline that makes the most sense to me.
Comparison of Volume, Persistent Volume and Provisioner

Name	Volume	Persistent Volume	Provisioner
AWS EBS	✓	✓	✓
Azure Disk	✓	✓	✓
Azure File	✓	✓	✓
CephFS	✓	✓
Cinder	✓	✓	✓
Fiber Channel	✓	✓
Flexvolume	✓	✓	✓
Flocker	✓	✓	✓
GCE Persistent Disk	✓	✓	✓
Glusterfs	✓	✓	✓
HostPath	✓	✓	✓
iSCSI	✓	✓
NFS	✓	✓
Photon PersistentDisk	✓	✓	✓
Portworx	✓	✓	✓
Quobyte	✓	✓	✓
Ceph RBD	✓	✓	✓
ScaleIO	✓	✓	✓
StorageOS	✓	✓	✓
vsphereVolume	✓	✓	✓
ConfigMap	✓
DownwardAPI	✓
EmptyDir	✓
Projected	✓
Secret	✓
Container Storage Interface(CSI)		✓	✓
Local		✓

Here I only covered the in-tree support from Kubernetes. There are some official out-of-tree Provisioners you can use as well.

As you see here, Volume, Persistent Volume and Provisioner are different in some nuanced ways.

1. Volume supports most of the volume plugins.
   1. It’s the only way to connect PVC to Pod.
   2. It’s also the only one that supports Config Map, Secret, Downward API, and Projected. All of those are closely related to the Kubernetes API server.
   3. And it’s the only one that supports EmptyDir, which will automatically allocate and clean up a temporary volume for Pod*.
2. PV’s supported plugins are the superset of what Provisioner supports. Because Provisioner needs to create PV before workloads can use it. However, there are a few plugins supported by PV but not supported by Provisioner, e.g. Local Volume (which is a work-in-progress).
3. There are two types that Volume doesn’t support. These are the two most recent feature, CSI and Local Volume. There are works-in-progress trying to bring them to Volume.

* A side note about EmptyDir with PV:

Back in 2015, there was an issue raised by Clayton Coleman to support EmptyDir with PV. It can be very helpful for the workloads needing persistent storage but only have local volumes available. But it didn’t get much traction. Without scheduler supports, it was too hard to do it at the time. Now, in 2018, scheduler and PV node affinity support have been added for Local Volume in Kubernetes v1.11. But there is still no EmptyDir PV. And Local Volume feature is not exactly what I expected since it doesn’t have the ability to create new volumes with new directories on the node. So I’ve written Local Path Provisioner, which utilized the scheduler and PV node affinity changes, to dynamically provision Host Path type PV for the workload.

Guideline for choosing between Volume, Persistent Volume and Provisioner

So which way should users choose?

In my opinion, users should stick to one principle:

Choose Provisioner over Persistent Volume, Persistent Volume over Volume when possible.

To elaborate:

1. For Config Map, Downward API, Secret or Projected, use Volume since PV doesn’t support those.
2. For EmptyDir, use Volume directly. Or use Host Path instead.
3. For Host Path, use Volume directly in general, since it’s bound to a specific node and normally homogeneous across the node.
   1. If you want to have heterogeneous Host Path volumes, it didn’t work until Kubernetes v1.11 due to lack of node affinity knowledge for PV. With v1.11+, you can create Host Path PV with node affinity using my Local Path Provisioner.
4. For all other cases, unless you need to hook up with existing volumes (in which case you should use PV), use Provisioner instead. Some of Provisioners are not made into built-in options, but you should able to find them here or at vendor’s official repositories.

The rationale behind this guideline is simple. While operating inside Kubernetes, an object (PV) is easier to manage than a property (Volume), and creating PV automatically (Provisioner) is much easier than creating it manually.

There is an exception: if you prefer to operate storages outside of Kubernetes, it’s better to stick with Volume. Though in this way, you will need to do creation/deletion using another set of API. Also, you will lose the ability to scale storage automatically with StatefulSet due to the lack of VolumeClaimTemplate. I don’t think it will be the choice for most Kubernetes users.

Why are there so many options to do the same thing?

This question was one of the first things that came to my mind when I started working with Kubernetes storage. The lack of consistent and intuitive design makes Kubernetes storage look like an afterthought. I’ve tried to research the history behind those design decisions, but it’s hard to find anything before 2016.

In the end, I tend to believe those are due to a few initial design decision made very early, which may be combined with the urgent need for vendor support, resulting in Volume gets way more responsibility than it should have. In my opinion, all those built-in volume plugins duplicated with PV shouldn’t be there.

While researching the history, I realized dynamic provisioning was already an alpha feature in Kubernetes v1.2 release in early 2016. It took two release cycles to become beta, another two to become stable, which is very reasonable.

There is also a huge ongoing effort by SIG Storage (which drives Kubernetes storage development) to move Volume plugins to out of tree using Provisioner and CSI. I think it will be a big step towards a more consistent and less complex system.

Unfortunately, I don’t think different Volume types will go away. It’s kinda like the flipside of Silicon Valley’s unofficial motto: move fast and break things. Sometimes, it’s just too hard to fix the legacy design left by a fast-moving project. We can only live with them, work around them cautiously, and don’t herald them in a wrong way.

What’s next

We will talk about the mechanism to extend Kubernetes storage system in the next part of the series, namely Flexvolume and CSI. A hint: as you may have noticed already, I am not a fan of Flexvolume. And it’s not storage subsystem’s fault.

[To be continued]

[You can join the discussion here]

Sheng Yang

Principal Engineer

Sheng Yang currently leads Project Longhorn in Rancher Labs, Rancher’s open source microservices-based, distributed block storage solution. He is also the author of Convoy, an open source persistent storage solution for Docker. Before Rancher Labs, he joined Citrix through the Cloud.com acquisition, where he worked on CloudStack project and CloudPlatform product. Before that, he was a kernel developer at Intel focused on KVM and Xen development. He has worked in the fields of virtualization and cloud computing for the last eleven years.

Source