Category: Docker

Last few months, I have been looking at Docker forums(https://forums.docker.com/, https://stackoverflow.com/questions/tagged/docker) and trying to understand some of the common questions/issues faced in the Docker Networking area. This prompted me to do 2 presentations:

• Docker Networking overview
• Docker Networking – Common issues and troubleshooting techniques

I received positive feedback to these 2 presentations. As a next step, I thought preparing each Docker Networking tip as a video can help some folks to get a better picture. As a first attempt, I prepared Macvlan driver as my first Docker Networking video tip. Following is the associated Youtube video and presentation.

If you think this is useful and would like to see more videos, please let me know. Based on the feedback received, I will try to create more Docker Networking tips in video format.

Source

Docker containers provides an isolated sandbox for the containerized program to execute. One-shot containers accomplishes a particular task and stops. Long running containers runs for an indefinite period till it either gets stopped by the user or when the root process inside container crashes. It is necessary to gracefully handle container’s death and to make sure that the Job running as container does not get impacted in an unexpected manner. When containers are run with Swarm orchestration, Swarm monitors the containers health, exit status and the entire lifecycle including upgrade and rollback. This will be a pretty long blog. I did not want to split it since it makes sense to look at this holistically. You can jump to specific sections by clicking on the links below if needed. In this blog, I will cover the following topics with examples:

• Handling Signals and exit codes
• Container restart policy
• Container health check
• Service restart with Swarm
• Service health check
• Service rolling upgrade and rollback

Handling Signals and exit codes

When we pass a signal to container using Docker CLI, Docker passes the signal to the main process running inside container(PID-1). This link has the list of all Linux signals. Docker exit codes follow the chroot exit standard for Docker defined exit codes. Other standard exit codes can come from the program running inside container. Container exit code can be seen from container events coming from Docker daemon when the container exits. For containers that have not been cleaned up, exit code can be found from “docker ps -a”.
Following is a sample “docker ps -a” output where nginx container exited with exit code 0. Here, I used “docker stop” to stop the container.

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
32d675260384 nginx “nginx -g ‘daemon …” 18 seconds ago Exited (0) 7 seconds ago web

Following is a sample “docker ps -a” output where nginx container exited with exit code 137. Here, I used “docker kill” to stop the container.

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b5d8348cb89 nginx “nginx -g ‘daemon …” 11 seconds ago Exited (137) 2 seconds ago web

Following is the list of standard and Docker defined exit codes:

0: Success
125: Docker run itself fails
126: Contained command cannot be invoked
127: Containerd command cannot be found
128 + n: Fatal error signal n:
130: (128+2) Container terminated by Control-C
137: (128+9) Container received a SIGKILL
143: (128+15) Container received a SIGTERM
255: Exit status out of range(-1)

Following is a simple Python program that handles Signals. This program will be run as Docker container to illustrate Docker signals and exit codes.

#!/usr/bin/python

import sys
import signal
import time

def signal_handler_int(sigid, frame):
print “signal”, sigid, “,”, “Handling Ctrl+C/SIGINT!”
sys.exit(signal.SIGINT)

def signal_handler_term(sigid, frame):
print “signal”, sigid, “,”, “Handling SIGTERM!”
sys.exit(signal.SIGTERM)

def signal_handler_usr(sigid, frame):
print “signal”, sigid, “,”, “Handling SIGUSR1!”
sys.exit(0)

def main():
# Register signal handler
signal.signal(signal.SIGINT, signal_handler_int)
signal.signal(signal.SIGTERM, signal_handler_term)
signal.signal(signal.SIGUSR1, signal_handler_usr)

while True:
print “I am alive”
sys.stdout.flush()
time.sleep(1)

# This is the standard boilerplate that calls the main() function.
if __name__ == ‘__main__’:
main()

Following is the Dockerfile to convert this to container:

FROM python:2.7
COPY ./signalexample.py ./signalexample.py
ENTRYPOINT [“python”, “signalexample.py”]

Lets build the container:

docker build –no-cache -t smakam/signaltest:v1 .

Lets start the container:

docker run -d –name signaltest smakam/signaltest:v1

We can watch the logs from container using docker logs:

docker logs -f signaltest

The Python program above handles SIGINT, SIGTERM and SIGUSR1. We can pass these signals to the container using Docker CLI.
Following command sends SIGINT to the container:

docker kill –signal=SIGINT signaltest

In the Docker logs, we can see the following to show that this signal is handled:

signal 2 , Handling Ctrl+C/SIGINT!

Following output shows the container exit status:

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c06266e79a43 smakam/signaltest:v1 “python signalexam…” 36 seconds ago Exited (2) 3 seconds ago signaltest

Following command sends SIGTERM to the container:

docker kill –signal=SIGTERM signaltest

In the Docker logs, we can see the following to show that this signal is handled:

signal 15 , Handling SIGTERM!

Following output shows the container exit status:

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0149708f42b2 smakam/signaltest:v1 “python signalexam…” 10 seconds ago Exited (15) 2 seconds ago signaltest

Following command sends SIGUSR1 to the container:

docker kill –signal=SIGUSR1 signaltest

In the Docker logs, we can see the following to show that this signal is handled:

signal 15 , Handling SIGUSR1!

Following output shows the container exit status:

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c92f7b4dd45b smakam/signaltest:v1 “python signalexam…” 12 seconds ago Exited (0) 2 seconds ago signaltest

When we execute “docker stop “, Docker first sends SIGTERM signal to the container, waits for some time and then sends SIGKILL. This is done so that the program executing inside the container can use the SIGTERM signal to do the graceful shutdown of the program.

Common mistake in Docker signal handling

In the above example, the python program runs as PID 1 inside container since we used the EXEC form of ENTRYPOINT in Dockerfile. If we use the background method of ENTRYPOINT, shell process runs as PID 1 and the python program runs as another process. Following is a sample Dockerfile for starting the program as background process.

FROM python:2.7
COPY ./signalexample.py ./signalexample.py
ENTRYPOINT python signalexample.py

In this example, Docker passes the signal to the shell process instead of to the Python program. This causes the python program to not see the signal sent to the container. If there are multiple processes running inside the container and we need to pass the signal, 1 possible approach is to run the ENTRYPOINT as a script, handle the signal in the script and pass it to the correct process. 1 example using this approach is mentioned here.

Difference between “docker stop”, “docker rm” and “docker kill”

“docker stop” – Sends SIGTERM to container, waits some time for process to handle it and then sends SIGKILL. Container filesystem remains intact.
“docker kill” – Sends SIGKILL directly. Container filesystem remains intact.
“docker rm” – Removes container filesystem. “docker rm -f” will send SIGKILL and then remove container filesystem.
Using “docker run” with “–rm” option will automatically remove containers including container filesystem when the container exits.

When container exits without the container filesystem getting removed, we can still restart the container.

Container restart policy

Container restart policy controls the restart actions when Container exits. Following are the supported restart options:

• no – This is default. Containers do not get restarted when they exit.
• on-failure – Containers restart only when there is a failure exit code. Any exit code other than 0 is treated as failure.
• unless-stopped – Containers restart as long as it was not manually stopped by user.
• always – Always restart container irrespective of exit status.

Following is an example of starting “signaltest” container with restart policy of “on-failure” and retry count of 3. Retry count 3 is the number of restarts that will be done by Docker before giving up.

docker run -d –name=signaltest –restart=on-failure:3 smakam/signaltest:v1

To show the restart happening, we can manually try to send signals to the container. In the “signaltest” example, signals SIGTERM, SIGINT and SIGKILL will cause non-zero exit code and SIGUSR1 will cause zero exit code. 1 thing to remember is that restart does not work if we stop the container or send signals using “docker kill”. I think this is because there must be an explicit check added by Docker to prevent restart in these cases since the action is triggered by user.
Lets send SIGINT to the container by passing the signal to the process. We can find the process id by doing “ps -eaf | grep signalexample” in host machine.

kill -s SIGINT

Lets check the “docker ps” output. We can see that the “created” time is 50 seconds. Uptime is less than a second because container restarted.

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b867543b110c smakam/signaltest:v1 “python signalexam…” 50 seconds ago Up Less than a second

Following command shows the restart policy and the restart count for the running container. In this example, container restart happened once.

$ docker inspect signaltest | grep -i -A 2 -B 2 restart
“Name”: “/signaltest”,
“RestartCount”: 1,
“RestartPolicy”: {
“Name”: “on-failure”,
“MaximumRetryCount”: 3

To illustrate that restart does not work on exit code 0, lets send SIGUSR1 to the container that will cause exit code 0.

sudo kill -s SIGUSR1

In this case, container exits, but it does not get restarted.

Container restart does not work with “–rm” option. This is because “–rm” option causes container to be removed as soon as the container exit happens.

Container health check

It is possible that container does not exit but it not performing as per the requirement. Health check probes can be used to identify misbehaving containers and take action rather than waiting till the end when container dies. Health check probes are used to accomplish the specific task of checking container health. For a container like webserver, health check probe can be as simple as sending curl request to webserver port. By using container’s health, we can restart the container if health check fails.
To illustrate health check feature, I have used the container described here.
Following command starts the webserver container with health check capability enabled.

docker run -p 8080:8080 -d –rm –name health-check –health-interval=1s –health-timeout=3s –health-retries=3 –health-cmd “curl -f http://localhost:8080/health || exit 1” effectivetrainings/docker-health

Following are all parameters related to healthcheck:

Following “docker ps” output shows container health status:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
947dad1c1412 effectivetrainings/docker-health “java -jar /app.jar” 28 seconds ago Up 26 seconds (healthy) 0.0.0.0:8080->8080/tcp health-check

This container has a backdoor approach to mark container health as unhealthy. Lets use the backdoor approach to mark container as unhealthy like below:

curl “http://localhost:8080/environment/health?status=false”

Now, lets check the “docker ps” output. The container’s health has now become unhealthy.

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
947dad1c1412 effectivetrainings/docker-health “java -jar /app.jar” 3 minutes ago Up 3 minutes (unhealthy) 0.0.0.0:8080->8080/tcp health-check

Service restart with Swarm

Docker Swarm mode introduces a higher level of abstraction called Service and containers are part of the service. When we create a service, we specify the number of containers that needs to be part of the service using “replicas” parameter. Docker swarm will monitor the number of replicas and if any container dies, Swarm will create new container to keep the replica count as requested by the user.
Below command can be used to create signal service with 2 container replicas:

docker service create –name signaltest –replicas=2 smakam/signaltest:v1

Following command output shows the 2 containers that are part of “signaltest” service:

$ docker service ps signaltest
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
vsgtopkkxi55 signaltest.1 smakam/signaltest:v1 ubuntu Running Running 36 seconds ago
dbbm05w91wv7 signaltest.2 smakam/signaltest:v1 ubuntu Running Running 36 seconds ago

Following parameters control the container restart policy in a service:

Lets start the “signaltest” service with restart-condition of “on-failure”:

docker service create –name signaltest –replicas=2 –restart-condition=on-failure –restart-delay=3s smakam/signaltest:v1

Remember that sending signal “SIGTERM”, “SIGINT”, “SIGKILL” causes non-zero container exit codes and sending “SIGUSR1” causes zero container exit code.
Lets first send SIGTERM to 1 of the 2 containers:

docker kill –signal=SIGTERM

Following is the “signaltest” service output that shows the 3 containers including the one that has exited with non-zero status:

$ docker service ps signaltest
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
35ndmu3jbpdb signaltest.1 smakam/signaltest:v1 ubuntu Running Running 4 seconds ago
ullnsqio5151 _ signaltest.1 smakam/signaltest:v1 ubuntu Shutdown Failed 11 seconds ago “task: non-zero exit (15)”
2rfwgq0388mt signaltest.2 smakam/signaltest:v1 ubuntu Running Running 49 seconds ago

Following command sends SIGUSR1 signal to 1 of the containers which causes container to exit with status 0.

docker kill –signal=SIGUSR1

Following command shows that the container did not restart since the container exit code is 0.

$ docker service ps signaltest
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
35ndmu3jbpdb signaltest.1 smakam/signaltest:v1 ubuntu Running Running 52 seconds ago
ullnsqio5151 _ signaltest.1 smakam/signaltest:v1 ubuntu Shutdown Failed 59 seconds ago “task: non-zero exit (15)”
2rfwgq0388mt signaltest.2 smakam/signaltest:v1 ubuntu Shutdown Complete 3 seconds ago

$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
xs8lzbqlr69n signaltest replicated 1/2 smakam/signaltest:v1

I don’t see a real need to change the default Swarm service restart policy from “any”.

Service health check

In the previous sections, we saw how to use container health check with “effectivetrainings/docker-health” container. Even though we could detect the container as unhealthy, we could not restart the container automatically. For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.

“docker service” command provides following options for health check and associated behavior.

Lets create “swarmhealth” service with 2 replicas of “docker-health” containers.

docker service create –name swarmhealth –replicas 2 -p 8080:8080 –health-interval=2s –health-timeout=10s –health-retries=10 –health-cmd “curl -f http://localhost:8080/health || exit 1” effectivetrainings/docker-health

Following output shows the “swarmhealth” service output and the 2 healthy containers:

$ docker service ps swarmhealth
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
jg8d78inw97n swarmhealth.1 effectivetrainings/docker-health:latest ubuntu Running Running 21 seconds ago
l3fdz5awv4u0 swarmhealth.2 effectivetrainings/docker-health:latest ubuntu Running Running 19 seconds ago

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9b1f1b0a9b0 effectivetrainings/docker-health:latest “java -jar /app.jar” About a minute ago Up About a minute (healthy) swarmhealth.1.jg8d78inw97nmmbdtjzrscg1q
bb15bfc6e588 effectivetrainings/docker-health:latest “java -jar /app.jar” About a minute ago Up About a minute (healthy) swarmhealth.2.l3fdz5awv4u045g2xiyrbpe2u

Lets mark 1 of the container unhealthy using backdoor command:

curl “http://:8080/environment/health?status=false”

Following output shows that 1 of the containers that has been shutdown which is the unhealthy container and 2 more running replicas. 1 of the replicas got restarted after the other container became unhealthy.

$ docker service ps swarmhealth
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ixxvzyuyqmcq swarmhealth.1 effectivetrainings/docker-health:latest ubuntu Running Running 4 seconds ago
jg8d78inw97n _ swarmhealth.1 effectivetrainings/docker-health:latest ubuntu Shutdown Failed 23 seconds ago “task: non-zero exit (143): do…”
l3fdz5awv4u0 swarmhealth.2 effectivetrainings/docker-health:latest ubuntu Running Running 5 minutes ago

Service upgrade and rollback

When we have new versions of service to be updated without taking service downtime, Docker provides many controls to do the upgrade and rollback. For example, we can control parameters like number of tasks to upgrade at a single time, actions on upgrade failure, delay between task upgrades etc. This helps us achieve release patterns like Blue green and Canary deployment patterns.

Following options are provided by Docker in “docker service” command to control rolling upgrade and rollback.

Rolling upgrade:

Rollback:

To illustrate service upgrade, I have a simple python webserver program running as container.
Following is the Python program:

#!/usr/bin/python

import sys
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import urlparse
import json

class GetHandler(BaseHTTPRequestHandler):

def do_GET(self):
message = “You are using version 1n”
self.send_response(200)
self.end_headers()
self.wfile.write(message)
return

def main():
server = HTTPServer((”, 8000), GetHandler)
print ‘Starting server at http://localhost:8000’
server.serve_forever()

# This is the standard boilerplate that calls the main() function.
if __name__ == ‘__main__’:
main()

This is the Dockerfile to create the Container:

FROM python:2.7
COPY ./webserver.py ./webserver.py
ENTRYPOINT [“python”, “webserver.py”]

I have 2 versions of Container, smakam/webserver:v1 and smakam/webserver:v2. The only difference is the message output that either shows “You are using version 1” or “You are using version 2”.

Lets create version 1 of the service with 2 replicas:

docker service create –name webserver –replicas=2 -p 8000:8000 smakam/webserver:v1

We can access the service using script. The service request will get load balanced between the 2 replicas.

while true; do curl -s “localhost:8000”;sleep 1;done

Following is the service request output that shows we are using version 1 of the service:

You are using version 1
You are using version 1
You are using version 1

Lets upgrade to version 2 of the web service. Since we have specified update-delay of 3 seconds, there will be a 3 second gap between upgrades of 2 replicas. Since the “update-parallelism” default is 1, only 1 task will be upgraded at 1 time.

docker service update –update-delay=3s –image=smakam/webserver:v2 webserver

Following is the service request output output that shows the request slowly getting migrated to version 2 as the upgrade happens 1 replica at a time.

You are using version 1
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 2
You are using version 2

Now, lets rollback to version 1 of the webserver:

docker service update –rollback webserver

Following is the service request output output that shows the request slowly getting downgraded from version 2 to version 1.

You are using version 2
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 2
You are using version 1
You are using version 1

Please let me know your feedback and if you want to see more details on any specific topic related to this. I have put the code associated with this blog here. The containers used in this blog(smakam/signaltest, smakam/webserver) are in Docker hub.

References

• Blog – Life and Death of a Container
• Blog – Trapping signals in Docker containers
• Blog – Gracefully stopping Docker containers
• Docker container restart policies
• Docker container healthcheck
• Docker service rolling update

Source

The people at the Armbian team have been doing an impressive work at optimizing and generating Debian images for many ARM boards.

While they have documentation on the build process, it focuses mostly on setting up a build environment on VirtualBox with Vagrant.

They do support building in a Docker environment, but I found the documentation lacking and I saw myself asking in the forums.

It is easier to set up Docker than virtualization, plus the containers spawn so much faster so we can better iterate and make modifications in the build.

I will just reflect here the commands to produce a fully automated build for an Odroid XU4, headless Debian Stretch image

git clone https://github.com/armbian/build armbian-build cd armbian-build ./compile.sh docker BOARD=odroidxu4 BRANCH=next KERNEL_ONLY=no KERNEL_CONFIGURE=no RELEASE=stretch BUILD_DESKTOP=no

 

All the build options are here. I personally add these ones

• CLEAN_LEVEL=”” to avoid recompiling the kernel, u-boot and the rest every build, unless they have changed.
• NO_APT_CACHER=no to use apt-cacher-ng for faster builds.

Source

Today I want to share a new project with you called OpenFaaS Cloud.

Announcing @openfaas cloud at @DevNetCreate #TeamServerless #Serverless pic.twitter.com/n6hhcRK0I5

— Jock Reed (@JockDaRock) April 10, 2018

Sharing my conceptual design at Cisco DevNet Create (developer conference) in Mountain View

OpenFaaS Cloud makes it even easier for developers to build and ship functions with Docker using a Git-based workflow with native integrations for GitHub and more integrations planned in the future.

I wrote my first public commit to OpenFaaS (then called FaaS) in late 2016, but since then I’ve focused on building a community and a set of stable and extensible features for Making Serverless Functions Simple with Docker. Feedback is important for an Open Source project and so are values. OpenFaaS now has half a dozen strong case-studies, almost 100 contributors and 11k GitHub stars.

Last year before KubeCon in Austin the community worked really hard to be the first Open Source FaaS project to release a Function Store. OpenFaaS Cloud is another example of innovation which has come about from listening to the growing community and stepping out to lead in this space.

Back to values

I began working on OpenFaaS Cloud late last year after noticing a trend from OpenFaaS case-studies from industry. I found that DevOps engineers and early adopters had bought into functions, but wanted a way to make an accessible pipeline for their developers. Each of them had created their own version of this pipeline, but with varying levels of stability and functionality.

The values of the OpenFaaS community are: developer-first, operational simplicity and community-centric. These values align well with building OpenFaaS Cloud and provides value for existing users in industry.

Developer-first

putting your needs first with a familiar UI/CLI and Docker/Git workflow

At BT Research a team of DevOps professionals in (BetaLab) are supporting research experiments for 140 research engineers. OpenFaaS provided a great way for them to make functions available at scale to the whole team, but they needed a way for functions to be built quickly from Git using Docker.

Operational Simplicity

easy to use and deploy, built on battle-tested cloud-native tech

Anisha Keshavan is a post-doctorate at the University of Washington and is using machine-learning with Python to push the boundaries of neuro-informatics. If you’ve ever used a machine-learning framework then you’ll already know that it’s impossible to work with a cloud functions framework due to the meagre 50-60mb file-size limit applied. One of the first things she said to me was: “Why can’t this work like Heroku? I just want to push code and not worry about HTTP certificates”

Community-centric

investing in people and creating an open platform together – #TeamServerless

One of the earliest requests I had from the community was for a shared cluster where functions could be deployed on OpenFaaS and left running for a period of time. Initially I pushed back on this to focus on building the platform, but there is now a great way to make exactly what they asked for available.

GitOps a history

Some people have coined this way of working as “GitOps”, but it was pioneered long ago by projects like Heroku. More recently start-ups like Weaveworks have provided CI/CD for Kubernetes in a tool called Flux. Netlify have provided GitOps for docs – you push code to your repo and in a few minutes mkdocs or some other tool will generate and publish your project’s documentation site.

GitOps is really about using a Git repository as a single source of truth for application code and configuration. Lifecycle events such as builds and deployments are triggered by commits to Git rather than by mouse clicks or API calls.

OpenFaaS Cloud applies the principles of GitOps to build and deploy your OpenFaaS Functions from a public GitHub repository. As soon as you type in git push a GitHub App on GitHub.com will receive a notification that triggers a push event to be sent to a pipeline in OpenFaaS. The pipeline of functions will then build your function with Docker and deploy it with a rolling update to a Kubernetes or Docker Swarm cluster. Your GitHub commit is then updated with a passed/failed status and a HTTPS URL where you can access your function will be made available.

This is the process from the moment OpenFaaS Cloud receives a message from the GitHub App hosted at GitHub.com:

• OpenFaaS Cloud verifies the JSON digest with HMAC
• Clones the repository and checks out the SHA
• Builds your functions in stack.yml using buildkit and publishes to a registry
• During the build – OpenFaaS function templates such as the Golang template will execute any unit-tests present, if the tests fail then the image will not be deployed
• Deploys your functions to OpenFaaS using a rolling update
• Updates GitHub with a status of pass/fail and a URL endpoint

Let’s look at an example function wihch you may be familiar with if you’ve already tried OpenFaaS.

Example

I set up a public GitHub repo which I use for demos called kubecon-tester. It contains a Python and a Golang function. The Go function will echo a string back when called with a HTTP POST.

• The code as pushed to a GitHub repo

def handle(req):
msg = req
if len(req) == 0:
msg = “pass a value to the function”

return str.format(“You said: {}”, msg)

handler.py

This function was generated with the following command: faas-cli new –lang python3 kubecon.

We also get a kubecon.yml file generated which we need to rename to stack.yml so that OpenFaaS Cloud will know which functions to build.

provider:
name: faas
gateway: http://localhost:8080

functions:
kubecon:
lang: python
handler: ./kubecon
image: alexellis/kubecon

stack.yml

• GitHub status updates:

Here we see that the function was deployed and the GitHub status was updated with a successful indicator.

• GitHub badge

You can get a GitHub badge for your functions to show the build status at a glance:

• User overview page

The user overview page shows your public HTTPS endpoints, the deployment date and will soon include R.E.D. metrics from Prometheus too.

• Public endpoints

A public endpoint it set up with the following scheme where each user gets their own sub-domain with HTTPS via a LetsEncrypt wildcard certificate.

https://github-owner.get-faas.com/function-name

So I will now have a public URL of:

https://alexellis.get-faas.com/kubecon

The URL is subject to rate-limiting and some other policies, but this can be now invoked in the following way by curl:

$ curl –data “Let’s get started” https://alexellis.get-faas.com/kubecon
You said: Let’s get started

Q&A

What languages are supported?

The public trial is limited to the official templates repository which currently contains: Go, Node.js, Python, Ruby and C#. This is an artificial limit imposed along with some other policies for the trial period and can be removed with configuration at any point.

Did you know that any existing container or binary can be turned into an OpenFaaS Function in a few minutes? Read more here

How do I get OpenFaaS Cloud?

OpenFaaS Cloud comes in two flavours, both are free and open-source. For total control over the pipeline, or for development you can deploy it to your existing OpenFaaS cluster on Swarm or Kubernetes. There is also a fully-hosted Kubernetes cluster which I was using in the example for my kubecon-tester functions. This environment is being hosted by Packet.net on their baremetal cloud.

The public environment is free of charge subject to availibility and makes a great place for experimenting, especially if you want to run functions without having to pay for your own VMs. Having public endpoints with HTTPS enabled is really useful if you are unable to get get inbound traffic at home or at work – I’ve even used it to host an Alexa skill.

If you’re a contributor to OpenFaaS (there are around 100 now) then ask me about short-listed access to the public environment.

How does this differ from rolling my own?

OpenFaaS Cloud delivers a Git-based workflow for building and deploying your Serverless Functions with a tight integration into GitHub, but you may have an existing cluster and Docker CI/CD pipeline.

The market is saturated with tooling that you can run on-premise or in the cloud to build your Docker images. Here are a few products that you could use to create a similar flow on your own: GitLab, Travis CI, CircleCI, Flux, CodeFresh, CodeShip, Visual Studio Team Services or Docker EE.

The reason this is possible is because OpenFaaS builds functions as Docker images so that you get the same result in dev, staging and production. The good news is that if you already have a cluster then you can start benefiting from functions with very little investment – either with OpenFaaS Cloud or with your own existing container pipeline.

What about testing?

There are currently two types of testing taking place automatically, but you can expect to see support for additional types of testing in the future. Unit-tests can be executed as part of the function build. The OpenFaaS language template for Golang includes a step for running linting gofmt and unit tests – if either of these steps fail then your code will not be deployed. The second kind of test taking place is the rolling update provided by Docker Swarm and Kubernetes – if your Pod cannot be scheduled or started then the endpoint will not switch over.

In my personal test repo you can also see a Go function which generates and verifies bcrypt hashed passwords.

Wrapping up

OpenFaaS Cloud is still early work, but from the example above you can see that it is functional for use on premise or with the community-hosted, public cluster. The roadmap is available on GitHub and the community is seeking contributions. So if you see value in your team having easy access to functions please kick the tires, contribute some fixes and help me with the wider community to build OpenFaaS Cloud into a great project.

Let’s make it even easier to deploy functions with GitOps to any cloud.

You can star or fork the project here: openfaas/openfaas-cloud.

If you’d like to get started with learning OpenFaaS check out: Three ways to learn Serverless with OpenFaaS this season.

Please share the post and follow OpenFaaS on Twitter @openfaas:

Introducing @OpenFaaS Cloud – making it even easier to build and ship your Serverless Functions with a git-based workflow – https://t.co/C4L1REhIYq @github @GitOps @docker pic.twitter.com/Frr6IawFWu

— Alex Ellis (@alexellisuk) April 30, 2018

Source

 

Red Hat Container Development Kit (CDK) With Nested KVM

By Scott McCarty February 13, 2018February 12, 2018

Why

If you are like me, you probably prefer to install new and exploratory software in a fresh virtual machine (VM) or container to insulate your laptop/desktop from software pollution (TM). Red Hat Container Development Kit (CDK) relies on virtualization to create a Red Hat Enterprise Linux (RHEL) virtual machine to run OpenShift (based on Kubernetes). Red Hat specifically supports installation of the CDK on Windows, macOS, and RHEL Server, but if you are running Fedora, RHEL Workstation, or even CentOS, you will run into trouble. If you are not running a supported desktop, you can always use a RHEL Server virtual machine, and this tutorial is for you.

This tutorial is specific to running RHEL Server as a VM on RHEL Workstation, but these instructions should work for Fedora and CentOS. With a modification of the first step – creating a virtual machine with nested virtualization enabled (vmware, hyper-v) – you should be able to make these instructions work on Windows and macOS as well.

How

Create a Virtual Machine

First, create a new virtual machine and do a RHEL Server installation. Personally, I use virt-manager because it makes it easy to create ephemeral virtual machines for testing. I gave my VM 8192 MB of RAM, and 1 vCPU. While creating the VM, remember to configure the CPU to copy the host configuration. This will enable Nested KVM which will allow you to run virtual machines inside of your new virtual machine – mind….blown….

Install RHEL

Download and install RHEL Server, because that’s one of the supported platforms for CDK. I won’t rewrite the instructions on this because most people can do this without documentation.

On the new RHEL Installation, install and configure virtualization and a few other tools to make things easier:

yum install -y qemu-kvm qemu-img libvirt virt-manager xauth firefox

Install CDK

In the newly created virtual machine enable, download, and install CDK. Remember to:

subscription-manager repos –enable rhel-7-server-devtools-rpms
subscription-manager repos –enable rhel-server-rhscl-7-rpms
cd /etc/pki/rpm-gpg
wget -O RPM-GPG-KEY-redhat-devel https://www.redhat.com/security/data/a5787476.txt
rpm –import RPM-GPG-KEY-redhat-devel
yum install cdk-minishift docker-machine-kvm

Now, setup the CDK. This will do everything for you, including putting the OC binary where it needs to be.

ln -s /root/.minishift/cache/oc/v3.7.14/linux/oc /usr/bin/oc
minishift setup-cdk
minishift start

Up and Running

These are instructions usually missed by tutorials. Notice, the oc command is automatically configured to connect to the Kubernetes/OpenShift environment in the virtual machine (which is inside the virtual machine you created – mic drop)

oc get pods
oc get pv
oc get node

You can also get directly into the CDK virtual, virtual machine by using the following command. From here you can run docker commands, look at underlying storage, etc:

minishift ssh
docker ps
docker images

Or, go into the browser console with this command. This will show you the OpenShift web console in a browser, displayed over X11 to your laptop (that’s why we installed xauth). Warning, you have to disable SELinux:

setenforce 0
minishift console

Now, you have a fully functional OpenShift environment up and running, and you are ready explore just about any Kubernetes or OpenShift tutorial on the Internet. You even have persistent volumes set up for storage tests.

Tips & Tricks

Tip: if the minishift setup fails, you can always delete and start again with:

minishift delete
minishift cdk-setup

Tip: Sometimes you have to manually delete the subscription on the Red Hat Portal so that you can run the CDK setup again. Just use the Remove System button:

Tip: Since we are doing nested virtualization, every now and then you will end up with some funky network problems or other problems. Just, delete the CDK and reboot the virtual machine:

minishift delete
reboot

Take advantage of your Red Hat Developers membership and download RHEL today at no cost.

Source

I had promised earlier that I will continue my Docker Networking Tip series. This is second in that series. The first one on Macvlan driver can be found here. In this presentation, I have covered different load balance options with Docker. Following topics are covered in this presentation:

• Overview of Service Discovery and Load balancing
• Service Discovery and Load balancing implementation in Docker
• Docker Load balancing use cases with Demo
  • Internal Load balancing
  • Ingress routing mesh Load balancing
  • Proxy load balancing with nginx
  • L7 load balancing with Traefik

Following is the associated Youtube video and presentation:

I have put the use cases in github if you want to try it out.

If you think this is useful and would like to see more videos, please let me know. Based on the feedback received, I will try to create more Docker Networking tips in video format.

Source

The latest release of NextCloudPi is out!

Work has been focused on improving user experience during the first setup, and bringing more power to the docker container and ncp-web. Also there is some news!

NextCloudPlus improves everyday thanks to your feedback. Please report any problems here. Also, you can discuss anything in the forums.

Last but not least, please download through bitorrent and share it for a while to help keep hosting costs down.

NextCloudPlus

While originally NCP was only oriented towards running Nextcloud in the Raspberry Pi, it has since extended its scope to include all kinds of boards and architectures. We keep running into people online under the impression that we can’t enjoy ncp-web and the rest in an x86 server or an Odroid HC2, just because of the name.

So we decided for NextCloudPlus: we are still NCP and well, it’s Nextcloud with a plus 😉

New web

It was getting confusing to have the information scattered throughout the blog and the wiki, so a single landing page with a simple design has been created, nextcloudplus.com

No more default passwords

The initial setup process have been improved to make it lazy proof. There are no default passwords anymore, and less steps to follow.

If we navigate to nextcloudplus.local, we will be greeted with an activation page

We can now copy the credentials, or print this page and click the activation button. Our instance will be initialized with these random passwords, and we will be redirected to the first run wizard.

ncp-web

The web panel has received a polish, and a number of new features.

Dashboard

An informative landing page that presents an overview of the system, and some configuration advice.

Nextcloud configuration

An easy way to inspect config.php

Language bar

The language bar has been made more visible

Docker container

The docker container has received a well needed push to make it on par with the SD card version. All remaining ncp-apps have been ported to the docker version, plus we now have in-container upgrades just like the SD card version.

Just set nc-autoupdate-ncp and the container will receive all improvements and fixes just like all other NCP users.

Also, it is now supported to specify any custom external folder to use as a volume, so we can have our data and database in any folder of your choice. For instance, in order to save our files to /media/USBdrive/ncdata, just pass the folder as a volume argument

docker run -d -p 4443:4443 -p 443:443 -p 80:80 -v /media/USBdrive/ncdata:/data –name nextcloudpi ownyourbits/nextcloudplus-x86 $DOMAIN

 

New languages

Spanish has been added to ncp-web, and Chinese is also in the oven

Source

In this post I’ll round-up the highlights of the OpenFaaS community meeting from 25th May.

The all-hands community meetings are a great way to remain connected to the project and community where we have a chance to share our finest work and importantly get face-to-face time with each other. One of the things I like the most is hearing about the various backgrounds people have with our community covering both the users, developers and operators of OpenFaaS.

Highlights

I kicked-off the meeting with a project update and an announcement of a new core contributor Lucas Roesler. Lucas has shown a commitment to the project and willingness to take responsibility and ownership for features like the secrets management for functions.

The Core contributors group helps to build and maintain OpenFaaS, taking responsibility for and pride in the project.

We then had three demos from the community.

1. OpenFaaS on AWS Fargate/ECS

I’ve known Ed Wilde for some time and it was great to see him getting into the Cloud Native world with his start-up. Ed works in London and has been putting together a Proof-of-Concept provider for OpenFaaS that allows functions to be managed on AWS Fargate/ECS. He’s going to release his code when it’s feature complete, but here’s a preview:

Community meeting – excited to see the progress @ewilde is making with faas-fargate – @OpenFaaS on AWS Fargate. Why is Ed interested in building this? Because he’s in a small team and they want to manage all their services in the same way. #TeamServerless #FaaSFriday pic.twitter.com/zUnjx90xBe

— Alex Ellis (@alexellisuk) May 25, 2018

2. CloudEvents integration

CloudEvents is a designed-by-committee spec for how clouds should publish events about their managed services such as S3 CRUD events or changes to DB rows such as when a new document is added to DynamoDB.

John McCabe gave us background on the CloudEvents 0.1 spec and showed us a demo of how he was able to integrate with events from Azure EventGrid in the CloudEvents 0.1 format without changing any code in OpenFaaS.

We’re learning about how @mccabejohn was able to consume CloudEvents 0.1 within @openfaas without changing any code on the #FaaSFriday community call. Good attendance for a bank-holiday weekend! #teamserverless pic.twitter.com/suL29AM71Y

— Alex Ellis (@alexellisuk) May 25, 2018

3. The OpenFaaS Operator with Weave Flux

Stefan Prodan showed us the culmination of a week’s work on a blog post that combines Flux and Weave Cloud from Weaveworks with Heptio Contour, Bitnami SealedSecrets and the OpenFaaS Kubernetes Operator to deliver hardened CD with GitOps on GKE.

In our community call for @OpenFaaS – we’re getting a demo of OpenFaaS Operator, @HeptioContour, @bitnami SealedSecrets, @weaveworks Flux and Cloud – for a hardened GitOps on GKE with NetworkPolicy. Thanks @stefanprodan #FaaSFriday https://t.co/CXDDZsPHj3 pic.twitter.com/thuSWncTtI

— Alex Ellis (@alexellisuk) May 25, 2018

Bonus community material

I want to give a shout-out for Robert Malmstein from Holberton School in San Francisco. We met in April earlier this year at Cisco’s DevNet Create conference, since I’ve been meeting with and mentoring Robert with his studies. We got together for coffee to talk about his latest project and that got me thinking about how functions could be applied.

Great to meet up with my new friends from @holbertonschool @fallenicicle and @RobertMalmstein – we met at @DevNetCreate in April. Today had coffee today, started some mentoring and hacking for Robert’s project – locate a trash can in SF – keep SF tidy. pic.twitter.com/wQ2qMPQQdB

— Alex Ellis (@alexellisuk) May 29, 2018

Robert’s idea is simple – making public asset data available on mobile devices such as the location of the nearest trash can or park bench for some R&R in the city between busy meetings. Robert’s project can be taken further to crowd-source and vote up new locations to gamify the experience.

Check out this GitHub repository for the sample code implemented with OpenFaaS functions.

Video recording

You can view the whole video recording here on YouTube:

Get involved

• Try the OpenFaaS Operator

OpenFaaS supports Kubernetes through the faas-netes project, but the OpenFaaS Operator offers a deeper extension into the Kubernetes API meaning you can manage your functions both through the RESTful API and with kubectl.

Try it out on GitHub

• Experiment with CloudEvents

If you’re an Azure user then there’s already support for publishing events from EventGrid in the CloudEvents JSON format with inbound webhooks. Check out John’s GitHub account.

• Join the Community

Whether you’re new to this space, have questions or want to start contributing to Open Source you’re welcome to join our community.

The easiest way to get connected into the project is to join us over on the OpenFaaS Slack workspace and on GitHub: https://github.com/openfaas/

Source

By Scott McCarty  February 22, 2018July 16, 2018

You might think containers seem like a pretty straightforward concept, so why do I need to read about container terminology? In my work as a container technology evangelist, I’ve encountered misuse of container terminology that causes people to stumble on the road to mastering containers. Terms like containers and images are used interchangeably, but there are important conceptual differences. In the world of containers, repository has a different meaning than what you’d expect. Additionally, the landscape for container technologies is larger than just docker. Without a good handle on the terminology, It can be difficult to grasp the key differences between docker and (pick your favorites, CRI-O, rkt, lxc/lxd) or understand what the Open Container Initiative is doing to standardize container technology.

It is deceptively simple to get started with Linux Containers. It takes only a few minutes to install a container engine like docker and run your first commands. Within another few minutes, you are building your first container image and sharing it. Next, you begin the familiar process of architecting a production-like container environment, and have the epiphany that it’s necessary to understand a lot of terminology and technology behind the scenes. Worse, many of the following terms are used interchangeably… often causing quite a bit of confusion for newcomers.

• Container
• Image
• Container Image
• Image Layer
• Registry
• Repository
• Tag
• Base Image
• Platform Image
• Layer

Understanding the terminology laid out in this technical dictionary will provide you a deeper understanding of the underlying technologies. This will help you and your teams speak the same language and also provide insight into how to better architect your container environment for the goals you have. As an industry and wider community, this deeper understanding will enable us to build new architectures and solutions. Note, this technical dictionary assumes that the reader already has an understanding of how to run containers. If you need a primer, try starting with A Practical Introduction to Docker Containers on the Red Hat Developer Blog.

Containers 101

To understand container terminology, it’s important to understand exactly what a container is – with technical precision. A container is really two different things. Like a normal Linux program, containers really have two states – rest and running. When at rest, a container is a file (or set of files) that is saved on disk. This is referred to as a Container Image or Container Repository. When you type the command to start a container, the Container Engine unpacks the required files and meta-data, then hands them off to the the Linux kernel. Starting a container is very similar to starting a normal Linux process and requires making an API call to the Linux kernel. This API call typically initiates extra isolation and mounts a copy of the files that were in the container image. Once running, Containers are just a Linux process. The process for starting containers, as well as the image format on disk, are defined and governed by standards.

There are several competing Container Image formats (Docker, Appc, LXD), but the industry is moving forward with a standard governed under the Open Container Initiative – sometimes referred to simply as Open Containers or the OCI. The scope of the OCI includes a Container Image Format Specification, which defines the on-disk format for container images as well as the meta-data which defines things like hardware architecture and the operating system (Linux, Windows, etc). An industry wide container image format enables ecosystems of software to flourish – different individual contributors, projects, and vendors are able to build images and tooling, which are interoperable. Users want interoperability between tools for signing, scanning, building, running, moving and managing container images.

There are also several competing Container Engines including Docker, CRI-O, Railcar, RKT, LXC. These Container Engines take a Container Image and turn it into a Container (aka running processes). How this happens is governed by the scope of the OCI which includes a Container Runtime Specification and a Reference Runtime Implementation called RunC. This reference implementation is open source, governed by a community development model, and commonly used by many container engines to communicate with the host kernel when creating containers.

Tools which target the OCI Container Image Format Specification and Container Runtime Specification ensure portability between a broad ecosystem of container platforms, container engines, and supporting tools across cloud providers and on premise architectures. Understanding the nomenclature, container standards, and the architecture of the building blocks of containers, will ensure that you can communicate with other architects to build scalable & supportable containerized applications and environments to productively run containers for years to come.

Background

Containers 101

Basic Vocabulary

Container Image

Container Image Format

Container Engine

Container

Container Host

Registry Server

Container Orchestration

Advanced Vocabulary

Container Runtime

Image Layer

Tag

Repository

Namespace

Kernel Namespace

Graph Driver

Container Use Cases

Application Containers

Operating System Containers

Pet Containers

Super Privileged Containers

Tools & Operating System Software

Architecture of Containers

Application Images

Base Images

Builder Images

Containerized Components

Deployer Images

Intermediate Images

Intermodal Container Images

System Containers

Conclusion

Container Image

See also Repository.

A container image, in its simplest definition, is a file which is pulled down from a Registry Server and used locally as a mount point when starting Containers. The container community uses “container image” quite a bit, but this nomenclature can be quite confusing. Docker, RKT, and even LXD, operate on the concept of pulling remote files and running them as a Container. Each of these technologies treats container images in different ways. LXD pulls a single container image (single layer), while docker and RKT use OCI-based images which can be made up of multiple layers.

Technically, it is much more complicated than a single file on a Registry Server. When people use the term “container image,” they often mean to imply Repository and referring to a bundle of multiple container Image Layers as well as metadata which provides extra information about the layers.

Implicit in the concept of a container image is the concept of a Container Image Format.

Container Image Format

See Container Image and Background.

Historically, each Container Engine had its container images format. LXD, RKT, and Docker all had their own image formats. Some were made up of a single layer, while others were made up of a bunch of layers in a tree structure. Today, almost all major tools and engines have moved to a format defined by the Open Container Initiative (OCI).This image format defines the layers and metadata within a container image. Essentially, the OCI image format defines a container image composed of tar files for each layer, and a manifest.json file with the metadata.

The Open Container Initiative (OCI), which was originally based on the Docker V2 image format, has successfully unified a wide ecosystem of container engines, cloud providers and tools providers (security scanning, signing, building and moving). This will help protect users as the invest in knowledge, and tooling in their environments.

Container Engine

See also Container Runtime.

A container engine is a piece of software that accepts user requests, including command line options, pulls images, and from the end user’s perspective runs the container. There are many container engines, including docker, RKT, CRI-O, and LXD. Also, many cloud providers, Platforms as a Service (PaaS), and Container Platforms have their own built-in container engines which consume Docker or OCI compliant Container Images. Having an industry standard Container Image Format allows interoperability between all of these different platforms.

Going one layer deeper, most container engines don’t actually run the containers, they rely on an OCI compliant runtime like runc. Typically, the container runtime is responsible:

• Handling user input
• Handling input over an API often from a Container Orchestrator
• Pulling the Container Images from the Registry Server
• Expanding decompressing and expanding the container image on disk using a Graph Driver (block, or file depending on driver)
• Preparing a container mount point, typically on copy-on-write storage (again block or file depending on driver)
• Preparing the metadata which will be passed to the container Container Runtime to start the Container correctly
  • Using some defaults from the container image (ex.ArchX86)
  • Using user input to override defaults in the container image (ex. CMD, ENTRYPOINT)
  • Using defaults specified by the container image (ex. SECCOM rules)
• Calling the Container Runtime

For an even deeper understanding, please see Understanding the Container Standards. See also Container Runtime.

Container

Containers have existed within operating systems for quite a long time. A container is the runtime instantiation of a Container Image. A container is a standard Linux process typically created through a clone() system call instead of fork() or exec(). Also, containers are often isolated further through the use of cgroups, SELinux or AppArmor.

Container Host

The container host is the system that runs the containerized processes, often simply called containers. This could be, for example, RHEL Atomic Host running in a VM, as an instance in the public cloud, or on bare metal in your data center. Once a container image (aka repository) is pulled from a Registry Server to the local container host, it is said to be in the local cache.

Determining which repositories are synchronized to the local cache can be done with the following command:

[root@rhel7 ~]# docker images -a

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
registry.access.redhat.com/rhel7 latest 6883d5422f4e 3 weeks ago 201.7 MB

Registry Server

A registry server is essentially a fancy file server that is used to store docker repositories. Typically, the registry server is specified as a normal DNS name and optionally a port number to connect to. Much of the value in the docker ecosystem comes from the ability to push and pull repositories from registry servers.

When a docker daemon does not have a locally cached copy of a repository, it will automatically pull it from a registry server. Most Linux distributions have the docker daemon configured to pull from docker.io but it is configurable on some Linux distributions. For example, Red Hat Enterprise Linux is configured to pull repositories from registry.access.redhat.com first, then it will try docker.io (Docker Hub).

It is important to stress that there is implicit trust in the registry server. You must determine how much you trust the content provided by the registry and you may want to allow or block certain registries. In addition to security, there are other concerns such as users having access to licensed software and compliance issues. The simplicity with which docker allows users to pull software makes it critical that you trust upstream content.

In Red Hat Enterprise Linux, the default docker registry is configurable. Specific registry servers can be added or blocked in RHEL7 and RHEL7 Atomic by modifying the configuration file:

vi /etc/sysconfig/docker

In RHEL7 and RHEL 7 Atomic, Red Hat’s registry server is configured out of the box:

ADD_REGISTRY=’–add-registry registry.access.redhat.com’

As a matter of security, it may be useful to block public docker repositories such as DockerHub:

# BLOCK_REGISTRY=’–block-registry’

Red Hat also offers an integrated Registry Server with OpenShift Container Platform, a standalone enterprise Registry Server with Quay Enterprise, as well as cloud based, public and private repositories on Quay.io.

Container Orchestration

Often teams start with installing a Container Host, then pulling some Container Images. Then they move on to building some new Container Images and pushing them to a Registry Server to share with others on their team. After a while they want to wire a few containers together and deploy them as a unit. Finally, at some point, they want to push that unit into a pipeline (Dev/QA/Prod) leading towards production. This is the path towards the realization that orchestration is needed.

A container orchestrator really does two things:

1. Dynamically schedules container workloads within a cluster of computers. This is often referred to as distributed computing.
2. Provides a standardized application definition file (kube yaml, docker compose, etc)

The above two features provide many capabilities:

1. Allows containers within an application to be scheduled completely separately. This is useful if:
   • Allows the utilization of large clusters of Container Hosts
   • Individual containers fail (process hang, out of memory (OOM))
   • Container Hosts fail (disk, network, reboot)
   • Container Engines fail (corruption, restart)
   • Individual containers need to be scaled up, or scaled down
2. It’s easy to deploy new instances of the same application into new environments. In a cloud native world, or traditional world, there are many reasons why you might want to do this including:
   • On developers laptop with a container orchestrator running
   • On a shared development environment in a private namespace
   • On a shared development environment in an internally public namespace for visibility and testing
   • On a quality assurance (QA) environment internally
   • On a load test environment dynamically provisioned and deprovisioned in the cloud
   • On a gold environment to test compatibility with production
   • On a production environment
   • On a disaster recovery environment
   • On a new production environment which has upgraded Container Hosts, Container Engines, or container orchestrators
   • On a new production environment with the same versions of Container Host, Container Engine, and container orchestrator in a new geolocation (APAC, EMEA, etc)

There are many container schedulers being developed in the community and by vendors. Historically, Swarm, Mesos, and Kubernetes were the big three, but recently even Docker and Mesosphere have announced support for Kubernetes – as has almost every major cloud service provider. Kubernetes has become the defacto standard in container orchestration, similar to Linux before it. If you are looking at container orchestration, Red Hat recommends our enterprise distribution called OpenShift.

Container Runtime

A container runtime a lower level component typically used in a Container Engine but can also be used by hand for testing. The Open Containers Initiative (OCI) Runtime Standard reference implementation is runc. This is the most widely used container runtime, but there are others OCI compliant runtimes, such as crun, railcar, and katacontainers. Docker, CRI-O, and many other Container Engines rely on runc.

The container runtime is responsible for:

• Consuming the container mount point provided by the Container Engine (can also be a plain directory for testing)
• Consuming the container metadata provided by the Container Engine (can be a also be a manually crafted config.json for testing)
• Communicating with the kernel to start containerized processes (clone system call)
• Setting up cgroups
• Setting up SELinux Policy
• Setting up App Armor rules

To provide a bit of history, when the Docker engine was first created it relied on LXC as the container runtime. Later, the Docker team developed their own library called libcontainer to start containers. This library was written in Golang, and compiled into the original Docker engines. Finally, when the OCI was created, Docker donated the libcontainer code and turned it into a standalone utility called runc. Now, runc is the reference implementation and used by other Container Engines such as CRI-O. At the lowest level, this provides the ability to start a container consistently, no matter the container engine. Runc is a very terse utility and expects a mount point (directory) and meta-data (config.json) to be provided to it. See this tutorial for more on runc.

For an even deeper understanding, please see Understanding the Container Standards. See also Container Runtime.

Image Layer

Repositories are often referred to as images or container images, but actually they are made up of one or more layers. Image layers in a repository are connected together in a parent-child relationship. Each image layer represents changes between itself and the parent layer.

Below, we are going to inspect the layers of a repository on the local container host. Since Docker 1.7, there is no native tooling to inspect image layers in a local repository (there are tools for online registries). With the help of a tool called Dockviz, you can quickly inspect all of the layers. Notice that each layer has tag and a Universally Unique Identifier (UUID). The following command will returned shortened versions of the UUID that are typically unique enough to work with on a single machine. If you need to the full UUID, use the –no-trunc option.

docker run –rm –privileged -v /var/run/docker.sock:/var/run/docker.sock nate/dockviz images -t

├─2332d8973c93 Virtual Size: 187.7 MB
│ └─ea358092da77 Virtual Size: 187.9 MB
│ └─a467a7c6794f Virtual Size: 187.9 MB
│ └─ca4d7b1b9a51 Virtual Size: 187.9 MB
│ └─4084976dd96d Virtual Size: 384.2 MB
│ └─943128b20e28 Virtual Size: 386.7 MB
│ └─db20cc018f56 Virtual Size: 386.7 MB
│ └─45b3c59b9130 Virtual Size: 398.2 MB
│ └─91275de1a5d7 Virtual Size: 422.8 MB
│ └─e7a97058d51f Virtual Size: 422.8 MB
│ └─d5c963edfcb2 Virtual Size: 422.8 MB
│ └─5cfc0ce98e02 Virtual Size: 422.8 MB
│ └─7728f71a4bcd Virtual Size: 422.8 MB
│ └─0542f67da01b Virtual Size: 422.8 MB Tags: docker.io/registry:latest

Notice, that the “docker.io/registry” repository is actually made up of many images layers. More importantly, notice that a user could potentially “run” a container based off of any one of these layers. The following command is perfectly valid, though not guaranteed to have been tested or actually even work correctly. Typically, an image builder will tag (create a name for) specific layers that you should use:

docker run -it 45b3c59b9130 bash

Repositories are constructed this way because whenever an image builder creates a new image, the differences are saved as a layer. There are two main ways that new layers are created in a repository. First, if building an image manually, each “commit” creates a new layer. If the image builder is building an image with a Dockerfile, each directive in the file creates a new layer. It is useful to have visibility into what has changed in a container repository between each layer.

Tag

Even though a user can designate that a container mount and start from any layer in a repository, they shouldn’t necessarily do that. When an image builder creates a new repository, they will typically label the best image layers to use. These are called tags and are a tool for container image builders to communicate to container image consumers which layers are best to consume. Typically, tags are used to designate versions of software within in the repository. This is by convention only – in reality, the OCI nor any other standard mandates what tags can be used for, and they can be abused for anything that a user wants. Be careful dong this because it can create a lot of confusion in development, operations and architecture teams, so document it it well if you use it for anything other than software version.

There is one special tag – latest – which typically points to the layer containing the latest version of software in the repository. This special tag still just points to an image layer, like any other tag, so it can be abused.

To remotely view the tags available in a repository, run the following command (the jq utility makes the output a lot more readable):

curl -s registry.access.redhat.com/v1/repositories/rhel7/tags | jq
{
“7.0-21”: “e1f5733f050b2488a17b7630cb038bfbea8b7bdfa9bdfb99e63a33117e28d02f”,
“7.0-23”: “bef54b8f8a2fdd221734f1da404d4c0a7d07ee9169b1443a338ab54236c8c91a”,
“7.0-27”: “8e6704f39a3d4a0c82ec7262ad683a9d1d9a281e3c1ebbb64c045b9af39b3940”,
“7.1-11”: “d0a516b529ab1adda28429cae5985cab9db93bfd8d301b3a94d22299af72914b”,
“7.1-12”: “275be1d3d0709a06ff1ae38d0d5402bc8f0eeac44812e5ec1df4a9e99214eb9a”,
“7.1-16”: “82ad5fa11820c2889c60f7f748d67aab04400700c581843db0d1e68735327443”,
“7.1-24”: “c4f590bbcbe329a77c00fea33a3a960063072041489012061ec3a134baba50d6”,
“7.1-4”: “10acc31def5d6f249b548e01e8ffbaccfd61af0240c17315a7ad393d022c5ca2”,
“7.1-6”: “65de4a13fc7cf28b4376e65efa31c5c3805e18da4eb01ad0c8b8801f4a10bc16”,
“7.1-9”: “e3c92c6cff3543d19d0c9a24c72cd3840f8ba3ee00357f997b786e8939efef2f”,
“7.2”: “6c3a84d798dc449313787502060b6d5b4694d7527d64a7c99ba199e3b2df834e”,
“7.2-2”: “58958c7fafb7e1a71650bc7bdbb9f5fd634f3545b00ec7d390b2075db511327d”,
“7.2-35”: “6883d5422f4ec2810e1312c0e3e5a902142e2a8185cd3a1124b459a7c38dc55b”,
“7.2-38”: “6c3a84d798dc449313787502060b6d5b4694d7527d64a7c99ba199e3b2df834e”,
“latest”: “6c3a84d798dc449313787502060b6d5b4694d7527d64a7c99ba199e3b2df834e”
}

Repository

When using the docker command, a repository is what is specified on the command line, not an image. In the following command, “rhel7” is the repository.

docker pull rhel7

This is actually expanded automatically to:

docker pull registry.access.redhat.com/rhel7:latest

This can be confusing, and many people refer to this as an image or a container image. In fact, the docker images sub-command is what is used to list the locally available repositories. Conceptually, these repositories can be thought of as container images, but it’s important to realize that these repositories are actually made up of layers and include metadata about in a file referred to as the manifest (manifest.json):

docker images

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
registry.access.redhat.com/rhel7 latest 6883d5422f4e 4 weeks ago 201.7 MB
registry.access.redhat.com/rhel latest 6883d5422f4e 4 weeks ago 201.7 MB
registry.access.redhat.com/rhel6 latest 05c3d56ba777 4 weeks ago 166.1 MB
registry.access.redhat.com/rhel6/rhel latest 05c3d56ba777 4 weeks ago 166.1 MB
…

When we specify the repository on the command line, the Container Engine is doing some extra work for you. In this case, the docker daemon (not the client tool) is configured with a list of servers to search. In our example above, the daemon will search for the “rhel7” repository on each of the configured servers.

In the above command, only the repository name was specified, but it’s also possible to specify a full URL with the docker client. To highlight this, let’s start with dissecting a full URL.

Another way you will often see this specified is:

REGISTRY/NAMESPACE/REPOSITORY[:TAG]

The full URL is made up of a standard server name, a namespace, and optionally a tag. There are actually many permutations of how to specify a URL and as you explore the docker ecosystem, you will find that many pieces are optional. The following commands are all valid and all pull some permutation of the same repository:

docker pull registry.access.redhat.com/rhel7/rhel:latest
docker pull registry.access.redhat.com/rhel7/rhel
docker pull registry.access.redhat.com/rhel7
docker pull rhel7/rhel:latest

Namespace

A namespace is a tool for separating groups of repositories. On the public DockerHub, the namespace is typically the username of the person sharing the image, but can also be a group name, or a logical name.

Red Hat uses the namespace to separate groups of repositories based on products listed on the Red Hat Federated Registry server. Here are some example results returned by registry.access.redhat.com. Notice, the last result is actually listed on another registry server. This is because Red Hat works to also list repositories on our partner’s registry servers:

registry.access.redhat.com/rhel7/rhel
registry.access.redhat.com/openshift3/mongodb-24-rhel7
registry.access.redhat.com/rhscl/mongodb-26-rhel7
registry.access.redhat.com/rhscl_beta/mongodb-26-rhel7
registry-mariadbcorp.rhcloud.com/rhel7/mariadb-enterprise-server:10.0

Notice, that sometimes the full URL does not need to be specified. In this case, there is a default repository for a given namespace. If a user only specifies the fedora namespace, the latest tag from the default repository will be pulled to the local server. So, running the following commands is essentially the same, each one more specific:

docker pull fedora
docker pull docker.io/fedora
docker pull docker.io/library/fedora:latest

Kernel Namespace

A kernel namespace is completely different than the namespace we are referring to when discussing Repositories and Registry Servers. When discussing containers, Kernel namespaces are perhaps the most important data structure, because they enable containers as we know them today. Kernel namespaces enable each container to have it’s own mount points, network interfaces, user identifiers, process identifiers, etc.

When you type a command in a Bash terminal and hit enter, Bash makes a request to the kernel to create a normal Linux process using a version of the exec() system call. A container is special because when you send a request to a container engine like docker, the docker daemon makes a request to the kernel to create a containerized process using a different system call called clone(). This clone() system call is special because it can create a process with its own virtual mount points, process ids, user ids, network interfaces, hostname, etc

While, technically, there is no single data structure in Linux that represents a container, kernel namespaces and the clone() system call are as close as it comes.

Graph Driver

When the end user specifies the Tag of a container image to run – by default this is the latest Tag – the graph driver unpacks all of the dependent Image Layers necessary to construct the data in the selected Tag. The graph driver is the piece of software that maps the necessary image layers in the Repository to a piece of local storage. The container image layers can be mapped to a directory using a driver like Overlay2 or in block storage using a driver like Device Mapper. Drivers include: aufs, devicemapper, btrfs, zfs, and overlayfs.

When the container is started, the image layers are mounted read-only with a kernel namespace. The Image Layers from the Repository are always mounted read only but by default, a separate copy-on-write layer is also set up. This allows the containerized process to write data within the container. When data is written, it is stored in the copy-on-write layer, on the underlying host. This copy-on-write layer can be disabled by running the container with an option such as –readonly.

The docker daemon has it’s own set of Graph Drivers and there are other open source libraries which provide Graph Drivers such as containers/images which is used in tools like CRI-O, Skopeo and other container engines.

Determining which graph driver you are using can be done with the docker info command:

[root@rhel7 ~]# docker info

…
Storage Driver: devicemapper
Pool Name: docker-253:1-884266-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.037 GB
Data Space Total: 107.4 GB
Data Space Available: 2.56 GB
Metadata Space Used: 2.707 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)

There are many types of Container design patterns forming. Since containers are the run time version of a container image, the way it is built is tightly coupled to how it is run.

Some Container Images are designed to be run without privilege, while others are more specialized and require root-like privileges. There are many dimensions in which patterns can be evaluated and often users will see multiple patterns or use cases tackled together in one container image/container.

This section will delve into some of the common use cases that users are tackling with containers.

Application Containers

Application containers are the most popular form of container. These are what developers and application owner’s care about. Application containers contain the code that developers work on. They also include things like MySQL, Apache, MongoDB, and or Node.js.

There is a great ecosystem of application containers forming. Projects like Software Collections are providing secure and supportable applications container images for use with Red Hat Enterprise Linux. At the same time, Red Hat community members are driving some great cutting edge applications containers.

Red Hat believes that Application Containers should not typically require special privileges to run their workloads. That said, production container environments typically require much more than just non-privileged application containers to provide other supporting services.

Operating System Containers

See also System Containers

Operating System Containers are containers which are treated more like a full virtual operating system. Operating System Containers still share a host kernel, but run a full init system which allows them to easily run multiple processes. LXC and LXD are examples of Operating System Containers because they are treated much like a full virtual machine.

It is also possible to approximate an Operating System Container with docker/OCI based containers, but requires running systemd inside the container. This allows an end user to install software like they normally would and treat the container much more like a full operating system.

yum install mysql
systemctl enable mysql

This makes it easier to migrate existing applications. Red Hat is working hard to make Operating System Containers easier by enabling systemd to run inside a container and by enabling management with machined. While many customers aren’t (yet) ready to adopt micro-services, they can still gain benefits from adopting image based containers as a software delivery model.

Pet Containers

While Red Hat certainly recommends, supports and evangelizes the use cloud native patterns for new application development, in reality not all existing applications will be rewritten to take advantage of new patterns. Many existing applications are one of a kind, and one of kind applications are often referred to as Pets. Containers built specifically to handle these pet applications are sometimes referred to as Pet Containers

Pet containers provide users with the portability and convenience of a standardized container infrastructure relying on registry servers, container images, and standard container hosts for infrastructure, but provide the flexibility of a traditional environment within the container. The idea is to make things easier to containerize existing applications, such as using systemd in a container. The goal is to reuse existing automation, installers, and tools to easily create a container image that just runs.

Super Privileged Containers

When building container infrastructure on dedicated container hosts such as Red Hat Enterprise Linux Atomic Host, systems administrators still need to perform administrative tasks. Whether used with distributed systems, such as Kubernetes or OpenShift or standalone container hosts, Super Privileged Containers (SPC) are a powerful tool. SPCs can even do things like load specialized kernel modules, such as with systemtap.

In an infrastructure that is built to run containers, administrators will most likely need SPCs to do things like monitoring, backups, etc. It’s important to realize that there is typically a tighter coupling between SPCs and the host kernel, so administrators need to choose a rock solid container host and standardize on it, especially in a large clustered/distributed environment where things are more difficult to troubleshoot. They then need to select a user space in the SPC that is compatible with the host kernel.

Tools & Operating System Software

Linux distributions have always provided users with system software such as Rsyslogd, SSSD, sadc, etc. Historically, these pieces of system software were installed through RPM or DEB packages. But with the advent of containers as a packaging format, it has become both convenient and easy to install system software through containers images. Red Hat provides some pre-packaged containers for things like the Red Hat Virtualization tools, rsyslog, sssd, and sadc.

New design patterns are forming as more and more people deliver software with containers. Red Hat engineering is leveraging and driving many of these patterns in the community. The goal of this section is to help highlight and define some of these patterns.

The way a container is saved on disk (i.e. its image format) can have a dramatic affect on how it is run. For example, a container which is designed to run sssd needs to have special privileges whenever it’s run, or it can’t do its job. The following is a short list of patterns that are forming in the container community:

Application Images

These images are what end users consume. Use cases range from databases and web servers, to applications and services buses. These can be built in house or delivered to a customer from an ISV. Often end users will investigate and care about what bits were used to create a standalone image. Standalone images are the easiest kind of image to consume, but the hardest to design, build, and patch.

Base Images

A base image is one of the simplest types of images, but you will find a lot of definitions. Sometimes users will refer to corporate standard build, or even an application image as the “base image.” Technically this is not a base image. These are Intermediate images.

Simply put, a base image is an image that has no parent layer. Typically, a base image contains a fresh copy of an operating system. Base images normally include the tools (yum, rpm, apt-get, dnf, microdnf) necessary to install packages / make updates to the image over time. While base images can be “hand crafted”, in practice they are typically produced and published by open source projects (like Debian, Fedora or CentOS) and vendors (like Red Hat). The provenance of base images is critical for security. In short, the sole purpose of a base image is to provide a starting place for creating your derivative images. When using a dockerfile, the choice of which base image you are using is explicit:

FROM registry.access.redhat.com/rhel7-atomic

Builder Images

These are a specialized form of container image which produce application container images as offspring. They include everything but a developer’s source code. Builder images include operating system libraries, language runtimes, middleware, and the source-to-image tooling.

When a builder image is run, it injects the developers source code and produces a ready-to-run offspring application container image. This newly created application container image can then be run in development or production.

For example, if a developer has PHP code and they want to run it in a container, they can use a PHP builder image to produce a ready to run application container image. The developer passes the GitHub URL where the code is stored and the builder image does the rest of the work for them. The output of a Builder container is an Application container image which includes Red Hat Enterprise Linux, PHP from Software Collections, and the developer’s code, all together, ready to run.

Builder images provide a powerful way to go from code to container quickly and easily, building off of trusted components.

Containerized Components

A container is meant to be deployed as part of a larger software system, not on its own. Two major trends are driving this.

First, microservices are driving the use of best of breed components – this is also driving the use of more components combined together to build a single application. Containerized components are meeting the need to deploy an expanding quantity of complex software more quickly and easily. Each of these components can have different revisions, and container images help enable this. Application definitions such as Kubernetes/OpenShift deployments yaml/json, open service broker, OpenShift Templates, and Helm Charts are all making it possible to define applications at a higher level.

Second, not all pieces of software are easy to deploy as containers. Sometimes, it makes sense to containerize only certain components which are easier to move to containers or provide more value to the overall project. With multi-service application, some services may be deployed as containers, while others may be deployed through traditional a traditional methodology such as an RPM or installer script – see Pet Containers. But, other components might be difficult to put into containers because they are too tightly coupled to break up, need access to special hardware, or perhaps requires lower level kernel APIs, etc. Within a larger application there will likely be parts of the application that can be containerized, and parts that can’t. Containerized components represent the parts that can and are containerized. Containerized components are intended to be ran as part of a specific application, not standalone. It’s important to understand that containerized components are not designed to function on their own. They provide value to a larger piece of software, but provide very little value on their own.

For example, when OpenShift Enterprise 3.0 was released, most of the core code was deployed using RPMs, but after installation administrators deployed the router and registry as containers. With the release of OpenShift 3.1 an option was added to the installer to deploy the master, node, openvswitch and etcd components as containers – after installation, administrators were given the option to deploy elasticsearch, fluentd, and kibana as containers.

While the OpenShift installer still makes modifications to a server’s file system, all of the major software components can now be installed using container images. What makes these containerized components is that, for example, an instance of the etcd image built into OpenShift should and would never be used to store data for your customer facing application code, because it is a containerized component designed to be run as part of OpenShift Container Platform.

With the latest releases of OpenShift, there is a trend towards more and more containerized components. The containerized component pattern is becoming more and more common and other software vendors are seeing an advantage to deploying as containerized components.

Deployer Images

A deployer image is a specialized kind of container which, when run, deploys or manages other containers. This pattern enables sophisticated deployment techniques such as mandating the start order of containers, or first run logic such as populating schema or data.

As an example, the “image/container type” pattern is used to deploy the logging and metrics in OpenShift. Deploying these components with a deployer container allows the OpenShift engineering team to control start order of the different components and make sure they are all up and running together.

Intermediate Images

An Intermediate image is any container image that relies on a base image. Typically, core builds, middleware and language runtimes are built as layers on “top of” a base image. These images are then referenced in the FROM directive of another image. These images are not used on their own, they are typically used as a building block to build a standalone image.

It is common to have different teams of specialists own different layers of an image. Systems administrators may own the core build layer, while “developer experience” may own the middleware layer. Intermediate Images are built to be consumed by other teams building images, but can sometimes be run standalone too, especially for testing.

Intermodal Container Images

Intermodal container images, analogous to intermodal shipping containers, are images that have hybrid architectures. For example, many Red Hat Software Collections images can be used in two ways. First, they can be used as simple Application Containers running a fully contained Ruby on Rails and Apache server. Second, they can be used as Builder Images inside of OpenShift Container Platform. In this case, the output child images which contain Ruby on Rails, Apache, and the application code which the source to image process was pointed towards during the build phase.

The intermodal pattern is becoming more and more common to solve two business problems with one container image.

System Containers

When system software is distributed as a container, it often needs to run super privileged. To make this deployment easier, and to allow these containers to start before the container runtime or orchestration, Red Hat developed a special container pattern called System Containers. System Containers start early in the boot process and rely on the atomic command and systemd to be started independent of any container runtime or orchestration. Red Hat provides System Containers for many pieces of software including rsyslog, cockpit, etcd, and flanneld. In the future, Red Hat will expand the list.

This design pattern will make it easier for administrators to add these services to Red Hat Enterprise Linux and Atomic Host in a modular way.

Containers are quite easy to consume, but when building a production container environment, it shifts the complexity behind the scenes. To be able to discuss architectures, and how you will build your environment, it’s important to have shared nomenclature. There are a lot of pitfalls as you dig deeper into building and architecting your environment. We leave you with a couple of critical ones to remember.

People often use the words container image and repository interchangeably and the docker sub-commands don’t make a distinction between an image and a repository. The commands are quite easy to use, but once architecture discussions start, it’s important to understand that a repository is really the central data structure.

It’s also quite easy to misunderstand the difference between a namespace, repository, image layer, and tag. Each of these has an architectural purpose. While different vendors, and users are using them for different purposes, they are tools in our toolbox.

The goal of this article is to leave you with the ability to command this nomenclature so that more sophisticated architectures can be created. For example, imagine that you have just been charged with building an infrastructure that limits, based on role, which namespaces, repositories, and even which image layers and tags can be pushed and pulled based on business rules. Finally, remember that how a container image is built will have profound effect on how it is to be run (orchestrated, privileged, etc).

For further reading, check out the Architecting Containers series:

• Architecting Containers Part 1: Why Understanding User Space vs. Kernel Space Matters
• Architecting Containers Part 2: Why the User Space Matters
• Architecting Containers Part 3: How the User Space Affects Your Application

As always, if you have comments or questions, please leave a message below.

Source

Debugging Container and Docker Networking issues can be daunting at first considering that containers do not contain any debug tools inside the container. I normally see a lot of questions around Docker networking issues in Docker and stackoverflow forum. All the usual networking tools can be used to debug Docker networking, it is just that the approach taken is slightly different. I have captured my troubleshooting steps in a video and a presentation.
Following is the video and presentation of my Docker Networking troubleshooting tips.
I would appreciate if you can provide me feedback if the Networking tip videos were useful to you. Also, if there are any other Docker Networking topics that you would like to see as a tip video, please let me know.
I have also put few Docker Networking videos and presentations that I did over last 3 months below for completeness.
Following are the 2 previous Networking tip videos.
Following are 2 Docker Networking deep dive presentations:
Source