Category: Kubernetes

So last week I finally got out from my “tech” comfort zone, and tried
to set up a Node.js application which uses a MongoDB database, and to
add an extra layer of fun I used Rancher to set up the
whole application stack using Docker containers.

I designed a small application with Node, its only function is to
calculate the number of hits on the website, you can find the code at
Github

The setup was to add an Nginx container as a load balancer at the
front-end to serve two back-end Node application containers, and then
have the two Node servers connect to a MongoDB database container. In
this setup I will use 5 machines from Digital Ocean, 4 to build the
application stack with the highest availability, and the 5th as a
Rancher server.

[]Set Up A Rancher Server

On a Digital Ocean machine with Docker 1.4 installed we will apply the
following command to set up a Rancher platform on the port 8000:

root@Rancher-Mngmt:~# docker run -d –name rancher-server -p 8080:8080 rancher/server

The previous command will run a docker instance with rancher platform,
and proxy the port 8080 on the instance to the same port on the Digital
Ocean machine. To make sure that the server is running type this
command:

root@Rancher-io-Mngmt:~# docker logs rancher-server

You should see something like the following output:

20:02:41.943 [main] INFO ConsoleStatus – [DONE ] [68461ms] Startup Succeeded, Listening on port 8080

To access Rancher now, type the following url in your browser:

http://DO-ip-address:8080/
you should see something like the following:

[]Register Digital Ocean’s instances With Rancher

To register the Digital Ocean machines with docker 1.4 installed with
Rancher, type the following on each machine:

root@Rancher-Test-Instance-X# docker run -it –privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent http://rancher-server-ip:8080

where rancher-server-ip is the ip address of the
Rancher server we just installed, or you can click on “Register a New
Host “ on Rancher platform and copy the command shown.

After applying the previous command on each machine you should see
something like the following when you access the Rancher management
server:

If you are familiar with Ansible as a configuration management
tool, you can use it to register the Digital Ocean machines with Rancher
in one command:

• First, add the ips of the Digital Ocean machines in
  /etc/ansible/hosts under one group name:

[DO]
178.62.101.243
178.62.27.24
178.62.98.242
178.62.11.154

• Now, run the following command to register all machines at
  once:

$ ansible DO -u root -a “docker run -it –privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent http://rancher-server-ip:8080”

MongoDB Docker Container

After Registering the 4 machines with Rancher, its time to start
building our application stack.

The node.js application will calculate the number of hits on a
website, so it needs to store this data somewhere. I will use MongoDB
container to store the number of hits.

The Dockerfile will be like the following:

FROM ubuntu:14.04
MAINTAINER hussein.galal.ahmed.11@gmail.com
ENV chached_FLAG 0
RUN apt-get -qq update && apt-get -yqq upgrade
RUN apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv 7F0CEB10
RUN echo ‘deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen’ | tee /etc/apt/sources.list.d/10gen.list
RUN apt-get update && apt-get install -yqq mongodb-org
RUN mkdir -p /data/db
EXPOSE 27017
ADD run.sh /tmp/run.sh
ADD init.json /tmp/init.json
ENTRYPOINT [“/bin/bash”, “/tmp/run.sh”]

The previous Docker file is really simple, lets explain it line by
line:

• First update the apt cache and install latest updates:

RUN apt-get -qq update && apt-get -yqq upgrade

• Add the key and the mongodb repo to apt sources.list:

RUN apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv 7F0CEB10
RUN echo ‘deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen’ | tee /etc/apt/sources.list.d/10gen.list

• Install the MongoDB package which installs the server and the
  client:

RUN apt-get update && apt-get install -yqq mongodb-org

• Create the directory which will store the MongoDB files:

• Expose the port 27017, which is the default port to connect to
  MongoDB:

• Add two files to the container:

• init.json: the initial database to start the
  application.
• run.sh: will import the init.json database to the
  MongoDB server and ran the server.

ADD run.sh /tmp/run.sh
ADD init.json /tmp/init.json

• Finally, it will add entrypoint to the container to be started with
  executing the run.sh file:

ENTRYPOINT [“/bin/bash”, “/tmp/run.sh”]

Let’s take a look at the run.sh file:

#!/bin/bash
/usr/bin/mongod &
sleep 3
mongoimport –db countdb –collection hits –type json –file /tmp/init.json
/usr/bin/mongod –shutdown
sleep 3
/usr/bin/mongod

The server started first to be able to import the init.json database to
the countdb database and hits collection, then shutdown the server and
start it up again but in the foreground this time.

The init.json database file:

Node.js Application Container

The Node.js container will install node.js and git packages, and
then will run a simple script to update the /etc/hosts file with the ip of the MongoDB container provided by the
environment variable: $MONGO_IP.

FROM ubuntu:14.04
MAINTAINER hussein.galal.ahmed.11@gmail.com
ENV CACHED_FLAG 1

# Install node
RUN apt-get update -qq && apt-get -y upgrade
RUN apt-get install -yqq nodejs git git-core
VOLUME [ “/var/www/nodeapp” ]
ADD ./run.sh /tmp/run.sh# Install Dependencies
WORKDIR /var/www/nodeapp

# Run The App
ENTRYPOINT [“/bin/bash”, “/tmp/run.sh”]

The ENTRYPOINT of the Docker container is executing the
/tmp/run.sh script:

MONGO_DN=mongo
if [ -n “$MONGO_IP” ]
then
echo “$MONGO_IP $MONGO_DN” >> /etc/hosts
fi

# Fetch the application
git clone https://github.com/galal-hussein/hitcntr-nodejs.git
mv hitcntr-nodejs/* .
rm -rf hitcntr-nodejs

# Run the Application
nodejs index.js

The previous script will check for the MONGO_IP environment variable and if it is set, it will add the content of
this variable to /etc/hosts, then pull the code from
Github Repo, and finally run the node application.

Nginx Container

The Dockerfile of the Nginx container will install nginx webserver and
add the configuration files, and ran a script to update /etc/hosts file
like the Node.js container, and finally run the web server.

Nginx Dockerfile:

#dockerfile for nginx/nodejs
FROM ubuntu:14.04
MAINTAINER hussein.galal.ahmed.11@gmail.com
ENV CACHED_FLAG 0

# Install nginx
RUN apt-get update -qq && apt-get -y upgrade
RUN apt-get -y -qq install nginx

# Adding the configuration files
ADD conf/nginx.conf /etc/nginx/nginx.conf
ADD conf/default /etc/nginx/conf.d/default
ADD ./run.sh /tmp/run.sh

# Expose the port 80
EXPOSE 80

# Run nginx
ENTRYPOINT [ “/bin/bash”, “/tmp/run.sh” ]

The Dockerfile is very simple and use the same commands like the
previous images.

run.sh:

NODE_1_DN=node_app1
NODE_2_DN=node_app2
if [ -n “$NODE_APP1_IP” ]
then
echo “$NODE_APP1_IP $NODE_1_DN” >> /etc/hosts
fi
if [ -n “$NODE_APP2_IP” ]
then
echo “$NODE_APP2_IP $NODE_2_DN” >> /etc/hosts
fi
# Run Nginx
/usr/sbin/nginx

Since we are using two Node application servers, we need to proxy the
http requests received by Nginx to those servers and to do that we need
to add the ips of the Node.js containers to the hosts file.

The ips of the Node.js containers are defined by the two environment
variables (NODE_APP1_IP, and NODE_APP2_IP).

Build And Push The Images

Now for the final step, build and then push the images to Docker
hup:

~/rancher_vm# docker build -t husseingalal/nodeapp_mongo mongo/
~/rancher_vm# docker build -t husseingalal/nodeapp_node node/
~/rancher_vm# docker build -t husseingalal/nodeapp_nginx nginx/
~/rancher_vm# docker push husseingalal/nodeapp_mongo
~/rancher_vm# docker push husseingalal/nodeapp_node
~/rancher_vm# docker push husseingalal/nodeapp_nginx

Now Docker will ask you for your account credentials, then the images
will be pushed to the Docker hub to be used later with Rancher.

Set Up The Application Stack

1. At Rancher platform, create at the first host a Docker container
   using the MongoDB image we just created:

Note that the option “Manage Network on docker0“
was chosen to make sure that we will enable one of the unique features
of Rancher which is cross container networking, this feature enables
Docker containers on different hosts to communicate in a virtual private
network.

After clicking Create, you should see that the machine is started to
download the image and install it along with another docker instance
called Network Agent which is used to create the virtual private network
we just talked about.

1. The second step is to add the the two Node.js Application servers
   which are connected to the MongoDB database:

Note that we used the Node.js image we just created, before creating
the container make sure to add the MONGO_IP environment variable to add the ip of the MongoDB server, you can
get the private ip of the MongoDB server from the Rancher panel:

After that click Create to begin the creation process of the Node.js
container. On the second host create the second Node.js Application
container using the same steps.

1. The final step is to create the Nginx webserver container on the
   last host:

Since the nginx instance will be facing the internet, we should proxy
the port 80 from inside the container to the port 80 of the Digital
Ocean machine:

We need also to add the ips of the two Node.js application servers
which are connected to Nginx, you can add the ips through creating two
environment variables (NODE_APP1_IP, NODE_APP2_IP):

Now wecan access the application using the ip address of the Host
machine http://<the-ip-address>.

Conclusion

In part 1 of this series, I created a Node.js application stack using
Docker containers and Rancher platform. The stack consists of Nginx
container which balances the load between two Node.js application
containers and using MongoDB as our database.

In part 2
I introduce one of the newest features of Rancher: Github
Authentication, also I will use Github WebHooks feature for automatic deployment of the web application.

If you’d like to learn more about Rancher, please schedule a
demo:

Hussein Galal is a Linux System Administrator, with experience in
Linux, Unix, Networking, and open source technologies like Nginx,
Apache, PHP-FPM, Passenger, MySQL, LXC, and Docker. You can follow
Hussein
on Twitter @galal_hussein.*

Source

In the first part of
this post,
I created a full Node.js application stack using MongoDB as the
application’s database and Nginx as a load balancer that distributed
incoming requests to two Node.js application servers. I created the
environment on Rancher and using Docker containers.

In this post I will go through setting up Rancher authentication with
GitHub, and creating a webhook with GitHub for automatic
deployments.

[]Rancher Access Control

Starting from version 0.5, Rancher can be configured to restrict
access to a set of GitHub users and organization members (you can read a
blog about it
here).
Using this feature ensures that no one other than authorized users can
access Rancher server through the web UI.

After setting up the rancher server, you should see message that says
“Access Control is not configured” :

Click on settings and on the Access Control panel you will be
instructed on how to setup and register new application with GitHub. The
instructions will provide you with a
link to GitHub application settings.

Now on GitHub Application Settings page, click on Register new
application:

Now you will put some information about Rancher’s server:

Application name: any name you choose

Homepage URL: Rancher server url

Application description: any description

Authorization callback URL: also Rancher server url.

After clicking on Register Application, you will be provided with
a Client ID and Client Secret, which are both used to register the user
to the Rancher server:

Now add the Client ID and Client Secret to the Rancher management
server, click on Authenticate with Github:

If everything went well, you should see something like the
following:

Now you have authorized a GitHub user account to your Rancher
management server, and can start adding users and organizations from
GitHub to Rancher projects.

[]Automatic Deployment Using Webhooks

Webhooks can provide an efficient way for changing the application’s
content using HTTP callbacks for specific events, in this configuration
I will register a couple of webhooks with GitHub to send a POST request
to a custom URL.

There are a number of ways to create an automatic deployment setup for
your app, I decided to use the following approach:

• Create a webhook on Github for each push.
• Modify the Node.js Docker instances with:
• A webhook handler in Node.js. – A script that pulls the new
  pushed repo.
• Start the Application with Nodemon, supervisor, or PM2 to restart on
  each modification.
• Start the Handler with any port, and proxy this port to the
  corresponding port of the host machine.

Let’s go through our solution in more detail:

The new Node.js Application Container

First we need to modify the Node.js Docker image which i created in the
first post. Now it has to contain the Hook handler program plus the
re-deploy script, also we should start the main application using
Nodemon, the new Dockerfile:

# Dockerfile For Node.js App
FROM ubuntu:14.04
MAINTAINER hussein.galal.ahmed.11@gmail.com
ENV CACHED_FLAG 1

# Install node and npm
RUN apt-get update -qq && apt-get -y upgrade
RUN apt-get install -yqq nodejs npm git git-core

# Install nodemon
RUN npm install -g nodemon
VOLUME [ “/var/www/nodeapp” ]

# Add redeploy script and hook handler
ADD ./run.sh /tmp/run.sh
ADD ./redeploy.sh /tmp/redeploy.sh
ADD ./webhook.js /tmp/webhook.js
WORKDIR /var/www/nodeapp
# Expose both ports (app port and the hook handler port)
EXPOSE 8000
EXPOSE 9000

# Run The App
ENTRYPOINT [“/b2in/bash”, “/tmp/run.sh”]

You should notice that a two new files were added to this Dockerfile:
the webhook.js which is the hook handler, and redeploy.sh script which
is basically a git pull from the GitHub repo.

The webhook.js handler

I wrote the webhook handle in NodeJS:

var http = require(‘http’)
var createHandler = require(‘github-webhook-handler’)
var handler = createHandler({ path: ‘/’, secret: ‘secret’ })
var execFile = require(‘child_process’).execFile;
//Create Server That Listen On Port 9000
http.createServer(function (req, res) {
handler(req, res, function (err) {
res.statusCode = 404
res.end(‘no such location’)
})
}).listen(9000)

//Hook Handler on Error
handler.on(‘error’, function (err) {
console.error(‘Error:’, err.message)
})

//Hook Handler on Push
handler.on(‘push’, function (event) {
console.log(‘Received a push event for %s to %s’,
event.payload.repository.name,
event.payload.ref)
execFile(‘/tmp/redeploy.sh’, function(error, stdout, stderr) {
console.log(‘Error: ‘+error)
console.log( ‘Redeploy Completed’ );
});
})

I won’t go into the details of the code, but here are some notes that
you should consider:

• I used
  github-webhook-handler library.
• The handler will use a secret string that will be configured later
  using GitHub.
• The handler will listen on port 9000.
• The handler will execute redeploy.sh.

The redeploy.sh script:

sleep 5
cd /var/www/nodeapp
git pull

The last script is the run script which used to start the handler and
the application:

MONGO_DN=mongo
if [ -n “$MONGO_IP” ]
then
echo “$MONGO_IP $MONGO_DN” >> /etc/hosts
fi
ln -s /usr/bin/nodejs /usr/bin/node
chmod a+x /tmp/redeploy.sh

#fetch the app
git clone https://github.com/galal-hussein/hitcntr-nodejs.git .
cd /tmp
npm install github-webhook-handler
nodejs webhook.js &

# Run the Application
cd /var/www/nodeapp
nodemon index.js

Now build and push the image like I did in the previous post.

Add Webhook With Github

To create a webhook on Github, open the repository → settings →
Webhooks & Services then Add Webhook:

Now add a custom url which will be notified when the specified events
happen:

You should add the secret token which we specified previously in the
handler’s code. Add a second webhook but this time with the url of
the second application, then build the application stack like we did in
the previous post, but this time proxy port 9000 at the Node container:

After building the stack check the Github webhooks, and you should see
something like this:

Now let’s test the webhooks, if you accessed the url of the Nginx web
server you will see something like this:

Now commit any changes to your code and push it on Github, and the
changes will be applied immediately to the app servers, in our case I
changed the “hits” to be “Webhooks Worked, Hits”:

Conclusion

In this two post series, I created a simple Node.js application with
MongoDB as a NoSQL database and used Rancher to build the whole stack
with Docker containers. In the second post I used the authentication
feature of Rancher with GitHub accounts, then I used webhooks to build
an automatic deployment solution.

I hope this helps you understand how to leverage Rancher, Docker and
GitHub to better manage application deployments.

If you’d like to learn more about using Rancher, please don’t hesitate
to schedule a demo and discussion with one of our
engineers.

Source

This
week we released Rancher 0.12, which adds support for provisioning hosts
using Docker Machine. We’re really excited to get this feature out,
because it makes launching Rancher-enabled Docker hosts easier than
ever. If you’re not familiar with Docker Machine, it is a project that
allows cloud providers to develop standard “drivers” for provisioning
cloud infrastructure on the fly. You can learn more about it on the
Docker
website.
The first cloud we’re supporting with Docker Machine is Digital Ocean.
For our initial release, we chose Digital Ocean, because it is an
excellent implementation of the machine driver. As always, the Digital
Ocean team has focused on simplicity and user experience, and were
fantastic to work with during our testing. Docker machine drivers are
already available for many public cloud providers, as well as vCenter,
CloudStack, OpenStack and other private cloud platforms. We will be
adding support for additional drivers over the next few weeks, and
documenting how you can use any driver you like. Please feel free to
let us know if there
are drivers you would like us to prioritize. Now, let me walk you
through using Docker Machine with Rancher. To get started, click on the
“Regsiter a New Host” link in the Hosts tab within Rancher.

If this is the first time you’ve added a host, you’ll be presented
with a Host Setup dialog that asks you to confirm the DNS host name or
IP address that hosts should use to connect to the Rancher API. Confirm
this setting and click Save.

Once that is completed, you’ll be taken to the Add Host page,
where you’ll see a new tab for provisioning Digital Ocean hosts.

To provision a Digital Ocean machine, fill out the relevant
information about the host you want to provision, inlcuding the OS
image, size and Digital Ocean region. You’ll need to have a Digital
Ocean access token, which you can get by creating an
account on their
site. Once you hit create, you’ll be returned to the hosts page where
you will see your new host being created.

Creating the host will take a few minutes, as the VM needs to be
provisioned, configured with Docker, and bootstrapped as a Rancher host.
But once it’s done, the UI will automatically update to show the new
host. At this point, you have a fully enabled Docker host. You can click
the Add Container link to start adding containers. We hope you find this
feature useful and welcome your feedback. As always, you can submit any
feature requests or other issues to the Rancher GitHub
repo. In the next few weeks,
we’ll be adding the ability to export the Docker machine configuration
so that you can deploy containers outside of Rancher, more verbose
status updates during machine creation, and (of course) more Machine
drivers. If you’d like to talk with one of our engineers and learn more
about Rancher, please feel free to request a demo, and we’ll walk you
through Rancher and answer all of your questions.

Source

As you may have seen, Rancher recently announced our integration
with docker-machine. This
integration will allow users to spin up Rancher compute nodes across
multiple cloud providers right from the Rancher UI. In our initial
release, we supported Digital Ocean. Amazon EC2 is soon to follow and
we’ll continue to add more cloud providers as interest dictates. We
believe this feature will really help the Zero-to-Docker _(and
Zero-to-Rancher)_ experience. But the feature itself is not the focus
of this post. In this post, I want to detail the software architerture
employed to achieve this integration. First, it’s important to
understand that everyhting in Rancher is an API resource with a process
lifecycle. Containers, images, networks, and accounts are all API
resources with their own process lifecycles. When you deploy a machine
in the Rancher UI, you’re creating a machine resource. It has three
life cycle processes: 1. Create 2. Bootstrap 3. Delete The create
process is kicked off when the user creates a machine in the UI. When
the create process completes, it auotmatically kicks off the bootstrap
process. Delete (perhaps obviously) occurs when the user chooses to
delete or destroy the host. Our integration with machine is achieved
through a microservice that hooks into Rancher machine lifecycle events
and execs out to the docker-machine binary accordingly. You can check
out the source code for this service here:
https://github.com/rancherio/go-machine-service.
Logically, the interaction looks like this:

…Sorry for the bad graphic. Anyway… When you spin up Rancher
with docker run rancher/server … with the default configuration, the
Rancher API, Rancher Process Server, DB, and Machine Microservice are
all processes living inside that container (and in fact, the API and
process server are the same process). The docker-machine binary is in
the container as well but only runs when it is called. You may at this
point be wondering about that event bus. In Rancher, we keep eventing
dead-simple and above all follow this principle:

There is no such thing as reliable messaging.

So, that “event bus” consists of the microservice making a POST
request to the /subsribe API endpoint. The response is a stream of
newline-terminated json events, similar in concept to the docker event
stream. The
process server is responsible for firing (and refiring) events until it
receives a reply event (another API POST) indicating the event was
handled. Further event handlers are blocked until the current event
handler replies successfully. The microservice is responsible for
handling the events, replying, and acting idempotently so that refires
can occur without ill-effect. So when the machine microservie receives a
create event, it translate the machine API resource’s prooperties into
a docker-machine cli command and execs out to it. Since the machine
creation process is long lived, the service monitors the standard out
and error of the call and sends corresponding status updates to the
Rancher server. These are then presented to the user in the UI. When
docker-machine reports that the machine was successfully created, the
microservice will reply to the original event it received from the
Rancher server. The successful end of the create event will cause the
process server to automatically kick off the bootstrap event, which
makes it way right back down to the machine microservice. When that
event is received, we’ll again exec out to docker-machine to get the
details needed to connect to the machine’s docker daemon. We do this by
executing the docker-machine config command and parsing the response.
With the connection parameters in hand, the service fires up a rancher
agent on the machine via docker run … rancher/agent …. This is the
exact same command that a user would run if they wanted to manaully join
a server to Rancher. When that container is up and running, it will
report into the Rancher server and start hooking into container
lifecycle events in much the same way that this service hooks into
machine lifecycle events. From there, it’s business as normal for the
Rancher server and the machine’s rancher-agent. That about does it for
the technical architecture of our docker-machine integration. There are
a lot more interesting but minor technical detail to share, but I
didn’t want to go too far off into the weeds in this post. I’ll write
up some follow up post sharing those details in the not-too-distant
future. Finally, shout out (and thanks) to Evan
Haslett, Ben
Firshman, and the rest of the docker-machine
team and community for the help along the way. We look forward to more
exciting work with the docker-machine, including getting RancherOS in
there. If you’d like to learn more about Rancher, please schedule a
demo and we’ll walk you through the latest features, and our
future roadmap. Note: This post also appears
on Craig’s personal blog
here.
Feel free to check out that blog for more software engineering
insights.

Source

When we shipped Rancher 0.12 last week we added one of the more
frequently requested features, support for private Docker registries.
Rancher had always allowed users to provision containers from
DockerHub, but many organizations run their own registries, or use
private hosted registries such as Quay.io, and
private DockerHub accounts. Beginning with
this release, users will be able to connect their private registry
directly to their Rancher environment, and deploy containers
from private Docker images. To use this new feature navigate to the new
“Registries” tab on your Rancher instance. You’ll see that you now
have the option to “Add a Privae Registry,” then fill out the form,
add your credentials and you are done. Credentials aren’t required
unless the registry or images you want to use require a password.
Once you’ve set up your private registry, you’ll be able to launch
containers from images hosted in your private registry from the launch
container workflow. You can access your private registry by clicking on
the “docker” image, and selecting the name of your private registry.
From that point, simply provide the name of your Docker image on the
private registry and continue the provisioning process.
The video below gives a more complete explanation of how to register a
private registry and credentials within Rancher. Hope you enjoy the new
feature. If you would like to set up some time to talk with us about
getting started with Rancher, please request a
demonstration.

Source

Recently I have been playing around with Riak and I wanted to get it
running with Docker, using RancherOS and Rancher. If you’re not
familiar with Riak, it is a distributed key/value store which is
designed for high availability, fault tolerance, simplicity, and
near-linear scalability. Riak is written in Erlang programming language
and it runs on an Erlang virtual machine. Riak provides availability
through replication and faster operations and more capacity through
partitions, using the ring design to its cluster, hashed keys
are partitioned by default to 64 partitions (or vnodes), each vnode will
be assigned to one physical node as following:
From Relational to Riak
Whitepaper
For example, if the cluster consists of 4 nodes: Node1, Node2, Node3,
and Node4, we will count around the nodes assigning each vnode to a
physical node until the all vnodes are accounted for, so in the previous
figure, Riak used 32 partition with 4 node cluster so we get:

Node0 : [1, 5, 9, 13, 17, 21, 25, 29]
Node1 : [2, 6, 10, 14, 18, 22, 26, 30]
Node3 : [3, 7, 11, 15, 19, 23, 27, 31]
Node4 : [4, 8, 12, 16, 20, 24, 28, 32]

So how about replication? Every time a write process happens Raik will
replicate the value to the next N vnodes, where N is the value of
the n_val setting in Riak cluster. By default, N is 3. To explain
this, assume we will use the default n_val value and we will use
the previous cluster setup with 4 nodes and 32 partitions, now lets
assume we will write a key/value to partition (vnode) 2 which is
assigned to the second node then the value will be replicated to vnode 3
and vnode 4 which are assigned to the 3rd and 4th nodes respectively.
For more information about Riak cluster, visit the official riak
documentation. In this post, I am
going to deploy Riak cluster using Docker on RancherOS, the setup will
include:

• Five Docker containers as Riak nodes.
• Each Container will be on separate EC2 Instance.
• RancherOS will be installed on each EC2 instance.
• The whole setup will be managed using Rancher platform.

##

The Riak Docker Image

Before launching your EC2 instances and the Rancher platform, you should
create the Riak Docker image that will run each instance. I used the
implementation of Riak Docker image of
hectcastro, although I
added and removed some parts to become suitable to run on RancherOS.
First the Dockerfile:

FROM phusion/baseimage:latest
MAINTAINER Hussein Galal hussein.galal.ahmed.11@gmail.com

RUN sed -i.bak ‘s/main$/main universe/’ /etc/apt/sources.list
RUN apt-get update -qq && apt-get install -y software-properties-common &&
apt-add-repository ppa:webupd8team/java -y && apt-get update -qq &&
echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections &&
apt-get install -y oracle-java7-installer

# Install Riak
RUN curl https://packagecloud.io/install/repositories/basho/riak/script.deb | bash
RUN apt-get install -y riak

# Setup the Riak service
RUN mkdir -p /etc/service/riak
ADD scripts/riak.sh /etc/service/riak/run

RUN sed -i.bak ‘s/listener.http.internal = 127.0.0.1/listener.http.internal = 0.0.0.0/’ /etc/riak/riak.conf && sed -i.bak ‘s/listener.protobuf.internal = 127.0.0.1/listener.protobuf.internal = 0.0.0.0/’ /etc/riak/riak.conf &&
echo “anti_entropy.concurrency_limit = 1” >> /etc/riak/riak.conf &&
echo “javascript.map_pool_size = 0” >> /etc/riak/riak.conf &&
echo “javascript.reduce_pool_size = 0” >> /etc/riak/riak.conf &&
echo “javascript.hook_pool_size = 0” >> /etc/riak/riak.conf

# Add Automatic cluster support
ADD scripts/run.sh /etc/my_init.d/99_automatic_cluster.sh
RUN chmod u+x /etc/my_init.d/99_automatic_cluster.sh
RUN chmod u+x /etc/service/riak/run

# Enable insecure SSH key
RUN /usr/sbin/enable_insecure_key.sh

EXPOSE 22 8098 8087
CMD [“/sbin/my_init”]

A couple of notes on the previous Dockerfile. The phusion/baseimage is
used as the Docker base image, 2 important scripts were added to the
image (riak.sh, automatic_cluster.sh) which I will explain in a second,
the ports 8098 and 8087 are used for HTTP and Protocol Buffers and
finally ssh support through insecure key was added. The purpose of the
riak.sh script is to start the Riak service and ensure that the node
name is set correctly, while the automatic_cluster.sh script is used to
join the node to the cluster only if the RIAK_JOINING_IP is set
during the starting of the contianer. riak.sh

#! /bin/sh

# Ensure correct ownership and permissions on volumes
chown riak:riak /var/lib/riak /var/log/riak
chmod 755 /var/lib/riak /var/log/riak

# Open file descriptor limit
ulimit -n 4096
IP_ADDRESS=$(ip -o -4 addr list eth0 | awk ” | cut -d/ -f1 | sed -n 2p)

# Ensure the Erlang node name is set correctly
sed -i.bak “s/riak@127.0.0.1/riak@$/” /etc/riak/riak.conf
rm -rf /var/lib/riak/ring/*

# Start Riak
exec /sbin/setuser riak “$(ls -d /usr/lib/riak/erts*)/bin/run_erl” “/tmp/riak”
“/var/log/riak” “exec /usr/sbin/riak console”

automatic_cluster.sh

#!/bin/sh
sleep 10
if env | grep -q “RIAK_JOINING_IP”; then
# Join node to the cluster
(sleep 5;riak-admin cluster join “riak@$” && echo -e “Node Joined The Cluster”) &

# Are we the last node to join?
(sleep 8; if riak-admin member-status | egrep “joining|valid” | wc -l | grep -q “$”; then
riak-admin cluster plan && riak-admin cluster commit && echo -e “nCommiting The Changes…”
fi) &
fi

Also note that RIAK_CLUSTER_SIZE is used to specify the size of the
cluster used in this setup. We don’t need more than that to start the
cluster, now build the image and push it to Docker Hub to be used later.

# docker build -t husseingalal/riak2 .
# docker push husseingalal/riak2

Launch Rancher Platform

The Rancher Management platform will be used manage the Docker
containers on RancherOS instances. First you need to run Rancher
platform on a machine using the following command:

# docker run -d -p 8080:8080 rancher/server

Create RancherOS EC2 Instances

RancherOS is available as an Amazon Web Services AMI, and can be easily
run on EC2, the next step is to create 5 EC2 instances to setup the
cluster:

You will get something like that after creating five instances with
Amazon AWS:

After creating the five instances, its time to register each instance
with Rancher by running the following command on each server:

[rancher@rancher ~]$ sudo docker run –rm -it –privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent http://<ip-address>:8080/v1/scripts/4E1D4A26B07A1539CD33:1426626000000:jZskPi71YEPSJo1uMISMEOpbUo

After running the previous command on each server you will see that the
servers have been registered with Rancher:

Running The Riak cluster

The RIAK_CLUSTER_SIZE will provide the number of instances needed
to be added to the cluster before committing the changes, its
recommended to add 5 Riak nodes to the cluster of a production
environment, although you can set the RIAK_CLUSTER_SIZE to more or
fewer as needed. **** **** To create a Docker container using the
Rancher platform, on any instance click on “Add Container”:

On the first node you just need to specify the name of the container
and select the Riak image, but for other Riak nodes you need to specify
two more environment variables which will help the node to connect to
the cluster the RIAK_JOINING_IP which tells the Riak node to
connect to a node in the cluster and RIAK_CLUSTER_SIZE which used
to specify the number of nodes joining the cluster:

Testing The Riak Cluster

From Rancher we can view the logs of the running containers, similar to
using docker logs -f container-name. This allows us to see the logs of
the Riak containers and ensure that everything is running as planned:


At the last node you will see something different. Since the number of
the node that joined the cluster matches the value of the environment
variable RIAK_CLUSTER_SIZE, so the changes will be committed and the
cluster will be up and running:
To see that the nodes are connected to the cluster, you can write
the following command inside the shell of any of the Riak containers:

# riak-admin member-status

And you will get the following output:
This indicates that each node is a valid member of the cluster and
acquire a roughly equal percentage of the ring. Now to test the cluster
from outside the environment, you should map the ports of the Docker
containers to the host’s ports, this can be achieved dynamically using
Rancher platform:

I already created and activated a bucket-type called “cluster,” which I
used to test via the Riak HTTP API. You can see from below the
environment is up and running now.

$ export RIAK=http://52.0.119.255:8098
$ curl -XPUT “$RIAK/types/cluster/buckets/rancher/keys/hello”
-H “Content-Type:text/plain”
-d “World.. Riak”

$ curl -i “$RIAK/types/cluster/buckets/rancher/keys/hello”
HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzGDKBVIcqZfePk3k6vPOYEpkzGNlYAroOseXBQA=
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.5 (jokes are better explained)
url: </buckets/rancher>; rel=”up”
Last-Modified: Fri, 27 Mar 2015 22:04:50 GMT
ETag: “4flAtEZ59hdYsKhSGVhKpZ”
Date: Fri, 27 Mar 2015 22:11:23 GMT
Content-Type: text/plain
Content-Length: 5

World.. Riak

Conclusion

Riak cluster provides a distributed, high available, and simple
key-value store. Building the Riak cluster using RancherOS and Rancher
platform provide docker management and networking capabilities, making
installation quick and making it simple to upgrade and scale the
environment in the future. You can download Riak
here. To download Rancher
or RancherOS please visit our GitHub
site. You can find a detailed getting
started
guide for
RancherOS on GitHub as well. If you would like to learn more, please
join our next online meetup to meet the team and learn about the latest
with Rancher and RancherOS. Hussein Galal is
a Linux System Administrator, with experience in Linux, Unix,
Networking, and open source technologies like Nginx, Apache, PHP-FPM,
Passenger, MySQL, LXC, and Docker. You can follow Hussein
on Twitter @galal_hussein.

Source

Spinnaker is a cloud-native continuous delivery tool created at Netflix and was originally designed and built to help internal development teams release software changes with confidence. Since then it has been open-sourced and has gained the support of a growing number of mainstream cloud providers including Google, Amazon, Microsoft, IBM and Oracle.

At Jetstack we receive questions almost on a daily basis from our customers about how to deploy to Kubernetes across different environments and in some cases to clusters in multiple cloud providers/on-prem. Since Spinnaker runs natively on Kubernetes and has first-class support for Kubernetes manifests, it is a strong candidate as a tool for this purpose. However, being able to demonstrate the tool in action and more importantly how it might integrate with other tooling is vital for making a decision. For this reason we have been working on a series of demonstrators with various best-of-breed cloud-native technologies to help inform our customers. In this post, we’ll describe the architecture of the demo and how these cloud-native technologies can be used together and with Spinnaker.

Overview

The primary aim of the demo is to show how Spinnaker could be used to automate the deployment of a new version of an application to production with confidence. The chosen application is a simple webserver called the goldengoose that we use for our advanced wargaming training course. The techniques described below could of course be applied to a more complex application, but in order to keep the focus on Spinnaker’s capabilities rather than the intricacies of managing a particular application, we chose to keep the application simple.

The demo configures two pipelines within Spinnaker: Build and Deploy. When a commit is pushed to the master branch of the goldengoose GitHub repository, a GitHub webhook triggers the Build pipeline which builds an image on-cluster and pushes the result to Docker Hub. If successful, the Deploy pipeline is then triggered which deploys the new image in a controlled way to production.

One of the main components of the demo that provides the confidence and control mentioned above is the use of Spinnaker’s automated canary analysis (ACA) feature. This feature leverages Kayenta, a component responsible for querying relevant metrics from a configured sink and performing statistical analysis on the data to decide how to proceed (in this case, whether a canary deployment should be promoted to production or not). Deciding which metrics should be used to make such a decision can be challenging, however this feature provides operators with an incredibly flexible way of describing to Spinnaker what it means for a new version of their application to be ‘better’ than the previous version.

The whole demo (except for the GitHub and Docker Hub components, load balancers and disks) runs on a single GKE cluster. This cluster does not have any special requirements except that we have enabled autoscaling and made the nodes larger than the default (n1-standard-4).

More detail on how the tools used within the demo interact with each other will be described below, but the high-level steps are as follows:

1. Make a local change to the goldengoose codebase and push to GitHub
2. GitHub webhook triggers Spinnaker Build pipeline which applies Knative Build custom resource to the cluster
3. Knative build controller triggers a build of the goldengoose image which is pushed to Docker Hub
4. If the build is successful, the Spinnaker Deploy pipeline is triggered
5. Canary deployment is deployed from the newly built image
6. Baseline deployment is deployed using the image from the current production deployment
7. Spinnkaker performs ACA on performance metrics collected from both the canary and baseline deployments
8. If ACA is deemed successful, the canary image is promoted to production by performing a rolling update of the production deployment
9. Canary and baseline deployments are cleaned up

The reason for deploying a baseline using the current production image rather than just using the production deployment itself is to avoid differences in performance metrics due to how long the deployment has been running. Heap size is one such metric that could be affected by this.

These steps could of course be extended to a more complex pipeline involving more environments and more testing, perhaps with a final manual promotion to production; a single Spinnaker deployment can interact with multiple Kubernetes clusters other than the cluster Spinnaker is running on by installing credentials for these other clusters.

Here we list the main tools that have been used and their purpose within the demo and how they relate to Spinnaker:

• Knative: when a code change is pushed to our goldengoose repository, we want to trigger a build so that a new canary deployment can be rolled out. Knative’s build component worked nicely for this and allowed Spinnaker to apply a Build custom resource to the cluster whenever a commit was pushed to our master branch. This CI component of the demo is not strictly within Spinnaker’s domain as a CD tool, however by having Knative controllers handle the logic involved in building a new image we could still make use of Spinnaker’s first-class support for Kubernetes resources.
• Prometheus: Spinnaker’s ACA requires access to a set of metrics from both a canary deployment and a baseline deployment. Spinnaker supports a number of metrics sinks but some of the reasons we chose Prometheus was its ubiquity in the cloud-native space and the fact that it integrates with Istio out of the box. By configuring Spinnaker to talk to our in-cluster Prometheus instance we were able to automate the decision to promote canary images to production.
• Istio: as we were only making use of Knative’s build component, we did not have a strict dependency on Istio; however, by using Istio’s traffic shifting capabilities we were able to easily route equal and weighted production traffic to both our baseline and canary deployments, producing performance metrics to be used by Spinnaker’s ACA feature. Istio’s traffic mirroring feature could also be used if you did not want responses from the canary to be seen by users. We also made use of the Prometheus adapter to describe to Istio which goldengoose metrics we wanted to make available in Prometheus. Finally, the Istio Gateway was used to allow traffic to reach our goldengoose deployments.
• cert-manager: to secure Spinnaker’s UI and API endpoints we needed TLS certificates; what else would we use?
• nginx-ingress: the NGINX ingress controller was used to allow traffic to reach both the Spinnaker UI and API endpoints as well as for cert-manager Let’s Encrypt ACME HTTP challenges.
• GitHub: used as both a source code respository and as an OAuth identity provider for Spinnaker. There are other authentication options available.
• OpenLDAP: used for authorisation within Spinnaker. There are other authorisation options available.

Summary

We have described how Spinnaker can be used for continuous delivery (and integration) and how it can be integrated with other cloud-native tooling to provide powerful capabilities within your organisation.

It is still relatively early days for the Spinnaker project and we can expect to see lots of future development; the documentation that exists today is clean and easy to follow, however there are a number of undocumented features that I would like to see around exposing the internals of the various microservices that make up a Spinnaker deployment. Some interesting ones exist today for example writing custom stages and adding first-class support for particular Kubernetes custom resources but other changes such as letting Spinnaker know that a new CRD exists in the cluster and the recommended way of manually adding to generated Halyard configuration (for component sizing for example) would be nice to see. Fortunately the Spinnaker community is strong and responsive and has clearly outlined how best to get in touch here.

One potential barrier to Spinnaker adoption for some users is the amount it lays on top of Kubernetes; authentication, authorisation and configuration validation (e.g. for Spinnaker pipelines) are all handled by various Spinnaker components or external services, however upstream Kubernetes already has a lot of machinery to handle these exact problems which Spinnaker does not make use of. The ability to apply a pipeline custom resource for example that Spinnaker watches for would be very powerful, allowing RBAC rules to be configured to control which users are allowed to manage pipelines. Not relying on Kubernetes for these features does of course allow for more granular authentication for example and additionally makes Spinnaker’s deployment options more wide than just Kubernetes, however since the only production installation instructions require Kubernetes and since Kubernetes is becoming increasingly ubiquitous, it might ease adoption by working towards making that coupling tighter. Projects such as k8s-pipeliner do try to provide some of that glue but deeper integrations would be greatly valued for users already familiar with Kubernetes.

For more information on anything covered in this post please reach out to our team at hello@jetstack.io.

Source

Rancher Server has recently added Docker Machine support,
enabling us to easily deploy new Docker hosts on multiple cloud
providers via Rancher’s UI/API and automatically have those hosts
registered with Rancher. For now Rancher supports DigitalOcean and
Amazon EC2 clouds, and more providers will be supported in the future.
Another significant feature of Rancher is its networking implementation,
because it enhances and facilitates the way you connect Docker
containers and those services running on them. Rancher creates a private
network across all Docker hosts that allows containers to communicate as
if they were in the same subnet. In this post we will see how to use the
new Docker Machine support and Rancher networking by deploying a Grafana
dashboard installation on Amazon EC2. We are creating EC2 instances directly from
Rancher UI and all our containers are being connected through the
Rancher network. If you have never heard of Grafana, it is an open
source rich metric web dashboard and graph editor for Graphite, influxDB
and OpenTSBD metric storages. To set this up we are using these docker
images:

• tutum/influxdb for storing metrics and grafana dashboards
• tutum/grafana for graphing influxDB metrics and serving
  dashboards
• a custom linux image that will send linux O.S. metrics to influxDB
  using sysinfo_influxdb (CPU, memory, load, disks I/O, network
  traffic).

In a test environment you may want to deploy docker images in the same
host, but we are using a total of 4 AWS instances listed below in order
to mimic a large-scale production deployment and also to see how Rancher
networking works.

• 1 as a Rancher Server to provision and manage application stack AWS
  instances,
• 1 running influxDB docker image (tutum/influxdb)
• 1 running grafana docker image (tutum/grafana)
• 1 running sysinfo docker image (nixel/sysinfo_influxdb)

Preparing AWS Environment

First you will need to create the following in AWS Console: a Key Pair
for connecting to your servers, a Security Group to give you access to
Rancher Console, and a Access Key for Rancher to provision EC2
instances. Creating a Key Pair Enter AWS EC2 Console, go to Key
Pairs section, click Create Key Pair button and then enter a name for
your Key Pair. Once created, your browser downloads a pem certificate.
You will need it if you want to connect to your AWS instances.
Creating a Security Group First of all go to VPC Console and enter
Subnets section. You will get a list of available subnets in default
VPC, choose one for deploying AWS instances and copy its ID and CIDR.
Also copy VPC ID, you will need all this data later when creating Docker
hosts with Machine integration. I am using subnet 172.31.32.0/20 for
this tutorial.

Then enter AWS EC2 Console, go to Security Groups section and click
Create Security Group button. Enter the following data:

• Security Group Name: Rancher and Grafana
• Description: Open Rancher and Grafana ports
• VPC: select the default one
• Add a new inbound rule to allow 22 TCP port to be accessible only
  from your IP
• Add a new inbound rule to allow 8080 TCP port to be accessible only
  from your IP
• Add a new inbound rule to allow 9345-9346 TCP ports to be accessible
  from anywhere
• Add a new inbound rule to allow all traffic from your VPC network.
  In this case source is 172.31.32.0/20, change it accordingly to your
  environment.

Creating an Access Key Enter EC2 Console and click your user name in
the top menu bar, click Security Credentials and then expand Access
Keys option. Click Create New Access Key button and after it has been
created you will click Show Access Key to get the ID and Secret Key.
Save them because you are needing them later to create Docker hosts.

Rancher Server Setup

For launching Rancher Server you will need an AWS instance. I am using
the t1.micro instance for writing this guide, but it is recommended to
use a larger instance for real environments. Enter AWS EC2 Console and
go to Instances section, click Launch Instance button, click
Community AMIs and then search for RancherOS and select last version,
for example rancheros-v0.2.1. Choose an instance type and click Next:
Configure instance details button. In configuration screen be sure to
select the same subnet you chose for Security Group. Expand Advanced
Details section and enter this user data to initialize your instance
and get Rancher Server installed and running.

#!/bin/bash
docker run -d -p 8080:8080 rancher/server:v0.14.1

You may keep default options for all steps excepting Security Group
(choose Security Group named Rancher and Grafana). When launching AWS
instance you are asked to choose a Key Pair, be sure to select the one
that we created before. Go to Instances section and click your Rancher
Server instance to know its private and public IPs. Wait a few minutes
and then browse to
http://RANCHER_SERVER_PUBLIC_IP:8080
to enter Rancher Web Console and click Add Host. You will be asked to
confirm Rancher Server IP address, click Something else and enter
RANCHER_SERVER_PRIVATE_IP:8080, finally click Save button.

Docker hosts setup

Go to Rancher Console, click Add Host and select Amazon EC2. Here
you will need to enter the new host name, the Access Key and the
Secret Key. Also be sure to set the same Region, Zone, and VPC
ID as those used by Rancher Server. Leave all other parameters with
their default values. Repeat this process to create our three Docker
hosts that will appear up and running after a while.

Security Group for grafana Rancher Server has created a Security
Group named docker-machine for your Docker hosts. Now in order to be
able to connect to grafana you must go to VPC Console and add the
following Inbound rules:

• Add a new inbound rule to allow 80 TCP port to be accessible only
  from your IP
• Add a new inbound rule to allow 8083-8084 TCP ports to be accessible
  only from your IP
• Add a new inbound rule to allow 8086 TCP port to be accessible only
  from your IP
• Add a new inbound rule to allow all traffic from your VPC network.
  In this case source is 172.31.32.0/20, change it accordingly to your
  environment.

Installing application containers

This step consists of installing and configuring influxDB, grafana, and
an ubuntu container running sysinfo_influxdb. This container will send
O.S. metrics to influxDB which will be graphed in grafana.
Installing influxDB container Go to Rancher Web Console and click +
Add Container button at your first host, enter a container name like
influxdb and tutum/influxdb in Select Image field. Add these
three port mappings, all of them are TCP:

• 8083 (on host) to 8083 (in container)
• 8084 (on host) to 8084 (in container)
• 8086 (on host) to 8086 (in container)

Expand Advanced Options section an add an environment variable named
PRE_CREATE_DB which value is grafana, so influxDB will create
an empty database for grafana metrics. Now go to Networking section
and enter a hostname like influxdb for this container. Be sure that
Network type is Managed Network on docker0 so this container can be
reached by grafana and sysinfo_influxdb. You can leave other options
with their default values. After a few minutes you will see your
influxDB container launched and running in your host. Note that influxdb
container has a private IP address, copy it to configure
sysinfo_influxdb later. Copy also the public IP of host that is running
this container, you will need it later to configure grafana.

Installing grafana container Go to Rancher Web Console and click +
Add Container button at your second host, enter a container name like
grafana and tutum/grafana in Select Image field. Add this TCP
port mapping:

• 80 (on host) to 80 (in container)

Expand Advanced Options section and enter the following environment
variables needed by grafana:

Variable name Variable value Used for
HTTP_USER admin User login for grafana basic HTTP authentication
HTTP_PASS Some password User password for grafana basic HTTP authentication
INFLUXDB_HOST 52.11.32.51 InfluxDB host’s public IP. Adapt this to your environment
INFLUXDB_PORT 8086 InfluxDB port
INFLUXDB_NAME grafana Name of previously created database
INFLUXDB_USER root InfluxDB user credentials
INFLUXDB_PASS root InfluxDB user credentials
INFLUXDB_IS_GRAFANADB true Tell grafana to use InfluxDB for storing dashboards

Grafana makes your browser to connect to influxDB directly. This is why
we need to configure a public IP in INFLUXDB_HOST variable here. If
not, your browser could not reach influxDB when reading metric values.
Go to Networking section and enter a hostname like grafana for this
container. Be sure that Network type is Managed Network on docker0 so
this container can connect to influxdb. You can leave other options with
their default values and after a few minutes you will see your grafana
container launched and running in your host.

Now go to Instances section in EC2 Console, click on the instance which
is running grafana container and copy its public IP. Type the following
url in your browser:
http://GRAFANA_HOST_PUBLIC_IP, use
HTTP_USER and HTTP_PASS credentials to log in.

Installing sysinfo_influxdb container Go to Rancher Web Console and
click + Add Container button at your third host, enter sysinfo in
container name and nixel/sysinfo_influxdb in Select Image
field. No port mapping is needed. Expand Advanced Options section and
enter these environment variables which are needed by this container:

Variable name Variable value Used for
INFLUXDB_HOST 10.42.169.239 InfluxDB container private IP. Adapt this to your environment
INFLUXDB_PORT 8086 InfluxDB port
INFLUXDB_NAME grafana Name of previously created database
INFLUXDB_USER root InfluxDB user credentials
INFLUXDB_PASS root InfluxDB user credentials
SYSINFO_INTERVAL 5m Sysinfo frequency to update metric values. Default is 5m

Note that in this case INFLUXDB_HOST contains influxDB container
private IP. This is because sysinfo_influxdb will directly connect to
influxDB, using the VPN created by Rancher. Go to Networking section
and be sure the container hostname is sysinfo because you will later
import a sample grafana dashboard which needs this. Be sure that Network
type is Managed Network on docker0 so this container can connect to
influxdb. You can leave other options with their default values and
after a few minutes you will see your sysinfo container launched and
running in your host.

Graph metrics with grafana

At this point sysinfo container is collecting O.S. metrics and sending
them to influxDB every 5 minutes using Rancher networking. In this final
step we are graphing those metrics in grafana. First let’s import a
sample grafana dashboard that is already configured. Execute the
following command to download the dashboard definition:

curl -o https://raw.githubusercontent.com/nixelsolutions/sysinfo_influxdb/master/grafana_dashboard.json

Then open grafana web, browse
to http://GRAFANA_HOST_PUBLIC_IP and
click folder icon on top.

Click import button and upload the file you have just downloaded. Click
save button on top and now you will be able to see CPU, Load Average,
RAM, Swap and Disks metrics that are being collected in your sysinfo
container.

Conclusion

Rancher implements a networking solution that really simplifies the way
you bring connectivity to those services running in your containers.
Instead of managing port mappings it automatically puts all your
containers into the same network without requiring any configuration
from you. This is an important feature because, in fact, it brings
containers closer to enterprise production platforms because it makes
easier to deploy complex scenarios where some containers need to connect
with others. With Rancher you can deploy any container on any host at
any time without reconfiguring your environment, and there is no need to
worry about defining, configuring or maintaining port mappings when
interconnecting containers. To get more information on Rancher, feel
free at any time to request a demonstration from one of our
engineers, or sign up
for an upcoming online meetup.

Manel Martinez
is a Linux systems engineer with experience in the design and management
of scalable, distributable and highly available open source web
infrastructures based on products like KVM, Docker, Apache, Nginx,
Tomcat, Jboss, RabbitMQ, HAProxy, MySQL and XtraDB. He lives in spain,
and you can find him on Twitter
@manel_martinezg.

Source

Nagios is a fantastic monitoring tool, and I wanted to see if I could
get the agent to run as a system container on RancherOS, in order to
monitor the host and any Docker containers running on it. It turned out
to be incredibly easy. In this blog post, I’ll walk through how to
launch the Nagios agent as system container in RancherOS. Specifically,
I’ll use two vagrant boxes to cover:

1. Provisioning a server with the Rancher control plane
2. Adding a second server running Rancher OS
3. Installing a Nagios agent as system container on the second server
4. Connecting the Nagios agent to the Nagios management server

System Containers in RancherOS

First, for anyone who isn’t familiar with RancherOS, it is a minimal
distribution of Linux designed specifically to run Docker. RancherOS
runs a Docker daemon as PID 1, a role typically occupied by the init
system or systemd in most distributions. This daemon runs essential
system services like SSH, syslog or NTP as containers, and is called
system docker.

A second Docker daemon, called user docker, is launched as a
container. This is where any new containers started by the user are
created, as well as containers placed by Rancher or other management
services.

To give the Nagios agent access to all of the data from the server, as
well as the system and user containers, it should run in the system
docker instance. I will run this setup in 2 Vagrant virtual machines.

Set up Rancher

Even though we could monitor RancherOS with Nagios directly, I’m going
to set up Rancher in this deployment to manage the containers we create.
The Rancher team provides a Vagrantfile to run RancherOS in a VM here:
https://github.com/rancher/os-vagrant and
another Vagrantfile for Rancher here:
https://github.com/rancher/rancher
But, since I want to have both in one Vagrant setup, I merged both
Vagrantfiles into one and added the option to run multiple RancherOS
instances in one.

You can find my new Vagrant file here:
[https://github.com/buster/rancher-tutorial]{.c10}

The first step (after installing Vagrant, of course) is to clone this
repository and edit the Vagrantfile to match your IP addresses in the
lines:

# The number of VMs will be added to the following string,
# so Rancher will be on 192.168.0.200, the first RancherOS instance on 192.168.0.201, etc.
$rancher_ip_start = “192.168.0.20”
$rancherui_ip = $rancher_ip_start + “0”
# the number of rancher instances
$n_rancher = 1

* *

Leave $n_rancher at 1 for now.

After editing this file, run `vagrant up’.

Vagrant will now first setup the Rancher VM, which means Vagrant will
download the Virtualbox image, start it and Docker will then download
and run the Rancher Server and the Rancher Agent. Afterwards, the second
VM, which will host our RancherOS instance, will be started and the
RancherOS instance will register itself at the Rancher Server.

When finished, browse to the Rancher IP (http://192.168.0.200:8080/ in
my case) and observe your new and shiny VMs:

Adding a System Container to Rancher

The next task is to set up the Nagios Agent on the RancherOS instance.

For that you will need to log in to the server, which you do by running
`vagrant ssh rancher1`.

There you will have access to the user docker (by calling `docker`)
and to the system docker by calling `sudo system-docker`.

A system container is not different from your usual docker container,
except that it is run by the system docker and that has no networking by
default. Thus, it needs to inherit the network of the host (–net=host
parameter):

sudo system-docker run -d –net=host –name nagios-agent buster/nagios-agent

This nagios agent container comes with a minimal configuration to check
the load on the second RancherOS
instance.

[]Deploying the Nagios Server to Rancher

In order for the Nagios agent to make any sense, we will also need a
Nagios Server which polls the Nagios Agent.

This is as easy as any other Rancher deployment, by clicking on “Add
Container” in the Rancher UI.

There we will make use of the already existing Nagios Server docker
container from
https://registry.hub.docker.com/u/cpuguy83/nagios/
Also don’t forget to go to the `Ports` tab and map port 80 to port
8081 so that you can login on nagios.

Add this container and after a while, the Nagios Server will be up and
running! Browse to
http://192.168.0.200:8081/ and
observe the Nagios UI running. The default username is
nagiosadmin and the password is nagios.

[]Configure Nagios Server

The Nagios Server only knows itself right now, so we will need to
configure it to poll the Nagios Agent.

This can be done in /opt/nagios/etc/conf.d/rancher1.cfg, for example.

Rancher offers a very nice terminal into the running containers, which
you can reach by click on the container and afterwards on the “execute
shell” url:

Now, you can edit the config file by running `nano
/opt/nagios/etc/conf.d/rancher1.cfg`.

Add the following lines to the file:

define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}
define host{
use linux-server
host_name rancher1
address 192.168.0.201
}
define service{
use linux-server
host_name rancher1
service_description Current Users
check_command check_nrpe!check_users
}
define service{
use generic-service
host_name rancher1
service_description Current Load
check_command check_nrpe!check_load
}

Afterwards you can check if the configuration file format is correct by
running `nagios -v /opt/nagios/etc/nagios.cfg`.

To check that the nrpe server on the second host is running you can also
run a check manually: `/opt/nagios/libexec/check_nrpe -H
192.168.0.201 -c check_load`

After you have verified the working Nagios setup you simply need to
restart the Nagios Server container by clicking on the symbol:

Now, you can login to Nagios again and see the Nagios Plugins doing
their work:

Conclusion

Using Nagios to monitor multiple RancherOS servers is as easy as running
a preconfigured publicly available Docker container from
https://registry.hub.docker.com

Starting a system docker container requires a few additional steps
compared to running a user container, but hopefully we’ve explained
them clearly here.

In the next few weeks RancherOS will ship 0.3, which includes support
for predefined system services. That will make configuration of new
agents in the Nagios server as easy as executing a docker run command.

If you’d like to get started with RancherOS, you can download it from
GitHub here. Also, we’re always demoing new features and answering lots
of questions at each months Rancher Meetup, which you can find a link
to for below.

Sebastian Schulze is a Technology Consultant from Germany, with
experience in Linux, Solaris, Docker, and Vagrant. You can contact him
via github at:
https://github.com/buster

Source

A Detailed Overview of Rancher’s Architecture

This newly-updated, in-depth guidebook provides a detailed overview of the features and functionality of the new Rancher: an open-source enterprise Kubernetes platform.

Get the eBook

Update (October 2017): Gord Sissons revisited this topic and compared
the top 10 container-monitoring solutions for Rancher in a recent blog
post.

*Update (October 2016): Our October online meetup demonstrated and
compared Sysdig, Datadog, and Prometheus in one go. Check
out
the recording. *

As Docker is used for larger deployments it becomes more important to
get visibility into the status and health of docker environments. In
this article, I aim to go over some of the common tools used to monitor
containers. I will be evaluating these tools based on the following

criteria:

1. ease of deployment,
2. level of detail of information presented,
3. level of aggregation of information from entire deployment,
4. ability to raise alerts from the data,
5. ability to monitor non-docker resources, and
6. cost. This list is by no means comprehensive however I have tried to highlight the most common tools and tools that optimize our six evaluation criteria.

A Detailed Overview of Rancher’s Architecture

This newly-updated, in-depth guidebook provides a detailed overview of the features and functionality of the new Rancher: an open-source enterprise Kubernetes platform.

Get the eBook

Docker Stats

All commands in this article have been specifically tested on
a RancherOS instance running on Amazon Web Services EC2. However, all
tools presented today should be usable on any Docker deployment.

The first tool I will talk about is Docker itself, yes you may not be
aware that docker client already provides a rudimentary command line
tool to inspect containers’ resource consumption. To look at the
container stats run docker stats with the name(s) of the running
container(s) for which you would like to see stats. This will present
the CPU utilization for each container, the memory used and total memory
available to the container. Note that if you have not limited memory for
containers this command will post total memory of your host. This does
not mean each of your container has access to that much memory. In
addition you will also be able to see total data sent and received over
the network by the container.

$ docker stats determined_shockley determined_wozniak prickly_hypatia
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
determined_shockley 0.00% 884 KiB/1.961 GiB 0.04% 648 B/648 B
determined_wozniak 0.00% 1.723 MiB/1.961 GiB 0.09% 1.266 KiB/648 B
prickly_hypatia 0.00% 740 KiB/1.961 GiB 0.04% 1.898 KiB/648 B

For a more detailed look at container stats you may also use the Docker
Remote API via netcat (See below). Send an http get request for
/containers/[CONTAINER_NAME]/stats where CONTAINER_NAME is name of
the container for which you want to see stats. You can see an example of
the complete response for a container stats request
here. This
will present details of the metrics shown above for example you will get
details of caches, swap space and other details about memory. You may
want to peruse the Run
Metrics section of the
Docker documentation to get an idea of what the metrics mean.

echo -e “GET /containers/[CONTAINER_NAME]/stats HTTP/1.0rn” | nc -U /var/run/docker.sock

Score Card:

1. Easy of deployment: *****
2. Level of detail: *****
3. Level of aggregation: none
4. Ability to raise alerts: none
5. Ability to monitor non-docker resources: none
6. Cost: Free

CAdvisor

The docker stats command and the remote API are useful for getting
information on the command line, however, if you would like to access
the information in a graphical interface you will need a tool such as
CAdvisor. CAdvisor provides a
visual representation of the data shown by the docker stats command
earlier. Run the docker command below and go to
http://<your-hostname>:8080/ in the browser of your choice to see
the CAdvisor interface. You will be shown graphs for overall CPU usage,
Memory usage, Network throughput and disk space utilization. You can
then drill down into the usage statistics for a specific container by
clicking the Docker Containers link at the top of the page and then
selecting the container of your choice. In addition to these statistics
CAdvisor also shows the limits, if any, that are placed on container,
using the Isolation section.

docker run
–volume=/:/rootfs:ro
–volume=/var/run:/var/run:rw
–volume=/sys:/sys:ro
–volume=/var/lib/docker/:/var/lib/docker:ro
–publish=8080:8080
–detach=true
–name=cadvisor
google/cadvisor:latest

CAdvisor is a useful tool that is trivially easy to setup, it saves us
from having to ssh into the server to look at resource consumption and
also produces graphs for us. In addition the pressure gauges provide a
quick overview of when a your cluster needs additional resources.
Furthermore, unlike other options in this article CAdvisor is free as it
is open source and also it runs on hardware already provisioned for your
cluster, other than some processing resources there is no additional
cost of running CAdvisor. However, it has it limitations; it can only
monitor one docker host and hence if you have a multi-node deployment
your stats will be disjoint and spread though out your cluster. Note
that you can use
heapster to monitor
multiple nodes if you are running Kubernetes. The data in the charts is
a moving window of one minute only and there is no way to look at longer
term trends. There is no mechanism to kick-off alerting if the resource
usage is at dangerous levels. If you currently do not have any
visibility in to the resource consumption of your docker node/cluster
then CAdvisor is a good first step into container monitoring however, if
you intend to run any critical tasks on your containers a more robust
tool or approach is needed. Note that
Rancher runs CAdvisor on
each connected host, and exposes a limited set of stats through the UI,
and all of the system stats through the API.

Score Card (Ignoring heapster because only supported on Kubernetes):

1. Easy of deployment: *****
2. Level of detail: **
3. Level of aggregation: *
4. Ability to raise alerts: none
5. Ability to monitor non-docker resources: none
6. Cost: Free

Scout

The next approach for docker monitoring is Scout and it addresses
several of limitations of CAdvisor. Scout is a hosted monitoring service
which can aggregate metrics from many hosts and containers and present
the data over longer time-scales. It can also create alerts based on
those metrics. The first step to getting scout running is to sign up
for a Scout account at https://scoutapp.com/, the free trial account
should be suitable for testing out integration. Once you have created
your account and logged in, click on your account name in the top right
corner and then Account Basics and take note of your Account Key as
you will need this to send metrics from our docker server.

Now
on your host, create a file called scoutd.yml and copy the following
text into the the file, replacing the account_key with the key you
took note of earlier. You can specify any values that make sense for the
host, display_name, environment and roles properties. These will be
used to separate out the metrics when they are presented in the scout
dashboard. I am assuming an array of web-servers is run on docker so
will use the values shown below.

# account_key is the only required value
account_key: YOUR_ACCOUNT_KEY
hostname: web01-host
display_name: web01
environment: production
roles: web

You can now bring up your scout agent with the scout configuration file
by using the docker scout plugin.

docker run -d –name scout-agent
-v /proc:/host/proc:ro
-v /etc/mtab:/host/etc/mtab:ro
-v /var/run/docker.sock:/host/var/run/docker.sock:ro
-v `pwd`/scoutd.yml:/etc/scout/scoutd.yml
-v /sys/fs/cgroup/:/host/sys/fs/cgroup/
–net=host –privileged
soutapp/docker-scout

Now go back to the Scout web view and you should see an entry for your
agent which will be keyed by the display_name parameter (web01) that
you specified in your scoutd.yml earlier.

If
you click the display name it will display detailed metrics for the
host. This includes the process count, CPU usage and memory utilization
for everything running on your host. Note these are not limited to
processes running inside docker.

To add docker monitoring to your servers click the Roles tab and then
select All Servers. Now click the + Plugin Template Button and then
Docker Monitor from the following screen to load the details view.
Once you have the details view up select Install Plugin to add the
plugin to your hosts. In the following screen give a name to the plugin
installation and specify which containers you want to monitor. If you
leave the field blank the plugin will monitor all of the containers on
the host. Click complete installation and after a minute or so you can
go to [Server Name] > Plugins to see details from the docker monitor
plugin. The plugin shows the CPU usage, memory usage network throughput
and the number of containers for each host.

If
you click on any of the graphs you can pull a detailed view of the
metrics and this view allows you to see the trends in the metric values
across a longer time span. This view also allows you to filter the
metrics based on environment and server role. In addition you can create
“Triggers” or alerts to send emails to you if metrics go above or
below a configured threshold. This allows you to setup automated alerts
to notify you if for example some of your containers die and the
container count falls below a certain number. You can also setup alerts
for average CPU utilization so if for example if your containers are
running hot you will get an alert and you can launch more add more hosts
to your docker cluster. To create a trigger select Roles
> All Servers from the top menu and then docker monitor from the
plugins section. Then select triggers from the Plugin template
Administration menu on the right hand side of the screen. You should
now see an option to ”Add a Trigger” which will apply to the entire
deployment. Below is an example of a trigger which will send out an
alert if the number of containers in the deployment falls below 3. The
alert was created for “All Servers” however you could tag your hosts
with different roles using the scoutd.yml created on the server. Using
the roles you can apply triggers to a sub-set of the servers on your
deployment. For example you could setup an alert for when the number of
containers on your web nodes falls below a certain number. Even with the
role based triggers I still feel that Scout alerting could be better.
This is because many docker deployments have heterogeneous containers on
the same host. In such a scenario it would be impossible to setup
triggers for specific types of containers as roles are applied to all
containers on the host.

Another advantage of using Scout over CAdvisor is that it has a large
set of plugins which can pull in
other data about your deployment in addition to docker information. This
allows Scout to be your one stop monitoring system instead of having a
different monitoring system for various resources in your system.

One drawback of Scout is that it does not present detailed information
about individual containers on each host like CAdvisor can. This is
problematic, if your are running heterogeneous containers on the same
server. For example if you want a trigger to alert you about issues in
your web containers but not about your Jenkins containers Scout will not
be able to support that use case. Despite the drawbacks Scout is a
significantly more useful tool for monitoring your docker deployments.
However this does come at a cost, ten dollars per monitored host. The
cost could be a factor if you are running a large deployment with many
hosts.

Score Card:

1. Easy of deployment: ****
2. Level of detail: **
3. Level of aggregation: ***
4. Ability to raise alerts: ***
5. Ability to monitor non-docker resources: Supported
6. Cost: $10 / host

Data Dog

From Scout lets move to another monitoring service, DataDog, which
addresses several of the short-comings of Scout as well as all of the
limitations of CAdvisor. To get started with DataDog, first sign up for
a DataDog account at https://www.datadoghq.com/. Once you are signed
into your account you will be presented with list of supported
integrations with instructions for each type. Select docker from the
list and you will be given a docker run command (show below) to copy
into your host. The command will have your API key preconfigured and
hence can be run the command as listed. After about 45 seconds your
agent will start reporting metrics to the DataDog system.

docker run -d –privileged –name dd-agent
-h `hostname`
-v /var/run/docker.sock:/var/run/docker.sock
-v /proc/mounts:/host/proc/mounts:ro
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
-e API_KEY=YOUR_API_KEY datadog/docker-dd-agent

Now that your containers are connected you can go to the Events tab in
the DataDog web console and see all events pertaining to your cluster.
All container launches and terminations will be part of this event
stream.

You can also click the Dashboards tab and hit create dashboards to
aggregate metrics across your entire cluster. Datadog collects metrics
about CPU usage, memory and I/O for all containers running in the
system. In addition you get counts of running and stopped containers as
well as counts of docker images. The dashboard view allows you to create
graphs for any metric or set of metrics across the entire deployment or
grouped by host or container image. For example the graph below shows
the number of running containers broken down by the image type, I am
running 9 ubuntu:14.04 containers in my cluster at the moment.

You could also split the same data by Hosts, as the second graph shows,
7 of the containers are running on my Rancher host and the remaining
ones on my local laptop.

Data Dog also supports alerting using a feature called *Monitors. *A
monitor is DataDog’s equivalent to a Scout trigger and allows you to
define thresholds for various metrics. DataDog’s alerting system is a
lot more flexible and detailed then Scout’s. The example below shows
how to specify that you are concerned about Ubuntu containers
terminating hence you would monitor the docker.containers.running metric
for containers created from the ubuntu:14.04 docker image.

Then specify the alert conditions to say that if there are fewer than
ten ubuntu containers in our deployment (on average) for the last 5
minutes, you would like to be alerted. Although not shown here you will
also be asked to specify the text of the message which is sent out when
this alert is triggered as well as the target audience for this alert.
In the current example I am using a simple absolute threshold. You can
also specify a delta based alert which triggers if say the avg stopped
container count was four over the last five minutes raise an alert.

Lastly, using the Metrics Explorer tab you can make ad-hoc
aggregations over your metrics to help debug issues or extract specific
information from your data. This view allows your to graph any metric
over a slice based on container image or host. You may combine output
into a single graph or generate a set of graphs by grouping across
images or hosts.

DataDog is a significant improvement over scout in terms feature set,
easy of use and user friendly design. However this level of polish comes
with additional cost as each DataDog agent costs $15.

Score Card:

1. Easy of deployment: *****
2. Level of detail: *****
3. Level of aggregation: *****
4. Ability to raise alerts: Supported
5. Ability to monitor non-docker resources: *****
6. Cost: $15 /
   host

Sensu Monitoring Framework

Scout and Datadog provide centralized monitoring and alerting however
both are hosted services that can get expensive for large deployments.
If you need a self-hosted, centralized metrics service, you may
consider the sensu open source monitoring
framework. To run the Sensu server you can use
the hiroakis/docker-sensu-server
container. This container installs sensu-server, the uchiwa web
interface, redis, rabbitmq-server, and the sensu-api. Unfortunately
sensu does not have any docker support out of the box. However, using
the plugin system you can configure support for both container metrics
as well as status checks.

Before launch your sensu server container you must define a check that
you can load into the server. Create a file called check-docker.json
and add the following contents into the file. In this file you are
telling the Sensu server to run a script called load-docker-metrics.sh
every ten seconds on all clients which are subscribed to the docker tag.
You will define this script a little later.

{
“checks”: {
“load_docker_metrics”: {
“type”: “metric”,
“command”: “load-docker-metrics.sh”,
“subscribers”: [
“docker”
],
“interval”: 10
}
}
}

Now you can run the sensu server docker container with our check
configuration file using the command below. Once you run the command you
should be able to launch the uchiwa dashboard at
http://YOUR_SERVER_IP:3000 in your browser.

docker run -d –name sensu-server
-p 3000:3000
-p 4567:4567
-p 5671:5671
-p 15672:15672
-v $PWD/check-docker.json:/etc/sensu/conf.d/check-docker.json
hiroakis/docker-sensu-server

Now that the sensu server is up you can launch sensu clients on each of
the hosts running our docker containers. You told the server that the
containers will have a script called load-docker-metrics.sh so lets
create the script and insert it into our client containers. Create the
file and add the text shown below into the file, replacing HOST_NAME
with a logical name for your host . The script below is using the
Docker Remote
APIto
pull in the meta data for running containers, all containers and all
images on the host. It then prints the values out using sensu’s key
value notation. The sensu server will read the output values from the
STDOUT and collect those metrics. This example only pulls these three
values but you could make the script as detailed as required. Note that
you could also add multiple check scripts such as thos, as long as you
reference them in the server configuration file you created earlier. You
can also define that you want the check to fail if the number of running
containers ever falls below three. You can make a check fail by
returning a non-zero value from the check script.

#!/bin/bash
set -e

# Count all running containers
running_containers=$(echo -e “GET /containers/json HTTP/1.0rn” | nc -U /var/run/docker.sock
| tail -n +5
| python -m json.tool
| grep “Id”
| wc -l)
# Count all containers
total_containers=$(echo -e “GET /containers/json?all=1 HTTP/1.0rn” | nc -U /var/run/docker.sock
| tail -n +5
| python -m json.tool
| grep “Id”
| wc -l)

# Count all images
total_images=$(echo -e “GET /images/json HTTP/1.0rn” | nc -U /var/run/docker.sock
| tail -n +5
| python -m json.tool
| grep “Id”
| wc -l)

echo “docker.HOST_NAME.running_containers $”
echo “docker.HOST_NAME.total_containers $”
echo “docker.HOST_NAME.total_images $”

if [ $ -lt 3 ]; then
exit 1;
fi

Now that you have defined your load docker metrics check you need to to
start the sensu client using the
usman/sensu-client
container I defined for this purpose. You can use the command shown
below to launch sensu client. Note that the container must run as
privileged in order to be able to access unix sockets, it must have the
docker socket mounted in as a volume as well as the
load-docker-metrics.sh script you defined above. Make sure the
load-docker-metrics.sh script is marked as executable in your host
machine as the permissions carry through into the container. The
container also takes in SENSU_SERVER_IP, RABIT_MQ_USER,
RABIT_MQ_PASSWORD, CLIENT_NAME and CLIENT_IP as parameters, please
specify the value of these parameters for your setup. The default values
for the RABIT_MQ_USER RABIT_MQ_PASSWORD are sensu and password.

docker run -d –name sensu-client –privileged
-v $PWD/load-docker-metrics.sh:/etc/sensu/plugins/load-docker-metrics.sh
-v /var/run/docker.sock:/var/run/docker.sock
usman/sensu-client SENSU_SERVER_IP RABIT_MQ_USER RABIT_MQ_PASSWORD CLIENT_NAME CLIENT_IP

A few seconds after running this command you should see the client
count increase to 1 in the uchiwa dashboard. If you click the clients
icon you should see a list of your clients including the client that you
just added. I named my client client-1 and specified the host IP as
192.168.1.1.

If you click on the client name your should get further details of the
checks. You can see that the load_docker_metrics check was run at
10:22 on the 28th of March.

If
you Click on the check name you can see further details of check runs.
The zeros indicate that there were no errors, if the script had failed
(if for example your docker Daemon dies) you would see an error code
(non zero) value. Although it is not covered this in the current article
you can also setup sensu to alert you when these checks fail using
Handlers. Furthermore,
uchiwa only shows the values of checks and not the metrics collected.
Note that sensu does not store the collected metrics, they have to be
forwarded to a time series database such as InfluxDB or Graphite. This
is also done through Handlers. Please find details of how to configure
metric forwarding to graphite
here.

Sensu ticks all the boxes in our evaluation criteria; you can collect as
much detail about our docker containers and hosts as ypu want. In
addition you are able to aggregate the values of all of out hosts in one
place and raise alerts over those checks. The alerting is not as
advanced as DataDog or Scout, as you are only able to alert on checks
failing on individual hosts. However, the big drawback of Sensu is
difficulty of deployment. Although I have automated many steps in the
deployment using docker containers, Sensu remains a complicated system
requiring us to install, launch and maintain separate processes for
Redis, RabitMQ, Sensu API, uchiwa and Sensu Core. Furthermore, you would
require still more tools such as Graphite to present metric values and a
production deployment would require customizing the containers I have
used today for secure passwords and custom ssl certificates. In addition
were you to add more checks after launching the container you would have
to restart the Sensu server as that is the only way for it start
collecting new metrics. For these reasons I rate Sensu fairly low for
ease of deployment.

Easy of deployment: * Level of detail: **** Level of aggregation:
**** Ability to raise alerts: Supported but limited Ability to
monitor non-docker resources: ***** Cost: $Free

I also evaluated two other monitoring services, Prometheus and Sysdig
Cloud in a
second article,
and have included them in this post for simplicity.

Prometheus

First lets take a look at Prometheus; it is a self-hosted set of tools
which collectively provide metrics storage, aggregation, visualization
and alerting. Most of the tools and services we have looked at so far
have been push based, i.e. agents on the monitored servers talk to a
central server (or set of servers) and send out their metrics.
Prometheus on the other hand is a pull based server which expects
monitored servers to provide a web interface from which it can scrape
data. There are several exporters
available for
Prometheus which will capture metrics and then expose them over http for
Prometheus to scrape. In addition there are
libraries which
can be used to create custom exporters. As we are concerned with
monitoring docker containers we will use the
container_exporter
capture metrics. Use the command shown below to bring up the
container-exporter docker container and browse to
http://MONITORED_SERVER_IP:9104/metrics to see the metrics it has
collected for you. You should launch exporters on all servers in your
deployment. Keep track of the respective *MONITORED_SERVER_IP*s as we
will be using them later in the configuration for Prometheus.

docker run -p 9104:9104 -v /sys/fs/cgroup:/cgroup -v /var/run/docker.sock:/var/run/docker.sock prom/container-exporter

Once we have got all our exporters running we are can launch Prometheus
server. However, before we do we need to create a configuration file for
Prometheus that tells the server where to scrape the metrics from.
Create a file called prometheus.conf and then add the following text
inside it.

global:
scrape_interval: 15s
evaluation_interval: 15s
labels:
monitor: exporter-metrics

rule_files:

scrape_configs:
– job_name: prometheus
scrape_interval: 5s

target_groups:
# These endpoints are scraped via HTTP.
– targets: [‘localhost:9090′,’MONITORED_SERVER_IP:9104’]

In this file there are two sections, global and job(s). In the global
section we set defaults for configuration properties such as data
collection interval (scrape_interval). We can also add labels which
will be appended to all metrics. In the jobs section we can define one
or more jobs that each have a name, an optional override scraping
interval as well as one or more targets from which to scrape metrics. We
are adding two targets, one is the Prometheus server itself and the
second is the container-exporter we setup earlier. If you setup more
than one exporter your can setup additional targets to pull metrics from
all of them. Note that the job name is available as a label on the
metric hence you may want to setup separate jobs for your various types
of servers. Now that we have a configuration file we can start a
Prometheus server using the
prom/prometheus
docker image.

docker run -d –name prometheus-server -p 9090:9090 -v $PWD/prometheus.conf:/prometheus.conf prom/prometheus -config.file=/prometheus.conf

After launching the container, Prometheus server should be available in
your browser on the port 9090 in a few moments. Select Graph from the
top menu and select a metric from the drop down box to view its latest
value. You can also write queries in the expression box which can find
matching metrics. Queries take the form
METRIC_NAME. You can find more
details of the query syntax here.

We are able to drill down into the data using queries to filter out data
from specific server types (jobs) and containers. All metrics from
containers are labeled with the image name, container name and the host
on which the container is running. Since metric names do not encompass
container or server name we are able to easily aggregate data across
our deployment. For example we can filter for the
container_memory_usage_bytes to get
information about the memory usage of all ubuntu containers in our
deployment. Using the built in functions we can also aggregate the
resulting set of of metrics. For example
average_over_time(container_memory_usage_bytes
[5m]) will show the memory used by ubuntu
containers, averaged over the last five minutes. Once you are happy with
with a query you can click over to the Graph tab and see the variation
of the metric over time.

Temporary graphs are great for ad-hoc investigations but you also need
to have persistent graphs for dashboards. For this you can use the
Prometheus Dashboard
Builder. To launch
Prometheus Dashboard Builder you need access to an SQL database which
you can create using the official MySQL Docker
image. The command to launch
the MySQL container is shown below, note that you may select any value
for database name, user name, user password and root password however
keep track of these values as they will be needed later.

docker run -p 3306:3306 –name promdash-mysql
-e MYSQL_DATABASE=<database-name>
-e MYSQL_USER=<database-user>
-e MYSQL_PASSWORD=<user-password>
-e MYSQL_ROOT_PASSWORD=<root-password>
-d mysql

Once you have the database setup, use the rake installation inside the
promdash container to initialize the database. You can then run the
Dashboard builder by running the same container. The command to
initialize the database and bring up the Prometheus Dashboard Builder
are shown below.

# Initialize Database
docker run –rm -it –link promdash-mysql:db
-e DATABASE_URL=mysql2://<database-user>:<user-password>@db:3306/<database-name> prom/promdash ./bin/rake db:migrate

# Run Dashboard
docker run -d –link promdash-mysql:db -p 3000:3000 –name prometheus-dash
-e DATABASE_URL=mysql2://<database-user>:<user-password>@db:3306/<database-name> prom/promdash

Once your container is running you can browse to port 3000 and load up
the dashboard builder UI. In the UI you need to click Servers in the
top menu and New Server to add your Prometheus Server as a datasource
for the dashboard builder. Add http://PROMETHEUS_SERVER_IP:9090 to
the list of servers and hit Create Server.

Now click Dashboards in the top menu, here you can create
Directories (Groups of Dashboards) and Dashboards. For example we
created a directory for Web Nodes and one for Database Nodes and in each
we create a dashboard as shown below.

Once you have created a dashboard you can add metrics by mousing over
the title bar of a graph and selecting the data sources icon (Three
Horizontal lines with an addition sign following them ). You can then
select the server which you added earlier, and a query expression which
you tested in the Prometheus Server UI. You can add multiple data
sources into the same graph in order to see a comparative view.

You can add multiple graphs (each with possibly multiple data sources)
by clicking the Add Graph button. In addition you may select the
time range over which your dashboard displays data as well as a refresh
interval for auto-loading data. The dashboard is not as polished as the
ones from Scout and DataDog, for example there is no easy way to explore
metrics or build a query in the dashboard view. Since the dashboard runs
independently of the Prometheus server we can’t ‘pin’ graphs
generated in the Prometheus server into a dashboard. Furthermore several
times we noticed that the UI would not update based on selected data
until we refreshed the page. However, despite its issues the dashboard
is feature competitive with DataDog and because Prometheus is under
heavy development, we expect the bugs to be resolved over time. In
comparison to other self-hosted solutions Prometheus is a lot more user
friendly than Sensu and allows you present metric data as graphs without
using third party visualizations. It also is able to provide much better
analytical capabilities than CAdvisor.

Prometheus also has the ability to apply alerting rules over the input
data and displaying those on the UI. However, to be able to do something
useful with alerts such send emails or notify
pagerduty we need to run the the Alert
Manager. To run
the Alert Manager you first need to create a configuration file. Create
a file called alertmanager.conf and add the following text into it:

notification_config {
name: “ubuntu_notification”
pagerduty_config {
service_key: “<PAGER_DUTY_API_KEY>”
}
email_config {
email: “<TARGET_EMAIL_ADDRESS>”
}
hipchat_config {
auth_token: “<HIPCHAT_AUTH_TOKEN>”
room_id: 123456
}
}
aggregation_rule {
filter {
name_re: “image”
value_re: “ubuntu:14.04”
}
repeat_rate_seconds: 300
notification_config_name: “ubuntu_notification”
}

In this configuration we are creating a notification configuration
called ubuntu_notification, which specifies that alerts must go to
the PagerDuty, Email and HipChat. We need to specify the relevant API
keys and/or access tokens for the HipChat and PagerDutyNotifications to
work. We are also specifying that the alert configuration should only
apply to alerts on metrics where the label image has the value
ubuntu:14.04. We specify that a triggered alert should not retrigger
for at least 300 seconds after the first alert is raised. We can bring
up the Alert Manager using the docker image by volume mounting our
configuration file into the container using the command shown below.

docker run -d -p 9093:9093 -v $PWD:/alertmanager prom/alertmanager -logtostderr -config.file=/alertmanager/alertmanager.conf

Once the container is running you should be able to point your browser
to port 9093 and load up the Alarm Manger UI. You will be able to see
all the alerts raised here, you can ‘silence’ them or delete them once
the issue is resolved. In addition to setting up the Alert Manager we
also need to create a few alerts. Add rule_file:
“/prometheus.rules” in a new line into the global section of the
prometheus.conf file you created earlier. This line tells Prometheus
to look for alerting rules in the prometheus.rules file. We now need
to create the rules file and load it into our server container. To do so
create a file called prometheus.rules in the same directory where you
created prometheus.conf. and add the following text to it:

ALERT HighMemoryAlert
IF container_memory_usage_bytes > 1000000000
FOR 1m
WITH {}
SUMMARY “High Memory usage for Ubuntu container”
DESCRIPTION “High Memory usage for Ubuntu container on {{$labels.instance}} for container {{$labels.name}} (current value: {{$value}})”

In this configuration we are telling Prometheus to raise an alert called
HighMemoryAlert if the container_memory_usage_bytes metric
for containers using the Ubuntu:14.04 image goes above 1 GB for 1
minute. The summary and the description of the alerts is also specified
in the rules file. Both of these fields can contain placeholders for
label values which are replaced by Prometheus. For example our
description will specify the server instance (IP) and the container name
for metric raising the alert. After launching the Alert Manager and
defining your Alert rules, you will need to re-run your Prometheus
server with new parameters. The commands to do so are below:

# stop and remove current container
docker stop prometheus-server && docker rm prometheus-server

# start new container
docker run -d –name prometheus-server -p 9090:9090
-v $PWD/prometheus.conf:/prometheus.conf
-v $PWD/prometheus.rules:/prometheus.rules
prom/prometheus
-config.file=/prometheus.conf
-alertmanager.url=http://ALERT_MANAGER_IP:9093

Once the Prometheus Server is up again you can click Alerts in the top
menu of the Prometheus Server UI to bring up a list of alerts and their
statuses. If and when an alert is fired you will also be able to see it
in the Alert Manager UI and any external service defined in the
alertmanager.conf file.

Collectively the Prometheus tool-set’s feature set is on par with
DataDog which has been our best rated Monitoring tool so far. Prometheus
uses a very simple format for input data and can ingest from any web
endpoint which presents the data. Therefore we can monitor more or less
any resource with Prometheus, and there are already several libraries
defined to monitor common resources. Where Prometheus is lacking is in
level of polish and ease of deployment. The fact that all components are
dockerized is a major plus however, we had to launch 4 different
containers each with their own configuration files to support the
Prometheus server. The project is also lacking detailed, comprehensive
documentation for these various components. However, in caparison to
self-hosted services such as CAdvisor and Sensu, Prometheus is a much
better toolset. It is significantly easier setup than sensu and has the
ability to provide visualization of metrics without third party tools.
It is able has much more detailed metrics than CAdvisor and is also able
to monitor non-docker resources. The choice of using pull based metric
aggregation rather than push is less than ideal as you would have to
restart your server when adding new data sources. This could get
cumbersome in a dynamic environment such as cloud based deployments.
Prometheus does offer the Push
Gateway to bridge the
disconnect. However, running yet another service will add to the
complexity of the setup. For these reasons I still think DataDog is
probably easier for most users, however, with some polish and better
packaging Prometheus could be a very compelling alternative, and out of
self-hosted solutions Prometheus is my pick.

Score Card:

1. Easy of deployment: **
2. Level of detail: *****
3. Level of aggregation: *****
4. Ability to raise alerts: ****
5. Ability to monitor non-docker resources: Supported
6. Cost: Free

Sysdig Cloud

Sysdig cloud is a hosted service that provides metrics storage,
aggregation, visualization and alerting. To get started with sysdig sign
up for a trial account at https://app.sysdigcloud.com. and complete
the registration form. Once you complete the registration form and log
in to the account, you will be asked to Setup your Environment and be
given a curl command similar to the shown below. Your command will have
your own secret key after the -s switch. You can run this command on the
host running docker and which you need to monitor. Note that you should
replace the [TAGS] place holder with tags to group your metrics. The
tags are in the format TAG_NAME:VALUE so you may want to add a tag
role:web or deployment:production. You may also use the containerized
sysdig agent.

# Host install of sysdig agent
curl -s https://s3.amazonaws.com/download.draios.com/stable/install-agent | sudo bash -s 12345678-1234-1234-1234-123456789abc [TAGS]

# Docker based sysdig agent
docker run –name sysdig-agent –privileged –net host
-e ACCESS_KEY=12345678-1234-1234-1234-123456789abc
-e TAGS=os:rancher
-v /var/run/docker.sock:/host/var/run/docker.sock
-v /dev:/host/dev -v /proc:/host/proc:ro
-v /boot:/host/boot:ro
-v /lib/modules:/host/lib/modules:ro
-v /usr:/host/usr:ro sysdig/agent

Even if you use docker you will still need to install Kernel headers in
the host OS. This goes against Docker’s philosophy of isolated micro
services. However, installing kernel headers is fairly benign.
Installing the headers and getting sysdig running is trivial if you are
using a mainstream kernel such us CentOS, Ubuntu or Debian. Even the
Amazon’s custom kernels are supported however RancherOS’s custom
kernel presented problems for sysdig as did the tinycore kernel. So be
warned if you would like to use Sysdig cloud on non-mainstream kernels
you may have to get your hands dirty with some system hacking.

After you run the agent you should see the Host in the Sysdig cloud
console in the Explore tab. Once you launch docker containers on the
host those will also be shown. You can see basic stats about the CPU
usage, memory consumption, network usage. The metrics are aggregated for
the host as well as broken down per container.

By
selecting one of the hosts or containers you can get a whole host of
other metrics including everything provided by the docker stats API. Out
of all the systems we have seen so far sysdig certainly has the most
comprehensive set of metrics out of the box. You can also select from
several pre-configured dashboards which present a graphical or tabular
representation of your deployment.

You can see live metrics, by selecting Real-time Mode (Target Icon)
or select a window of time over which to average values. Furthermore,
you can also setup comparisons which will highlight the delta of current
values and values at a point in the past. For example the table below
shows values compared with those from ten minutes ago. If the CPU usage
is significantly higher than 10 minutes ago you may be experiencing load
spikes and need to scale out. The UI is at par with, if not better than
DataDog for identifying and exploring trends in the data.

In addition to exploring data on an ad-hoc basis you can also create
persistent dashboards. Simply click the pin icon on any graph in the
explore view and save it to a named dashboard. You can view all the
dashboards and their associated graphs by clicking the Dashboards
tab. You can also select the bell icon on any graph and create an
alert from the data. The Sysdig cloud supports detailed alerting
criteria and is again one of the best we have seen. The example below
shows an alert which triggers if the count of containers labeled web
falls below three on average for the last ten minutes. We are also
segmenting the data by the region tag, so there will be a separate
check for web nodes in North America and Europe. Lastly, we also specify
a Name, description and Severity for the alerts. You can control where
alerts go by going to Settings (Gear Icon) > Notifications and add
email addresses or SNS Topics to send alerts too. Note all alerts go to
all notification endpoints which may be problematic if you want to wake
up different people for different alerts.

I am very impressed with Sysdig cloud as it was trivially easy to setup,
provides detailed metrics with great visualization tools for real-time
and historical data. The requirement to install kernel headers on the
host OS is troublesome though and lack of documentation and support for
non-standard kernels could be problematic in some scenarios. The
alerting system in the Sysdig cloud is among the best we have seen so
far, however, the inability to target different email addresses for
different alerts is problematic. In a larger team for example you would
want to alert a different team for database issues vs web server issues.
Lastly, since it is in beta the pricing for Sysdig cloud is not easily
available. I have reached out to their sales team and will update this
article if and when they get back to me. If sysdig is price competitive
then Datadog has serious competition in the hosted service category.

Score Card:

1. Easy of deployment: ***
2. Level of detail: *****
3. Level of aggregation: *****
4. Ability to raise alerts: ****
5. Ability to monitor non-docker resources: Supported
6. Cost: Must Contact Support

Conclusion

Today’s article has covered several options for monitoring docker
containers, ranging from free options; docker stats, CAdvisor,
Prometheus or Sensu to paid services such as Scout, Sysdig Cloud and
DataDog. From my research so far DataDog seems to be the best-in-class
system for monitoring docker deployments. The setup was complete in
seconds with a one-line command, all hosts were reporting metrics in one
place, historical trends were apparent in the UI and Datadog supports
deep diving into metrics as well as alerting. However, at $15 per host
the system can get expensive for large deployments. For larger scale,
self-hosted deployments Sensu is able to fulfill most requirements
however the complexity in setting up and managing a Sensu cluster may be
prohibitive. Obviously, there are plenty of other self-hosted options,
such as Nagios or Icinga, which are similar to Sensu.

Hopefully this gives you an idea of some of the options for monitoring
containers available today. I am continuing to investigate other
options, including a more streamlined self-managed container monitoring
system using CollectD, Graphite or InfluxDB and Grafana. Stay tuned for
more details.

ADDITIONAL INFORMATION: After publishing this article I had some
suggestions to also evaluate Prometheus and Sysdig Cloud, two other very
good options for monitoring Docker. We’ve now included them in this
article, for ease of discovery. You can find the original second part
of my
posthere.

To learn more about monitoring and managing Docker, please join us for
our next Rancher online meetup.

Usman is a server and infrastructure engineer, with experience in
building large scale distributed services on top of various cloud
platforms. You can read more of his work at
techtraits.com, or follow him on twitter
@usman_ismailor
onGitHub.

Source