Update (October 2017): Gord Sissons revisited this topic and compared *Update (October 2016): Our October online meetup demonstrated and As Docker is used for larger deployments it becomes more important to criteria:<\/p>\n This newly-updated, in-depth guidebook provides a detailed overview of the features and functionality of the new Rancher: an open-source enterprise Kubernetes platform.<\/p>\n Get the eBook<\/a><\/p>\n All commands in this article have been specifically tested on The first tool I will talk about is Docker itself, yes you may not be $ docker stats determined_shockley determined_wozniak prickly_hypatia For a more detailed look at container stats you may also use the Docker echo -e “GET \/containers\/[CONTAINER_NAME]\/stats HTTP\/1.0rn” | nc -U \/var\/run\/docker.sock<\/p>\n Score Card:<\/p>\n The docker stats command and the remote API are useful for getting docker run CAdvisor is a useful tool that is trivially easy to setup, it saves us Score Card (Ignoring heapster because only supported on Kubernetes):<\/p>\n The next approach for docker monitoring is Scout and it addresses # account_key is the only required value You can now bring up your scout agent with the scout configuration file docker run -d –name scout-agent Now go back to the Scout web view and you should see an entry for your To add docker monitoring to your servers click the Roles tab and then Another advantage of using Scout over CAdvisor is that it has a large One drawback of Scout is that it does not present detailed information Score Card:<\/p>\n From Scout lets move to another monitoring service, DataDog, which docker run -d –privileged –name dd-agent Now that your containers are connected you can go to the Events<\/em> tab in You can also click the Dashboards<\/em> tab and hit create dashboards to Data Dog also supports alerting using a feature called *Monitors. *A Then specify the alert conditions to say that if there are fewer than Lastly, using the Metrics Explorer<\/em> tab you can make ad-hoc Score Card:<\/p>\n Scout and Datadog provide centralized monitoring and alerting however Before launch your sensu server container you must define a check that { Now you can run the sensu server docker container with our check docker run -d –name sensu-server Now that the sensu server is up you can launch sensu clients on each of #!\/bin\/bash # Count all running containers # Count all images echo “docker.HOST_NAME.running_containers $” if [ $ -lt 3 ]; then Now that you have defined your load docker metrics check you need to to docker run -d –name sensu-client –privileged A few seconds after running this command you should see the client If you click on the client name your should get further details of the Sensu ticks all the boxes in our evaluation criteria; you can collect as Easy of deployment: * Level of detail: **** Level of aggregation: I also evaluated two other monitoring services, Prometheus and Sysdig First lets take a look at Prometheus; it is a self-hosted set of tools docker run -p 9104:9104 -v \/sys\/fs\/cgroup:\/cgroup -v \/var\/run\/docker.sock:\/var\/run\/docker.sock prom\/container-exporter<\/p>\n Once we have got all our exporters running we are can launch Prometheus global: rule_files:<\/p>\n scrape_configs: target_groups: In this file there are two sections, global and job(s). In the global docker run -d –name prometheus-server -p 9090:9090 -v $PWD\/prometheus.conf:\/prometheus.conf prom\/prometheus -config.file=\/prometheus.conf<\/p>\n After launching the container, Prometheus server should be available in We are able to drill down into the data using queries to filter out data Temporary graphs are great for ad-hoc investigations but you also need docker run -p 3306:3306 –name promdash-mysql Once you have the database setup, use the rake installation inside the # Initialize Database # Run Dashboard Once your container is running you can browse to port 3000 and load up Now click Dashboards<\/em> in the top menu, here you can create Once you have created a dashboard you can add metrics by mousing over You can add multiple graphs (each with possibly multiple data sources) Prometheus also has the ability to apply alerting rules over the input notification_config { In this configuration we are creating a notification configuration docker run -d -p 9093:9093 -v $PWD:\/alertmanager prom\/alertmanager -logtostderr -config.file=\/alertmanager\/alertmanager.conf<\/p>\n Once the container is running you should be able to point your browser ALERT HighMemoryAlert In this configuration we are telling Prometheus to raise an alert called # stop and remove current container # start new container Once the Prometheus Server is up again you can click Alerts in the top Collectively the Prometheus tool-set\u2019s feature set is on par with Score Card:<\/p>\n Sysdig cloud is a hosted service that provides metrics storage, # Host install of sysdig agent # Docker based sysdig agent Even if you use docker you will still need to install Kernel headers in After you run the agent you should see the Host in the Sysdig cloud You can see live metrics, by selecting Real-time Mode (Target Icon) In addition to exploring data on an ad-hoc basis you can also create I am very impressed with Sysdig cloud as it was trivially easy to setup, Score Card:<\/p>\n Today\u2019s article has covered several options for monitoring docker Hopefully this gives you an idea of some of the options for monitoring ADDITIONAL INFORMATION: After publishing this article I had some To learn more about monitoring and managing Docker, please join us for Usman is a server and infrastructure engineer, with experience in
\nthe top 10 container-monitoring solutions for Rancher in a recent blog
\npost<\/a>.<\/em><\/p>\n
\ncompared Sysdig, Datadog, and Prometheus in one go. Check
\nout<\/a>
\nthe recording. *<\/p>\n
\nget visibility into the status and health of docker environments. In
\nthis article, I aim to go over some of the common tools used to monitor
\ncontainers. I will be evaluating these tools based on the following<\/p>\n\n
A Detailed Overview of Rancher’s Architecture<\/h5>\n
Docker Stats<\/h2>\n
\na RancherOS instance running on Amazon Web Services EC2. However, all
\ntools presented today should be usable on any Docker deployment.<\/p>\n
\naware that docker client already provides a rudimentary command line
\ntool to inspect containers\u2019 resource consumption. To look at the
\ncontainer stats run docker stats<\/em> with the name(s) of the running
\ncontainer(s) for which you would like to see stats. This will present
\nthe CPU utilization for each container, the memory used and total memory
\navailable to the container. Note that if you have not limited memory for
\ncontainers this command will post total memory of your host. This does
\nnot mean each of your container has access to that much memory. In
\naddition you will also be able to see total data sent and received over
\nthe network by the container.<\/p>\n
\nCONTAINER CPU % MEM USAGE\/LIMIT MEM % NET I\/O
\ndetermined_shockley 0.00% 884 KiB\/1.961 GiB 0.04% 648 B\/648 B
\ndetermined_wozniak 0.00% 1.723 MiB\/1.961 GiB 0.09% 1.266 KiB\/648 B
\nprickly_hypatia 0.00% 740 KiB\/1.961 GiB 0.04% 1.898 KiB\/648 B<\/p>\n
\nRemote API via netcat (See below). Send an http get request for
\n\/containers\/[CONTAINER_NAME]\/stats<\/em> where CONTAINER_NAME is name of
\nthe container for which you want to see stats. You can see an example of
\nthe complete response for a container stats request
\nhere<\/a>. This
\nwill present details of the metrics shown above for example you will get
\ndetails of caches, swap space and other details about memory. You may
\nwant to peruse the Run
\nMetrics<\/a> section of the
\nDocker documentation to get an idea of what the metrics mean.<\/p>\n\n
CAdvisor<\/h2>\n
\ninformation on the command line, however, if you would like to access
\nthe information in a graphical interface you will need a tool such as
\nCAdvisor<\/a>. CAdvisor provides a
\nvisual representation of the data shown by the docker stats command
\nearlier. Run the docker command below and go to
\nhttp:\/\/<your-hostname>:8080\/<\/em> in the browser of your choice to see
\nthe CAdvisor interface. You will be shown graphs for overall CPU usage,
\nMemory usage, Network throughput and disk space utilization. You can
\nthen drill down into the usage statistics for a specific container by
\nclicking the Docker Containers<\/em> link at the top of the page and then
\nselecting the container of your choice. In addition to these statistics
\nCAdvisor also shows the limits, if any, that are placed on container,
\nusing the Isolation section.<\/p>\n
\n–volume=\/:\/rootfs:ro
\n–volume=\/var\/run:\/var\/run:rw
\n–volume=\/sys:\/sys:ro
\n–volume=\/var\/lib\/docker\/:\/var\/lib\/docker:ro
\n–publish=8080:8080
\n–detach=true
\n–name=cadvisor
\ngoogle\/cadvisor:latest<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-19-at-11.50.29-PM.png\")
<\/a><\/p>\n
\nfrom having to ssh into the server to look at resource consumption and
\nalso produces graphs for us. In addition the pressure gauges provide a
\nquick overview of when a your cluster needs additional resources.
\nFurthermore, unlike other options in this article CAdvisor is free as it
\nis open source and also it runs on hardware already provisioned for your
\ncluster, other than some processing resources there is no additional
\ncost of running CAdvisor. However, it has it limitations; it can only
\nmonitor one docker host and hence if you have a multi-node deployment
\nyour stats will be disjoint and spread though out your cluster. Note
\nthat you can use
\nheapster<\/a> to monitor
\nmultiple nodes if you are running Kubernetes. The data in the charts is
\na moving window of one minute only and there is no way to look at longer
\nterm trends. There is no mechanism to kick-off alerting if the resource
\nusage is at dangerous levels. If you currently do not have any
\nvisibility in to the resource consumption of your docker node\/cluster
\nthen CAdvisor is a good first step into container monitoring however, if
\nyou intend to run any critical tasks on your containers a more robust
\ntool or approach is needed. Note that
\nRancher<\/a> runs CAdvisor on
\neach connected host, and exposes a limited set of stats through the UI,
\nand all of the system stats through the API.<\/p>\n\n
Scout
![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-9.30.08-AM.png\")
<\/a><\/h2>\n
\nseveral of limitations of CAdvisor. Scout is a hosted monitoring service
\nwhich can aggregate metrics from many hosts and containers and present
\nthe data over longer time-scales. It can also create alerts based on
\nthose metrics. The first step to getting scout running is to sign up
\nfor a Scout account at https:\/\/scoutapp.com\/<\/a>, the free trial account
\nshould be suitable for testing out integration. Once you have created
\nyour account and logged in, click on your account name in the top right
\ncorner and then Account Basics<\/em> and take note of your Account Key as
\nyou will need this to send metrics from our docker server.<\/p>\n![\"accountid\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/accountid.png\")
<\/a>Now
\non your host, create a file called scoutd.yml<\/em> and copy the following
\ntext into the the file, replacing the account_key<\/em> with the key you
\ntook note of earlier. You can specify any values that make sense for the
\nhost, display_name, environment and roles properties. These will be
\nused to separate out the metrics when they are presented in the scout
\ndashboard. I am assuming an array of web-servers is run on docker so
\nwill use the values shown below.<\/p>\n
\naccount_key: YOUR_ACCOUNT_KEY
\nhostname: web01-host
\ndisplay_name: web01
\nenvironment: production
\nroles: web<\/p>\n
\nby using the docker scout plugin.<\/p>\n
\n-v \/proc:\/host\/proc:ro
\n-v \/etc\/mtab:\/host\/etc\/mtab:ro
\n-v \/var\/run\/docker.sock:\/host\/var\/run\/docker.sock:ro
\n-v `pwd`\/scoutd.yml:\/etc\/scout\/scoutd.yml
\n-v \/sys\/fs\/cgroup\/:\/host\/sys\/fs\/cgroup\/
\n–net=host –privileged
\nsoutapp\/docker-scout<\/p>\n
\nagent which will be keyed by the display_name parameter (web01) that
\nyou specified in your scoutd.yml earlier.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-9.58.40-AM.png\")
<\/a>If
\nyou click the display name it will display detailed metrics for the
\nhost. This includes the process count, CPU usage and memory utilization
\nfor everything running on your host. Note these are not limited to
\nprocesses running inside docker.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-10.00.47-AM.png\")
<\/a><\/p>\n
\nselect All Servers.<\/em> Now click the + Plugin Template<\/em> Button and then
\nDocker Monitor<\/em> from the following screen to load the details view.
\nOnce you have the details view up select Install Plugin to add the
\nplugin to your hosts. In the following screen give a name to the plugin
\ninstallation and specify which containers you want to monitor. If you
\nleave the field blank the plugin will monitor all of the containers on
\nthe host. Click complete installation and after a minute or so you can
\ngo to [Server Name] > Plugins to see details from the docker monitor
\nplugin. The plugin shows the CPU usage, memory usage network throughput
\nand the number of containers for each host.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-20-at-10.11.06-PM.png\")
<\/a><\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-20-at-10.11.39-PM.png\")
<\/a>If
\nyou click on any of the graphs you can pull a detailed view of the
\nmetrics and this view allows you to see the trends in the metric values
\nacross a longer time span. This view also allows you to filter the
\nmetrics based on environment and server role. In addition you can create
\n\u201cTriggers\u201d or alerts to send emails to you if metrics go above or
\nbelow a configured threshold. This allows you to setup automated alerts
\nto notify you if for example some of your containers die and the
\ncontainer count falls below a certain number. You can also setup alerts
\nfor average CPU utilization so if for example if your containers are
\nrunning hot you will get an alert and you can launch more add more hosts
\nto your docker cluster. To create a trigger select ![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-22-at-6.30.25-PM.png\")
<\/a>Roles<\/em>
\n> All Servers<\/em> from the top menu and then docker monitor<\/em> from the
\nplugins section. Then select triggers<\/em> from the Plugin template
\nAdministration<\/em> menu on the right hand side of the screen. You should
\nnow see an option to \u201dAdd a Trigger\u201d<\/em> which will apply to the entire
\ndeployment. Below is an example of a trigger which will send out an
\nalert if the number of containers in the deployment falls below 3. The
\nalert was created for \u201cAll Servers\u201d however you could tag your hosts
\nwith different roles using the scoutd.yml created on the server. Using
\nthe roles you can apply triggers to a sub-set of the servers on your
\ndeployment. For example you could setup an alert for when the number of
\ncontainers on your web nodes falls below a certain number. Even with the
\nrole based triggers I still feel that Scout alerting could be better.
\nThis is because many docker deployments have heterogeneous containers on
\nthe same host. In such a scenario it would be impossible to setup
\ntriggers for specific types of containers as roles are applied to all
\ncontainers on the host.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-22-at-6.33.12-PM.png\")
<\/a><\/p>\n
\nset of plugins<\/a> which can pull in
\nother data about your deployment in addition to docker information. This
\nallows Scout to be your one stop monitoring system instead of having a
\ndifferent monitoring system for various resources in your system.<\/p>\n
\nabout individual containers on each host like CAdvisor can. This is
\nproblematic, if your are running heterogeneous containers on the same
\nserver. For example if you want a trigger to alert you about issues in
\nyour web containers but not about your Jenkins containers Scout will not
\nbe able to support that use case. Despite the drawbacks Scout is a
\nsignificantly more useful tool for monitoring your docker deployments.
\nHowever this does come at a cost, ten dollars per monitored host. The
\ncost could be a factor if you are running a large deployment with many
\nhosts.<\/p>\n\n
Data Dog<\/h2>\n
\naddresses several of the short-comings of Scout as well as all of the
\nlimitations of CAdvisor. To get started with DataDog, first sign up for
\na DataDog account at https:\/\/www.datadoghq.com\/<\/a>. Once you are signed
\ninto your account you will be presented with list of supported
\nintegrations with instructions for each type. Select docker<\/em> from the
\nlist and you will be given a docker run command (show below) to copy
\ninto your host. The command will have your API key preconfigured and
\nhence can be run the command as listed. After about 45 seconds your
\nagent will start reporting metrics to the DataDog system.<\/p>\n
\n-h `hostname`
\n-v \/var\/run\/docker.sock:\/var\/run\/docker.sock
\n-v \/proc\/mounts:\/host\/proc\/mounts:ro
\n-v \/sys\/fs\/cgroup\/:\/host\/sys\/fs\/cgroup:ro
\n-e API_KEY=YOUR_API_KEY datadog\/docker-dd-agent<\/p>\n
\nthe DataDog web console and see all events pertaining to your cluster.
\nAll container launches and terminations will be part of this event
\nstream.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-2.56.04-PM.png\")
<\/a><\/p>\n
\naggregate metrics across your entire cluster. Datadog collects metrics
\nabout CPU usage, memory and I\/O for all containers running in the
\nsystem. In addition you get counts of running and stopped containers as
\nwell as counts of docker images. The dashboard view allows you to create
\ngraphs for any metric or set of metrics across the entire deployment or
\ngrouped by host or container image. For example the graph below shows
\nthe number of running containers broken down by the image type, I am
\nrunning 9 ubuntu:14.04 containers in my cluster at the moment.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-2.35.21-PM.png\")
<\/a>
\nYou could also split the same data by Hosts, as the second graph shows,
\n7 of the containers are running on my Rancher host and the remaining
\nones on my local laptop.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-3.14.10-PM.png\")
<\/a><\/p>\n
\nmonitor is DataDog\u2019s equivalent to a Scout trigger and allows you to
\ndefine thresholds for various metrics. DataDog\u2019s alerting system is a
\nlot more flexible and detailed then Scout\u2019s. The example below shows
\nhow to specify that you are concerned about Ubuntu containers
\nterminating hence you would monitor the docker.containers.running metric
\nfor containers created from the ubuntu:14.04 docker image.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-22-at-6.49.53-PM.png\")
<\/a><\/p>\n
\nten ubuntu containers in our deployment (on average) for the last 5
\nminutes, you would like to be alerted. Although not shown here you will
\nalso be asked to specify the text of the message which is sent out when
\nthis alert is triggered as well as the target audience for this alert.
\nIn the current example I am using a simple absolute threshold. You can
\nalso specify a delta based alert which triggers if say the avg stopped
\ncontainer count was four over the last five minutes raise an alert.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-22-at-6.49.58-PM.png\")
<\/a><\/p>\n
\naggregations over your metrics to help debug issues or extract specific
\ninformation from your data. This view allows your to graph any metric
\nover a slice based on container image or host. You may combine output
\ninto a single graph or generate a set of graphs by grouping across
\nimages or hosts.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-21-at-2.40.30-PM.png\")
<\/a>
\nDataDog is a significant improvement over scout in terms feature set,
\neasy of use and user friendly design. However this level of polish comes
\nwith additional cost as each DataDog agent costs $15.<\/p>\n\n
\nhost![\"sensu\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/sensu.png\")
<\/a><\/li>\n<\/ol>\nSensu Monitoring Framework<\/h2>\n
\nboth are hosted services that can get expensive for large deployments.
\nIf you need a self-hosted, centralized metrics service, you may
\nconsider the sensu open source monitoring
\nframework<\/a>. To run the Sensu server you can use
\nthe hiroakis\/docker-sensu-server<\/a>
\ncontainer. This container installs sensu-server, the uchiwa web
\ninterface, redis, rabbitmq-server, and the sensu-api. Unfortunately
\nsensu does not have any docker support out of the box. However, using
\nthe plugin system you can configure support for both container metrics
\nas well as status checks.<\/p>\n
\nyou can load into the server. Create a file called check-docker.json<\/em>
\nand add the following contents into the file. In this file you are
\ntelling the Sensu server to run a script called load-docker-metrics.sh<\/em>
\nevery ten seconds on all clients which are subscribed to the docker tag.
\nYou will define this script a little later.<\/p>\n
\n“checks”: {
\n“load_docker_metrics”: {
\n“type”: “metric”,
\n“command”: “load-docker-metrics.sh”,
\n“subscribers”: [
\n“docker”
\n],
\n“interval”: 10
\n}
\n}
\n}<\/p>\n
\nconfiguration file using the command below. Once you run the command you
\nshould be able to launch the uchiwa dashboard at
\nhttp:\/\/YOUR_SERVER_IP:3000<\/a> in your browser.<\/p>\n
\n-p 3000:3000
\n-p 4567:4567
\n-p 5671:5671
\n-p 15672:15672
\n-v $PWD\/check-docker.json:\/etc\/sensu\/conf.d\/check-docker.json
\nhiroakis\/docker-sensu-server<\/p>\n
\nthe hosts running our docker containers. You told the server that the
\ncontainers will have a script called load-docker-metrics.sh<\/em> so lets
\ncreate the script and insert it into our client containers. Create the
\nfile and add the text shown below into the file, replacing HOST_NAME
\nwith a logical name for your host . The script below is using the
\nDocker Remote
\nAPI<\/a>to
\npull in the meta data for running containers, all containers and all
\nimages on the host. It then prints the values out using sensu\u2019s key
\nvalue notation. The sensu server will read the output values from the
\nSTDOUT and collect those metrics. This example only pulls these three
\nvalues but you could make the script as detailed as required. Note that
\nyou could also add multiple check scripts such as thos, as long as you
\nreference them in the server configuration file you created earlier. You
\ncan also define that you want the check to fail if the number of running
\ncontainers ever falls below three. You can make a check fail by
\nreturning a non-zero value from the check script.<\/p>\n
\nset -e<\/p>\n
\nrunning_containers=$(echo -e “GET \/containers\/json HTTP\/1.0rn” | nc -U \/var\/run\/docker.sock
\n| tail -n +5
\n| python -m json.tool
\n| grep “Id”
\n| wc -l)
\n# Count all containers
\ntotal_containers=$(echo -e “GET \/containers\/json?all=1 HTTP\/1.0rn” | nc -U \/var\/run\/docker.sock
\n| tail -n +5
\n| python -m json.tool
\n| grep “Id”
\n| wc -l)<\/p>\n
\ntotal_images=$(echo -e “GET \/images\/json HTTP\/1.0rn” | nc -U \/var\/run\/docker.sock
\n| tail -n +5
\n| python -m json.tool
\n| grep “Id”
\n| wc -l)<\/p>\n
\necho “docker.HOST_NAME.total_containers $”
\necho “docker.HOST_NAME.total_images $”<\/p>\n
\nexit 1;
\nfi<\/p>\n
\nstart the sensu client using the
\nusman\/sensu-client<\/a>
\ncontainer I defined for this purpose. You can use the command shown
\nbelow to launch sensu client. Note that the container must run as
\nprivileged in order to be able to access unix sockets, it must have the
\ndocker socket mounted in as a volume as well as the
\nload-docker-metrics.sh<\/em> script you defined above. Make sure the
\nload-docker-metrics.sh script is marked as executable in your host
\nmachine as the permissions carry through into the container. The
\ncontainer also takes in SENSU_SERVER_IP, RABIT_MQ_USER,
\nRABIT_MQ_PASSWORD, CLIENT_NAME and CLIENT_IP as parameters, please
\nspecify the value of these parameters for your setup. The default values
\nfor the RABIT_MQ_USER RABIT_MQ_PASSWORD are sensu<\/em> and password<\/em>.<\/p>\n
\n-v $PWD\/load-docker-metrics.sh:\/etc\/sensu\/plugins\/load-docker-metrics.sh
\n-v \/var\/run\/docker.sock:\/var\/run\/docker.sock
\nusman\/sensu-client SENSU_SERVER_IP RABIT_MQ_USER RABIT_MQ_PASSWORD CLIENT_NAME CLIENT_IP<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-28-at-10.13.48-PM.png\")
<\/a><\/p>\n
\ncount increase to 1 in the uchiwa dashboard. If you click the clients
\nicon you should see a list of your clients including the client that you
\njust added. I named my client client-1<\/em> and specified the host IP as
\n192.168.1.1.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-28-at-10.13.54-PM.png\")
<\/a><\/p>\n
\nchecks. You can see that the load_docker_metrics check was run at
\n10:22 on the 28th of March.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-28-at-10.14.00-PM.png\")
<\/a>If
\nyou Click on the check name you can see further details of check runs.
\nThe zeros indicate that there were no errors, if the script had failed
\n(if for example your docker Daemon dies) you would see an error code
\n(non zero) value. Although it is not covered this in the current article
\nyou can also setup sensu to alert you when these checks fail using
\nHandlers<\/a>. Furthermore,
\nuchiwa only shows the values of checks and not the metrics collected.
\nNote that sensu does not store the collected metrics, they have to be
\nforwarded to a time series database such as InfluxDB or Graphite. This
\nis also done through Handlers. Please find details of how to configure
\nmetric forwarding to graphite
\nhere<\/a>.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/03\/13030215\/Screen-Shot-2015-03-28-at-10.27.59-PM.png\")
<\/a><\/p>\n
\nmuch detail about our docker containers and hosts as ypu want. In
\naddition you are able to aggregate the values of all of out hosts in one
\nplace and raise alerts over those checks. The alerting is not as
\nadvanced as DataDog or Scout, as you are only able to alert on checks
\nfailing on individual hosts. However, the big drawback of Sensu is
\ndifficulty of deployment. Although I have automated many steps in the
\ndeployment using docker containers, Sensu remains a complicated system
\nrequiring us to install, launch and maintain separate processes for
\nRedis, RabitMQ, Sensu API, uchiwa and Sensu Core. Furthermore, you would
\nrequire still more tools such as Graphite to present metric values and a
\nproduction deployment would require customizing the containers I have
\nused today for secure passwords and custom ssl certificates. In addition
\nwere you to add more checks after launching the container you would have
\nto restart the Sensu server as that is the only way for it start
\ncollecting new metrics. For these reasons I rate Sensu fairly low for
\nease of deployment.<\/p>\n
\n**** Ability to raise alerts: Supported but limited Ability to
\nmonitor non-docker resources: ***** Cost: $Free<\/p>\n
\nCloud in a
\nsecond article<\/a>,
\nand have included them in this post for simplicity.<\/em><\/p>\nPrometheus<\/h2>\n
\nwhich collectively provide metrics storage, aggregation, visualization
\nand alerting. Most of the tools and services we have looked at so far
\nhave been push based, i.e. agents on the monitored servers talk to a
\ncentral server (or set of servers) and send out their metrics.
\nPrometheus on the other hand is a pull based server which expects
\nmonitored servers to provide a web interface from which it can scrape
\ndata. There are several exporters
\navailable<\/a> for
\nPrometheus which will capture metrics and then expose them over http for
\nPrometheus to scrape. In addition there are
\nlibraries<\/a> which
\ncan be used to create custom exporters. As we are concerned with
\nmonitoring docker containers we will use the
\ncontainer_exporter<\/a>
\ncapture metrics. Use the command shown below to bring up the
\ncontainer-exporter docker container and browse to
\nhttp:\/\/MONITORED_SERVER_IP:9104\/metrics<\/a><\/em> to see the metrics it has
\ncollected for you. You should launch exporters on all servers in your
\ndeployment. Keep track of the respective *MONITORED_SERVER_IP*s as we
\nwill be using them later in the configuration for Prometheus.<\/p>\n
\nserver. However, before we do we need to create a configuration file for
\nPrometheus that tells the server where to scrape the metrics from.
\nCreate a file called prometheus.conf<\/em> and then add the following text
\ninside it.<\/p>\n
\nscrape_interval: 15s
\nevaluation_interval: 15s
\nlabels:
\nmonitor: exporter-metrics<\/p>\n
\n– job_name: prometheus
\nscrape_interval: 5s<\/p>\n
\n# These endpoints are scraped via HTTP.
\n– targets: [‘localhost:9090′,’MONITORED_SERVER_IP:9104’]<\/p>\n
\nsection we set defaults for configuration properties such as data
\ncollection interval (scrape_interval). We can also add labels which
\nwill be appended to all metrics. In the jobs section we can define one
\nor more jobs that each have a name, an optional override scraping
\ninterval as well as one or more targets from which to scrape metrics. We
\nare adding two targets, one is the Prometheus server itself and the
\nsecond is the container-exporter we setup earlier. If you setup more
\nthan one exporter your can setup additional targets to pull metrics from
\nall of them. Note that the job name is available as a label on the
\nmetric hence you may want to setup separate jobs for your various types
\nof servers. Now that we have a configuration file we can start a
\nPrometheus server using the
\nprom\/prometheus<\/a>
\ndocker image.<\/p>\n
\nyour browser on the port 9090 in a few moments. Select Graph<\/em> from the
\ntop menu and select a metric from the drop down box to view its latest
\nvalue. You can also write queries in the expression box which can find
\nmatching metrics. Queries take the form
\nMETRIC_NAME. You can find more
\ndetails of the query syntax here<\/a>.<\/p>\n![\"\"](\"http:\/\/i.imgur.com\/k0n8b9k.png\")
<\/p>\n
\nfrom specific server types (jobs) and containers. All metrics from
\ncontainers are labeled with the image name, container name and the host
\non which the container is running. Since metric names do not encompass
\ncontainer or server name we are able to easily aggregate data across
\nour deployment. For example we can filter for the
\ncontainer_memory_usage_bytes to get
\ninformation about the memory usage of all ubuntu containers in our
\ndeployment. Using the built in functions we can also aggregate the
\nresulting set of of metrics. For example
\naverage_over_time(container_memory_usage_bytes
\n[5m]) will show the memory used by ubuntu
\ncontainers, averaged over the last five minutes. Once you are happy with
\nwith a query you can click over to the Graph tab and see the variation
\nof the metric over time.<\/p>\n![\"\"](\"http:\/\/i.imgur.com\/DgrLYLl.png\")
<\/p>\n
\nto have persistent graphs for dashboards. For this you can use the
\nPrometheus Dashboard
\nBuilder<\/a>. To launch
\nPrometheus Dashboard Builder you need access to an SQL database which
\nyou can create using the official MySQL Docker
\nimage<\/a>. The command to launch
\nthe MySQL container is shown below, note that you may select any value
\nfor database name, user name, user password and root password however
\nkeep track of these values as they will be needed later.<\/p>\n
\n-e MYSQL_DATABASE=<database-name>
\n-e MYSQL_USER=<database-user>
\n-e MYSQL_PASSWORD=<user-password>
\n-e MYSQL_ROOT_PASSWORD=<root-password>
\n-d mysql<\/p>\n
\npromdash container to initialize the database. You can then run the
\nDashboard builder by running the same container. The command to
\ninitialize the database and bring up the Prometheus Dashboard Builder
\nare shown below.<\/p>\n
\ndocker run –rm -it –link promdash-mysql:db
\n-e DATABASE_URL=mysql2:\/\/<database-user>:<user-password>@db:3306\/<database-name> prom\/promdash .\/bin\/rake db:migrate<\/p>\n
\ndocker run -d –link promdash-mysql:db -p 3000:3000 –name prometheus-dash
\n-e DATABASE_URL=mysql2:\/\/<database-user>:<user-password>@db:3306\/<database-name> prom\/promdash<\/p>\n![\"\"](\"http:\/\/i.imgur.com\/JCwRhwx.png\")
<\/p>\n
\nthe dashboard builder UI. In the UI you need to click Servers<\/em> in the
\ntop menu and New Server<\/em> to add your Prometheus Server as a datasource
\nfor the dashboard builder. Add http:\/\/PROMETHEUS_SERVER_IP:9090<\/a><\/em> to
\nthe list of servers and hit Create Server<\/em>.<\/p>\n
\nDirectories<\/em> (Groups of Dashboards) and Dashboards<\/em>. For example we
\ncreated a directory for Web Nodes and one for Database Nodes and in each
\nwe create a dashboard as shown below.<\/p>\n![\"\"](\"http:\/\/i.imgur.com\/ntOQORp.png\")
<\/p>\n
\nthe title bar of a graph and selecting the data sources icon (Three
\nHorizontal lines with an addition sign following them ). You can then
\nselect the server which you added earlier, and a query expression which
\nyou tested in the Prometheus Server UI. You can add multiple data
\nsources into the same graph in order to see a comparative view.<\/p>\n![\"\"](\"http:\/\/i.imgur.com\/BQM3rkG.png\")
<\/p>\n
\nby clicking the Add Graph button. In addition you may select the
\ntime range over which your dashboard displays data as well as a refresh
\ninterval for auto-loading data. The dashboard is not as polished as the
\nones from Scout and DataDog, for example there is no easy way to explore
\nmetrics or build a query in the dashboard view. Since the dashboard runs
\nindependently of the Prometheus server we can\u2019t \u2018pin\u2019 graphs
\ngenerated in the Prometheus server into a dashboard. Furthermore several
\ntimes we noticed that the UI would not update based on selected data
\nuntil we refreshed the page. However, despite its issues the dashboard
\nis feature competitive with DataDog and because Prometheus is under
\nheavy development, we expect the bugs to be resolved over time. In
\ncomparison to other self-hosted solutions Prometheus is a lot more user
\nfriendly than Sensu and allows you present metric data as graphs without
\nusing third party visualizations. It also is able to provide much better
\nanalytical capabilities than CAdvisor.<\/p>\n
\ndata and displaying those on the UI. However, to be able to do something
\nuseful with alerts such send emails or notify
\npagerduty<\/a> we need to run the the Alert
\nManager<\/a>. To run
\nthe Alert Manager you first need to create a configuration file. Create
\na file called alertmanager.conf<\/em> and add the following text into it:<\/p>\n
\nname: “ubuntu_notification”
\npagerduty_config {
\nservice_key: “<PAGER_DUTY_API_KEY>”
\n}
\nemail_config {
\nemail: “<TARGET_EMAIL_ADDRESS>”
\n}
\nhipchat_config {
\nauth_token: “<HIPCHAT_AUTH_TOKEN>”
\nroom_id: 123456
\n}
\n}
\naggregation_rule {
\nfilter {
\nname_re: “image”
\nvalue_re: “ubuntu:14.04”
\n}
\nrepeat_rate_seconds: 300
\nnotification_config_name: “ubuntu_notification”
\n}<\/p>\n
\ncalled ubuntu_notification<\/em>, which specifies that alerts must go to
\nthe PagerDuty, Email and HipChat. We need to specify the relevant API
\nkeys and\/or access tokens for the HipChat and PagerDutyNotifications to
\nwork. We are also specifying that the alert configuration should only
\napply to alerts on metrics where the label image has the value
\nubuntu:14.04. We specify that a triggered alert should not retrigger
\nfor at least 300 seconds after the first alert is raised. We can bring
\nup the Alert Manager using the docker image by volume mounting our
\nconfiguration file into the container using the command shown below.<\/p>\n
\nto port 9093 and load up the Alarm Manger UI. You will be able to see
\nall the alerts raised here, you can \u2018silence\u2019 them or delete them once
\nthe issue is resolved. In addition to setting up the Alert Manager we
\nalso need to create a few alerts. Add rule_file:
\n\u201c\/prometheus.rules\u201d in a new line into the global section of the
\nprometheus.conf<\/em> file you created earlier. This line tells Prometheus
\nto look for alerting rules in the prometheus.rules<\/em> file. We now need
\nto create the rules file and load it into our server container. To do so
\ncreate a file called prometheus.rules<\/em> in the same directory where you
\ncreated prometheus.conf<\/em>. and add the following text to it:<\/p>\n
\nIF container_memory_usage_bytes > 1000000000
\nFOR 1m
\nWITH {}
\nSUMMARY “High Memory usage for Ubuntu container”
\nDESCRIPTION “High Memory usage for Ubuntu container on {{$labels.instance}} for container {{$labels.name}} (current value: {{$value}})”<\/p>\n
\nHighMemoryAlert if the container_memory_usage_bytes metric
\nfor containers using the Ubuntu:14.04 image goes above 1 GB for 1
\nminute. The summary and the description of the alerts is also specified
\nin the rules file. Both of these fields can contain placeholders for
\nlabel values which are replaced by Prometheus. For example our
\ndescription will specify the server instance (IP) and the container name
\nfor metric raising the alert. After launching the Alert Manager and
\ndefining your Alert rules, you will need to re-run your Prometheus
\nserver with new parameters. The commands to do so are below:<\/p>\n
\ndocker stop prometheus-server && docker rm prometheus-server<\/p>\n
\ndocker run -d –name prometheus-server -p 9090:9090
\n-v $PWD\/prometheus.conf:\/prometheus.conf
\n-v $PWD\/prometheus.rules:\/prometheus.rules
\nprom\/prometheus
\n-config.file=\/prometheus.conf
\n-alertmanager.url=http:\/\/ALERT_MANAGER_IP:9093<\/p>\n
\nmenu of the Prometheus Server UI to bring up a list of alerts and their
\nstatuses. If and when an alert is fired you will also be able to see it
\nin the Alert Manager UI and any external service defined in the
\nalertmanager.conf<\/em> file.<\/p>\n![\"\"](\"http:\/\/i.imgur.com\/uC483G5.png\")
<\/p>\n
\nDataDog which has been our best rated Monitoring tool so far. Prometheus
\nuses a very simple format for input data and can ingest from any web
\nendpoint which presents the data. Therefore we can monitor more or less
\nany resource with Prometheus, and there are already several libraries
\ndefined to monitor common resources. Where Prometheus is lacking is in
\nlevel of polish and ease of deployment. The fact that all components are
\ndockerized is a major plus however, we had to launch 4 different
\ncontainers each with their own configuration files to support the
\nPrometheus server. The project is also lacking detailed, comprehensive
\ndocumentation for these various components. However, in caparison to
\nself-hosted services such as CAdvisor and Sensu, Prometheus is a much
\nbetter toolset. It is significantly easier setup than sensu and has the
\nability to provide visualization of metrics without third party tools.
\nIt is able has much more detailed metrics than CAdvisor and is also able
\nto monitor non-docker resources. The choice of using pull based metric
\naggregation rather than push is less than ideal as you would have to
\nrestart your server when adding new data sources. This could get
\ncumbersome in a dynamic environment such as cloud based deployments.
\nPrometheus does offer the Push
\nGateway<\/a> to bridge the
\ndisconnect. However, running yet another service will add to the
\ncomplexity of the setup. For these reasons I still think DataDog is
\nprobably easier for most users, however, with some polish and better
\npackaging Prometheus could be a very compelling alternative, and out of
\nself-hosted solutions Prometheus is my pick.<\/p>\n\n
Sysdig Cloud<\/h2>\n
\naggregation, visualization and alerting. To get started with sysdig sign
\nup for a trial account at https:\/\/app.sysdigcloud.com<\/a>. and complete
\nthe registration form. Once you complete the registration form and log
\nin to the account, you will be asked to Setup your Environment<\/em> and be
\ngiven a curl command similar to the shown below. Your command will have
\nyour own secret key after the -s switch. You can run this command on the
\nhost running docker and which you need to monitor. Note that you should
\nreplace the [TAGS] place holder with tags to group your metrics. The
\ntags are in the format TAG_NAME:VALUE so you may want to add a tag
\nrole:web or deployment:production. You may also use the containerized
\nsysdig agent.<\/p>\n
\ncurl -s https:\/\/s3.amazonaws.com\/download.draios.com\/stable\/install-agent | sudo bash -s 12345678-1234-1234-1234-123456789abc [TAGS]<\/p>\n
\ndocker run –name sysdig-agent –privileged –net host
\n-e ACCESS_KEY=12345678-1234-1234-1234-123456789abc
\n-e TAGS=os:rancher
\n-v \/var\/run\/docker.sock:\/host\/var\/run\/docker.sock
\n-v \/dev:\/host\/dev -v \/proc:\/host\/proc:ro
\n-v \/boot:\/host\/boot:ro
\n-v \/lib\/modules:\/host\/lib\/modules:ro
\n-v \/usr:\/host\/usr:ro sysdig\/agent<\/p>\n
\nthe host OS. This goes against Docker\u2019s philosophy of isolated micro
\nservices. However, installing kernel headers is fairly benign.
\nInstalling the headers and getting sysdig running is trivial if you are
\nusing a mainstream kernel such us CentOS, Ubuntu or Debian. Even the
\nAmazon\u2019s custom kernels are supported however RancherOS\u2019s custom
\nkernel presented problems for sysdig as did the tinycore kernel. So be
\nwarned if you would like to use Sysdig cloud on non-mainstream kernels
\nyou may have to get your hands dirty with some system hacking.<\/p>\n
\nconsole in the Explore tab. Once you launch docker containers on the
\nhost those will also be shown. You can see basic stats about the CPU
\nusage, memory consumption, network usage. The metrics are aggregated for
\nthe host as well as broken down per container.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/04\/20181622\/Screen-Shot-2015-04-14-at-12.06.36-PM.png\")
<\/a>By
\nselecting one of the hosts or containers you can get a whole host of
\nother metrics including everything provided by the docker stats API. Out
\nof all the systems we have seen so far sysdig certainly has the most
\ncomprehensive set of metrics out of the box. You can also select from
\nseveral pre-configured dashboards which present a graphical or tabular
\nrepresentation of your deployment.<\/p>\n![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/04\/20181622\/Screen-Shot-2015-04-16-at-11.26.53-AM.png\")
<\/a><\/p>\n
\nor select a window of time over which to average values. Furthermore,
\nyou can also setup comparisons which will highlight the delta of current
\nvalues and values at a point in the past. For example the table below
\nshows values compared with those from ten minutes ago. If the CPU usage
\nis significantly higher than 10 minutes ago you may be experiencing load
\nspikes and need to scale out. The UI is at par with, if not better than
\nDataDog for identifying and exploring trends in the data.![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/04\/20181622\/Screen-Shot-2015-04-19-at-4.59.09-PM.png\")
<\/a><\/p>\n
\npersistent dashboards. Simply click the pin icon on any graph in the
\nexplore view and save it to a named dashboard. You can view all the
\ndashboards and their associated graphs by clicking the Dashboards
\ntab. You can also select the bell icon on any graph and create an
\nalert from the data. The Sysdig cloud supports detailed alerting
\ncriteria and is again one of the best we have seen. The example below
\nshows an alert which triggers if the count of containers labeled web<\/em>
\nfalls below three on average for the last ten minutes. We are also
\nsegmenting the data by the region<\/em> tag, so there will be a separate
\ncheck for web nodes in North America and Europe. Lastly, we also specify
\na Name, description and Severity for the alerts. You can control where
\nalerts go by going to Settings (Gear Icon) > Notifications and add
\nemail addresses or SNS Topics to send alerts too. Note all alerts go to
\nall notification endpoints which may be problematic if you want to wake
\nup different people for different alerts.![\"Screen](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/04\/20181622\/Screen-Shot-2015-04-19-at-4.55.35-PM.png\")
<\/a><\/p>\n
\nprovides detailed metrics with great visualization tools for real-time
\nand historical data. The requirement to install kernel headers on the
\nhost OS is troublesome though and lack of documentation and support for
\nnon-standard kernels could be problematic in some scenarios. The
\nalerting system in the Sysdig cloud is among the best we have seen so
\nfar, however, the inability to target different email addresses for
\ndifferent alerts is problematic. In a larger team for example you would
\nwant to alert a different team for database issues vs web server issues.
\nLastly, since it is in beta the pricing for Sysdig cloud is not easily
\navailable. I have reached out to their sales team and will update this
\narticle if and when they get back to me. If sysdig is price competitive
\nthen Datadog has serious competition in the hosted service category.<\/p>\n\n
Conclusion<\/h2>\n
\ncontainers, ranging from free options; docker stats, CAdvisor,
\nPrometheus or Sensu to paid services such as Scout, Sysdig Cloud and
\nDataDog. From my research so far DataDog seems to be the best-in-class
\nsystem for monitoring docker deployments. The setup was complete in
\nseconds with a one-line command, all hosts were reporting metrics in one
\nplace, historical trends were apparent in the UI and Datadog supports
\ndeep diving into metrics as well as alerting. However, at $15 per host
\nthe system can get expensive for large deployments. For larger scale,
\nself-hosted deployments Sensu is able to fulfill most requirements
\nhowever the complexity in setting up and managing a Sensu cluster may be
\nprohibitive. Obviously, there are plenty of other self-hosted options,
\nsuch as Nagios or Icinga, which are similar to Sensu.<\/p>\n
\ncontainers available today. I am continuing to investigate other
\noptions, including a more streamlined self-managed container monitoring
\nsystem using CollectD, Graphite or InfluxDB and Grafana. Stay tuned for
\nmore details.<\/p>\n
\nsuggestions to also evaluate Prometheus and Sysdig Cloud, two other very
\ngood options for monitoring Docker. We\u2019ve now included them in this
\narticle, for ease of discovery. You can find the original second part
\nof my
\nposthere<\/a>.<\/p>\n
\nour next Rancher online meetup.<\/p>\n
\nbuilding large scale distributed services on top of various cloud
\nplatforms. You can read more of his work at
\ntechtraits.com<\/a>, or follow him on twitter
\n@usman_ismail<\/a>or
\nonGitHub<\/a>.<\/em><\/p>\n