Container monitoring environments come in all shapes and sizes. Some are open source while others are commercial. Some are in the Rancher Catalog while others require manual configuration. Some are general purpose while others are aimed specifically at container environments. Some are hosted in the cloud while others require installation on own cluster hosts. In this post, I take an updated look at 10 container monitoring solutions. This effort builds on earlier work including Ismail Usman\u2019s The monitoring solutions covered here include:<\/p>\n In the following sections, I suggest a framework for comparing monitoring solutions, present a high-level comparison of each, and then discuss each solution in more detail by addressing how each solution works with Rancher. I also cover a few additional solutions you may have come across that did not make my top 10.<\/p>\n A challenge with objectively comparing monitoring solutions is that architectures, capabilities, deployment models, and costs can vary widely. One solution may extract and graph Docker-related metrics from a single host while another aggregates data from many hosts, measures application response times, and sends automated alerts under particular conditions. Having a framework is useful when comparing solutions. I\u2019ve somewhat arbitrarily proposed the following tiers of functionality that most monitoring solutions have in common as a basis for my comparison. Like any self-respecting architectural stack, this one has seven layers.<\/p>\n Figure 1: A seven-layer model for comparing monitoring solutions<\/p>\n Beyond understanding how each monitoring solution implements the basic capabilities above, users will be interested in other aspects of the monitoring solution as well:<\/p>\n The diagram below shows a high-level view of how our 10 monitoring solutions map to our seven-layer model, which components implement the capabilities at each layer, and where the components reside. Each framework is complicated, and this is a simplification to be sure, but it provides a useful view of which component does what. Read on for Figure 2 – 10 monitoring solutions at a glance Additional attributes of each monitoring solution are presented in a summary fashion below. For some solutions, there are multiple deployment options, so the comparisons become a little more nuanced.<\/p>\n Heapster is another solution that often comes up related to monitoring-container A popular stack is comprised of Heapster, InfluxDB, and Grafana, and An unsung member of the Elastic stack is Gord Sissons, Principal Consultant at StoryTek<\/p>\n
\nComparing 7 Monitoring Options for Docker<\/a> from 2015 and The Great Container Monitoring Bake Off Meetup<\/a> in October of 2016. The number of monitoring solutions is daunting. New solutions are coming on the scene continuously, and existing solutions evolve in functionality. Rather than looking at each solution in depth, I\u2019ve taken the approach of drawing high-level comparisons. With this approach, readers can hopefully \u201cnarrow the list\u201d and do more serious
\nevaluations of solutions best suited to their own needs.<\/p>\n\n
\nDocker<\/a><\/li>\nA Framework for Comparison<\/h2>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17114746\/seven_layers_of_docker_monitoring.png\")
<\/p>\n\n
\n
Comparing Our 10 Monitoring Solutions<\/h2>\n
\nadditional detail.<\/p>\n![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17115425\/gord_comparing_ten_monitoring_solutions-1024x585.png\")
<\/p>\n![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17115647\/gord_monitoring_functionality_comparison-1024x640.png\")
<\/p>\nLooking at Each Solution in More Depth<\/h2>\n
Docker Stats<\/h3>\n
![\"docker](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17122404\/gord_docker_logo-150x150.png\"){kind=logo}
\nAt the most basic level, Docker<\/a> provides built-in command monitoring for Docker
\nhosts via the docker stats<\/a> command. Administrators can query the Docker daemon and obtain detailed, real-time information about container resource consumption metrics, including CPU and memory usage, disk and network I\/O, and the number of running processes. Docker stats leverages the Docker Engine API<\/a> to retrieve this information. Docker stats has no notion of history, and it can only monitor a single host, but clever administrators can write scripts to gather metrics from multiple hosts. Docker stats is of limited use on its own, but docker stats data can be combined with other data sources like Docker log files<\/a> and docker events<\/a> to feed higher level monitoring services. Docker only knows about metrics reported by a single host, so Docker stats is of limited use monitoring Kubernetes or Swarm clusters with multi-host application services. With no visualization interface, no aggregation, no datastore, and no ability to collect data from multiple hosts, Docker stats does not fare well against our seven-layer model. Because Rancher<\/a> runs on Docker, basic docker stats functionality is automatically available to Rancher users.<\/p>\ncAdvisor<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17122432\/gord_cadvisor_logo-150x150.png\"){kind=logo}
cAdvisor<\/a> (container advisor) is an open-source project<\/a> that like Docker stats provides users with resource usage information about running containers. cAdvisor was originally developed by Google to manage its lmctfy<\/a> containers, but it now supports Docker as well. It is implemented as a daemon process that collects, aggregates, processes, and exports information about running containers. cAdvisor exposes a web interface and can generate multiple graphs but, like Docker stats, it monitors only a single Docker host. It can be installed on a Docker machine either as a container or natively
\non the Docker host itself. cAdvisor itself only retains information for 60 seconds. cAdvisor needs to be configured to log data to an external datastore. Datastores commonly used with cAdvisor data include Prometheus<\/a> and
\nInfluxDB<\/a>. While cAdvisor itself is not a complete monitoring solution, it is often a component of other monitoring solutions. Before Rancher version 1.2 (late December), Rancher embedded cAdvisor in the rancher-agent (for internal use by Rancher), but this is no longer the case. More recent versions of Rancher use Docker stats to gather information exposed through the Rancher UI because they can do so with less overhead. Administrators can
\neasily deploy cAdvisor on Rancher, and it is part of several comprehensive monitoring stacks, but cAdvisor is no longer part of Rancher itself.<\/p>\nScout<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17122457\/gord_scout_logo-150x150.png\"){kind=logo}
Scout<\/a> is a Colorado-based company that provides a cloud-based application and database-monitoring service aimed mainly at Ruby and Elixir environments. One of many use cases it supports is monitoring Docker containers leveraging its existing monitoring and alerting framework. We mention Scout because it was covered in previous comparisons as a solution for monitoring Docker. Scout provides comprehensive data gathering, filtering, and monitoring functionality with flexible alerts and integrations to third-party alerting services. The team at Scout provides guidance on how to write scripts using Ruby and StatsD<\/a> to tap the Docker Stats
\nAPI<\/a> (above), the Docker Event API, and relay metrics to Scout for monitoring. They\u2019ve also packaged a docker-scout container, available
\non Docker Hub (scoutapp\/docker-scout<\/a>), that makes installing and configuring the scout agent simple. The ease of use will depend on whether users configure the StatsD agent themselves or leverage the packaged docker-scout container. As a hosted cloud service, ScoutApp can save a lot of headaches when it comes to getting a container-monitoring solution up and running quickly. If you\u2019re deploying Ruby apps or running the database environments supported by Scout, it probably makes good sense to consolidate your Docker, application, and database-level monitoring and use the Scout solution. Users might want to watch out for a few things, however. At most service levels, the platform only allows for 30 days of data retention, and rather than being priced month per monitored host,
\nstandard packages are priced per transaction ranging from $99 to $299
\nper month. The solution out of the box is not Kubernetes-aware, and
\nextracts and relays a limited set of metrics. Also, while docker-scout
\nis available on Docker Hub, development is by Pingdom, and there have
\nbeen only minor updates in the last two years to the agent component.
\nScout is not natively supported in Rancher but, because it is a cloud
\nservice, it is easy to deploy and use, particularly when the
\ncontainer-based agent is used. At present, the docker-scout agent is
\nnot in the Rancher Catalog.<\/p>\nPingdom<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17122523\/gord_pingdom_logo-150x150.png\"){kind=logo}
Because we\u2019ve mentioned Scout as a cloud-hosted app, we also need to mention a similar solution called Pingdom<\/a>. Pingdom
\nis a hosted-cloud service operated by
\nSolarWinds<\/a>, an Austin, TX,
\ncompany focused on monitoring IT infrastructure. While the main use case
\nfor Pingdom is website monitoring, as a part of its server monitor
\nplatform, Pingdom offers approximately 90 plug-ins. In fact, Pingdom
\nmaintains
\ndocker-scout<\/a>,
\nthe same StatsD agent used by Scout. Pingdom is worth a look because its
\npricing scheme appears better suited to monitoring Docker
\nenvironments. Pricing is flexible, and users can choose between
\nper-server based plans and plans based on the number of StatsD metrics
\ncollected ($1 per 10 metrics per month). Pingdom makes sense for users
\nwho need a full-stack monitoring solution that is easy to set up and
\nmanage, and who want to monitor additional services beyond the container
\nmanagement platform. Like Scout, Pingdom is a cloud service that can be
\neasily used with Rancher.<\/p>\nDatadog<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17122544\/gord_datadog_logo-150x150.png\"){kind=logo}
Datadog is another commercial hosted-cloud monitoring service similar to Scout and Pingdom. Datadog also provides a Dockerized agent for installation on each Docker host; however, rather than using StatsD like the
\ncloud-monitoring solutions mentioned previously, Datadog has developed
\nan enhanced StatsD called
\nDogStatsD<\/a>. The Datadog
\nagent collects and relays the full set of metrics available from the
\nDocker API providing more detailed, granular monitoring. While Datadog
\ndoes not have native support for Rancher, a Datadog catalog entry in the
\nRancher UI makes the Datadog agent easy to install and configure on
\nRancher. Rancher tags can be used as well so that reporting in Datadog
\nreflects labels you\u2019ve used for hosts and applications in Rancher.
\nDatadog provides better access to metrics and more granularity in
\ndefining alert conditions than the cloud services mentioned earlier.
\nLike the other services, Datadog can be used to monitor other services
\nand applications as well, and it boasts a library of over 200
\nintegrations. Datadog also retains data at full resolution for 18
\nmonths, which is longer than the cloud services above. An advantage of
\nDatadog over some of other cloud services is that it has integrations
\nbeyond Docker and can collect metrics from Kubernetes, Mesos, etcd, and
\nother services that you may be running in your Rancher environment. This
\nversatility is important to users running Kubernetes on Rancher because
\nthey want to be able to monitor metrics for things like Kubernetes pods,
\nservices, namespaces, and kubelet health. The Datadog-Kubernetes
\nmonitoring solution uses DaemonSets in Kubernetes to automatically
\ndeploy the data collection agent to each cluster node. Pricing for
\nDatadog starts at approximately $15 per host per month and goes up from
\nthere depending services required and the number of monitored containers
\nper host.<\/p>\nSysdig<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/18103837\/gord_sysdig_logo_square-150x150.png\"){kind=logo}
Sysdig is a California company that provides a cloud-based monitoring solution. Unlike some of the cloud-based monitoring solutions described so far, Sysdig focuses more narrowly on monitoring container environments including Docker, Swarm, Mesos, and Kubernetes. Sysdig also makes some of its functionality available in open-source projects, and they provide
\nthe option of either cloud or on-premises deployments of the Sysdig
\nmonitoring service. In these respects, Sysdig is different than the
\ncloud-based solutions looked at so far. Like Datadog, catalog entries
\nare available for Rancher, but for Sysdig there are separate entries for
\non-premises and cloud installations. Automated installation from the
\nRancher Catalog is not available for Kubernetes; however, it can be
\ninstalled on Rancher outside of the catalog. The commercial Sysdig
\nMonitor has Docker monitoring, alerting, and troubleshooting facilities
\nand is also Kubernetes, Mesos, and Swarm-aware. Sysdig is automatically
\naware of Kubernetes pods and services, making it a good solution if
\nyou\u2019ve chosen Kubernetes as your orchestration framework on Rancher.
\nSysdig is priced monthly per host like Datadog. While the entry price is
\nslightly higher, Sysdig includes support for more containers per host,
\nso actual pricing will likely be very similar depending on the user\u2019s
\nenvironment. Sysdig also provides a comprehensive CLI, csysdig,
\ndifferentiating it from some of the offerings.<\/p>\nPrometheus<\/h3>\n
![\"\"](\"https:\/\/prometheus.io\/)\n![](http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/17122800\/gord_prometheus_logo-150x150.png\"){kind=logo}
\nPrometheus is a popular, open-source monitoring and alerting toolkit originally built at SoundCloud. It is now a CNCF project, the company\u2019s second hosted project after Kubernetes. As a toolkit, it is substantially different
\nfrom monitoring solutions described thus far. A first major difference
\nis that rather being offered as a cloud service, Prometheus is modular
\nand self-hosted, meaning that users deploy Prometheus on their clusters
\nwhether on-premises or cloud-resident. Rather than pushing data to a
\ncloud service, Prometheus installs on each Docker host and pulls or
\n\u201cscrapes\u201d data from an extensive variety of
\nexporters<\/a>
\navailable to Prometheus via HTTP. Some exporters are officially
\nmaintained as a part of the Prometheus GitHub project, while others are
\nexternal contributions. Some projects expose Prometheus metrics natively
\nso that exporters are not needed. Prometheus is highly extensible. Users
\nneed to mind the number of exporters and configure polling intervals
\nappropriately depending on the amount of data they are collecting. The
\nPrometheus server retrieves time-series data from various sources and
\nstores data in its internal datastore. Prometheus provides features like
\nservice discovery, a separate push gateway for specific types of metrics
\nand has an embedded query language (PromQL) that excels at querying
\nmultidimensional data. It also has an embedded web UI and API. The web
\nUI in Prometheus provides good functionality but relies on users knowing
\nPromQL, so some sites prefer to use Grafana as an interface for charting
\nand viewing cluster-related metrics. Prometheus has a discrete Alert
\nManager with a distinct UI that can work with data stored in Prometheus.
\nLike other alert managers, it works with a variety of external alerting
\nservices including email, Hipchat, Pagerduty, #Slack, OpsGenie,
\nVictorOps, and others. Because Prometheus is comprised of many
\ncomponents, and exporters need to be selected and installed depending on
\nthe services monitored, it is more difficult to install; but as a free
\noffering, the price is right. While not quite as refined as tools like
\nDatadog or Sysdig, Prometheus offers similar functionality, extensive
\nthird-party software integrations, and best-in-class cloud monitoring
\nsolutions. Prometheus is aware of Kubernetes and other container
\nmanagement frameworks. An entry in the Rancher Catalog developed by
\nInfinityworks<\/a> makes getting started
\nwith Prometheus easier when Cattle is used as the Rancher orchestrator
\nbut, because of the wide variety of configuration options,
\nadministrators need to spend some time to get it properly installed and
\nconfigured. Infinityworks have contributed useful add-ons including the
\nprometheus-rancher-exporter<\/a> that
\nexposes the health of Rancher stacks and hosts obtained from the Rancher
\nAPI to a Prometheus compatible endpoint. For administrators who don\u2019t
\nmind going to a little more effort, Prometheus is one of the most
\ncapable monitoring solutions and should be on your shortlist for
\nconsideration.<\/p>\nHeapster<\/h3>\n
\nenvironments. Heapster is a project under the Kubernetes umbrella that
\nhelps enable container-cluster monitoring and performance analysis.
\nHeapster specifically supports Kubernetes and OpenShift and is most
\nrelevant for Rancher users running Kuberenetes as their orchestrator. It
\nis not typically be used with Cattle or Swarm. People often describe
\nHeapster as a monitoring solution, but it is more precisely a
\n\u201ccluster-wide aggregator of monitoring and event data.\u201d Heapster is
\nnever deployed alone; rather, it is a part of a stack of open-source
\ncomponents. The Heapster monitoring stack is typically comprised of:<\/p>\n\n
\nkubelet on each cluster host<\/li>\n
\nKafka, Graphite, or roughly a dozen
\nothers<\/a><\/li>\n
\nMonitoring<\/li>\n<\/ul>\n
\nthis combination is installed by default on Rancher when users choose to
\ndeploy Kubernetes. Note that these components are considered add-ons to
\nKubernetes, so they may not be automatically deployed with all
\nKubernetes distributions. One of the reasons that InfluxDB is popular is
\nthat it is one of the few data backends that supports both Kubernetes
\nevents and metrics, allowing for more comprehensive monitoring of
\nKubernetes. Note that Heapster does not natively support alerting or
\nservices related to Application Performance Management (APM) found in
\ncommercial cloud-based solutions or Prometheus. Users that need
\nmonitoring services can supplement their Heapster installation using
\nHawkular<\/a>, but this is not automatically
\nconfigured as part of the Rancher deployment and will require extra user
\neffort.<\/p>\nELK Stack<\/h3>\n
![\"\"](\"http:\/\/rancher.com\/wp-content\/uploads\/2017\/10\/Elastic-Stack-Master4-150x150.jpg\")
Another
\nopen-source software stack available for monitoring container environments is ELK, comprised of three open-source projects contributed by Elastic<\/a>. The ELK stack is versatile and
\nis widely used for a variety of analytic applications, log file
\nmonitoring being a key one. ELK is named for its key components:<\/p>\n\n
\ndistributed search engine based on Lucene<\/li>\n
\ndata-processing pipeline that ingests data and sends it to
\nElastisearch (or other \u201cstashes\u201d)<\/li>\n
\ndashboard and analysis tool for Elasticsearch<\/li>\n<\/ul>\n
\nBeats<\/a>, described by the project
\ndevelopers as \u201clightweight data shippers.\u201d There are a variety of
\noff-the-shelf Beats shippers including Filebeat (used for log files),
\nMetricbeat (using for gathering data metrics from various sources), and
\nHeartbeat for simple uptime monitoring among others. Metricbeat is
\nDocker-aware, and the authors provide
\nguidance<\/a>
\non how to use it to extract host metrics and monitor services in Docker
\ncontainers. There are variations in how the ELK stack is
\ndeployed. Lorenzo Fontana of Kiratech explains in this
\narticle<\/a>
\nhow to use cAdvisor to collect metrics from Docker Swarm hosts for
\nstorage in ElasticSearch and analysis using Kibana. In another
\narticle<\/a>,
\nAboullaite Mohammed describes a different use case focused on collecting
\nDocker log files for analysis focusing on analyzing various Linux and
\nNGINX log files (error.log, access.log, and syslog). There are
\ncommercial ELK stack providers such as logz.io<\/a> and
\nElastic Co<\/a> themselves that offer \u201cELK as a
\nservice\u201d supplementing the stack\u2019s capabilities with alerting
\nfunctionality. Additional information about using ELK with Docker is
\navailable at https:\/\/elk-docker.readthedocs.io\/<\/a>. For Rancher users
\nthat wish to experiment with ELK, the stack is available as a Rancher
\nCatalog entry, and a tutorial by Rachid
\nZaroualli<\/a>
\nexplains how to deploy it. Zaroualli has contributed an additional
\narticle<\/a>
\non how the ELK stack can be used for monitoring Twitter data. While
\nknowledgeable administrators can use ELK for container monitoring, this
\nis a tougher solution to implement compared to Sysdig, Prometheus, or
\nDatadog, all of which are more directly aimed at container monitoring.<\/p>\nSensu<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/18103933\/gord_sensu_logo_square-150x150.png\"){kind=logo}
Sensu is a general-purpose, self-hosted monitoring solution that supports a variety of monitoring applications. A free Sensu Core edition is available under an MIT license, while an enterprise version with added functionality is available for $99 per month for 50 Sensu clients. Sensu uses the term client to refer to its monitoring agents, so depending on the number of hosts and application environments you are monitoring, the enterprise
\nedition can get expensive. Sensu has impressive capabilities outside of
\ncontainer management, but consistent with the other platforms I\u2019ve
\nlooked at it from the perspective of monitoring the container
\nenvironment and containerized applications. The number of Sensu
\nplug-ins<\/a> continues
\nto grow, and there are dozens of Sensu and community supported plug-ins
\nthat allow metrics to be extracted from various sources. In an earlier
\nevaluation of Sensu on Rancher in 2015, it was necessary for the author
\nto develop shell scripts to extract information from Docker, but an
\nactively developed Docker
\nplug-in<\/a> is now
\navailable for this purpose making Sensu easier to use with Rancher.
\nPlug-ins tend to be written in Ruby with gem-based installation scripts
\nthat need to run on the Docker host. Users can develop additional
\nplug-ins in the languages they choose. Sensu plug-ins are not deployed
\nin their own containers, as common with other monitoring solutions we\u2019ve
\nconsidered. (This is no doubt because Sensu does not come from a
\nheritage of monitoring containers.) Different users will want to mix and
\nmatch plug-ins depending on their monitoring requirements, so having
\nseparate containers for each plug-in would become unwieldy, and this is
\npossibly why containers are not used for deployment. Plug-ins are
\ndeployable using platforms like Chef, Puppet, and Ansible, however. For
\nDocker alone, for example, there are six separate
\nplug-ins<\/a>
\nthat gather Docker-related data from various sources, including Docker
\nstats, container counts, container health, Docker ps, and more. The
\nnumber of plug-ins is impressive and includes many of the application
\nstacks that users will likely be running in container environments
\n(ElasticSearch, Solr, Redis, MongoDB, RabbitMQ, Graphite, and Logstash,
\nto name a few). Plug-ins for management and orchestration frameworks
\nlike AWS services (EC2, RDS, ELB) are also provided with Sensi.
\nOpenStack and Mesos support is available in Sensu as well. Kubernetes
\nappears to be missing from the list of plug-ins a present. Sensu uses a
\nmessage bus implemented using RabbitMQ to facilitate communication
\nbetween the agents\/clients and the Sensu server. Sensu uses Redis to
\nstore data, but it is designed to route data to external time-series
\ndatabases. Among the databases supported are Graphite, Librato, and
\nInfluxDB. Installing and configuring
\nSensu<\/a>
\ntakes some effort. Pre-requisites to installing Sensu are Redis and
\nRabbitMQ. The Sensu server, Sensu clients, and the Sensu dashboard
\nrequire separate installation, and the process varies depending on
\nwhether you are deploying Sensu core or the enterprise version. Sensu as
\nmentioned, do not offer a container friendly deployment model. For
\nconvenience, a Docker image is available
\n(hiroakis\/docker-sensu-server<\/a>)
\nthat runs redis, rabbitmq-server, uchiwa (the open-source web tier) and
\nthe Sensu server components, but this package is more useful for
\nevaluation than a production deployment. Sensu has a large number of
\nfeatures, but a drawback for container users is that the framework is
\nharder to install, configure, and maintain because the components are
\nnot themselves Dockerized. Also, many of the alerting features like
\nsending alerts to services like PagerDuty, Slack, or HipChat, for
\nexample, that are available in competing cloud-based solutions or
\nopen-source solutions like Prometheus require that purchase of the Sensu
\nenterprise license. Particularly if you are running Kubernetes, there
\nare probably better choices out there.<\/p>\nThe Monitoring Solutions We Missed<\/h3>\n
\n
\nthat comes up when monitoring Docker. Like ELK, Graylog is suited to
\nDocker log file analysis. It can accept and parse logs and event
\ndata from multiple data sources and supports third-party collectors
\nlike Beats, Fluentd, and NXLog. There\u2019s a good
\ntutorial<\/a>
\non configuring Graylog for use with Rancher.<\/li>\n
\nfor monitoring cluster hosts rather than containers but, for those
\nof us who grew up monitoring clusters, Nagios is a crowd favorite.
\nFor those interested in using Nagios with
\nRancher<\/a>, some work has
\nbeen done here.<\/li>\n
\nmonitoring application with plugins for Docker, Kubernetes, Mesos,
\nand a variety of applications and cloud providers. Netsil\u2019s
\nApplication Operations Center (AOC) provides framework-aware
\nmonitoring for cloud application services. Like some of the other
\nmonitoring frameworks discussed, it is offered as a cloud\/SaaS or
\nself-hosted.<\/li>\n<\/ul>\n