{"id":1345,"date":"2019-02-18T12:38:08","date_gmt":"2019-02-18T12:38:08","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=1345"},"modified":"2019-03-07T20:05:39","modified_gmt":"2019-03-07T20:05:39","slug":"docker-monitoring-continued-prometheus-and-sysdig","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2019\/02\/18\/docker-monitoring-continued-prometheus-and-sysdig\/","title":{"rendered":"Docker Monitoring Continued: Prometheus and Sysdig"},"content":{"rendered":"

I recently compared\u00a0<\/a>several docker monitoring tools and services. Since the article went live we have gotten feedback about additional tools that should be included in our survey. I would like to highlight two such tools;
\nPrometheus and Sysdig cloud. Prometheus is a capable self-hosted
\nsolution which is easier to manage than sensu. Sysdig cloud on the other
\nhand provides us with another hosted service much like Scout and
\nDatadog. Collectively they help us add more choices to their respective
\nclasses. As before I will be using the following six criteria to
\nevaluate Prometheus and Sysdig cloud: 1) ease of deployment, 2) level of
\ndetail of information presented, 3) level of aggregation of information
\nfrom entire deployment, 4) ability to raise alerts from the data and 5)
\nAbility to monitor non-docker resources 6) cost.<\/p>\n

Prometheus<\/h2>\n

First lets take a look at Prometheus; it is a self-hosted set of tools
\nwhich collectively provide metrics storage, aggregation, visualization
\nand alerting. Most of the tools and services we have looked at so far
\nhave been push based, i.e. agents on the monitored servers talk to a
\ncentral server (or set of servers) and send out their metrics.
\nPrometheus on the other hand is a pull based server which expects
\nmonitored servers to provide a web interface from which it can scrape
\ndata. There are several exporters
\navailable<\/a> for
\nPrometheus which will capture metrics and then expose them over http for
\nPrometheus to scrape. In addition there are
\nlibraries<\/a> which
\ncan be used to create custom exporters. As we are concerned with
\nmonitoring docker containers we will use the
\ncontainer_exporter<\/a>
\ncapture metrics. Use the command shown below to bring up the
\ncontainer-exporter docker container and browse to
\nhttp:\/\/MONITORED_SERVER_IP:9104\/metrics<\/a><\/em> to see the metrics it has
\ncollected for you. You should launch exporters on all servers in your
\ndeployment. Keep track of the respective *MONITORED_SERVER_IP*s as we
\nwill be using them later in the configuration for Prometheus.<\/p>\n

docker run -p 9104:9104 -v \/sys\/fs\/cgroup:\/cgroup -v \/var\/run\/docker.sock:\/var\/run\/docker.sock prom\/container-exporter<\/p>\n

Once we have got all our exporters running we are can launch Prometheus
\nserver. However, before we do we need to create a configuration file for
\nPrometheus that tells the server where to scrape the metrics from.
\nCreate a file called prometheus.conf<\/em> and then add the following text
\ninside it.<\/p>\n

global:
\nscrape_interval: 15s
\nevaluation_interval: 15s
\nlabels:
\nmonitor: exporter-metrics<\/p>\n

rule_files:<\/p>\n

scrape_configs:
\n– job_name: prometheus
\nscrape_interval: 5s<\/p>\n

target_groups:
\n# These endpoints are scraped via HTTP.
\n– targets: [‘localhost:9090′,’MONITORED_SERVER_IP:9104’]<\/p>\n

In this file there are two sections, global and job(s). In the global
\nsection we set defaults for configuration properties such as data
\ncollection interval (scrape_interval). We can also add labels which
\nwill be appended to all metrics. In the jobs section we can define one
\nor more jobs that each have a name, an optional override scraping
\ninterval as well as one or more targets from which to scrape metrics. We
\nare adding two targets, one is the Prometheus server itself and the
\nsecond is the container-exporter we setup earlier. If you setup more
\nthan one exporter your can setup additional targets to pull metrics from
\nall of them. Note that the job name is available as a label on the
\nmetric hence you may want to setup separate jobs for your various types
\nof servers. Now that we have a configuration file we can start a
\nPrometheus server using the
\nprom\/prometheus<\/a>
\ndocker image.<\/p>\n

docker run -d –name prometheus-server -p 9090:9090 -v $PWD\/prometheus.conf:\/prometheus.conf prom\/prometheus -config.file=\/prometheus.conf<\/p>\n

After launching the container, Prometheus server should be available in
\nyour browser on the port 9090 in a few moments. Select Graph<\/em> from the
\ntop menu and select a metric from the drop down box to view its latest
\nvalue. You can also write queries in the expression box which can find
\nmatching metrics. Queries take the form
\nMETRIC_NAME. You can find more
\ndetails of the query syntax here<\/a>.<\/p>\n

\"\"<\/p>\n

We are able to drill down into the data using queries to filter out data
\nfrom specific server types (jobs) and containers. All metrics from
\ncontainers are labeled with the image name, container name and the host
\non which the container is running. Since metric names do not encompass
\ncontainer or server name we are able to easily aggregate data across
\nour deployment. For example we can filter for the
\ncontainer_memory_usage_bytes to get
\ninformation about the memory usage of all ubuntu containers in our
\ndeployment. Using the built in functions we can also aggregate the
\nresulting set of of metrics. For example
\naverage_over_time(container_memory_usage_bytes
\n[5m]) will show the memory used by ubuntu
\ncontainers, averaged over the last five minutes. Once you are happy with
\nwith a query you can click over to the Graph tab and see the variation
\nof the metric over time.<\/p>\n

\"\"<\/p>\n

Temporary graphs are great for ad-hoc investigations but you also need
\nto have persistent graphs for dashboards. For this you can use the
\nPrometheus Dashboard
\nBuilder<\/a>. To launch
\nPrometheus Dashboard Builder you need access to an SQL database which
\nyou can create using the official MySQL Docker
\nimage<\/a>. The command to launch
\nthe MySQL container is shown below, note that you may select any value
\nfor database name, user name, user password and root password however
\nkeep track of these values as they will be needed later.<\/p>\n

docker run -p 3306:3306 –name promdash-mysql
\n-e MYSQL_DATABASE=<database-name>
\n-e MYSQL_USER=<database-user>
\n-e MYSQL_PASSWORD=<user-password>
\n-e MYSQL_ROOT_PASSWORD=<root-password>
\n-d mysql<\/p>\n

Once you have the database setup, use the rake installation inside the
\npromdash container to initialize the database. You can then run the
\nDashboard builder by running the same container. The command to
\ninitialize the database and bring up the Prometheus Dashboard Builder
\nare shown below.<\/p>\n

# Initialize Database
\ndocker run –rm -it –link promdash-mysql:db
\n-e DATABASE_URL=mysql2:\/\/<database-user>:<user-password>@db:3306\/<database-name> prom\/promdash .\/bin\/rake db:migrate<\/p>\n

# Run Dashboard
\ndocker run -d –link promdash-mysql:db -p 3000:3000 –name prometheus-dash
\n-e DATABASE_URL=mysql2:\/\/<database-user>:<user-password>@db:3306\/<database-name> prom\/promdash<\/p>\n

\"\"<\/p>\n

Once your container is running you can browse to port 3000 and load up
\nthe dashboard builder UI. In the UI you need to click Servers<\/em> in the
\ntop menu and New Server<\/em> to add your Prometheus Server as a datasource
\nfor the dashboard builder. Add http:\/\/PROMETHEUS_SERVER_IP:9090<\/a><\/em> to
\nthe list of servers and hit Create Server<\/em>.<\/p>\n

Now click Dashboards<\/em> in the top menu, here you can create
\nDirectories<\/em> (Groups of Dashboards) and Dashboards<\/em>. For example we
\ncreated a directory for Web Nodes and one for Database Nodes and in each
\nwe create a dashboard as shown below.<\/p>\n

\"\"<\/p>\n

Once you have created a dashboard you can add metrics by mousing over
\nthe title bar of a graph and selecting the data sources icon (Three
\nHorizontal lines with an addition sign following them ). You can then
\nselect the server which you added earlier, and a query expression which
\nyou tested in the Prometheus Server UI. You can add multiple data
\nsources into the same graph in order to see a comparative view.<\/p>\n

\"\"<\/p>\n

You can add multiple graphs (each with possibly multiple data sources)
\nby clicking the Add Graph button. In addition you may select the
\ntime range over which your dashboard displays data as well as a refresh
\ninterval for auto-loading data. The dashboard is not as polished as the
\nones from Scout and DataDog, for example there is no easy way to explore
\nmetrics or build a query in the dashboard view. Since the dashboard runs
\nindependently of the Prometheus server we can\u2019t \u2018pin\u2019 graphs
\ngenerated in the Prometheus server into a dashboard. Furthermore several
\ntimes we noticed that the UI would not update based on selected data
\nuntil we refreshed the page. However, despite its issues the dashboard
\nis feature competitive with DataDog and because Prometheus is under
\nheavy development, we expect the bugs to be resolved over time. In
\ncomparison to other self-hosted solutions Prometheus is a lot more user
\nfriendly than Sensu and allows you present metric data as graphs without
\nusing third party visualizations. It also is able to provide much better
\nanalytical capabilities than CAdvisor.<\/p>\n

Prometheus also has the ability to apply alerting rules over the input
\ndata and displaying those on the UI. However, to be able to do something
\nuseful with alerts such send emails or notify
\npagerduty<\/a> we need to run the the Alert
\nManager<\/a>. To run
\nthe Alert Manager you first need to create a configuration file. Create
\na file called alertmanager.conf<\/em> and add the following text into it:<\/p>\n

notification_config {
\nname: “ubuntu_notification”
\npagerduty_config {
\nservice_key: “<PAGER_DUTY_API_KEY>”
\n}
\nemail_config {
\nemail: “<TARGET_EMAIL_ADDRESS>”
\n}
\nhipchat_config {
\nauth_token: “<HIPCHAT_AUTH_TOKEN>”
\nroom_id: 123456
\n}
\n}
\naggregation_rule {
\nfilter {
\nname_re: “image”
\nvalue_re: “ubuntu:14.04”
\n}
\nrepeat_rate_seconds: 300
\nnotification_config_name: “ubuntu_notification”
\n}<\/p>\n

In this configuration we are creating a notification configuration
\ncalled ubuntu_notification<\/em>, which specifies that alerts must go to
\nthe PagerDuty, Email and HipChat. We need to specify the relevant API
\nkeys and\/or access tokens for the HipChat and PagerDutyNotifications to
\nwork. We are also specifying that the alert configuration should only
\napply to alerts on metrics where the label image has the value
\nubuntu:14.04. We specify that a triggered alert should not retrigger
\nfor at least 300 seconds after the first alert is raised. We can bring
\nup the Alert Manager using the docker image by volume mounting our
\nconfiguration file into the container using the command shown below.<\/p>\n

docker run -d -p 9093:9093 -v $PWD:\/alertmanager prom\/alertmanager -logtostderr -config.file=\/alertmanager\/alertmanager.conf<\/p>\n

Once the container is running you should be able to point your browser
\nto port 9093 and load up the Alarm Manger UI. You will be able to see
\nall the alerts raised here, you can \u2018silence\u2019 them or delete them once
\nthe issue is resolved. In addition to setting up the Alert Manager we
\nalso need to create a few alerts. Add rule_file:
\n\u201c\/prometheus.rules\u201d in a new line into the global section of the
\nprometheus.conf<\/em> file you created earlier. This line tells Prometheus
\nto look for alerting rules in the prometheus.rules<\/em> file. We now need
\nto create the rules file and load it into our server container. To do so
\ncreate a file called prometheus.rules<\/em> in the same directory where you
\ncreated prometheus.conf<\/em>. and add the following text to it:<\/p>\n

ALERT HighMemoryAlert
\nIF container_memory_usage_bytes > 1000000000
\nFOR 1m
\nWITH {}
\nSUMMARY “High Memory usage for Ubuntu container”
\nDESCRIPTION “High Memory usage for Ubuntu container on {{$labels.instance}} for container {{$labels.name}} (current value: {{$value}})”<\/p>\n

In this configuration we are telling Prometheus to raise an alert called
\nHighMemoryAlert if the container_memory_usage_bytes metric
\nfor containers using the Ubuntu:14.04 image goes above 1 GB for 1
\nminute. The summary and the description of the alerts is also specified
\nin the rules file. Both of these fields can contain placeholders for
\nlabel values which are replaced by Prometheus. For example our
\ndescription will specify the server instance (IP) and the container name
\nfor metric raising the alert. After launching the Alert Manager and
\ndefining your Alert rules, you will need to re-run your Prometheus
\nserver with new parameters. The commands to do so are below:<\/p>\n

# stop and remove current container
\ndocker stop prometheus-server && docker rm prometheus-server<\/p>\n

# start new container
\ndocker run -d –name prometheus-server -p 9090:9090
\n-v $PWD\/prometheus.conf:\/prometheus.conf
\n-v $PWD\/prometheus.rules:\/prometheus.rules
\nprom\/prometheus
\n-config.file=\/prometheus.conf
\n-alertmanager.url=http:\/\/ALERT_MANAGER_IP:9093<\/p>\n

Once the Prometheus Server is up again you can click Alerts in the top
\nmenu of the Prometheus Server UI to bring up a list of alerts and their
\nstatuses. If and when an alert is fired you will also be able to see it
\nin the Alert Manager UI and any external service defined in the
\nalertmanager.conf<\/em> file.<\/p>\n

\"\"<\/p>\n

Collectively the Prometheus tool-set\u2019s feature set is on par with
\nDataDog which has been our best rated Monitoring tool so far. Prometheus
\nuses a very simple format for input data and can ingest from any web
\nendpoint which presents the data. Therefore we can monitor more or less
\nany resource with Prometheus, and there are already several libraries
\ndefined to monitor common resources. Where Prometheus is lacking is in
\nlevel of polish and ease of deployment. The fact that all components are
\ndockerized is a major plus however, we had to launch 4 different
\ncontainers each with their own configuration files to support the
\nPrometheus server. The project is also lacking detailed, comprehensive
\ndocumentation for these various components. However, in caparison to
\nself-hosted services such as CAdvisor and Sensu, Prometheus is a much
\nbetter toolset. It is significantly easier setup than sensu and has the
\nability to provide visualization of metrics without third party tools.
\nIt is able has much more detailed metrics than CAdvisor and is also able
\nto monitor non-docker resources. The choice of using pull based metric
\naggregation rather than push is less than ideal as you would have to
\nrestart your server when adding new data sources. This could get
\ncumbersome in a dynamic environment such as cloud based deployments.
\nPrometheus does offer the Push
\nGateway<\/a> to bridge the
\ndisconnect. However, running yet another service will add to the
\ncomplexity of the setup. For these reasons I still think DataDog is
\nprobably easier for most users, however, with some polish and better
\npackaging Prometheus could be a very compelling alternative, and out of
\nself-hosted solutions Prometheus is my pick.<\/p>\n

Score Card:<\/p>\n

    \n
  1. Easy of deployment: **<\/li>\n
  2. Level of detail: *****<\/li>\n
  3. Level of aggregation: *****<\/li>\n
  4. Ability to raise alerts: ****<\/li>\n
  5. Ability to monitor non-docker resources: Supported<\/li>\n
  6. Cost: Free<\/li>\n<\/ol>\n

    Sysdig Cloud<\/h2>\n

    Sysdig cloud is a hosted service that provides metrics storage,
    \naggregation, visualization and alerting. To get started with sysdig sign
    \nup for a trial account at     https:\/\/app.sysdigcloud.com<\/a>. and complete
    \nthe registration form. Once you complete the registration form and log
    \nin to the account, you will be asked to Setup your Environment<\/em> and be
    \ngiven a curl command similar to the shown below. Your command will have
    \nyour own secret key after the -s switch. You can run this command on the
    \nhost running docker and which you need to monitor. Note that you should
    \nreplace the [TAGS] place holder with tags to group your metrics. The
    \ntags are in the format TAG_NAME:VALUE so you may want to add a tag
    \nrole:web or deployment:production. You may also use the containerized
    \nsysdig agent.<\/p>\n

    # Host install of sysdig agent
    \ncurl -s https:\/\/s3.amazonaws.com\/download.draios.com\/stable\/install-agent | sudo bash -s 12345678-1234-1234-1234-123456789abc [TAGS]<\/p>\n

    # Docker based sysdig agent
    \ndocker run –name sysdig-agent –privileged –net host
    \n-e ACCESS_KEY=12345678-1234-1234-1234-123456789abc
    \n-e TAGS=os:rancher
    \n-v \/var\/run\/docker.sock:\/host\/var\/run\/docker.sock
    \n-v \/dev:\/host\/dev -v \/proc:\/host\/proc:ro
    \n-v \/boot:\/host\/boot:ro
    \n-v \/lib\/modules:\/host\/lib\/modules:ro
    \n-v \/usr:\/host\/usr:ro sysdig\/agent<\/p>\n

    Even if you use docker you will still need to install Kernel headers in
    \nthe host OS. This goes against Docker\u2019s philosophy of isolated micro
    \nservices. However, installing kernel headers is fairly benign.
    \nInstalling the headers and getting sysdig running is trivial if you are
    \nusing a mainstream kernel such us CentOS, Ubuntu or Debian. Even the
    \nAmazon\u2019s custom kernels are supported however RancherOS\u2019s custom
    \nkernel presented problems for sysdig as did the tinycore kernel. So be
    \nwarned if you would like to use Sysdig cloud on non-mainstream kernels
    \nyou may have to get your hands dirty with some system hacking.<\/p>\n

    After you run the agent you should see the Host in the Sysdig cloud
    \nconsole in the Explore tab. Once you launch docker containers on the
    \nhost those will also be shown. You can see basic stats about the CPU
    \nusage, memory consumption, network usage. The metrics are aggregated for
    \nthe host as well as broken down per container.<\/p>\n

    \"Screen<\/a>By
    \nselecting one of the hosts or containers you can get a whole host of
    \nother metrics including everything provided by the docker stats API. Out
    \nof all the systems we have seen so far sysdig certainly has the most
    \ncomprehensive set of metrics out of the box. You can also select from
    \nseveral pre-configured dashboards which present a graphical or tabular
    \nrepresentation of your deployment.<\/p>\n

    \"Screen<\/a><\/p>\n

    You can see live metrics, by selecting Real-time Mode (Target Icon)
    \nor select a window of time over which to average values. Furthermore,
    \nyou can also setup comparisons which will highlight the delta of current
    \nvalues and values at a point in the past. For example the table below
    \nshows values compared with those from ten minutes ago. If the CPU usage
    \nis significantly higher than 10 minutes ago you may be experiencing load
    \nspikes and need to scale out. The UI is at par with, if not better than
    \nDataDog for identifying and exploring trends in the data.    \"Screen<\/a><\/p>\n

    In addition to exploring data on an ad-hoc basis you can also create
    \npersistent dashboards. Simply click the pin icon on any graph in the
    \nexplore view and save it to a named dashboard. You can view all the
    \ndashboards and their associated graphs by clicking the Dashboards
    \ntab. You can also select the bell icon on any graph and create an
    \nalert from the data. The Sysdig cloud supports detailed alerting
    \ncriteria and is again one of the best we have seen. The example below
    \nshows an alert which triggers if the count of containers labeled web<\/em>
    \nfalls below three on average for the last ten minutes. We are also
    \nsegmenting the data by the region<\/em> tag, so there will be a separate
    \ncheck for web nodes in North America and Europe. Lastly, we also specify
    \na Name, description and Severity for the alerts. You can control where
    \nalerts go by going to Settings (Gear Icon) > Notifications and add
    \nemail addresses or SNS Topics to send alerts too. Note all alerts go to
    \nall notification endpoints which may be problematic if you want to wake
    \nup different people for different alerts.    \"Screen<\/a><\/p>\n

    I am very impressed with Sysdig cloud as it was trivially easy to setup,
    \nprovides detailed metrics with great visualization tools for real-time
    \nand historical data. The requirement to install kernel headers on the
    \nhost OS is troublesome though and lack of documentation and support for
    \nnon-standard kernels could be problematic in some scenarios. The
    \nalerting system in the Sysdig cloud is among the best we have seen so
    \nfar, however, the inability to target different email addresses for
    \ndifferent alerts is problematic. In a larger team for example you would
    \nwant to alert a different team for database issues vs web server issues.
    \nLastly, since it is in beta the pricing for Sysdig cloud is not easily
    \navailable. I have reached out to their sales team and will update this
    \narticle if and when they get back to me. If sysdig is price competitive
    \nthen Datadog has serious competition in the hosted service category.<\/p>\n

    Score Card:<\/p>\n

      \n
    1. Easy of deployment: ***<\/li>\n
    2. Level of detail: *****<\/li>\n
    3. Level of aggregation: *****<\/li>\n
    4. Ability to raise alerts: ****<\/li>\n
    5. Ability to monitor non-docker resources: Supported<\/li>\n
    6. Cost: Must Contact Support<\/li>\n<\/ol>\n

      Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

      I recently compared\u00a0several docker monitoring tools and services. Since the article went live we have gotten feedback about additional tools that should be included in our survey. I would like to highlight two such tools; Prometheus and Sysdig cloud. Prometheus is a capable self-hosted solution which is easier to manage than sensu. Sysdig cloud on … <\/p>\n

      Continue reading “Docker Monitoring Continued: Prometheus and Sysdig”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1345","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/comments?post=1345"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1345\/revisions"}],"predecessor-version":[{"id":1420,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1345\/revisions\/1420"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/media?parent=1345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/categories?post=1345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/tags?post=1345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}