Now that you have a fully functioning ELK stack on Rancher, you can![\"kibana2\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/09\/14203235\/kibana2-300x205.png\")
<\/a>At
\nRancher Labs we generate a lot of logs in our internal environments. As
\nwe conduct more and more testing on these environments we have found the
\nneed to centrally aggregate the logs from each environment. We decided
\nto use Rancher<\/a>to build and run a scalable
\nELK stack to manage all of these logs. For those that are unfamiliar
\nwith the ELK stack, it is made up of Elasticsearch, Logstash and Kibana.
\nLogstash provides a pipeline for shipping logs from various sources and
\ninput types, combining, massaging and moving them into Elasticsearch, or
\nseveral other stores. It is a really powerful tool in the logging
\narsenal. Elasticsearch is a document database that is really good at
\nsearch. It can take our processed output from Logstash, analyze and
\nprovides an interface to query all of our logging data. Together with
\nKibana, a powerful visualization tool that consumes Elasticsearch data,
\nyou have amazing ability to gain insights from your logging. Previously,
\nwe have been using Elastic\u2019s Found product and have been very impressed.
\nOne of the interesting things we realized while using Found for
\nElasticsearch is that the ELK stack really is made up of discrete parts.
\nEach part of the stack has its own needs and considerations. Found
\nprovided us Elasticsearch and Kibana. There was no Logstash end point
\nprovided, though it was sufficiently documented how to use Found with
\nLogstash. So, we have always had to run our own Logstash pipeline.
\nLogstash Our Logstash implementation includes three tiers, one each
\nfor collection, queueing and processing. Collection- responsible for
\nproviding remote endpoints for logging inputs. Like Syslog, Gelf,
\nLogstash. Once it receives these logs it places them quickly onto a
\nRedis Queue. Queuing tier – provided by Redis, a very fast in memory
\ndatabase. It acts as a buffer between the collection and processing
\ntier. Processing tier – removes messages from the queue, and applies
\nfilter plugins to the logs that manipulate the data to a desired format.
\nThis tier does the heavy lifting and is often a bottleneck in a log
\npipeline. Once it processes the data it forwards it along to the final
\ndestination, which is Elasticsearch. ![\"Logstash\nPipeline\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/09\/14203235\/Logstash-Pipeline.svg\")
<\/a>
\nEach Logstash container has a configuration sidekick that provides
\nconfiguration through a shared volume.<\/em> By breaking the stack into these
\ntiers, you can scale and adapt each part without major impact to the
\nother parts of the stack. As a user, you can also scale and adjust each
\ntier to suit your needs. A good read on how to scale Logstash can be
\nfound on Elastic\u2019s web page here: Deploying and Scaling
\nLogstash<\/a>.
\nTo build the Logstash stack we stared as we usually do. In general, we
\ntry to reuse as much as possible from the community. Looking at the
\nDockerHub registry, we found there is already an official Logstash image
\nmaintained by Docker. The real magic is in configuration of Logstash at
\neach of the tiers. To achieve maximum flexibility with configuration, we
\nbuilt a confd container that consumes KV, or Key Value, data for its
\nconfiguration values. The logstash configurations are the most volatile,
\nand unique to an organization as they provide the interfaces for the
\ncollection, indexing, and shipping of the logs. Each organization is
\ngoing to have different processing needs, formatting, tagging etc. To
\nachieve maximum flexibility we leveraged the confd tool and Rancher
\nsidekick containers. The sidekick creates an atomic scheduling unit
\nwithin Rancher. In this case, our configuration container exposes the
\nconfiguration files to our Logstash container through volume sharing. In
\ndoing this, there is no modification needed to the default Docker
\nLogstash image. How is that for reuse! Elasticsearch Elasticsearch
\nis built out in three tiers as well. When reading the production
\ndeployment recommendations, it discusses having nodes that are dedicated
\nmasters, data nodes and client nodes. We followed the same deployment
\nparadigm with this application as the logstash implementation. We deploy
\neach role as a service. Each service is composed of an official image
\nand paired with a Confd sidekick container to provide configuration. It
\nends up looking like this: ![\"Elastic](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2015\/09\/14203235\/Elastic-Search-Tier.svg\")
<\/a>
\nEach tier in the Elasticsearch stack has a confd container providing
\nconfigurations through a shared volume. These containers are scheduled
\ntogether inside of Rancher.<\/em> In the current configuration, we use the
\nmaster service to provide node discovery. When using the Rancher private
\nnetwork, we disable multicast and enable unicast. Since every node in
\nthe cluster points to the master they can talk to one another. The
\nRancher network also allows the nodes to talk to one another. As a part
\nof our stack, we also use the Kopf tool to quickly visualize our
\nclusters health and perform other maintenance tasks. Once you bring up
\nthe stack you will see that you can use Kopf to see that all the nodes
\ncame up in the cluster. Kibana 4 Finally, in order to view all of
\nthese logs and make sense of the data, we bring up Kibana to complete
\nour ELK stack. We have chosen to go with Kibana 4 in this stack. Kibana
\n4 is launched with an Nginx container to provide basic auth behind a
\nRancher load balancer. The Kibana 4 instance is the Official image which
\nis hosted on DockerHub. The Kibana 4 image talks to the Elasticsearch
\nclient nodes. So now we have a full ELK stack for taking logs and
\nshipping them to Elasticsearch for visualization in Kibana. The next
\nstep is getting the logs from the hosts running your application.
\nBringing up the Stack on Rancher So now you have the backstory on
\nhow we came up with our ELK stack configuration. Here are instructions
\nto run the ELK stack on Rancher. This assumes that you already have a
\nRancher environment running with at least one compute node. We will also
\nbe using the Rancher compose CLI tool. Rancher-compose can be found on
\nGitHub here
\nrancher\/rancher-compose<\/a>.
\nYou will need API keys from your Rancher deployment. In the instructions
\nbelow, we will bring up each component of the ELK stack, as its own
\nstack in Rancher. A stack in Rancher is a collection of services that
\nmake up an application, and are defined by a Docker Compose file. In
\nthis example, we will build the stacks in the same environment and use
\ncross stack linking to connect services. Cross stack linking allows
\nservices in different stacks to discover each other through a DNS name.<\/p>\n\n
\nhttps:\/\/github.com\/rancher\/compose-templates.git<\/a><\/li>\n
\na. cd compose-templates\/elasticsearch
\nb. rancher-compose -p es up (Other services assume es as the
\nelasticsearch stack name) This will bring up four services.
\n– elasticsearch-masters
\n– elasticsearch-datanodes
\n– elasticsearch-clients
\n– kopf
\nc. Once Kopf is up, click on the container in the Rancher UI, and
\nget the IP of the node it is running on.
\nd. Open a new tab in your browser and go to the IP. You should see
\none datanode on the page.<\/li>\n
\na. cd ..\/logstash
\nb. rancher-compose -p logstash up
\nc. This will bring up the following services
\n– Redis
\n– logstash-collector
\n– logstash-indexer
\nd. At this point, you can point your applications at
\nlogtstash:\/\/host:5000.<\/li>\n
\na. cd ..\/logspout
\nb. rancher-compose -p logspout up
\nc. This will bring up a logspout container on every node in your
\nRancher environment. Logs will start moving through the pipeline
\ninto Elasticsearch.<\/li>\n
\na. cd ..\/kibana
\nb. rancher-compose -p kibana up
\nc. This will bring up the following services
\n– kibana-vip
\n– nginx-proxy
\n– kibana4
\nd. Click the container in the kibana-vip service in the Rancher UI.
\nVisit the host ip in a separate browser tab. You will be
\ndirected to the Kibana 4 landing page to select your index.<\/li>\n<\/ol>\n
\nstart sending your logs through the Logstash collector. By default the
\ncollector is listening for Logstash inputs on UDP port 5000. If you are
\nrunning applications outside of Rancher, you can simply point them to
\nyour Logstash endpoint. If your application runs on Rancher you can use
\nthe optional Logspout-logstash service above. If your services run
\noutside of Rancher, you can configure your Logstash to use Gelf, and use
\nthe Docker log driver. Alternatively, you could setup a Syslog listener,
\nor any number of supported Logstash input plugins. Conclusion
\nRunning the ELK stack on Rancher in this way provides a lot of
\nflexibility to build and scale to meet any organization\u2019s needs. It
\nalso creates a simple way to introduce Rancher into your environment
\npiece by piece. As an operations team, you could quickly spin up
\npipelines from existing applications to existing Elasticsearch clusters.
\nUsing Rancher you can deploy applications following container best
\npractices by using sidekick containers to customize standard containers.
\nBy scheduling these containers as a single unit, you can separate your
\napplication out into separate concerns. On Wednesday, September 16th,
\nwe hosted an online meetup focused on container logging, where I
\ndemonstrated how to build and deploy your own ELK stack. If you\u2019d like
\nto view a recording of this you can view it
\nhere<\/a>.
\nIf you\u2019d like to learn more about using Rancher, please join us for an
\nupcoming online meetup<\/a>, or join our beta
\nprogram<\/a> or request a discussion with one
\nof our engineers.<\/p>\n