![\"Grafana](\"https:\/\/blog.giantswarm.io\/assets\/2019\/03\/grafana-logging-using-loki.jpg\")
<\/p>\n<\/p>\n
Loki<\/a> is a Prometheus-inspired logging service for cloud native infrastructure.<\/p>\n Open sourced by Grafana Labs during KubeCon Seattle 2018, Loki is a logging backend optimized for users running Prometheus and Kubernetes with great logs search and visualization in Grafana 6.0.<\/p>\n Loki was built for efficiency alongside the following goals:<\/p>\n As said, Loki is designed for efficiency to work well in the Kubernetes context in combination with Prometheus metrics.<\/p>\n The idea is to switch easily between metrics and logs based on Kubernetes labels you already use with Prometheus.<\/p>\n Unlike most logging solutions, Loki does not parse incoming logs or do full-text indexing.<\/p>\n Instead, it indexes and groups log streams using the same labels you\u2019re already using with Prometheus. This makes it significantly more efficient to scale and operate.<\/p>\n Loki is a TSDB (Time-series database), it stores logs as split and gzipped chunks of data.<\/p>\n The logs are ingested via the API and an agent, called Promtail (Tailing logs in Prometheus format), will scrape Kubernetes logs and add label metadata before sending it to Loki.<\/p>\n This metadata addition is exactly the same as Prometheus, so you will end up with the exact same labels for your resources.<\/p>\n https:\/\/grafana.com\/blog\/2018\/12\/12\/loki-prometheus-inspired-open-source-logging-for-cloud-natives\/<\/a><\/p>\n The easiest way to deploy Loki on your Kubernetes cluster is by using the Helm chart<\/a> available in the official repository.<\/p>\n You can follow the setup guide<\/a> from the official repo.<\/p>\n This will deploy Loki and Promtail.<\/p>\n Promtail is the metadata appender and log sending agent<\/p><\/blockquote>\n The Promtail configuration you get from the Helm chart is already configured to get all the logs from your Kubernetes cluster and append labels on it as Prometheus does for metrics.<\/p>\n However, you can tune the configuration for your needs.<\/p>\n Here are two examples:<\/p>\n You can use the action: keep for your namespace and add a new relabel_configs for each scrape_config in promtail\/configmap.yaml<\/a><\/p>\n For example, if you want to get logs only for the kube-system namespace:<\/p>\n scrape_configs: # […]<\/p>\n – job_name: kubernetes-pods-app For example, if you want to exclude logs from kube-system namespace:<\/p>\n You can use the action: drop for your namespace and add a new relabel_configs for each scrape_config in promtail\/configmap.yaml<\/a><\/p>\n scrape_configs: # […]<\/p>\n – job_name: kubernetes-pods-app For more info on the configuration, you can refer to the official Prometheus configuration documentation<\/a>.<\/p>\n Fluentd is a well-known and good log forwarder that is also a [CNCF project] (https:\/\/www.cncf.io\/projects\/). It has a lot of input plugins and good filtering built-in. So, if you want to for example, forward journald logs to Loki, it\u2019s not possible via Promtail so you can use the fluentd syslog input plugin with the fluentd Loki output plugin to get those logs into Loki.<\/p>\n You can refer to the installation guide<\/a> on how to use the fluentd Loki plugin.<\/p>\n There\u2019s also an example<\/a>, of how to forward API server audit logs to Loki with fluentd.<\/p>\n Here is the fluentd configuration:<\/p>\n <match fluent.**> By default, Promtail is configured to automatically scrape logs from containers and send them to Loki. Those logs come from stdout.<\/p>\n But sometimes, you may like to be able to send logs from an external file to Loki.<\/p>\n In this case, you can set up Promtail as a sidecar, i.e. a second container in your pod, share the log file with it through a shared volume, and scrape the data to send it to Loki<\/p>\n Assuming you have an application simple-logger. The application logs into \/home\/slog\/creator.log<\/p>\n Your kubernetes deployment will look like this :<\/p>\n apiVersion: apps\/v1 apiVersion: apps\/v1 apiVersion: apps\/v1 As Promtail uses the same config as Prometheus, you can use the scrape_config type static_configs to read the file you want.<\/p>\n scrape_configs: And you\u2019re done.<\/p>\n A running example can be found here<\/a><\/p>\n So Loki looks very promising. The footprint is very low. It integrates nicely with Grafana and Prometheus. Having the same labels as in Prometheus is very helpful to map incidents together and quickly find logs related to metrics. Another big point is the simple scalability, Loki is horizontally scalable by design.<\/p>\n As Loki is currently alpha software, install it and play with it. Then, join us on grafana.slack.com and add your feedback to make it better.<\/p>\n Interested in finding out how Giant Swarm handles the entire cloud native stack including Loki? Request your free trial of the Giant Swarm Infrastructure here<\/a>.<\/p>\nWhat is Loki?<\/strong><\/h2>\n
\n
Loki vs other logging solutions<\/strong><\/h2>\n
Loki components<\/strong><\/h2>\n
![\"\"](\"https:\/\/grafana.com\/blog\/assets\/img\/blog\/image1.png\")
<\/p>\nHow to deploy Loki on your Kubernetes cluster<\/strong><\/h2>\n
\n
\n
\n
\n
![\"\"](\"https:\/\/grafana.com\/blog\/assets\/img\/blog\/loki\/loki_grafana_ui.png\")
<\/p>\nPromtail configuration<\/strong><\/h2>\n
\n
\n– job_name: kubernetes-pods
\nkubernetes_sd_configs:
\n– role: pod
\nrelabel_configs:
\n– source_labels: [__meta_kubernetes_namespace]
\naction: keep
\nregex: kube-system<\/p>\n
\nkubernetes_sd_configs:
\n– role: pod
\nrelabel_configs:
\n– source_labels: [__meta_kubernetes_namespace]
\naction: keep
\nregex: kube-system<\/p>\n\n
\n– job_name: kubernetes-pods
\nkubernetes_sd_configs:
\n– role: pod
\nrelabel_configs:
\n– source_labels: [__meta_kubernetes_namespace]
\naction: drop
\nregex: kube-system<\/p>\n
\nkubernetes_sd_configs:
\n– role: pod
\nrelabel_configs:
\n– source_labels: [__meta_kubernetes_namespace]
\naction: drop
\nregex: kube-system<\/p>\nUse fluentd output plugin<\/strong><\/h2>\n
\ntype null
\n<\/match>
\n<source>
\n@type tail
\npath \/var\/log\/apiserver\/audit.log
\npos_file \/var\/log\/fluentd-audit.log.pos
\ntime_format %Y-%m-%dT%H:%M:%S.%NZ
\ntag audit.*
\nformat json
\nread_from_head true
\n<\/source>
\n<filter kubernetes.**>
\ntype kubernetes_metadata
\n<\/filter>
\n<match audit.**>
\n@type loki
\nurl “#”
\nusername “#”
\npassword “#”
\nextra_labels {“env”:”dev”}
\nflush_interval 10s
\nflush_at_shutdown true
\nbuffer_chunk_limit 1m
\n<\/match><\/p>\nPromtail as a sidecar<\/strong><\/h2>\n
\n
\nkind: Deployment
\nmetadata:
\nname: my-app
\nspec:
\ntemplate:
\nmetadata:
\nname: my-app
\nspec:
\ncontainers:
\n– name: simple-logger
\nimage: giantswarm\/simple-logger:latest<\/p>\n
\nkind: Deployment
\nmetadata:
\nname: my-app
\nspec:
\ntemplate:
\nmetadata:
\nname: my-app
\nspec:
\ncontainers:
\n– name: simple-logger
\nimage: giantswarm\/simple-logger:latest
\n– name: promtail
\nimage: grafana\/promtail:master
\nargs:
\n– “-config.file=\/etc\/promtail\/promtail.yaml”
\n– “-client.url=http:\/\/loki:3100\/api\/prom\/push”<\/li>\n
\nkind: Deployment
\nmetadata:
\nname: my-app
\nspec:
\ntemplate:
\nmetadata:
\nname: my-app
\nspec:
\ncontainers:
\n– name: simple-logger
\nimage: giantswarm\/simple-logger:latest
\nvolumeMounts:
\n– name: shared-data
\nmountPath: \/home\/slog
\n– name: promtail
\nimage: grafana\/promtail:master
\nargs:
\n– “-config.file=\/etc\/promtail\/promtail.yaml”
\n– “-client.url=http:\/\/loki:3100\/api\/prom\/push”
\nvolumeMounts:
\n– name: shared-data
\nmountPath: \/home\/slog
\nvolumes:
\n– name: shared-data
\nemptyDir: {}<\/li>\n
\n– job_name: system
\nentry_parser: raw
\nstatic_configs:
\n– targets:
\n– localhost
\nlabels:
\njob: my-app
\nmy-label: awesome
\n__path__: \/home\/slog\/creator.log<\/p>\nConclusion<\/strong><\/h2>\n