{"id":777,"date":"2018-11-12T06:15:09","date_gmt":"2018-11-12T06:15:09","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=777"},"modified":"2018-11-18T01:19:54","modified_gmt":"2018-11-18T01:19:54","slug":"grpc-load-balancing-on-kubernetes-without-tears","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2018\/11\/12\/grpc-load-balancing-on-kubernetes-without-tears\/","title":{"rendered":"gRPC Load Balancing on Kubernetes without Tears"},"content":{"rendered":"

gRPC Load Balancing on Kubernetes without Tears<\/a><\/h3>\n

Many new gRPC users are surprised to find that Kubernetes\u2019s default load
\nbalancing often doesn\u2019t work out of the box with gRPC. For example, here\u2019s what
\nhappens when you take a simple gRPC Node.js microservices
\napp<\/a> and deploy it on Kubernetes:<\/p>\n

\"\"<\/p>\n

While the voting service displayed here has several pods, it\u2019s clear from
\nKubernetes\u2019s CPU graphs that only one of the pods is actually doing any
\nwork\u2014because only one of the pods is receiving any traffic. Why?<\/p>\n

In this blog post, we describe why this happens, and how you can easily fix it
\nby adding gRPC load balancing to any Kubernetes app with
\nLinkerd<\/a>, a CNCF<\/a> service mesh and service sidecar.<\/p>\n

First, let\u2019s understand why we need to do something special for gRPC.<\/p>\n

gRPC is an increasingly common choice for application developers. Compared to
\nalternative protocols such as JSON-over-HTTP, gRPC can provide some significant
\nbenefits, including dramatically lower (de)serialization costs, automatic type
\nchecking, formalized APIs, and less TCP management overhead.<\/p>\n

However, gRPC also breaks the standard connection-level load balancing,
\nincluding what\u2019s provided by Kubernetes. This is because gRPC is built on
\nHTTP\/2, and HTTP\/2 is designed to have a single long-lived TCP connection,
\nacross which all requests are multiplexed<\/em>\u2014meaning multiple requests can be
\nactive on the same connection at any point in time. Normally, this is great, as
\nit reduces the overhead of connection management. However, it also means that
\n(as you might imagine) connection-level balancing isn\u2019t very useful. Once the
\nconnection is established, there\u2019s no more balancing to be done. All requests
\nwill get pinned to a single destination pod, as shown below:<\/p>\n

\"\"<\/p>\n

The reason why this problem doesn\u2019t occur in HTTP\/1.1, which also has the
\nconcept of long-lived connections, is because HTTP\/1.1 has several features
\nthat naturally result in cycling of TCP connections. Because of this,
\nconnection-level balancing is \u201cgood enough\u201d, and for most HTTP\/1.1 apps we
\ndon\u2019t need to do anything more.<\/p>\n

To understand why, let\u2019s take a deeper look at HTTP\/1.1. In contrast to HTTP\/2,
\nHTTP\/1.1 cannot multiplex requests. Only one HTTP request can be active at a
\ntime per TCP connection. The client makes a request, e.g. GET \/foo, and then
\nwaits until the server responds. While that request-response cycle is
\nhappening, no other requests can be issued on that connection.<\/p>\n

Usually, we want lots of requests happening in parallel. Therefore, to have
\nconcurrent HTTP\/1.1 requests, we need to make multiple HTTP\/1.1 connections,
\nand issue our requests across all of them. Additionally, long-lived HTTP\/1.1
\nconnections typically expire after some time, and are torn down by the client
\n(or server). These two factors combined mean that HTTP\/1.1 requests typically
\ncycle across multiple TCP connections, and so connection-level balancing works.<\/p>\n

Now back to gRPC. Since we can\u2019t balance at the connection level, in order to
\ndo gRPC load balancing, we need to shift from connection balancing to request<\/em>
\nbalancing. In other words, we need to open an HTTP\/2 connection to each
\ndestination, and balance requests<\/em> across these connections, as shown below:<\/p>\n

\"\"<\/p>\n

In network terms, this means we need to make decisions at L5\/L7 rather than
\nL3\/L4, i.e. we need to understand the protocol sent over the TCP connections.<\/p>\n

How do we accomplish this? There are a couple options. First, our application
\ncode could manually maintain its own load balancing pool of destinations, and
\nwe could configure our gRPC client to use this load balancing
\npool<\/a>. This approach gives
\nus the most control, but it can be very complex in environments like Kubernetes
\nwhere the pool changes over time as Kubernetes reschedules pods. Our
\napplication would have to watch the Kubernetes API and keep itself up to date
\nwith the pods.<\/p>\n

Alternatively, in Kubernetes, we could deploy our app as headless
\nservices<\/a>.
\nIn this case, Kubernetes will create multiple A
\nrecords<\/a>
\nin the DNS entry for the service. If our gRPC client is sufficiently advanced,
\nit can automatically maintain the load balancing pool from those DNS entries.
\nBut this approach restricts us to certain gRPC clients, and it\u2019s rarely
\npossible to only use headless services.<\/p>\n

Finally, we can take a third approach: use a lightweight proxy.<\/p>\n

Linkerd<\/a> is a CNCF<\/a>-hosted service
\nmesh<\/em> for Kubernetes. Most relevant to our purposes, Linkerd also functions as
\na service sidecar<\/em>, where it can be applied to a single service\u2014even without
\ncluster-wide permissions. What this means is that when we add Linkerd to our
\nservice, it adds a tiny, ultra-fast proxy to each pod, and these proxies watch
\nthe Kubernetes API and do gRPC load balancing automatically. Our deployment
\nthen looks like this:<\/p>\n

\"\"<\/p>\n

Using Linkerd has a couple advantages. First, it works with services written in
\nany language, with any gRPC client, and any deployment model (headless or not).
\nBecause Linkerd\u2019s proxies are completely transparent, they auto-detect HTTP\/2
\nand HTTP\/1.x and do L7 load balancing, and they pass through all other traffic
\nas pure TCP. This means that everything will just work.<\/em><\/p>\n

Second, Linkerd\u2019s load balancing is very sophisticated. Not only does Linkerd
\nmaintain a watch on the Kubernetes API and automatically update the load
\nbalancing pool as pods get rescheduled, Linkerd uses an exponentially-weighted
\nmoving average<\/em> of response latencies to automatically send requests to the
\nfastest pods. If one pod is slowing down, even momentarily, Linkerd will shift
\ntraffic away from it. This can reduce end-to-end tail latencies.<\/p>\n

Finally, Linkerd\u2019s Rust-based proxies are incredibly fast and small. They
\nintroduce <1ms of p99 latency and require <10mb of RSS per pod, meaning that
\nthe impact on system performance will be negligible.<\/p>\n

Linkerd is very easy to try. Just follow the steps in the Linkerd Getting
\nStarted Instructions<\/a>\u2014install the
\nCLI on your laptop, install the control plane on your cluster, and \u201cmesh\u201d your
\nservice (inject the proxies into each pod). You\u2019ll have Linkerd running on your
\nservice in no time, and should see proper gRPC balancing immediately.<\/p>\n

Let\u2019s take a look at our sample voting service again, this time after
\ninstalling Linkerd:<\/p>\n

\"\"<\/p>\n

As we can see, the CPU graphs for all pods are active, indicating that all pods
\nare now taking traffic\u2014without having to change a line of code. Voila,
\ngRPC load balancing as if by magic!<\/p>\n

Linkerd also gives us built-in traffic-level dashboards, so we don\u2019t even need
\nto guess what\u2019s happening from CPU charts any more. Here\u2019s a Linkerd graph
\nthat\u2019s showing the success rate, request volume, and latency percentiles of
\neach pod:<\/p>\n

\"\"<\/p>\n

We can see that each pod is getting around 5 RPS. We can also see that, while
\nwe\u2019ve solved our load balancing problem, we still have some work to do on our
\nsuccess rate for this service. (The demo app is built with an intentional
\nfailure\u2014as an exercise to the reader, see if you can figure it out by
\nusing the Linkerd dashboard!)<\/p>\n

If you\u2019re interested in a dead simple way to add gRPC load balancing to your
\nKubernetes services, regardless of what language it\u2019s written in, what gRPC
\nclient you\u2019re using, or how it\u2019s deployed, you can use Linkerd to add gRPC load
\nbalancing in a few commands.<\/p>\n

There\u2019s a lot more to Linkerd, including security, reliability, and debugging
\nand diagnostics features, but those are topics for future blog posts.<\/p>\n

Want to learn more? We\u2019d love to have you join our rapidly-growing community!
\nLinkerd is a CNCF<\/a> project, hosted on GitHub<\/a>, and has a thriving community
\non Slack<\/a>, Twitter<\/a>, and the mailing lists<\/a>. Come and join the fun!<\/p>\n

Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

gRPC Load Balancing on Kubernetes without Tears Many new gRPC users are surprised to find that Kubernetes\u2019s default load balancing often doesn\u2019t work out of the box with gRPC. For example, here\u2019s what happens when you take a simple gRPC Node.js microservices app and deploy it on Kubernetes: While the voting service displayed here has … <\/p>\n

Continue reading “gRPC Load Balancing on Kubernetes without Tears”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-777","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/777","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/comments?post=777"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/777\/revisions"}],"predecessor-version":[{"id":786,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/777\/revisions\/786"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/media?parent=777"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/categories?post=777"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/tags?post=777"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}