Kubernetes has a variety of primitives that make it a great platform for running workloads submitted by multiple teams. Features like Role Based Access Control (RBAC) and Namespaces make it possible to divide clusters across multiple teams in a safe way. There are some challenges however, and one of the most important ones our enterprise customers have encountered lies in the Ingress API. In this post, we will explore how a bad Ingress resource can break your ingress layer, and walk through our novel approach to multi-team ingress using Heptio Contour’s new IngressRoute resource.
Multi-team Ingress on Kubernetes
Most organizations typically have more than one team interacting with a given cluster. Cluster operators assign one or more namespaces to each team and use RBAC to ensure that no team can mess with another team’s resources.
Even though Ingress is a namespaced resource that can be locked down with RBAC, it poses a challenge in multi-team clusters because it controls cluster-level configuration: the hosts and paths on which to serve application traffic.
Let us imagine a scenario where the marketing team owns www.example.com/blog. They are responsible for the organization’s blog and they have configured an Ingress resource that looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: blog
namespace: marketing
spec:
rules:
– host: www.example.com
http:
paths:
– path: /blog
backend:
serviceName: blog
servicePort: 80
Now, the engineering team is looking to run their own engineering-focused blog, and they mistakenly apply the following Ingress resource into the engineering namespace:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: blog
namespace: engineering
spec:
rules:
– host: www.example.com
http:
paths:
– path: /blog
backend:
serviceName: engineering-blog
servicePort: 80
We now have two conflicting Ingress configurations that point www.example.com/blog to different services. The Ingress API does not define how to handle this conflict and the behavior of Ingress Controllers frequently differs — this results in a negative user experience affecting multiple parties. The engineering team is completely unaware that they have taken down the company blog, while the avid blog readers are unable to access their favorite blog.
As you can see in this example, the Ingress resource can become the Achilles’ heel of a multi-team cluster. We have heard from multiple customers that have been bitten by this in production, and thus we decided to address this issue in Contour.
IngressRoute delegation to the rescue
One of the most exciting features introduced in the latest version of Heptio Contour is the IngressRoute Custom Resource Definition (CRD). Among the many improvements available in this new custom resource is delegation support, which allow you to delegate the configuration of a specific host or path to another IngressRoute.
The crux of the problem with the Ingress resource in a multi-team cluster is that operators do not have a way to prevent teams from claiming hosts and paths at will. The ability to create root IngressRoutes in a specific namespace, as well as the ability to do cross-namespace delegation is our answer to this problem.
Using the delegation feature of the IngressRoute, cluster operators get full control of the roots of their ingress layer by limiting which namespaces are authorized to create root IngressRoutes. This eliminates the possibility for two teams to create configurations that collide. The IngressRoute roots specify the top level domains and TLS configuration, while delegating the configuration of specific subdomains or paths to other IngressRoutes in other namespaces. In this way, each team gets the ability to use and configure the slice of the ingress space that has been delegated to their team’s namespace.
Let us revisit the problematic scenario we outlined above. The cluster operator creates a “roots” namespace, and configures Contour to only accept root IngressRoutes from this namespace. Then, the cluster operator creates a root IngressRoute for www.example.com and delegates the /blog path to the marketing team:
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: example-com-root
namespace: roots
spec:
virtualhost:
fqdn: www.example.com
routes:
– match: /blog
delegate:
name: blog
namespace: marketing
The marketing team creates an IngressRoute that sets up the company blog. Note that the virtualhost is missing, as this is not a root IngressRoute.
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: blog
namespace: marketing
spec:
routes:
– match: /blog
services:
– name: blog
port: 80
As you might imagine, if the engineering team were to create a conflicting IngressRoute, the company’s blog would remain accessible as there is no delegation path that points to the engineering team IngressRoute. Instead of producing an outage, Contour ignores the orphaned route and sets its status field accordingly:
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: blog
namespace: engineering
spec:
routes:
– match: /blog
services:
– name: engineering-blog
port: 80
status:
currentStatus: orphaned
description: this IngressRoute is not part of a delegation chain from a root IngressRoute
What’s next?
We have explored the new IngressRoute and more specifically, the delegation model that enables you to run multi-team Kubernetes clusters in a safe way; this is one of the exciting features available in the latest version of Heptio Contour. But, there’s more.
In future posts, we will explore other patterns enabled by the IngressRoute, including blue/green deployments, canary deployments and load balancing strategies. If you have any questions, or are interested in learning more, feel to reach us via the #contour channel on the Kubernetes community Slack, or follow us on Twitter.
Source