![\"Securing](\"https:\/\/blog.giantswarm.io\/assets\/2018\/11\/securing-the-base-infrastructure-of-a-kubernetes-cluster.png\")
<\/p>\n<\/p>\n
The first article<\/a> in this series Securing Kubernetes for Cloud Native Applications<\/em>, provided a discussion on why it\u2019s difficult to secure Kubernetes, along with an overview of the various layers that require our attention, when we set about the task of securing that platform.<\/p>\n The very first layer in the stack, is the base infrastructure layer. We could define this in many different ways, but for the purposes of our discussion, it\u2019s the sum of the infrastructure components on top of which Kubernetes is deployed. It\u2019s the physical or abstracted hardware layer for compute, storage, and networking purposes, and the environment in which these resources exist. It also includes the operating system, most probably Linux, and a container runtime environment, such as Docker.<\/p>\n Much of what we\u2019ll discuss, applies equally well to infrastructure components that underpin systems other than Kubernetes, but we\u2019ll pay special attention to those factors that will enhance the security of Kubernetes.<\/p>\n The adoption of the cloud as the vehicle for workload deployment, whether its public, private, or a hybrid mix, continues apace. And whilst the need for specialist bare-metal server provisioning hasn\u2019t entirely gone away, the infrastructure that underpins the majority of today\u2019s compute resource, is the virtual machine. It doesn\u2019t really matter, however, if the machines we deploy are virtual (cloud-based or otherwise), or physical, the entity is going to reside in a data center, hosted by our own organisation, or a chosen third-party, such as a public cloud provider.<\/p>\n Data centers are complex, and there is a huge amount to think about when it comes to the consideration of security. It\u2019s a general resource for hosting the data processing requirements of an entire organisation, or even, co-tenanted workloads from a multitude of independent organisations from different industries and geographies. For this reason, applying security to the many different facets of infrastructure at this level, tends to be a full-blown corporate or supplier responsibility. It will be governed according to factors such as, national or international regulation (HIPAA<\/a>, GDPR<\/a>), industry compliance requirements (PCI DSS<\/a>), and often results in the pursuit of certified standards accreditation (ISO 27001<\/a>, FIPS<\/a>).<\/p>\n In the case of a public cloud environment, a supplier can and will provide the necessary adherence to regulatory and compliance standards at the infrastructure layer, but at some point, it comes down to the service consumer (you and me), to further build on this secure foundation. It\u2019s a shared responsibility. As a public cloud service consumer, this begs the question, \u201cwhat should I secure, and how should I go about it?\u201d There are a lot of people with a lot of views on the topic, but one credible entity is the Center for Internet Security (CIS)<\/a>, a non-profit organisation dedicated to safeguarding public and private entities from the threat of malign cyber activity.<\/p>\n The CIS provides a range of tools, techniques, and information for combating the potential threat to the systems and data we rely on. CIS Benchmarks<\/a>, for example, are per-platform best practice configuration guidelines for security, consensually compiled by security professionals and subject matter experts. In recognition of the ever increasing number of organisations embarking on transformation programmes, which involve migration to public and\/or hybrid cloud infrastructure, the CIS have made it their business to provide benchmarks for the major public cloud providers. The CIS Amazon Web Services Foundations Benchmark<\/a> is an example, and there are similar benchmarks for the other major public cloud providers.<\/p>\n These benchmarks provide foundational security configuration advice, covering identity and access management (IAM), ingress and egress, and logging and monitoring best practice, amongst other things. Implementing these benchmark recommendations is a great start, but it shouldn\u2019t be the end of the journey. Each public cloud provider will have their own set of detailed recommended best practices1<\/a>,2<\/a>,3<\/a>, and a lot of benefit can be taken from other expert voices in the domain, such as the Cloud Security Alliance<\/a>.<\/p>\n Let\u2019s take a moment to look at a typical cloud-based scenario that requires some careful planning from a security perspective.<\/p>\n How can we balance the need to keep a Kubernetes cluster secure by limiting access, whilst enabling the required access for external clients via the Internet, and also from within our own organisation?<\/p>\n This scenario demonstrates the need to carefully consider how to configure the infrastructure to be secure, whilst providing the capabilities required for delivering services to their intended audience. It\u2019s not a unique scenario, and there will be other situations that will require similar treatment.<\/p>\n Assuming we\u2019ve investigated and applied the necessary security configuration to make the machine-level infrastructure and its environment secure, the next task is to lock down the host operating system (OS) of each machine, and the container runtime that\u2019s responsible for managing the lifecycle of containers.<\/p>\n Whilst it\u2019s possible to run Microsoft Windows Server as the OS for Kubernetes worker nodes, more often than not, the control plane and worker nodes will run a variant of the Linux operating system. There might be many factors that govern the choice of Linux distribution to use (commercials, in-house skills, OS maturity), but if its possible, use a minimal distribution that has been designed just for the purpose of running containers. Examples include CoreOS Container Linux<\/a>, Ubuntu Core<\/a>, and the Atomic Host<\/a> variants. These operating systems have been stripped down to the bare minimum to facilitate running containers at scale, and as a consequence, have a significantly reduced attack surface.<\/p>\n Again, the CIS have a number of different benchmarks for different flavours of Linux, providing best practice recommendations for securing the OS. These benchmarks cover what might be considered the mainstream distributions of Linux, such as RHEL, Ubuntu, SLES, Oracle Linux and Debian. If your preferred distribution isn\u2019t covered, there is a distribution independent<\/a> CIS benchmark, and there are often distribution-specific guidelines, such as the CoreOS Container Linux Hardening Guide<\/a>.<\/p>\n The final component in the infrastructure layer is the container runtime. In the early days of Kubernetes, there was no choice available; the container runtime was necessarily the Docker engine. With the advent of the Kubernetes Container Runtime Interface<\/a>, however, it\u2019s possible to remove the Docker engine dependency in favour of a runtime such as CRI-O<\/a>, containerd<\/a> or Frakti<\/a>.4<\/a> In fact, as of Kubernetes version 1.12, an alpha feature (Runtime Class<\/a>) allows for running multiple container runtimes, side-by-side in a cluster. Whichever container runtimes are deployed, they need securing.<\/p>\n Despite the varied choice, the Docker engine remains the default container runtime for Kubernetes (although this may change to containerd in the near future), and we\u2019ll consider its security implications here. It\u2019s built with a large number of configurable security settings, some of which are turned on by default, but which can be bypassed on a per-container basis. One such example is the whitelist of Linux kernel capabilities<\/a> applied to each container on creation, which helps to diminish the privileges available inside a running container.<\/p>\n Once again, the CIS maintain a benchmark for the Docker platform, the CIS Docker Benchmark<\/a>. It provides best practice recommendations for configuring the Docker daemon for optimal security. There\u2019s even a handy open source tool (script) called Docker Bench for Security<\/a>, that can be run against a Docker engine, which evaluates the system for conformance to the CIS Docker Benchmark. The tool can be run periodically to expose any drift from the desired configuration.<\/p>\n Some care needs to be taken when considering and measuring the security configuration of the Docker engine when it\u2019s used as the container runtime for Kubernetes. Kubernetes ignores much of the available functions of the Docker daemon, in preference of its own security controls. For example, the Docker daemon is configured to apply a default whitelist of available Linux kernel system calls to every created container, using a seccomp profile<\/a>. Unless specified, Kubernetes will instruct Docker to create pod containers \u2018unconfined\u2019 from a seccomp perspective, giving containers access to each and every syscall available. In other words, what may get configured at the lower \u2018Docker layer\u2019, may get undone at a higher level in the platform stack. We\u2019ll cover how to mitigate these discrepancies with security contexts, in a future article.<\/p>\n It might be tempting to focus all our attention on the secure configuration of the Kubernetes components of a platform. But as we\u2019ve seen in this article, the lower layer infrastructure components are equally important, and are ignored at our peril. In fact, providing a secure infrastructure layer can even mitigate problems we might introduce in the cluster layer itself. Keeping our nodes private, for example, will prevent an inadequately secured kubelet from being exploited for nefarious purposes<\/a>. Infrastructure components deserve the same level of attention, as the Kubernetes components, themselves.<\/p>\n In the next article, we\u2019ll move on to discuss the implications of securing the next layer in the stack, the Kubernetes cluster components.<\/p>\nMachines, Data Centers, and the Public Cloud<\/h2>\n
CIS Benchmarks<\/h3>\n
Cloud Scenario: Private vs. Public Networks<\/h3>\n
\n
Locking Down the Operating System and Container Runtime<\/h2>\n
Linux OS<\/h3>\n
Docker Engine<\/h3>\n
Summary<\/h2>\n