18\/Jan 2018<\/p>\n
By Richard Wall<\/a><\/p>\n Navigator<\/a> is a Kubernetes extension for managing distributed databases. The Apache Cassandra database<\/a> is a leading NoSQL database designed for scalability and high availability without compromising performance. Enter Navigator! This is what\u2019s currently possible and our goal is to make it simple enough that any developer can request a secure, highly available, scalable Cassandra database and Navigator will take care of the rest, ensuring that:<\/p>\n Now for a demo.<\/p>\n In this short screen cast we demonstrate how to install Navigator and then install a Cassandra cluster, using Helm. The commands used in the demo<\/a> are available in the Navigator repository.<\/p>\n Elasticsearch was the first database supported by Navigator and we\u2019ve made dozens of improvements in the last six months, Here are some examples:<\/p>\n The Navigator Pilot is a process which is injected into the container of the target database and becomes the entry point for the container. We\u2019ve introduced a new Pilot API resource. The Navigator controller creates a Pilot resource for every pod under its control.<\/p>\n We\u2019ve made many improvements to the Pilot to ensure that:<\/p>\n And the Pilot now has a new REST endpoint (\/healthz ) through which it responds to Kubernetes Readiness and Liveness probes. Pilots now have a leader election mechanism. This is all groundwork which will allow us to safely scale down Elasticsearch clusters.<\/p>\n Navigator now runs in (and is tested in) Kubernetes environments where RBAC is enabled. And if you prefer, you can run a separate Navigator controller for each database namespace. Navigator now has its own API server which is aggregated into the Kubernetes API<\/a>.<\/p>\n The reason for this change was to overcome the limitations<\/a> of CRDs (Custom Resource Definitions)<\/a>. And while the Navigator architecture has become somewhat more complex, the installation of Navigator has been vastly simplified by the introduction of a Navigator Helm chart (see below).<\/p>\n Navigator now has a tried and tested Helm chart<\/a>.<\/p>\n This allows you to install and configure the Navigator API server, and the Navigator Controller, and all the supporting Kubernetes resources, in a single fool-proof step. We have also been working on a suite of end-to-end tests for Navigator.<\/p>\n These verify that the Navigator API server and the Navigator controller can be installed, using Helm, as documented. We now use Kubernetes test infrastructure<\/a> to run unit tests and end-to-end tests.<\/p>\n We initially tried running tests against Minikube on Travis CI<\/a> and it totally worked<\/a>! Additionally some of our end-to-end tests were attempting to launch (and scale-up) multi-node Elasticsearch and Cassandra databases. So we\u2019ve installed our own test infrastructure on Google Cloud and installed and tweaked the Kubernetes Test-Infra for our own purposes and it works great!<\/p>\n We\u2019ll write more about this in a future blog post but for now take a look at: \u2018Prow: Testing the way to Kubernetes Next\u2019<\/a> and \u2018Making Use of Kubernetes Test Infra Tools\u2019<\/a>, which give a great introduction to the Kubernetes Test-Infra tools.<\/p>\n The Navigator API is subject to change and the project is still in an alpha state so we ask that you do not use it in production, yet! So stay tuned!
\nIn this post we\u2019ll tell you about all the improvements we\u2019ve made since we unveiled it last year<\/a>, including:
\nexperimental support for Apache Cassandra clusters,
\nimproved support for Elasticsearch clusters,
\nand a Helm chart for easy installation!
\nWe\u2019ll also give you an overview of the Navigator roadmap for 2018.<\/p>\n
\nIt\u2019s an ideal candidate for running on a scalable, highly available, distributed platform such as Kubernetes.
\nFor that reason it was the database chosen to showcase the potential of Kubernetes StatefulSets<\/a> (or PetSet as it was known initially).
\nSince then a Cassandra example<\/a> has been added to the official Kubernetes documentation but it is only an example; it is not sufficient if you want to run a Cassandra cluster on Kubernetes in production<\/em>.<\/p>\n
\nNavigator now has experimental support the Apache Cassandra database.
\nNavigator codifies and automates many of the processes that would previously have been performed by a database administrator or SRE (Site Reliability Engineer) (the operator<\/a>).
\nFor example it will bootstrap a Cassandra cluster and create a load balancer for distributing CQL connections to all the cluster nodes.
\nIt performs regular node health checks which means that if a node becomes unresponsive, that node will be automatically bypassed by the loadbalancer and eventually the node will be restarted.
\nNavigator can also scale up your Cassandra cluster, and it\u2019s as simple as using Helm to increment the replicas field on the CassandraCluster API resource.
\nSee the demo below.<\/p>\n\n
Demo<\/h2>\n
\nWe also show how to examine the status and logs of Navigator components and of the Cassandra database nodes.
\nWe demonstrate how to scale up the Cassandra database and then connect to the database using cqlsh to create a key space, a table and insert some data.<\/p>\n
\nworking closely with customers and the community.<\/p>\nPilot Resources<\/h2>\n
\nSo instead of starting the database process immediately, Kubernetes will actually start a \/pilot process which first connects to the Navigator API to discover the desired database configuration.
\nIt then configures and starts up an Elasticsearch sub-process.<\/p>\n
\nThis is a place where the controller can publish the desired configuration (spec<\/em>) of each database node.
\nAnd it\u2019s where a Pilot process can publish the actual state of its database sub-process (status<\/em>).<\/p>\nSub-process Management<\/h2>\n
\n
Health Checks<\/h2>\n
\nWhile the database is running, the Pilot queries the Elasticsearch API to gather the current state of the database and publishes it via the \/healthz endpoint.<\/p>\nLeader Election<\/h2>\n
\nThis allows a single \u201cleader\u201d Pilot process to perform cluster-wide administrative functions.<\/p>\nScale Down Safely<\/h2>\n
\nThe Navigator API server, the Navigator controller, and the Pilots all run with the least necessary privilege<\/a>.<\/p>\n
\nWe\u2019ve implemented a new Filtered Informer mechanism<\/a> so that, in this mode, the Navigator controller will only be able to interact with API resources in a single namespace.<\/p>\n
\nMost importantly it allows us to assign versions to our Navigator API resource types, as they evolve.
\nAnd it allows seamless conversion between different versions of the resources.<\/p>\n
\nWe use that same Helm chart to install Navigator before running our end-to-end tests, so we (and you) can be confident that this installation mechanism is reliable.
\nThe Helm chart is quite configurable and allows you to tweak the logging level of the Navigator components for example.<\/p>\n
\nThey verify that the Navigator API server is successfully aggregated in all the versions of Kubernetes we support.
\nThat the Navigator controller starts an Elasticsearch or Cassandra database cluster matching the desired configuration in the API server.
\nAnd most importantly that the databases are actually usable after they have been launched.<\/p>\n
\nBut we soon encountered limitations. We needed to use the minikube start –bootstrapper=kubeadm option, in order to properly set up the Navigator API server aggregation; but that doesn\u2019t work on the Ubuntu 14.04 operating system provided by Travis-CI.<\/p>\n
\nA single Travis-CI virtual machine just doesn\u2019t cut the mustard.<\/p>\n
\nBut we\u2019re working flat out to add more features and to make Navigator as robust as possible.
\nHere are some of the features that we\u2019re working on:<\/p>\n\n
\nAnd join us if you can at QCon London, in March 2018<\/a>, where we plan to announce and demonstrate a new Navigator release.
\nHope to see you there!<\/p>\n