A Vitess overview<\/h2>\n
Vitess<\/a> is a database solution for deploying, scaling and managing large clusters of MySQL instances. It\u2019s architected to run as effectively in a public or private cloud architecture as it does on dedicated hardware. It combines and extends many important MySQL features with the scalability of a NoSQL database.<\/p>\n In a simple scenario with Relational Databases and MySQL in particular, you would have a master instance, then a couple of replicas. You would then direct read-only queries to the replicas and write queries (and possible critical read queries) to the master instance. Similar to the setup in the image below:<\/p>\n This all works fine, until you hit a limit: These are the same problems engineers at youtube dealt with, and they came up with Vitess to solve these problems. Fortunately for us, they made it open source, so everyone can benefit from it.<\/p>\n Now you\u2019ll see a lot of features of Vitess, that will seem like a list of buzzwords, but in this case, they are actually true:<\/p>\n It can be clearly seen that, at scale, the above features become incredibly valuable and necessary.<\/p>\n Now, let\u2019s try it out \ud83d\ude42<\/p>\n Vitess has come a long way and now it is relatively easy to deploy a Vitess cluster with Kubernetes.<\/p>\n The plan is as follows:<\/p>\n You should be able to create all the resources needed for the outlined steps with a free trial from Google Cloud Platform. A trial from GCP has a limit of 8 cores per project, keep that in mind when deploying your resources.<\/p>\n There\u2019s already a good guide on how to deploy a Rancher 2.0 instance here<\/a>, thus let\u2019s not get into many details here. One note – if you want to run this guide on Google Cloud Platform trial, then create a separate project for the Rancher instace, thus saving yourself a CPU core for the Kubernetes cluster (remember the 8 cores limit per project for the GCP trial?)<\/p>\n Now onto starting a Kubernetes Cluster. For this, there\u2019s again a pretty good guide here<\/a>. One note – Vitess documentation recommends a cluster of 5 n1-standard-4 instances, that will get you over the GCP trial limit. But, if you create a 4 nodes of n1-standard-2 type, this will get you exactly to the 8 cores limit per project for GCP trial and it\u2019s enough for the outlined tasks. There will be some changes to the Vitess scripts, to deploy less resources, so you can fit needed resources into the smaller cluster.<\/p>\n The Create Cluster part from Rancher should look like in the below image:<\/p>\n Note the name of the cluster vitess-demo, it will be referenced further. Once you have your cluster created and active the first thing to do is to create a control instance, where you will have all the necessary tools for this Vitess demo. To do that, go to your newly created vitess-demo cluster, onto to the Default project and click on Deploy. Fill in there the details:<\/p>\n It should look like in the below image: Click confidently on Launch!<\/p>\n In a moment you should have your vitess-control pod running. It will be used as a control plane for your Vitess cluster and thanks to Rancher, you will be able to access it from anywhere you have an internet access and a browser.<\/p>\n Ok, let\u2019s get to installing the control part. To do that, click on the right hand side menu of vitess-control pod and select Execute shell, just like in the screenshot below:<\/p>\n This will bring you the go command line that we need. Let\u2019s start typing commands there. First build and install vtctlclient by running:<\/p>\n go get vitess.io\/vitess\/go\/cmd\/vtctlclient<\/p>\n When successfull, this command doesn\u2019t provide any output and it will download and build Vitess source code at this location: $GOPATH\/src\/vitess.io\/vitess\/, it will also copy the built vtctlclient binary to $GOPATH\/bin.<\/p>\n Now let\u2019s navigate to the Vitess source code directory and configure site-local settings with the following commands:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes It should look as below: At the moment Vitess has out-of-the-box support for backups in Google Cloud Storage. If you want, you can create a bucket in Google Cloud to test the backup in this demo, but it\u2019s not very important for the demo purposes.<\/p>\n There will also be a need to have kubectl configured on the vitess-control instance, to interact directly with the cluster. To do this, run the following commands on the vitess-control command line:<\/p>\n curl -LO https:\/\/storage.googleapis.com\/kubernetes-release\/release\/$(curl -s https:\/\/storage.googleapis.com\/kubernetes-release\/release\/stable.txt)\/bin\/linux\/amd64\/kubectl Now you need the kubeconfig file, that you can get from Rancher. You can get that from the vitess-demo default dashboard. On the upper right corner you should see Launch kubectl and Kubeconfig file. Click on Kubeconfig file and you\u2019ll be presented with the contents of the kubeconfig file. On the bottom you will have the Copy to Clipboard option, click on that. Now you need to paste it into ~\/.kube\/config file inside vitess-control instance. To do that, execute shell in vitess-control instance and issue the below commands there:<\/p>\n apt-get update Once you opened the file with vim, paste the kubeconfig file inside that file (don\u2019t forget to enter insert mode in vim, by just pressing the I key, before pasting the contents).<\/p>\n Confirm that you have kubectl working properly, by issuing the following command:<\/p>\n You should see the only pod running, the exact one from which you\u2019re running the commands – vitess-control<\/p>\n The Vitess topology service<\/a> stores coordination data for all the servers in a Vitess cluster. It can store this data in one of several consistent storage systems. In this example, we\u2019ll use etcd. This will require a separate etcd cluster, from the Kubernetes one.<\/p>\n To easily create an etcd cluster, let\u2019s install etcd-operator, using Rancher\u2019s Catalog Apps. Go to Catalog Apps, click on Launch, search for etcd-operator. Once you\u2019ve found etcd-operator, click on View Details and from the provided options, change only the namespace where etcd cluster will be deployed, because Vitess\u2019 etcd-cluster will be deployed in the default namespace and etcd-operator has to be in the same namespace. It should look like this:<\/p>\n Click on Lauch and wait a couple of moments for the etcd-operator to deploy. Verify that etcd-operator is running, by checking the Workloads tab and inside it, check the Service Discovery tab. If you see there etcd-operator as a workload and etcd-cluster with etcd-cluster-client as services, then everything is fine.<\/p>\n Now let\u2019s start etcd cluster for Vitess. If you\u2019re running this demo from a Google Cloud Platform trial account, then at this step you need to change the number of replicas started by the etcd cluster. To do that go to the Vitess control pod, Execute Shell and run the following command:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes ### example output Check that pods are running:<\/p>\n # kubectl get pod This one will be used to accept RPC commands from vtctlclient to modify the Vitess cluster. To install it, you\u2019ll also need your vitess-control command line and issue there the following commands:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes ### example output # To access vtctld web UI, start kubectl proxy in another terminal: At this point you can run vtctlclient from the vitess-control pod, to issue commands to the vtctld service on your Kubernetes cluster.<\/p>\n To enable RPC access into the Kubernetes cluster, you\u2019ll need to use kubectl to set up an authenticated tunnel.<\/p>\n Since the tunnel needs to target a particular vtctld pod name, guys at Vitess provided the kvtctl.sh script, which uses kubectl to discover the pod name and set up the tunnel before running vtctlclient. To test this, let\u2019s run .\/kvtctl.sh help from where we left, this will test your connection to vtctld and list the vtctlclient commands that you can use to administer the Vitess cluster.<\/p>\n .\/kvtctl.sh help<\/p>\n ### example output The global etcd cluster is configured from command-line parameters, specified in the Kubernetes configuration files. The per-cell etcd cluster however needs to be configured, so it is reachable by Vitess. Run the following commands from vitess-control pod to set it up:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes If successful you should only see a INFO message that tells you that your command is proxied to vtctld.<\/p>\n Now starting with the interesting parts. A Vitess tablet<\/a> is the unit of scaling for the database. A tablet consists of the vttablet and mysqld processes, running on the same host. We enforce this coupling in Kubernetes by putting the respective containers for vttablet and mysqld inside a single pod.<\/p>\n To start a vttablet, guys from Vitess provided a script that will launch 5 vttablet pods. This can be configured, and if you want to fit into the CPU limit provided by GCP trial, you need to change those values. How to do that, will be presented below.<\/p>\n Now go to vitess-control command line and run the following commands:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes ### example output At this point you should see tablet pods appearing in Rancher\u2019s Workloads tab, just like shown below:<\/p>\n You can also check the the status of the tablets from the vitess-control command line:<\/p>\n .\/kvtctl.sh ListAllTablets test Once all tablets are up and running, it\u2019s a good time to initialise a underlying MySQL database.<\/p>\n Take note that many vtctlclient commands produce no output on success, so the saying \u201cNo news, is good news\u201d is valid here too.<\/p>\n First, designate one of the tablets to be the initial master. Vitess will automatically connect the other slaves\u2019 mysqld instances so that they start replicating from the master\u2019s mysqld. This is also when the default database is created. Since our keyspace is named test_keyspace, the MySQL database will be named vt_test_keyspace. As you can see from the ListAllTablets output, there are 2 replica and one rdonly tablets. Let\u2019s designate the first tablet test-0000000100 as master (again, running from vitess-control command line):<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes ### example output Since this is the first time the shard has been started, the tablets are not doing any replication yet, and there is no existing master. That is why the InitShardMaster command above uses the -force flag to bypass the usual sanity checks that would apply if this wasn\u2019t a brand new shard.<\/p>\n Now you should be able to see one master, one replica and one rdonly (if you didn\u2019t change the counts in the scripts, you should see several replicas and rdonly pods):<\/p>\n .\/kvtctl.sh ListAllTablets test<\/p>\n ### example output The replica tablets are used for serving live web traffic, while the rdonly tablets are used for offline processing, such as batch jobs and backups.<\/p>\n Now it\u2019s starting to get shape and we can apply a schema to the databases. For this we can use the vtctlclient tool that can apply the database schema across all tablets in a keyspace. There\u2019s already a provided create_test_table.sql file from vitess on your vitess-control pod, let\u2019s apply it:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes The SQL that was applied is the following:<\/p>\n CREATE TABLE messages ( And it can be viewed like this:<\/p>\n .\/kvtctl.sh GetSchema test-0000000100<\/p>\n ### example output The example above we have just a database, with no specific configuration and we need to make that (empty) configuration visible for serving with the following command:<\/p>\n .\/kvtctl.sh RebuildVSchemaGraph<\/p>\n As this command doesn\u2019t produce any output on success, you\u2019re good if you don\u2019t receive any messages.<\/p>\n Vitess uses vtgate<\/a> to route each client query to the correct vttablet. In Kubernetes, a vtgate service distributes connections to a pool of vtgate pods. There\u2019s again a script ready to start vtgate and it can also be changed to start a lower number of pods, to fit the GCP trial limit. Now go to vitess-control command line and run the following commands:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes .\/vtgate-up.sh<\/p>\n ### example output After a couple of moments, you should see vtgate-test Workload in Rancher\u2019s dashboard.<\/p>\n Now comes the most awaited moment – testing your Vitess cluster with an App. Guys from Vitess provided that too, that is also already present on your vitess-control pod. It is a GuestBook example that uses Vitess as a backend database. Apply it from the vitess-control pod:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes ### example output At this moment you should be able to access the GuestBook app in your browser. You need to get the public IP address of it. There are at least 2 ways to do that: “`c ### example output Now you can access the GuestBook by pointing your browser to that IP address. It should look like this:<\/p>\n If you click on random page, it will generate a random page and insert it into the above created schema inside Vitess. Do that a couple of times and also post some messages to those pages, so we can view them afterwards in Vitess.<\/p>\n You can see Vitess\u2019 replication capabilities by opening the app in multiple browser windows, with the same Guestbook page number. Each new entry is committed to the master database. In the meantime, JavaScript on the page continuously polls the app server to retrieve a list of GuestBook entries. The app serves read-only requests by querying Vitess in \u2018replica\u2019 mode, confirming that replication is working.<\/p>\n Now let\u2019s see how it looks inside Vitess, go to vitess-control command line and issue the following commands:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes ### example output Now as you\u2019ve seen what Vitess can do on the surface, the next step would be to try Vitess resharding<\/a> or dynamic resharding<\/a>, just note that for these you\u2019ll need more resources than those provided by Google Cloud Platform trial limits.<\/p>\n To clean up all the demo pods and services you\u2019ve created, there are also scripts provided for that, that you can run from the vitess-control pod:<\/p>\n cd $GOPATH\/src\/vitess.io\/vitess\/examples\/kubernetes The last cleaning steps would be to delete the cluster from Rancher and then terminate the Rancher instance from the Google Cloud Platform console.<\/p>\n Vitess is a very powerful and very useful MySQL backend. Although it requires a little bit of effort to understand and properly set up, it offers a huge advantage for the future management of massive MySQL databases and it will scale easily. If Vitess is the default MySQL backend for giants like YouTube, it certainly has it\u2019s merits.<\/p>\n Rancher helped a lot in this demo to have everything in one place, I only had to have one Rancher instance to be able to do all the work, just by accessing it in a browser. The control instance was there, and all cluster details were there, just at a click distance.<\/p>\n Roman Doroschevici<\/p>\n![\"MySQL](\"https:\/\/rancher.com\/img\/blog\/2018\/mysql_default_setup.png\")
<\/p>\n
\n– when your hardware doesn\u2019t cope with the load and you need a more expensive hardware;
\n– you need to manually shard the database;
\n– cluster management becomes a growing pain;
\n– query monitoring is increasingly difficult.<\/p>\n\n
\n
\n
\n
\n
\n
\n
Deploy a Rancher 2.0 instance and start a Kubernetes Cluster<\/h3>\n
![\"Rancher](\"https:\/\/rancher.com\/img\/blog\/2018\/rancher-start-kubernetes-cluster.png\")
<\/p>\nStart a Vitess control pod<\/h4>\n
\n
\n![\"Vitess](\"https:\/\/rancher.com\/img\/blog\/2018\/rancher-deploy-vitess-control.png\")
<\/p>\n![\"Vitess](\"https:\/\/rancher.com\/img\/blog\/2018\/vitess-control-execute-shell.png\")
<\/p>\nBuild, install vtctlclient and configure settings<\/h4>\n
\n.\/configure.sh<\/p>\n
\n![\"Vitess](\"https:\/\/rancher.com\/img\/blog\/2018\/vitess-control-begin-steps.png\")
<\/p>\n
\nchmod +x .\/kubectl
\nmv .\/kubectl \/usr\/local\/bin\/kubectl<\/p>\n
\napt-get install -y vim
\nvim ~\/.kube\/config<\/p>\nStart Vitess etcd cluster<\/h4>\n
![\"etcd-operator](\"https:\/\/rancher.com\/img\/blog\/2018\/etcd-operator-default-namespace.png\")
<\/p>\n
\nsed -i ‘s\/ETCD_REPLICAS:-3\/ETCD_REPLICAS:-1\/g’ etcd-up.sh
\n.\/etcd-up.sh<\/p>\n
\n# Creating etcd service for ‘global’ cell…
\n# etcdcluster.etcd.database.coreos.com\/etcd-global created
\n# Creating etcd service for ‘test’ cell…
\n# etcdcluster.etcd.database.coreos.com\/etcd-test created<\/p>\n
\nNAME READY STATUS RESTARTS AGE
\netcd-cluster-4ftlwdxbrp 1\/1 Running 0 3m
\netcd-cluster-bjjsrfkm4x 1\/1 Running 0 3m
\netcd-cluster-cxqkghhjt4 1\/1 Running 0 2m
\netcd-global-qbnvtfd66b 0\/1 PodInitializing 0 10s
\netcd-operator-etcd-operator-etcd-operator-7d467bd7fb-kpkvw 1\/1 Running 0 3m
\netcd-test-x4pn5jw99s 0\/1 Running 0 10s
\nvitess-control-fdb84cbc4-7l2mp 1\/1 Running 0 1h
\n#<\/p>\nStart vtctld<\/h4>\n
\n.\/vtctld-up.sh<\/p>\n
\n# Creating vtctld ClusterIP service…
\n# service\/vtctld created
\n# Creating vtctld replicationcontroller…
\n# replicationcontroller\/vtctld created<\/p>\n
\n# kubectl proxy –port=8001
\n# Then visit http:\/\/localhost:8001\/api\/v1\/proxy\/namespaces\/default\/services\/vtctld:web\/<\/p>\nTry vtctlclient to send commands to vtctld<\/h4>\n
\n# Starting port forwarding to vtctld…
\n# Available commands:
\n#
\n# Tablets:
\n# InitTablet [-allow_update] [-allow_different_shard] [-allow_master_override] [-parent] [-db_name_override=<db name>] [-hostname=<hostname>] [-mysql_port=<port>] [-port=<port>] [-grpc_port=<port>] -keyspace=<keyspace> -shard=<shard> <tablet alias> <tablet type><\/p>\nSetup Vitess cell in the topology (Vitess\u2019s etcd cluster)<\/h4>\n
\n.\/kvtctl.sh AddCellInfo –root \/test -server_address http:\/\/etcd-test-client:2379 test<\/p>\nStart a Vitess tablet<\/h4>\n
\n### run below commands only if you want to change the vttable count from 5 to 3
\n### to fit into the GCP trial CPU limit
\nsed -i ‘s\/TABLETS_PER_SHARD:-5\/TABLETS_PER_SHARD:-3\/g’ vttablet-up.sh
\nsed -i ‘s\/RDONLY_COUNT:-2\/RDONLY_COUNT:-1\/g’ vttablet-up.sh
\n### end of vttablet count change
\n.\/vttablet-up.sh<\/p>\n
\n# Creating test_keyspace.shard-0 pods in cell test…
\n# Creating pod for tablet test-0000000100…
\n# pod\/vttablet-100 created
\n# Creating pod for tablet test-0000000101…
\n# pod\/vttablet-101 created
\n# Creating pod for tablet test-0000000102…
\n# pod\/vttablet-102 created<\/p>\n![\"vttablets](\"https:\/\/rancher.com\/img\/blog\/2018\/vttablets-pods.png\")
<\/p>\n
\n### example output
\n# Starting port forwarding to vtctld…
\n# test-0000000100 test_keyspace 0 replica 10.12.0.8:15002 10.12.0.8:3306 []
\n# test-0000000101 test_keyspace 0 replica 10.12.3.11:15002 10.12.3.11:3306 []
\n# test-0000000102 test_keyspace 0 rdonly 10.12.4.8:15002 10.12.4.8:3306 []<\/p>\nInitialize a MySQL Database<\/h4>\n
\n.\/kvtctl.sh InitShardMaster -force test_keyspace\/0 test-0000000100<\/p>\n
\n# Starting port forwarding to vtctld…
\n# W0817 22:29:32.961277 2530 main.go:60] W0817 22:29:32.958629 reparent.go:181] master-elect tablet test-0000000100 is not the shard master, proceeding anyway as -force was used
\n# W0817 22:29:32.961988 2530 main.go:60] W0817 22:29:32.959145 reparent.go:187] master-elect tablet test-0000000100 is not a master in the shard, proceeding anyway as -force was used<\/p>\n
\n# Starting port forwarding to vtctld…
\n# test-0000000100 test_keyspace 0 master 10.12.0.8:15002 10.12.0.8:3306 []
\n# test-0000000101 test_keyspace 0 replica 10.12.3.11:15002 10.12.3.11:3306 []
\n# test-0000000102 test_keyspace 0 rdonly 10.12.4.8:15002 10.12.4.8:3306 []<\/p>\nCreate a table inside Vitess Cluster<\/h4>\n
\n.\/kvtctl.sh ApplySchema -sql “$(cat create_test_table.sql)” test_keyspace<\/p>\n
\npage BIGINT(20) UNSIGNED,
\ntime_created_ns BIGINT(20) UNSIGNED,
\nmessage VARCHAR(10000),
\nPRIMARY KEY (page, time_created_ns)
\n) ENGINE=InnoDB<\/p>\n
\n# Starting port forwarding to vtctld…
\n# {
\n# “database_schema”: “CREATE DATABASE \/*!32312 IF NOT EXISTS*\/ {{.DatabaseName}} \/*!40100 DEFAULT CHARACTER SET utf8 *\/”,
\n# “table_definitions”: [
\n# {
\n# “name”: “messages”,
\n# “schema”: “CREATE TABLE `messages` (n `page` bigint(20) unsigned NOT NULL,n `time_created_ns` bigint(20) unsigned NOT NULL,n `message` varchar(10000) DEFAULT NULL,n PRIMARY KEY (`page`,`time_created_ns`)n) ENGINE=InnoDB DEFAULT CHARSET=utf8”,
\n# “columns”: [
\n# “page”,
\n# “time_created_ns”,
\n# “message”
\n# ],
\n# “primary_key_columns”: [
\n# “page”,
\n# “time_created_ns”
\n# ],
\n# “type”: “BASE TABLE”,
\n# “data_length”: “16384”,
\n# “row_count”: “0”
\n# }
\n# ],
\n# “version”: “5b2e5dbcb5766b6c69fe55c81b6ea805”
\n# }<\/p>\nInitialise Vitess routing schema<\/h4>\n
Start Vitess routing proxy – vtgate<\/h4>\n
\n### run below command only if you want to change the vtgate count from 3 to 1
\n### to fit into the GCP trial CPU limit
\nsed -i ‘s\/VTGATE_REPLICAS:-3\/VTGATE_REPLICAS:-1\/g’ vtgate-up.sh
\n### end of vttablet count change<\/p>\n
\n# Creating vtgate service in cell test…
\n# service\/vtgate-test created
\n# Creating vtgate replicationcontroller in cell test…
\n# replicationcontroller\/vtgate-test created<\/p>\nTest your cluster with an App<\/h4>\n
\n.\/guestbook-up.sh<\/p>\n
\n# Creating guestbook service…
\n# service\/guestbook created
\n# Creating guestbook replicationcontroller…
\n# replicationcontroller\/guestbook created<\/p>\n
\n– first one is to view it in Rancher, by going to Rancher\u2019s workloads, then select Load Balancer, there you should see the GuestBook load balancer. From it\u2019s right hand side menu, select View in API and then in the new opened page, search for PublicEndpoints -> Addresses, there you should have the public IP address of your GuestBook app.
\n– second one is to get it from the vitess-control command line, by running the following command:<\/p>\n
\nkubectl get service guestbook -o=jsonpath='{.status.loadBalancer.ingress[].ip}’<\/p>\n
\n# 35.197.201.4
\n“`<\/p>\n![\"Guestbook](\"https:\/\/rancher.com\/img\/blog\/2018\/guestbook-demo.png\")
<\/p>\n
\n.\/kvtctl.sh ExecuteFetchAsDba test-0000000100 “SELECT * FROM messages”<\/p>\n
\n# Starting port forwarding to vtctld…
\n# +——+———————+———–+
\n# | page | time_created_ns | message |
\n# +——+———————+———–+
\n# | 0 | 1534547955692433152 | Hello |
\n# | 0 | 1534547959825954816 | World |
\n# | 65 | 1534547943325885184 | And |
\n# | 65 | 1534547947058206208 | again |
\n# +——+———————+———–+<\/p>\nNext steps<\/h4>\n
Tear down the app and the clusters<\/h4>\n
\n.\/guestbook-down.sh
\n.\/vtgate-down.sh
\n.\/vttablet-down.sh
\n.\/vtctld-down.sh
\n.\/etcd-down.sh<\/p>\nSome final thoughts<\/h3>\n
![\"Roman](\"https:\/\/rancher.com\/img\/bio\/roman-doroschevici.jpg\")
<\/p>\n![\"github\"](\"https:\/\/rancher.com\/img\/icon-github.svg\"){kind=icon}
<\/a><\/p>\n