WordPress is a popular platform for editing and publishing content for I\u2019ll now present an overview of the technologies we\u2019ll use and their The architecture is organized as shown below:<\/p>\n In Kubernetes, stateful sets offer a way to define the order of pod # storage-class.yaml Create the class and deploy with this # mysql-configmap.yaml Create the configmap and deploy with this # mysql-services.yaml With this service declaration, we lay the groundwork to have a multiple In this section, we\u2019ll be writing the YAML configuration for a MySQL The following configuration defines behavior for masters and slaves of apiVersion: apps\/v1beta1 # Determine binlog position of cloned data, if any. # Check if we need to complete a clone by starting replication. echo “Initializing replication from clone position” # Start a server to send backups when requested by peers. Save this file as mysql-statefulset.yaml. The final step in this procedure is to deploy our WordPress pods onto # nfs.yaml — — Deploy the NFS server using $ kubectl create -f nfs.yaml. Now, we need # wordpress.yaml —<\/p>\n apiVersion: v1 —<\/p>\n apiVersion: v1 —<\/p>\n apiVersion: apps\/v1beta1 # for versions before 1.8.0 use apps\/v1beta1 We\u2019ve now created a persistent volume claim that maps to the NFS OK, now that we\u2019ve deployed our services, let\u2019s start tearing them You\u2019ve now built and deployed an HA WordPress and MySQL installation on
\nthe web. In this tutorial, I\u2019m going to walk you through how to build
\nout a highly available (HA) WordPress deployment using Kubernetes.
\nWordPress consists of two major components: the WordPress PHP server,
\nand a database to store user information, posts, and site data. We need
\nto make both of these HA for the entire application to be fault
\ntolerant. Running HA services can be difficult when hardware and
\naddresses are changing; keeping up is tough. With Kubernetes and its
\npowerful networking components, we can deploy an HA WordPress site and
\nMySQL database without typing a single IP address (almost). In this
\ntutorial, I\u2019ll be showing you how to create storage classes, services,
\nconfiguration maps, and sets in Kubernetes; run HA MySQL; and hook up an
\nHA WordPress cluster to the database service. If you don\u2019t already have
\na Kubernetes cluster, you can spin one up easily on Amazon, Google, or
\nAzure, or by using Rancher Kubernetes Engine
\n(RKE)<\/a> on any servers.<\/p>\nArchitecture Overview<\/h2>\n
\nfunctions:<\/p>\n\n
\nDisk Backing<\/li>\n
\nStorage<\/li>\n
\nservice networking<\/li>\n<\/ul>\n![\"Diagram\"](\"https:\/\/i.imgur.com\/Urk4o79.png\")
<\/p>\nCreating Storage Classes, Services, and Configuration Maps in Kubernetes<\/h2>\n
\ninitialization. We\u2019ll use a stateful set for MySQL, because it ensures
\nour data nodes have enough time to replicate records from previous pods
\nwhen spinning up. The way we configure this stateful set will allow the
\nMySQL master to spin up before any of the slaves, so cloning can happen
\ndirectly from master to slave when we scale up. To start, we\u2019ll need to
\ncreate a persistent volume storage class and a configuration map to
\napply master and slave configurations as needed. We\u2019re using persistent
\nvolumes so that the data in our databases aren\u2019t tied to any specific
\npods in the cluster. This method protects the database from data loss in
\nthe event of a loss of the MySQL master pod. When a master pod is lost,
\nit can reconnect to the xtrabackup slaves on the slave nodes and
\nreplicate data from slave to master. MySQL\u2019s replication handles
\nmaster-to-slave replication but xtrabackup handles slave-to-master
\nbackward replication. To dynamically allocate persistent volumes, we
\ncreate the following storage class utilizing GCE Persistent Disks.
\nHowever, Kubernetes offers a variety of persistent volume storage
\nproviders:<\/p>\n
\nkind: StorageClass
\napiVersion: storage.k8s.io\/v1
\nmetadata:
\nname: slow
\nprovisioner: kubernetes.io\/gce-pd
\nparameters:
\ntype: pd-standard
\nzone: us-central1-a<\/p>\n
\ncommand: $ kubectl create -f storage-class.yaml. Next, we\u2019ll create
\nthe configmap, which specifies a few variables to set in the MySQL
\nconfiguration files. These different configurations are selected by the
\npods themselves, but they give us a handy way to manage potential
\nconfiguration variables. Create a YAML file named mysql-configmap.yaml
\nto handle this configuration as follows:<\/p>\n
\napiVersion: v1
\nkind: ConfigMap
\nmetadata:
\nname: mysql
\nlabels:
\napp: mysql
\ndata:
\nmaster.cnf: |
\n# Apply this config only on the master.
\n[mysqld]
\nlog-bin
\nskip-host-cache
\nskip-name-resolve
\nslave.cnf: |
\n# Apply this config only on slaves.
\n[mysqld]
\nskip-host-cache
\nskip-name-resolve<\/p>\n
\ncommand: $ kubectl create -f mysql-configmap.yaml. Next, we want to
\nset up the service such that MySQL pods can talk to one another and our
\nWordPress pods can talk to MySQL, using mysql-services.yaml. This also
\nenables a service load balancer for the MySQL service.<\/p>\n
\n# Headless service for stable DNS entries of StatefulSet members.
\napiVersion: v1
\nkind: Service
\nmetadata:
\nname: mysql
\nlabels:
\napp: mysql
\nspec:
\nports:
\n– name: mysql
\nport: 3306
\nclusterIP: None
\nselector:
\napp: mysql<\/p>\n
\nwrite, multiple read cluster of MySQL instances. This configuration is
\nnecessary because each WordPress instance can potentially write to the
\ndatabase, so each node must be ready to read and write. To create the
\nservices above, execute the following command:
\n$ kubectl create -f mysql-services.yaml At this point, we\u2019ve created
\nthe volume claim storage class which will hand persistent disks to all
\npods that request them, we\u2019ve configured the configmap that sets a few
\nvariables in the MySQL configuration files, and we\u2019ve configured a
\nnetwork-level service that will load balance requests to the MySQL
\nservers. This is all just framework for the stateful sets, where the
\nMySQL servers actually operate, which we\u2019ll explore next.<\/p>\nConfiguring MySQL with Stateful Sets<\/h2>\n
\ninstance using a stateful set. Let\u2019s define our stateful set:<\/p>\n\n
\nnamed init-mysql.<\/p>\n\n
\nnamed clone-mysql.<\/p>\n\n
\ncontainer.<\/li>\n
\nprevious peer.<\/li>\n
\nslave containers can pick it up.<\/li>\n<\/ul>\n<\/li>\n\n
\nthe MySQL master.<\/li>\n
\nconnect to the xtrabackup master.<\/li>\n<\/ul>\n<\/li>\n
\nas a 10GB persistent disk.<\/li>\n<\/ul>\n
\nour MySQL cluster, offering a bash configuration that runs the slave
\nclient and ensures proper operation of a master before cloning. Slaves
\nand masters each get their own 10GB volume which they request from the
\npersistent volume storage class we defined earlier.<\/p>\n
\nkind: StatefulSet
\nmetadata:
\nname: mysql
\nspec:
\nselector:
\nmatchLabels:
\napp: mysql
\nserviceName: mysql
\nreplicas: 3
\ntemplate:
\nmetadata:
\nlabels:
\napp: mysql
\nspec:
\ninitContainers:
\n– name: init-mysql
\nimage: mysql:5.7
\ncommand:
\n– bash
\n– “-c”
\n– |
\nset -ex
\n# Generate mysql server-id from pod ordinal index.
\n[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
\nordinal=$
\necho [mysqld] > \/mnt\/conf.d\/server-id.cnf
\n# Add an offset to avoid reserved server-id=0 value.
\necho server-id=$((100 + $ordinal)) >> \/mnt\/conf.d\/server-id.cnf
\n# Copy appropriate conf.d files from config-map to emptyDir.
\nif [[ $ordinal -eq 0 ]]; then
\ncp \/mnt\/config-map\/master.cnf \/mnt\/conf.d\/
\nelse
\ncp \/mnt\/config-map\/slave.cnf \/mnt\/conf.d\/
\nfi
\nvolumeMounts:
\n– name: conf
\nmountPath: \/mnt\/conf.d
\n– name: config-map
\nmountPath: \/mnt\/config-map
\n– name: clone-mysql
\nimage: gcr.io\/google-samples\/xtrabackup:1.0
\ncommand:
\n– bash
\n– “-c”
\n– |
\nset -ex
\n# Skip the clone if data already exists.
\n[[ -d \/var\/lib\/mysql\/mysql ]] && exit 0
\n# Skip the clone on master (ordinal index 0).
\n[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
\nordinal=$
\n[[ $ordinal -eq 0 ]] && exit 0
\n# Clone data from previous peer.
\nncat –recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C \/var\/lib\/mysql
\n# Prepare the backup.
\nxtrabackup –prepare –target-dir=\/var\/lib\/mysql
\nvolumeMounts:
\n– name: data
\nmountPath: \/var\/lib\/mysql
\nsubPath: mysql
\n– name: conf
\nmountPath: \/etc\/mysql\/conf.d
\ncontainers:
\n– name: mysql
\nimage: mysql:5.7
\nenv:
\n– name: MYSQL_ALLOW_EMPTY_PASSWORD
\nvalue: “1”
\nports:
\n– name: mysql
\ncontainerPort: 3306
\nvolumeMounts:
\n– name: data
\nmountPath: \/var\/lib\/mysql
\nsubPath: mysql
\n– name: conf
\nmountPath: \/etc\/mysql\/conf.d
\nresources:
\nrequests:
\ncpu: 500m
\nmemory: 1Gi
\nlivenessProbe:
\nexec:
\ncommand: [“mysqladmin”, “ping”]
\ninitialDelaySeconds: 30
\nperiodSeconds: 10
\ntimeoutSeconds: 5
\nreadinessProbe:
\nexec:
\n# Check we can execute queries over TCP (skip-networking is off).
\ncommand: [“mysql”, “-h”, “127.0.0.1”, “-e”, “SELECT 1”]
\ninitialDelaySeconds: 5
\nperiodSeconds: 2
\ntimeoutSeconds: 1
\n– name: xtrabackup
\nimage: gcr.io\/google-samples\/xtrabackup:1.0
\nports:
\n– name: xtrabackup
\ncontainerPort: 3307
\ncommand:
\n– bash
\n– “-c”
\n– |
\nset -ex
\ncd \/var\/lib\/mysql<\/p>\n
\nif [[ -f xtrabackup_slave_info ]]; then
\n# XtraBackup already generated a partial “CHANGE MASTER TO” query
\n# because we’re cloning from an existing slave.
\nmv xtrabackup_slave_info change_master_to.sql.in
\n# Ignore xtrabackup_binlog_info in this case (it’s useless).
\nrm -f xtrabackup_binlog_info
\nelif [[ -f xtrabackup_binlog_info ]]; then
\n# We’re cloning directly from master. Parse binlog position.
\n[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
\nrm xtrabackup_binlog_info
\necho “CHANGE MASTER TO MASTER_LOG_FILE=’$’,
\nMASTER_LOG_POS=$” > change_master_to.sql.in
\nfi<\/p>\n
\nif [[ -f change_master_to.sql.in ]]; then
\necho “Waiting for mysqld to be ready (accepting connections)”
\nuntil mysql -h 127.0.0.1 -e “SELECT 1”; do sleep 1; done<\/p>\n
\n# In case of container restart, attempt this at-most-once.
\nmv change_master_to.sql.in change_master_to.sql.orig
\nmysql -h 127.0.0.1 <<EOF
\n$(<change_master_to.sql.orig),
\nMASTER_HOST=’mysql-0.mysql’,
\nMASTER_USER=’root’,
\nMASTER_PASSWORD=”,
\nMASTER_CONNECT_RETRY=10;
\nSTART SLAVE;
\nEOF
\nfi<\/p>\n
\nexec ncat –listen –keep-open –send-only –max-conns=1 3307 -c
\n“xtrabackup –backup –slave-info –stream=xbstream –host=127.0.0.1 –user=root”
\nvolumeMounts:
\n– name: data
\nmountPath: \/var\/lib\/mysql
\nsubPath: mysql
\n– name: conf
\nmountPath: \/etc\/mysql\/conf.d
\nresources:
\nrequests:
\ncpu: 100m
\nmemory: 100Mi
\nvolumes:
\n– name: conf
\nemptyDir: {}
\n– name: config-map
\nconfigMap:
\nname: mysql
\nvolumeClaimTemplates:
\n– metadata:
\nname: data
\nspec:
\naccessModes: [“ReadWriteOnce”]
\nresources:
\nrequests:
\nstorage: 10Gi<\/p>\n
\nType kubectl create -f mysql-statefulset.yaml and let Kubernetes
\ndeploy your database. Now, when you call $ kubectl get pods, you
\nshould see three pods spinning up or ready that each have two containers
\non them. The master pod is denoted as mysql-0 and the slaves follow
\nas mysql-1 and mysql-2. Give the pods a few minutes to make sure
\nthe xtrabackup service is synced properly between pods, then move on
\nto the WordPress deployment. You can check the logs of the individual
\ncontainers to confirm that there are no error messages being thrown. To
\ndo this, run $ kubectl logs -f -c <container_name> The master
\nxtrabackup container should show the two connections from the slaves
\nand no errors should be visible in the logs.<\/p>\nDeploying Highly Available WordPress<\/h2>\n
\nthe cluster. To do this, we want to define a service for WordPress and a
\ndeployment. For WordPress to be HA, we want every container running the
\nserver to be fully replaceable, meaning we can terminate one and spin up
\nanother with no change to data or service availability. We also want to
\ntolerate at least one failed container, having a redundant container
\nthere to pick up the slack. WordPress stores important site-relevant
\ndata in the application directory \/var\/www\/html. For two instances of
\nWordPress to serve the same site, that folder has to contain identical
\ndata. When running WordPress in HA, we need to share
\nthe \/var\/www\/html folders between instances, so we\u2019ll define an NFS
\nservice that will be the mount point for these volumes. The following
\nconfiguration sets up the NFS services. I\u2019ve provided the plain English
\nversion below:<\/p>\n\n
\nGCE persistent disk at size 200GB.<\/li>\n
\nat least one instance of the NFS server is running at all times.<\/li>\n
\nshare accessible.<\/li>\n
\nserver.<\/li>\n
\n# Define the persistent volume claim
\napiVersion: v1
\nkind: PersistentVolumeClaim
\nmetadata:
\nname: nfs
\nlabels:
\ndemo: nfs
\nannotations:
\nvolume.alpha.kubernetes.io\/storage-class: any
\nspec:
\naccessModes: [ “ReadWriteOnce” ]
\nresources:
\nrequests:
\nstorage: 200Gi<\/p>\n
\n# Define the Replication Controller
\napiVersion: v1
\nkind: ReplicationController
\nmetadata:
\nname: nfs-server
\nspec:
\nreplicas: 1
\nselector:
\nrole: nfs-server
\ntemplate:
\nmetadata:
\nlabels:
\nrole: nfs-server
\nspec:
\ncontainers:
\n– name: nfs-server
\nimage: gcr.io\/google_containers\/volume-nfs:0.8
\nports:
\n– name: nfs
\ncontainerPort: 2049
\n– name: mountd
\ncontainerPort: 20048
\n– name: rpcbind
\ncontainerPort: 111
\nsecurityContext:
\nprivileged: true
\nvolumeMounts:
\n– mountPath: \/exports
\nname: nfs-pvc
\nvolumes:
\n– name: nfs-pvc
\npersistentVolumeClaim:
\nclaimName: nfs<\/p>\n
\n# Define the Service
\nkind: Service
\napiVersion: v1
\nmetadata:
\nname: nfs-server
\nspec:
\nports:
\n– name: nfs
\nport: 2049
\n– name: mountd
\nport: 20048
\n– name: rpcbind
\nport: 111
\nselector:
\nrole: nfs-server<\/p>\n
\nto run $ kubectl describe services nfs-server to gain the IP address
\nto use below. Note: In the future, we\u2019ll be able to tie these
\ntogether using the service names, but for now, you have to hardcode the
\nIP address.<\/p>\n
\napiVersion: v1
\nkind: Service
\nmetadata:
\nname: wordpress
\nlabels:
\napp: wordpress
\nspec:
\nports:
\n– port: 80
\nselector:
\napp: wordpress
\ntier: frontend
\ntype: LoadBalancer<\/p>\n
\nkind: PersistentVolume
\nmetadata:
\nname: nfs
\nspec:
\ncapacity:
\nstorage: 20G
\naccessModes:
\n– ReadWriteMany
\nnfs:
\n# FIXME: use the right IP
\nserver: <IP of the NFS Service>
\npath: “\/”<\/p>\n
\nkind: PersistentVolumeClaim
\nmetadata:
\nname: nfs
\nspec:
\naccessModes:
\n– ReadWriteMany
\nstorageClassName: “”
\nresources:
\nrequests:
\nstorage: 20G<\/p>\n
\nkind: Deployment
\nmetadata:
\nname: wordpress
\nlabels:
\napp: wordpress
\nspec:
\nselector:
\nmatchLabels:
\napp: wordpress
\ntier: frontend
\nstrategy:
\ntype: Recreate
\ntemplate:
\nmetadata:
\nlabels:
\napp: wordpress
\ntier: frontend
\nspec:
\ncontainers:
\n– image: wordpress:4.9-apache
\nname: wordpress
\nenv:
\n– name: WORDPRESS_DB_HOST
\nvalue: mysql
\n– name: WORDPRESS_DB_PASSWORD
\nvalue: “”
\nports:
\n– containerPort: 80
\nname: wordpress
\nvolumeMounts:
\n– name: wordpress-persistent-storage
\nmountPath: \/var\/www\/html
\nvolumes:
\n– name: wordpress-persistent-storage
\npersistentVolumeClaim:
\nclaimName: nfs<\/p>\n
\nservice we created earlier. It then attaches the volume to the WordPress
\npod at the \/var\/www\/html root, where WordPress is installed. This
\npreserves all installation and environments across WordPress pods in the
\ncluster. With this configuration, we can spin up and tear down any
\nWordPress node and the data will remain. Because the NFS service is
\nconstantly using the physical volume, it will retain the volume and
\nwon\u2019t recycle or misallocate it. Deploy the WordPress instances
\nusing $ kubectl create -f wordpress.yaml. The default deployment only
\nruns a single instance of WordPress, so feel free to scale up the number
\nof WordPress instances
\nusing $ kubectl scale –replicas=<number of replicas> deployment\/wordpress.
\nTo obtain the address of the WordPress service load balancer,
\ntype $ kubectl get services wordpress and grab the EXTERNAL-IP field
\nfrom the result to navigate to WordPress.<\/p>\nResilience Testing<\/h2>\n
\ndown to see how well our HA architecture handles some chaos. In this
\napproach, the only single point of failure left is the NFS service (for
\nreasons explained in the Conclusion). You should be able to demonstrate
\ntesting the failure of any other services to see how the application
\nresponds. I\u2019ve started with three replicas of the WordPress service and
\nthe one master and two slaves on the MySQL service. First, let\u2019s kill
\nall but one WordPress node and see how the application reacts:
\n$ kubectl scale –replicas=1 deployment\/wordpress Now, we should see a
\ndrop in pod count for the WordPress deployment. $ kubectl get pods We
\nshould see that the WordPress pods are running only 1\/1 now. When
\nhitting the WordPress service IP, we\u2019ll see the same site and same
\ndatabase as before. To scale back up, we can
\nuse $ kubectl scale –replicas=3 deployment\/wordpress. We\u2019ll again
\nsee that data is preserved across all three instances. To test the MySQL
\nStatefulSet, we can scale down the number of replicas using the
\nfollowing: $ kubectl scale statefulsets mysql –replicas=1 We\u2019ll see
\na loss of both slaves in this instance and, in the event of a loss of
\nthe master in this moment, the data it has will be preserved on the GCE
\nPersistent Disk. However, we\u2019ll have to manually recover the data from
\nthe disk. If all three MySQL nodes go down, you\u2019ll not be able to
\nreplicate when new nodes come up. However, if a master node goes down, a
\nnew master will be spun up and via xtrabackup, it will repopulate with
\nthe data from a slave. Therefore, I don\u2019t recommend ever running with a
\nreplication factor of less than three when running production databases.
\nTo conclude, let\u2019s talk about some better solutions for your stateful
\ndata, as Kubernetes isn\u2019t really designed for state.<\/p>\nConclusions and Caveats<\/h2>\n
\nKubernetes! Despite this great achievement, your journey may be far from
\nover. If you haven\u2019t noticed, our installation still has a single point
\nof failure: the NFS server sharing the \/var\/www\/html directory between
\nWordPress pods. This service represents a single point of failure
\nbecause without it running, the html folder disappears on the pods
\nusing it. The image I\u2019ve selected for the server is incredibly stable
\nand production ready, but for a true production deployment, you may
\nconsider
\nusing GlusterFS<\/a> to
\nenable multi-read multi-write to the directory shared by WordPress
\ninstances. This process involves running a distributed storage cluster
\non Kubernetes, which isn\u2019t really what Kubernetes is built for, so
\ndespite it working<\/em>, it isn\u2019t a great option for long-term
\ndeployments. For the database, I\u2019d personally recommend using a managed
\nRelational Database service to host the MySQL instance, be it Google\u2019s
\nCloudSQL or AWS\u2019s RDS, as they provide HA and redundancy at a more
\nsensible price and keep you from worrying about data integrity.
\nKubernetes isn\u2019t really designed around stateful applications and any
\nstate built into it is more of an afterthought. Plenty of solutions
\nexist that offer much more of the assurances one would look for when
\npicking a database service. That being said, the configuration presented
\nabove is a labor of love, a hodgepodge of Kubernetes tutorials and
\nexamples found across the web to create a cohesive, realistic use case
\nfor Kubernetes and all the new features in Kubernetes 1.8.x. I hope your
\nexperiences deploying WordPress and MySQL using the guide I\u2019ve prepared
\nfor you are a bit less exciting than the ones I had ironing out bugs in
\nthe configurations, and of course, I wish you eternal uptime. That\u2019s
\nall for now. Tune in next time when I teach you to drive a boat using
\nonly a Myo gesture band and a cluster of Linode instances running Tails
\nLinux.<\/p>\nAbout the Author<\/h3>\n
![\"\"](\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2017\/10\/02185842\/Eric-300x300.png\")
Eric Volpert is a
\nstudent at the University of Chicago and works as an evangelist, growth
\nhacker, and writer for Rancher Labs. He enjoys any engineering
\nchallenge. He\u2019s spent the last three summers as an internal tools
\nengineer at Bloomberg and a year building DNS services for the Secure
\nDomain Foundation with CrowdStrike. Eric enjoys many forms of music
\nranging from EDM to High Baroque, playing MOBAs and other action-packed
\ngames on his PC, and late-night hacking sessions, duct taping APIs
\ntogether so he can make coffee with a voice command.<\/p>\n