Etcd is a highly available distributed key-value store that provides a reliable way to store data across machines, more importantly it is used as a Kubernetes\u2019 backing store for all of a cluster\u2019s data.<\/p>\n
In this post we are going to discuss how to backup etcd and how to recover from a backup to restore operations to a Kubernetes cluster.<\/p>\n
Etcd in Rancher 1.6<\/h2>\n
In Rancher 1.6 we use our own Docker image<\/a> for etcd which basically pulls the official etcd and adds some scripts and go binaries for orchestration, backup, disaster recovery, and healthcheck.<\/p>\n The scripts communicate with Rancher\u2019s metadata service to get important information, such as: how many etcd are running in the cluster, who is the etcd leader, etc. In Rancher 1.6, we introduced etcd backup, which works besides the main etcd in the background. This service is responsible for backup operations.<\/p>\n The backup operations work by performing rolling backups of etcd at specified intervals and also supports retention of old backups. Rancher-etcd does that by providing three environment variables to the Docker image:<\/p>\n Backups are taken at \/var\/etcd\/backups on the host and are taken using the following command:<\/p>\n etcdctl backup –data-dir <dataDir> –backup-dir <backupDir><\/p>\n To configure the backup operations for etcd in Rancher 1.6, you must supply the mentioned environment variables in the Kubernetes configuration template:<\/p>\n After configuring and launching Kubernetes, etcd should automatically take backups every 15 minutes by default.<\/p>\n Recovering etcd from a backup in rancher 1.6 requires the user to have data in the etcd volume created for etcd. For example, if you have 3 nodes and you have backups created in the \/var\/etcd\/backup directory:<\/p>\n # ls \/var\/etcd\/backups\/ -l Then you should be able to restore operations to etcd. First of all you should only start with one node, so that only one etcd will restore from backup, and then the rest of etcd will join the cluster. To begin the restoration, use the following steps:<\/p>\n target=2018-04-09T15:23:54Z_etcd_1 The next step is to start Kubernetes on this node normally:<\/p>\n After that you can add new hosts to the setup. Note that you have to make sure that new hosts don\u2019t have etcd volumes.<\/p>\n It\u2019s also preferable to have etcd backup mounted to NFS mount point so that if the hosts are down for any reason, it won\u2019t affect the backups created for etcd.<\/p>\n Recently Rancher announced GA for Rancher 2.0<\/a> and became ready for production deployments. Rancher 2.0 provides unified cluster management for different cloud providers including GKE, AKS, EKS as well providers that do not yet support a managed Kubernetes service.<\/p>\n Starting from RKE v0.1.7, the user is allowed to enable regular etcd snapshots automatically. In addition, it lets the user restore etcd from a snapshot stored on cluster instances.<\/p>\n In this section we will explain how to backup\/restore your Rancher installation on an RKE managed cluster. The steps for this kind of Rancher installation is explained in the official documentation<\/a> in more detail.<\/p>\n After you install Rancher using RKE as explained in the documentation, you should see similar output when you execute the command:<\/p>\n # kubectl get pods –all-namespaces You will notice that cattle pod is up and running in cattle-system namespace; this pod is the rancher server installed as a Kubernetes deployment:<\/p>\n RKE introduced two commands to save and restore etcd snapshots of a running RKE cluster; the two commands are:<\/p>\n rke etcd snapshot-save –config <config-path> –name <snapshot-name><\/p>\n AND<\/p>\n rke etcd snapshot-restore –config <config-path> –name <snapshot-name><\/p>\n For more information about etcd snapshot save\/restore in RKE, please refer to the official documentation<\/a>.<\/p>\n First we will take a snapshot of etcd that is running on the cluster. To do that, lets run the following command:<\/p>\n # rke etcd snapshot-save –name rancher.snapshot –config cluster.yml Assuming the Kubernetes cluster failed for any reason, we can restore normally from the taken snapshot, using the following command:<\/p>\n # rke etcd snapshot-restore –name rancher.snapshot –config cluster.yml<\/p>\n INFO[0000] Starting restoring snapshot on etcd hosts Notes After restoring the cluster, you have to restart the Kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in etcd; this will include restart to Kubernetes components and the network components. For more information, please refer to Kubernetes documentation<\/a>. To restart the Kubernetes components, you can run the following on each node:<\/p>\n docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy If you are restoring etcd on a cluster with multiple etcd nodes, the same exact snapshot must be copied to \/opt\/rke\/etcd-snapshots, rke etcd snapshot-save will take different snapshots on each node, so you will need to copy one of the created snapshots manually to all nodes before restoring.<\/p>\n Restoring etcd on a new Kubernetes cluster with new certificates is not currently supported, because the new cluster will contain different private keys which are used to sign service tokens for all service accounts. This may cause a lot of problems for all pods that communicate directly with kube api.<\/p>\n In this post we saw how backups can be created and restored for etcd in Kubernetes clusters in both Rancher 1.6.x and 2.0.x. Etcd snapshots can be managed in 1.6 using Rancher\u2019s etcd image and in 2.0 using RKE CLI.<\/p>\n Hussein Galal<\/p>\n DevOps Engineer<\/p>\n\n
Restoring backup<\/h2>\n
\ntotal 44
\ndrwx—— 3 root root 4096 Apr 9 15:03 2018-04-09T15:03:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:05 2018-04-09T15:05:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:07 2018-04-09T15:07:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:09 2018-04-09T15:09:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:11 2018-04-09T15:11:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:13 2018-04-09T15:13:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:15 2018-04-09T15:15:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:17 2018-04-09T15:17:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:19 2018-04-09T15:19:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:21 2018-04-09T15:21:54Z_etcd_1
\ndrwx—— 3 root root 4096 Apr 9 15:23 2018-04-09T15:23:54Z_etcd_1<\/p>\n
\ndocker volume create –name etcd
\ndocker run -d -v etcd:\/data –name etcd-restore busybox
\ndocker cp \/var\/etcd\/backups\/$target etcd-restore:\/data\/data.current
\ndocker rm etcd-restore<\/p>\nEtcd in Rancher 2.0<\/h2>\n
After Rancher Installation<\/h3>\n
\nNAMESPACE NAME READY STATUS RESTARTS AGE
\ncattle-system cattle-859b6cdc6b-tns6g 1\/1 Running 0 19s
\ningress-nginx default-http-backend-564b9b6c5b-7wbkx 1\/1 Running 0 25s
\ningress-nginx nginx-ingress-controller-shpn4 1\/1 Running 0 25s
\nkube-system canal-5xj2r 3\/3 Running 0 37s
\nkube-system kube-dns-5ccb66df65-c72t9 3\/3 Running 0 31s
\nkube-system kube-dns-autoscaler-6c4b786f5-xtj26 1\/1 Running 0 30s<\/p>\nRKE etcd Snapshots<\/h3>\n
\nINFO[0000] Starting saving snapshot on etcd hosts
\nINFO[0000] [dialer] Setup tunnel for host [x.x.x.x]
\nINFO[0003] [etcd] Saving snapshot [rancher.snapshot] on host [x.x.x.x]
\nINFO[0004] [etcd] Successfully started [etcd-snapshot-once] container on host [x.x.x.x]
\nINFO[0010] Finished saving snapshot [rancher.snapshot] on all etcd hosts<\/p>\nRKE etcd snapshot restore<\/h3>\n
\nINFO[0000] [dialer] Setup tunnel for host [x.x.x.x]
\nINFO[0001] [remove\/etcd] Successfully removed container on host [x.x.x.x]
\nINFO[0001] [hosts] Cleaning up host [x.x.x.x]
\nINFO[0001] [hosts] Running cleaner container on host [x.x.x.x]
\nINFO[0002] [kube-cleaner] Successfully started [kube-cleaner] container on host [x.x.x.x]
\nINFO[0002] [hosts] Removing cleaner container on host [x.x.x.x]
\nINFO[0003] [hosts] Successfully cleaned up host [x.x.x.x]
\nINFO[0003] [etcd] Restoring [rancher.snapshot] snapshot on etcd host [x.x.x.x]
\nINFO[0003] [etcd] Successfully started [etcd-restore] container on host [x.x.x.x]
\nINFO[0004] [etcd] Building up etcd plane..
\nINFO[0004] [etcd] Successfully started [etcd] container on host [x.x.x.x]
\nINFO[0005] [etcd] Successfully started [rke-log-linker] container on host [x.x.x.x]
\nINFO[0006] [remove\/rke-log-linker] Successfully removed container on host [x.x.x.x]
\nINFO[0006] [etcd] Successfully started etcd plane..
\nINFO[0007] Finished restoring snapshot [rancher.snapshot] on all etcd hosts<\/p>\n
\nThere are some important notes for the etcd restore process in RKE:<\/p><\/blockquote>\n1. Restarting Kubernetes components<\/h4>\n
\ndocker ps | grep flannel | cut -f 1 -d ” ” | xargs docker restart
\ndocker ps | grep calico | cut -f 1 -d ” ” | xargs docker restart<\/p>\n2. Restoring etcd on a multi-node cluster<\/h4>\n
3. Invalidated service account tokens<\/h4>\n
Conclusion<\/h2>\n
![\"Hussein](\"https:\/\/rancher.com\/img\/bio\/bio-user.jpg\")
<\/p>\n