In this Hidden Gems blog post, Luke looks at the new volume snapshotting functionality in Kubernetes and how cluster administrators can use this feature to take and restore snapshots of their data.<\/p>\n
In Kubernetes 1.8, volume snapshotting has been released as a prototype. It is external to core Kubernetes whilst it is in the prototype phase, but you can find the project under the snapshot subdirectory of the kubernetes-incubator\/external-storage<\/a> repository. For a detailed explanation of the implementation of volume snapshotting, read the design proposal here<\/a>. The prototype currently supports<\/a> GCE PD, AWS EBS, OpenStack Cinder and Kubernetes hostPath<\/a> volumes. Note that aside from hostPath volumes, the logic for snapshotting a volume is implemented by cloud providers; the purpose of volume snapshotting in Kubernetes is to provide a common API for negotiating with different cloud providers in order to take and restore snapshots.<\/p>\n The best way to get an overview of volume snapshotting in Kubernetes is by going through an example. In this post, we are going to spin up a Kubernetes 1.8 cluster on GKE, deploy snapshot-controller and snapshot-provisioner and take and restore a snapshot of a GCE PD.<\/p>\n For reproducibility, I am using Git commit hash b1d5472a7b47777bf851cfb74bfaf860ad49ed7c of the kubernetes-incubator\/external-storage repository.<\/p>\n The first thing we need to do is compile and package both snapshot-controller and snapshot-provisioner into Docker containers. Make sure you have installed Go and configured your GOPATH correctly.<\/p>\n $ go get -d github.com\/kubernetes-incubator\/external-storage You can then use the following Dockerfiles. These will build both snapshot-controller and snapshot-provisioner. We run apk add –no-cache ca-certificates in order to add root certificates into the container images. To avoid using stale certificates, we could alternatively pass them into the containers by mounting the hostPath \/etc\/ssl\/certs to the same location in the containers.<\/p>\n FROM alpine:3.6<\/p>\n RUN apk add –no-cache ca-certificates<\/p>\n COPY _output\/bin\/snapshot-controller-linux-amd64 \/usr\/bin\/snapshot-controller<\/p>\n ENTRYPOINT [“\/usr\/bin\/snapshot-controller”]<\/p>\n FROM alpine:3.6<\/p>\n RUN apk add –no-cache ca-certificates<\/p>\n COPY _output\/bin\/snapshot-provisioner-linux-amd64 \/usr\/bin\/snapshot-provisioner<\/p>\n ENTRYPOINT [“\/usr\/bin\/snapshot-provisioner”]<\/p>\n $ docker build -t dippynark\/snapshot-controller:latest . -f Dockerfile.controller We will now create a cluster on GKE using gcloud.<\/p>\n $ gcloud container clusters create snapshot-demo –cluster-version 1.8.3-gke.0 Snapshotting requires two extra resources, VolumeSnapshot and VolumeSnapshotData. For an overview of the lifecyle of these two resources, take a look at the user guide<\/a> in the project itself. We will look at the functionality of each of these resources further down the page, but the first step is to register them with the API Server. This is done using CustomResourceDefinitions<\/a>. snapshot-controller will create a CustomeResourceDefinition for each of VolumeSnapshot and VolumeSnapshotData when it starts up so some of the work is taken care of for us. snapshot-controller will also watch for VolumeSnapshot resources and take snapshots of the volumes they reference. To allow us to restore our snapshots we will deploy snapshot-provisioner as well.<\/p>\n apiVersion: v1 In this case we have specified -cloudprovider=gce, but you can also use aws or openstack depending on your environment. For these other cloud providers there may be other parameters you need to set to configure the neccessary authorisation. Examples of how to do this can be found here<\/a>. hostPath is enabled by default, but requires you to run snapshot-controller and snapshot-provisioner on the same node as the hostPath volume that you want to snapshot and restore and should only be used on single node development clusters for testing purposes<\/a>. For an example of how to deploy snapshot-controller and snapshot-provisioner to take and restore hostPath volume snapshots for a particular directory, see here<\/a>. For a walkthrough of taking and restoring a hostPath volume snapshot see here<\/a>.<\/p>\n We have also defined a new ServiceAccount to which we have bound a custom ClusterRole<\/a>. This is only needed for RBAC<\/a> enabled clusters. If you have not enabled RBAC in your cluster, you can ignore the ServiceAccount, ClusterRole and ClusterRoleBinding and remove the serviceAccountName field from the snapshot-controller Deployment. If you have enabled RBAC in your cluster, notice that we have authorised the ServiceAccount to create, list, watch and delete CustomResourceDefinitions. This is so that snapshot-controller can set them up for our two new resources. Since snapshot-controller only needs these CustomResourceDefinition permissions temporarily on startup, it would be better to remove them and make administrators create the two CustomResourceDefinitions manually. Once snapshort-controller is running, you will be able to see the created CustomResourceDefinitions.<\/p>\n $ kubectl get crd To see the full definitions for these resources you can run kubectl get crd -o yaml. Note that VolumeSnapshot specifies a scope of Namespaced and VolumeSnapshotData is non namespaced. We can now interact with our new resource types.<\/p>\n $ kubectl get volumesnapshot,volumesnapshotdata Looking at the logs for both snapshot containers we can see that they are working correctly.<\/p>\n $ kubectl get pods -n kube-system Let\u2019s now create the PersistentVolumeClaim we are going to snapshot.<\/p>\n apiVersion: v1 Note that this is using the default StorageClass on GKE which will dynamically provision a GCE PD PersistentVolume. Let\u2019s now create a Pod that will create some data in the volume. We will take a snapshot of the data and restore it later.<\/p>\n apiVersion: v1 The Pod appends the current date and time to a file stored on our GCE PD every second. We can use cat to inspect the file.<\/p>\n $ kubectl exec -it busybox cat \/tmp\/pod-out.txt We are now ready to take a snapshot. Once we create the VolumeSnapshot resource below, snapshot-controller will attempt to create the actual snapshot by interacting with the configured cloud provider (GCE in our case). If successful, the VolumeSnapshot resource is bound to a corresponding VolumeSnapshotData resource. We need to reference the PersistentVolumeClaim that references the data we want to snapshot.<\/p>\n apiVersion: volumesnapshot.external-storage.k8s.io\/v1 $ kubectl create -f snapshot.yaml Notice the Snapshot Data Name field. This is a reference to the VolumeSnapshotData resource that was created by snapshot-controller when we created our VolumeSnapshot. The conditions towards the bottom of the output above show that our snapshot was created successfully. We can check snapshot-controller\u2019s logs to verify this.<\/p>\n $ kubectl logs snapshot-controller-66f7c56c4-ptjmb -n kube-system -c snapshot-controller We can also view the snapshot in GCE.<\/p>\n We can now look at the corresponding VolumeSnapshotData resource that was created.<\/p>\n $ kubectl get volumesnapshotdata Notice the reference to the GCE PD snapshot. It also references the VolumeSnapshot resource we created above and the PersistentVolume that the snapshot has been taken from. This was the PersistentVolume that was dynamically provisioned when we created our gcd-pvc PersistentVolumeClaim earlier. One thing to point out here is that snapshot-controller does not deal with pausing any applications that are interacting with the volume before the snapshot is taken, so the data may be inconsistent if you do not deal with this manually. This will be less of a problem for some applications than others.<\/p>\n The following diagram shows how the various resources discussed above reference each other. We can see how a VolumeSnapshot binds to a VolumeSnapshotData resource. This is analogous to PersistentVolumeClaims and PersistentVolumes. We can also see that VolumeSnapshotData references the actual snapshot taken by the volume provider, in the same way to how a PersistentVolume references the physical volume backing it.<\/p>\n Now that we have created a snapshot, we can restore it. To do this we need to create a special StorageClass implemented by snapshot-provisioner. We will then create a PersistentVolumeClaim referencing this StorageClass. An annotation on the PersistentVolumeClaim will inform snapshot-provisioner on where to find the information it needs to negotiate with the cloud provider to restore the snapshot. The StorageClass can be defined as follows.<\/p>\n kind: StorageClass Note the provisioner field which tells snapshot-provisioner it needs to implement the StorageClass. We can now create a PersistentVolumeClaim that will use the StorageClass to dynamically provision a PersistentVolume that contains the contents of our snapshot.<\/p>\n apiVersion: v1 Note the snapshot.alpha.kubernetes.io\/snapshot annotation which refers to the VolumeSnapshot we want to use. snapshot-provisioner can use this resource to get all the information it needs to perform the restore. We have also specified snapshot-promoter as the storageClassName which tells snapshot-provisioner that it needs to act. snapshot-provisioner will provision a PersistentVolume containing the contents of the snapshot-demo snapshot. We can see from the STORAGECLASS columns below that the snapshot-promoter StorageClass has been used.<\/p>\n $ kubectl get pvc Checking the snapshot-provisioner logs we can see that the snapshot was restored successfully.<\/p>\n $ kubectl logs snapshot-controller-66f7c56c4-ptjmb -n kube-system -c snapshot-provisioner Let\u2019s finally mount the busybox-snapshot PersistentVolumeClaim into a Pod to see that the snapshot was restored properly.<\/p>\n apiVersion: v1 We can use cat to see the data written to the volume by the busybox pod.<\/p>\n $ kubectl exec -it busybox-snapshot cat \/tmp\/pod-out.txt Notice that since the data is coming from a snapshot, the final date does not change if we run cat repeatedly.<\/p>\n $ kubectl exec -it busybox-snapshot cat \/tmp\/pod-out.txt Comparing the final date to the creation time of the snapshot in GCE, we can see that the snapshot took about 2 seconds to take.<\/p>\n We can delete the VolumeSnapshot resource which will also delete the corresponding VolumeSnapshotData resource and the snapshot in GCE. This will not affect any PersistentVolumeClaims or PersistentVolumes we have already provisioned using the snapshot. Conversely, deleting any PersistentVolumeClaims or PersistentVolumes that have been used to take a snapshot or have been provisioned using a snapshot will not delete the snapshot itself from GCE, however deleting the PersistentVolumeClaim or PersistentVolume that was used to take a snapshot will prevent you from restoring any further snapshots using snapshot-provisioner.<\/p>\n $ kubectl delete volumesnapshot snapshot-demo We should also delete the busybox Pods so they do not keep checking the date forever.<\/p>\n $ kubectl delete pods busybox busybox-snapshot For good measure we will also clean up the PersistentVolumeClaims and the cluster itself.<\/p>\n $ kubectl delete pvc busybox-snapshot gce-pvc Do you want to continue (Y\/n)? As usual, any GCE PDs you provisioned will not be deleted by deleting the cluster, so make sure to clear those up too if you do not want to be charged.<\/p>\n Although this project is in the early stages, you can instantly see its potential from this simple example and we will hopefully see support for other volume providers very soon as it matures. Together with CronJobs<\/a>, we now have the primitives we need within Kubernetes to perform automated backups of our data. For submitting any issues or project contributions, the best place to start is the external-storage issues tab<\/a>.<\/p>\n
\n$ cd $GOPATH\/src\/github.com\/kubernetes-incubator\/external-storage\/snapshot
\n$ # Checkout a fixed revision
\n$ #git checkout b1d5472a7b47777bf851cfb74bfaf860ad49ed7c
\n$ GOOS=linux GOARCH=amd64 go build -o _output\/bin\/snapshot-controller-linux-amd64 cmd\/snapshot-controller\/snapshot-controller.go
\n$ GOOS=linux GOARCH=amd64 go build -o _output\/bin\/snapshot-provisioner-linux-amd64 cmd\/snapshot-pv-provisioner\/snapshot-pv-provisioner.go<\/p>\n
\n$ docker build -t dippynark\/snapshot-provisioner:latest . -f Dockerfile.provisioner
\n$ docker push dippynark\/snapshot-controller:latest
\n$ docker push dippynark\/snapshot-provisioner:latest<\/p>\n
\nCreating cluster snapshot-demo…done.
\nCreated [https:\/\/container.googleapis.com\/v1\/projects\/jetstack-sandbox\/zones\/europe-west1-b\/clusters\/snapshot-demo].
\nkubeconfig entry generated for snapshot-demo.
\nNAME ZONE MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
\nsnapshot-demo europe-west1-b 1.8.3-gke.0 35.205.77.138 n1-standard-1 1.8.3-gke.0 3 RUNNING<\/p>\n
\nkind: ServiceAccount
\nmetadata:
\nname: snapshot-controller-runner
\nnamespace: kube-system
\n—
\nkind: ClusterRole
\napiVersion: rbac.authorization.k8s.io\/v1
\nmetadata:
\nname: snapshot-controller-role
\nrules:
\n– apiGroups: [“”]
\nresources: [“persistentvolumes”]
\nverbs: [“get”, “list”, “watch”, “create”, “delete”]
\n– apiGroups: [“”]
\nresources: [“persistentvolumeclaims”]
\nverbs: [“get”, “list”, “watch”, “update”]
\n– apiGroups: [“storage.k8s.io”]
\nresources: [“storageclasses”]
\nverbs: [“get”, “list”, “watch”]
\n– apiGroups: [“”]
\nresources: [“events”]
\nverbs: [“list”, “watch”, “create”, “update”, “patch”]
\n– apiGroups: [“apiextensions.k8s.io”]
\nresources: [“customresourcedefinitions”]
\nverbs: [“create”, “list”, “watch”, “delete”]
\n– apiGroups: [“volumesnapshot.external-storage.k8s.io”]
\nresources: [“volumesnapshots”]
\nverbs: [“get”, “list”, “watch”, “create”, “update”, “patch”, “delete”]
\n– apiGroups: [“volumesnapshot.external-storage.k8s.io”]
\nresources: [“volumesnapshotdatas”]
\nverbs: [“get”, “list”, “watch”, “create”, “update”, “patch”, “delete”]
\n—
\napiVersion: rbac.authorization.k8s.io\/v1
\nkind: ClusterRoleBinding
\nmetadata:
\nname: snapshot-controller
\nroleRef:
\napiGroup: rbac.authorization.k8s.io
\nkind: ClusterRole
\nname: snapshot-controller-role
\nsubjects:
\n– kind: ServiceAccount
\nname: snapshot-controller-runner
\nnamespace: kube-system
\n—
\napiVersion: apps\/v1beta1
\nkind: Deployment
\nmetadata:
\nname: snapshot-controller
\nnamespace: kube-system
\nspec:
\nreplicas: 1
\nstrategy:
\ntype: Recreate
\ntemplate:
\nmetadata:
\nlabels:
\napp: snapshot-controller
\nspec:
\nserviceAccountName: snapshot-controller-runner
\ncontainers:
\n– name: snapshot-controller
\nimage: dippynark\/snapshot-controller
\nimagePullPolicy: Always
\nargs:
\n– -cloudprovider=gce
\n– name: snapshot-provisioner
\nimage: dippynark\/snapshot-provisioner
\nimagePullPolicy: Always
\nargs:
\n– -cloudprovider=gce<\/p>\n
\nNAME AGE
\nvolumesnapshotdatas.volumesnapshot.external-storage.k8s.io 1m
\nvolumesnapshots.volumesnapshot.external-storage.k8s.io 1m<\/p>\n
\nNo resources found.<\/p>\n
\nNAME READY STATUS RESTARTS AGE
\n…
\nsnapshot-controller-66f7c56c4-h7cpf 2\/2 Running 0 1m
\n$ kubectl logs snapshot-controller-66f7c56c4-h7cpf -n kube-system -c snapshot-controller
\nI1104 11:38:53.551581 1 gce.go:348] Using existing Token Source &oauth2.reuseTokenSource, mu:sync.Mutex, t:(*oauth2.Token)(nil)}
\nI1104 11:38:53.553988 1 snapshot-controller.go:127] Register cloudprovider %sgce-pd
\nI1104 11:38:53.553998 1 snapshot-controller.go:93] starting snapshot controller
\nI1104 11:38:53.554050 1 snapshot-controller.go:168] Starting snapshot controller
\n$ kubectl logs snapshot-controller-66f7c56c4-h7cpf -n kube-system -c snapshot-provisioner
\nI1104 11:38:57.565797 1 gce.go:348] Using existing Token Source &oauth2.reuseTokenSource, mu:sync.Mutex, t:(*oauth2.Token)(nil)}
\nI1104 11:38:57.569374 1 snapshot-pv-provisioner.go:284] Register cloudprovider %sgce-pd
\nI1104 11:38:57.585940 1 snapshot-pv-provisioner.go:267] starting PV provisioner volumesnapshot.external-storage.k8s.io\/snapshot-promoter
\nI1104 11:38:57.586017 1 controller.go:407] Starting provisioner controller be8211fa-c154-11e7-a1ac-0a580a200004!<\/p>\n
\nkind: PersistentVolumeClaim
\nmetadata:
\nname: gce-pvc
\nspec:
\naccessModes:
\n– ReadWriteOnce
\nresources:
\nrequests:
\nstorage: 3Gi<\/p>\n
\nkind: Pod
\nmetadata:
\nname: busybox
\nspec:
\nrestartPolicy: Never
\ncontainers:
\n– name: busybox
\nimage: busybox
\ncommand:
\n– “\/bin\/sh”
\n– “-c”
\n– “while true; do date >> \/tmp\/pod-out.txt; sleep 1; done”
\nvolumeMounts:
\n– name: volume
\nmountPath: \/tmp
\nvolumes:
\n– name: volume
\npersistentVolumeClaim:
\nclaimName: gce-pvc<\/p>\n
\nSat Nov 4 11:41:30 UTC 2017
\nSat Nov 4 11:41:31 UTC 2017
\nSat Nov 4 11:41:32 UTC 2017
\nSat Nov 4 11:41:33 UTC 2017
\nSat Nov 4 11:41:34 UTC 2017
\nSat Nov 4 11:41:35 UTC 2017
\n$<\/p>\n
\nkind: VolumeSnapshot
\nmetadata:
\nname: snapshot-demo
\nspec:
\npersistentVolumeClaimName: gce-pvc<\/p>\n
\nvolumesnapshot “snapshot-demo” created
\n$ kubectl get volumesnapshot
\nNAME AGE
\nsnapshot-demo 18s
\n$ kubectl describe volumesnapshot snapshot-demo
\nName: snapshot-demo
\nNamespace: default
\nLabels: SnapshotMetadata-PVName=pvc-048bd424-c155-11e7-8910-42010a840164
\nSnapshotMetadata-Timestamp=1509796696232920051
\nAnnotations: <none>
\nAPI Version: volumesnapshot.external-storage.k8s.io\/v1
\nKind: VolumeSnapshot
\nMetadata:
\nCluster Name:
\nCreation Timestamp: 2017-11-04T11:58:16Z
\nGeneration: 0
\nResource Version: 2348
\nSelf Link: \/apis\/volumesnapshot.external-storage.k8s.io\/v1\/namespaces\/default\/volumesnapshots\/snapshot-demo
\nUID: 71256cf8-c157-11e7-8910-42010a840164
\nSpec:
\nPersistent Volume Claim Name: gce-pvc
\nSnapshot Data Name: k8s-volume-snapshot-7193cceb-c157-11e7-8e59-0a580a200004
\nStatus:
\nConditions:
\nLast Transition Time: 2017-11-04T11:58:22Z
\nMessage: Snapshot is uploading
\nReason:
\nStatus: True
\nType: Pending
\nLast Transition Time: 2017-11-04T11:58:34Z
\nMessage: Snapshot created successfully and it is ready
\nReason:
\nStatus: True
\nType: Ready
\nCreation Timestamp: <nil>
\nEvents: <none><\/p>\n
\n…
\nI1104 11:58:34.245845 1 snapshotter.go:239] waitForSnapshot: Snapshot default\/snapshot-demo created successfully. Adding it to Actual State of World.
\nI1104 11:58:34.245853 1 actual_state_of_world.go:74] Adding new snapshot to actual state of world: default\/snapshot-demo
\nI1104 11:58:34.245860 1 snapshotter.go:516] createSnapshot: Snapshot default\/snapshot-demo created successfully.<\/p>\n![\"gce](\"https:\/\/blog.jetstack.io\/blog\/kubernetes-1-8-hidden-gems-snapshotting\/gce-snapshot.png\")
<\/p>\n
\nNAME AGE
\nk8s-volume-snapshot-7193cceb-c157-11e7-8e59-0a580a200004 3m
\n$ kubectl describe volumesnapshotdata k8s-volume-snapshot-2a97d3f9-c155-11e7-8e59-0a580a200004
\nName: k8s-volume-snapshot-7193cceb-c157-11e7-8e59-0a580a200004
\nNamespace:
\nLabels: <none>
\nAnnotations: <none>
\nAPI Version: volumesnapshot.external-storage.k8s.io\/v1
\nKind: VolumeSnapshotData
\nMetadata:
\nCluster Name:
\nCreation Timestamp: 2017-11-04T11:58:17Z
\nDeletion Grace Period Seconds: <nil>
\nDeletion Timestamp: <nil>
\nResource Version: 2320
\nSelf Link: \/apis\/volumesnapshot.external-storage.k8s.io\/v1\/k8s-volume-snapshot-7193cceb-c157-11e7-8e59-0a580a200004
\nUID: 71a28267-c157-11e7-8910-42010a840164
\nSpec:
\nGce Persistent Disk:
\nSnapshot Id: pvc-048bd424-c155-11e7-8910-42010a8401641509796696237472729
\nPersistent Volume Ref:
\nKind: PersistentVolume
\nName: pvc-048bd424-c155-11e7-8910-42010a840164
\nVolume Snapshot Ref:
\nKind: VolumeSnapshot
\nName: default\/snapshot-demo
\nStatus:
\nConditions:
\nLast Transition Time: <nil>
\nMessage: Snapshot creation is triggered
\nReason:
\nStatus: Unknown
\nType: Pending
\nCreation Timestamp: <nil>
\nEvents: <none><\/p>\n![\"relationship](\"https:\/\/blog.jetstack.io\/blog\/kubernetes-1-8-hidden-gems-snapshotting\/relationship-diagram.png\")
<\/p>\n
\napiVersion: storage.k8s.io\/v1
\nmetadata:
\nname: snapshot-promoter
\nprovisioner: volumesnapshot.external-storage.k8s.io\/snapshot-promoter
\nparameters:
\ntype: pd-standard<\/p>\n
\nkind: PersistentVolumeClaim
\nmetadata:
\nname: busybox-snapshot
\nannotations:
\nsnapshot.alpha.kubernetes.io\/snapshot: snapshot-demo
\nspec:
\naccessModes:
\n– ReadWriteOnce
\nresources:
\nrequests:
\nstorage: 3Gi
\nstorageClassName: snapshot-promoter<\/p>\n
\nNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
\n…
\nbusybox-snapshot Bound pvc-8eed96e4-c157-11e7-8910-42010a840164 3Gi RWO snapshot-promoter 11s
\n…
\n$ kubectl get pv pvc-8eed96e4-c157-11e7-8910-42010a840164
\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
\npvc-8eed96e4-c157-11e7-8910-42010a840164 3Gi RWO Delete Bound default\/busybox-snapshot snapshot-promoter 21s<\/p>\n
\n…
\nProvisioning disk pvc-8eed96e4-c157-11e7-8910-42010a840164 from snapshot pvc-048bd424-c155-11e7-8910-42010a8401641509796696237472729, zone europe-west1-b requestGB 3 tags map[source:Created from snapshot pvc-048bd424-c155-11e7-8910-42010a8401641509796696237472729 -dynamic-pvc-8eed96e4-c157-11e7-8910-42010a840164]
\n…
\nI1104 11:59:10.563990 1 controller.go:813] volume “pvc-8eed96e4-c157-11e7-8910-42010a840164” for claim “default\/busybox-snapshot” created
\nI1104 11:59:10.987620 1 controller.go:830] volume “pvc-8eed96e4-c157-11e7-8910-42010a840164” for claim “default\/busybox-snapshot” saved
\nI1104 11:59:10.987740 1 controller.go:866] volume “pvc-8eed96e4-c157-11e7-8910-42010a840164” provisioned for claim “default\/busybox-snapshot”<\/p>\n
\nkind: Pod
\nmetadata:
\nname: busybox-snapshot
\nspec:
\nrestartPolicy: Never
\ncontainers:
\n– name: busybox
\nimage: busybox
\ncommand:
\n– “\/bin\/sh”
\n– “-c”
\n– “while true; do sleep 1; done”
\nvolumeMounts:
\n– name: volume
\nmountPath: \/tmp
\nvolumes:
\n– name: volume
\npersistentVolumeClaim:
\nclaimName: busybox-snapshot<\/p>\n
\nSat Nov 4 11:41:30 UTC 2017
\nSat Nov 4 11:41:31 UTC 2017
\nSat Nov 4 11:41:32 UTC 2017
\nSat Nov 4 11:41:33 UTC 2017
\nSat Nov 4 11:41:34 UTC 2017
\nSat Nov 4 11:41:35 UTC 2017
\n…
\nSat Nov 4 11:58:13 UTC 2017
\nSat Nov 4 11:58:14 UTC 2017
\nSat Nov 4 11:58:15 UTC 2017
\n$<\/p>\n
\n…
\nSat Nov 4 11:58:15 UTC 2017
\n$<\/p>\n
\nvolumesnapshot “snapshot-demo” deleted<\/p>\n
\npod “busybox” deleted
\npod “busybox-snapshot” deleted<\/p>\n
\npersistentvolumeclaim “busybox-snapshot” deleted
\npersistentvolumeclaim “gce-pvc” deleted
\n$ yes | gcloud container clusters delete snapshot-demo –async
\nThe following clusters will be deleted.
\n– [snapshot-demo] in [europe-west1-b]<\/p>\n
\n$<\/p>\n