Author: Thomas Phelan (BlueData)<\/p>\n
KubeDirector is an open source project designed to make it easy to run complex stateful scale-out application clusters on Kubernetes. KubeDirector is built using the custom resource definition (CRD) framework and leverages the native Kubernetes API extensions and design philosophy. This enables transparent integration with Kubernetes user\/resource management as well as existing clients and tools.<\/p>\n
We recently introduced the KubeDirector project<\/a>, as part of a broader open source Kubernetes initiative we call BlueK8s. I\u2019m happy to announce that the pre-alpha KubeDirector provides the following capabilities:<\/p>\n KubeDirector enables data scientists familiar with data-intensive distributed applications such as Hadoop, Spark, Cassandra, TensorFlow, Caffe2, etc. to run these applications on Kubernetes \u2013 with a minimal learning curve and no need to write GO code. The applications controlled by KubeDirector are defined by some basic metadata and an associated package of configuration artifacts. The application metadata is referred to as a KubeDirectorApp resource.<\/p>\n To understand the components of KubeDirector, clone the repository on GitHub<\/a> using a command similar to:<\/p>\n git clone http:\/\/<userid>@github.com\/bluek8s\/kubedirector.<\/p>\n The KubeDirectorApp definition for the Spark 2.2.1 application is located ~> cat kubedirector\/deploy\/example_catalog\/cr-app-spark221e2.json The configuration of an application cluster is referred to as a KubeDirectorCluster resource. The ~> cat kubedirector\/deploy\/example_clusters\/cr-cluster-spark221.e1.yaml With KubeDirector, it\u2019s easy to run Spark clusters on Kubernetes.<\/p>\n First, verify that Kubernetes (version 1.9 or later) is running, using the command kubectl version<\/p>\n ~> kubectl version Deploy the KubeDirector service and the example KubeDirectorApp resource definitions with the commands:<\/p>\n cd kubedirector These will start the KubeDirector pod:<\/p>\n ~> kubectl get pods List the installed KubeDirector applications with kubectl get KubeDirectorApp<\/p>\n ~> kubectl get KubeDirectorApp Now you can launch a Spark 2.2.1 cluster using the example KubeDirectorCluster file and the ~> kubectl get pods The running services now include the Spark services:<\/p>\n ~> kubectl get service Pointing the browser at port 31533 connects to the Spark Master UI:<\/p>\n That\u2019s all there is to it! To start another application (e.g. Cassandra), just specify another KubeDirectorApp file:<\/p>\n kubectl create -f deploy\/example_clusters\/cr-cluster-cassandra311.yaml<\/p>\n See the running Cassandra cluster:<\/p>\n ~> kubectl get pods Now you have a Spark cluster (with a Jupyter notebook) and a Cassandra cluster running on Kubernetes. ~> kubectl get service KubeDirector is a fully open source, Apache v2 licensed, project \u2013 the first of multiple open source projects within a broader initiative we call BlueK8s.
\ncode for KubeDirector<\/a> is now available. And in this blog post, I\u2019ll show how it works.<\/p>\n\n
\nin the file kubedirector\/deploy\/example_catalog\/cr-app-spark221e2.json.<\/p>\n
\n{
\n“apiVersion”: “kubedirector.bluedata.io\/v1alpha1”,
\n“kind”: “KubeDirectorApp”,
\n“metadata”: {
\n“name” : “spark221e2”
\n},
\n“spec” : {
\n“systemctlMounts”: true,
\n“config”: {
\n“node_services”: [
\n{
\n“service_ids”: [
\n“ssh”,
\n“spark”,
\n“spark_master”,
\n“spark_worker”
\n],
\n\u2026<\/p>\n
\nKubeDirectorCluster definition for a sample Spark 2.2.1 cluster is located in the file
\nkubedirector\/deploy\/example_clusters\/cr-cluster-spark221.e1.yaml.<\/p>\n
\napiVersion: “kubedirector.bluedata.io\/v1alpha1”
\nkind: “KubeDirectorCluster”
\nmetadata:
\nname: “spark221e2”
\nspec:
\napp: spark221e2
\nroles:
\n– name: controller
\nreplicas: 1
\nresources:
\nrequests:
\nmemory: “4Gi”
\ncpu: “2”
\nlimits:
\nmemory: “4Gi”
\ncpu: “2”
\n– name: worker
\nreplicas: 2
\nresources:
\nrequests:
\nmemory: “4Gi”
\ncpu: “2”
\nlimits:
\nmemory: “4Gi”
\ncpu: “2”
\n– name: jupyter
\n\u2026<\/p>\nRunning Spark on Kubernetes with KubeDirector<\/h2>\n
\nClient Version: version.Info
\nServer Version: version.Info<\/p>\n
\nmake deploy<\/p>\n
\nNAME READY STATUS RESTARTS AGE
\nkubedirector-58cf59869-qd9hb 1\/1 Running 0 1m<\/p>\n
\nNAME AGE
\ncassandra311 30m
\nspark211up 30m
\nspark221e2 30m<\/p>\n
\nkubectl create -f deploy\/example_clusters\/cr-cluster-spark211up.yaml command.
\nVerify that the Spark cluster has been started:<\/p>\n
\nNAME READY STATUS RESTARTS AGE
\nkubedirector-58cf59869-djdwl 1\/1 Running 0 19m
\nspark221e2-controller-zbg4d-0 1\/1 Running 0 23m
\nspark221e2-jupyter-2km7q-0 1\/1 Running 0 23m
\nspark221e2-worker-4gzbz-0 1\/1 Running 0 23m
\nspark221e2-worker-4gzbz-1 1\/1 Running 0 23m<\/p>\n
\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
\nkubedirector ClusterIP 10.98.234.194 <none> 60000\/TCP 1d
\nkubernetes ClusterIP 10.96.0.1 <none> 443\/TCP 1d
\nsvc-spark221e2-5tg48 ClusterIP None <none> 8888\/TCP 21s
\nsvc-spark221e2-controller-tq8d6-0 NodePort 10.104.181.123 <none> 22:30534\/TCP,8080:31533\/TCP,7077:32506\/TCP,8081:32099\/TCP 20s
\nsvc-spark221e2-jupyter-6989v-0 NodePort 10.105.227.249 <none> 22:30632\/TCP,8888:30355\/TCP 20s
\nsvc-spark221e2-worker-d9892-0 NodePort 10.107.131.165 <none> 22:30358\/TCP,8081:32144\/TCP 20s
\nsvc-spark221e2-worker-d9892-1 NodePort 10.110.88.221 <none> 22:30294\/TCP,8081:31436\/TCP 20s<\/p>\n![\"kubedirector\"](\"https:\/\/d33wubrfki0l68.cloudfront.net\/5410ad39a3205e8470dac3bd0f36aa4c704713f9\/af429\/images\/blog\/2018-10-03-kubedirector\/kubedirector.png\")
<\/p>\n
\nIn fact, in the example above we also deployed a Jupyter notebook along with the Spark cluster.<\/p>\n
\nNAME READY STATUS RESTARTS AGE
\ncassandra311-seed-v24r6-0 1\/1 Running 0 1m
\ncassandra311-seed-v24r6-1 1\/1 Running 0 1m
\ncassandra311-worker-rqrhl-0 1\/1 Running 0 1m
\ncassandra311-worker-rqrhl-1 1\/1 Running 0 1m
\nkubedirector-58cf59869-djdwl 1\/1 Running 0 1d
\nspark221e2-controller-tq8d6-0 1\/1 Running 0 22m
\nspark221e2-jupyter-6989v-0 1\/1 Running 0 22m
\nspark221e2-worker-d9892-0 1\/1 Running 0 22m
\nspark221e2-worker-d9892-1 1\/1 Running 0 22m<\/p>\n
\nUse kubectl get service to see the set of services.<\/p>\n
\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
\nkubedirector ClusterIP 10.98.234.194 <none> 60000\/TCP 1d
\nkubernetes ClusterIP 10.96.0.1 <none> 443\/TCP 1d
\nsvc-cassandra311-seed-v24r6-0 NodePort 10.96.94.204 <none> 22:31131\/TCP,9042:30739\/TCP 3m
\nsvc-cassandra311-seed-v24r6-1 NodePort 10.106.144.52 <none> 22:30373\/TCP,9042:32662\/TCP 3m
\nsvc-cassandra311-vhh29 ClusterIP None <none> 8888\/TCP 3m
\nsvc-cassandra311-worker-rqrhl-0 NodePort 10.109.61.194 <none> 22:31832\/TCP,9042:31962\/TCP 3m
\nsvc-cassandra311-worker-rqrhl-1 NodePort 10.97.147.131 <none> 22:31454\/TCP,9042:31170\/TCP 3m
\nsvc-spark221e2-5tg48 ClusterIP None <none> 8888\/TCP 24m
\nsvc-spark221e2-controller-tq8d6-0 NodePort 10.104.181.123 <none> 22:30534\/TCP,8080:31533\/TCP,7077:32506\/TCP,8081:32099\/TCP 24m
\nsvc-spark221e2-jupyter-6989v-0 NodePort 10.105.227.249 <none> 22:30632\/TCP,8888:30355\/TCP 24m
\nsvc-spark221e2-worker-d9892-0 NodePort 10.107.131.165 <none> 22:30358\/TCP,8081:32144\/TCP 24m
\nsvc-spark221e2-worker-d9892-1 NodePort 10.110.88.221 <none> 22:30294\/TCP,8081:31436\/TCP 24m<\/p>\nGet Involved<\/h2>\n
\nThe pre-alpha code for KubeDirector has just been released and we would love for you to join the growing community of developers, contributors, and adopters.
\nFollow @BlueK8s<\/a> on Twitter and get involved through these channels:<\/p>\n\n