Art2Dec SoftLab

We are excited to announce the release of Ark v0.9.0! This release brings two major new features: integration with restic to back up almost every type of Kubernetes volume, and initial support for exporting metrics in the Prometheus data format. There is also a critical bug fix to avoid potential backup/restore data corruption, so we encourage you to upgrade.

Volume snapshots with restic

With Ark v0.9.0, it’s now possible to snapshot almost every type of Kubernetes volume (hostPath is not supported). This feature complements Ark’s ability to snapshot disks from AWS, Google, and Azure (as well as any custom plugin integrations, such as PortWorx). You could use native snapshotting for these types of disks and restic snapshots for everything else, or you could use restic for everything. Ark’s restic integration is a good option if your type of PersistentVolume doesn’t currently have a plugin for Ark, or if it doesn’t offer a native snapshot concept, such as emptyDir and NFS.

For more details, including setup instructions, see our restic documentation and our initial annoucement.

Prometheus metrics

We’d like to offer a huge thanks to community contributor Ashish Amarnath for providing the initial support for exposing Prometheus metrics. Ark v0.9.0 includes the following metrics:

• total # of backup attempts
• # of successful backups
• # of failed backups
• backup duration histogram
• backup tarball size

All of these metrics are grouped by schedule.

We are working to add additional metrics for backups as well as restores. If you’re interested in discussing what kinds of metrics you’d like us to add, please feel free to share your thoughts on our open issue.

Additional v0.9.0 highlights

Ark now restores any image pull secrets or other secrets that you added to the default service account.

Ark also automatically backs up all cluster roles and cluster role bindings that reference a given service account. If you’re backing up one or more specific namespaces and not including all cluster-scoped resources, this feature ensures your backup isn’t missing any relevant cluster-scoped RBAC resources.

This release includes several other improvements and bug fixes:

• Ark no longer tries to restore completed jobs, completed pods, or mirror pods
• Ark no longer backs up terminating resources
• Ark no longer backs up the same replica set or daemon set twice
• Ark no longer restores a PV with a reclaim policy of “delete” when there is no associated snapshot
• Ark works more smoothly with OpenShift
• We have improved our error handling, especially when backing up pods, their PVCs, and the associated PVs; as well as marking a backup as “failed” when uploading to Google Cloud Storage fails
• All logging from the Ark server now writes to stdout instead of stderr

See the release notes for full details on all improvements and bug fixes.

What’s next?

We are actively working on designing our next big feature — replication. This will ensure that your backed up Kubernetes resources and your persistent data are available in multiple locations, avoiding single points of failure.

We are also continuing to plot the roadmap to Ark 1.0. We’ll be discussing our plans with the community in the following weeks and encourage you to join the our Google group and Slack channel.

Finally, if you’re interested in contributing, you’ll find several GitHub issues labeled as Good First Issue and Help Wanted. Take a look — we would welcome your participation.

Source

Settled In as Their Marketing Guy

The Organizational Structure

To repeat, Giant Swarm is a remote-first company and with that, has a remote-first culture and organizational structure. This offers a certain kind of flexibility and pace of work that someone fresh off the agency boat has to come to terms with. That doesn’t mean taking massive breaks and getting work done when you want. It means work smart, rest, repeat. And get the job done. It is a style and culture of working that I don’t think would work for everyone.

Flexibility of Work

Working in a remote-first company offers the kind of flexibility owning your own company has without having to worry about the insane cost of insurance or answering the phone on Christmas day. This kind of flexibility gives you time to work on what you need to get done while bettering yourself in any number of different ways.

Here are some examples from seeing the daily AFK on #Slack:

• Piano lessons
• Travel to hometown
• German lessons
• Daughter’s track and field competition
• D&D
• Jazz festival

And then there are your basic TO DO’s and WTFs:

• Grocery shopping
• Kid is sick
• Flooded basement
• Lost my favorite keychain in Gdańsk, going back

This means when you are in front of the screen hacking away, your mind is clearer than wondering how am I going to get my keychain back or I need to pick up my son from piano lessons and I’m going to catch flack for it.

Pace of Team and Tech

I’ve always believed in move at the pace of client. This means that if the client is slow and they need more time to deliberate over content or a logo or don’t often have time for meetings, you have to keep cool and understand while still getting meaningful work done. You also have the clients who need everything done yesterday and you have to move mountains on a regular basis. The pace at Giant Swarm is both Client and Technology, so it adds a very exciting layer to the mix. The tech is rather new and growing in spades daily. This sets a different pace – it’s on the tech side, which means if you’re a couple of days late with info and updates, you’re the one behind. So content always needs to be fresh.

The Team

Meeting the team takes time. Working remotely, I may not run into my colleague from London or one of the two from Barcelona but maybe 3 times per year. And although Brno is now on my list of places to visit, I can’t just hop up and go for a beer on a Wednesday evening to the Czech Republic. The onboarding process at Giant Swarm consists of 1 on 1’s with everyone from the company. This was something that we needed to set up as new employees and had to get through everyone within the first couple of weeks. Just like any company, you’re going to have your different types of people but when the team represents 17 different countries, you have about 17 different types of people – even if in some regions you know you’d fit right in.

With technology as impressive as is is today, using Google Meet and keeping in close contact on #Slack, GitHub and email allows for very easy cross-departmental communication as well as cross-cultural communication. In conclusion, working remotely with others who thrive in this environment is a team experience like no other, and I think it’ll be adopted even more over the years.

Team building

Giant Swarm does team building gatherings twice a year, and of course they are a blast. But not focusing on all of the super-fun things we do, to be a remote-first company, you have to have close interaction with your colleagues whenever possible and these gatherings allow for that. The schedule is mixed with having fun as well as brainstorming sessions to better our processes and product. A very well organized trip to get everyone together is coordinated internally and the schedule is very full but of course, flexible based on what everyone wants to do or wants to work on. These are very necessary and have kept the team close although we are spread out across Europe.

Day 100

Conclusion: The Marketer at a Remote-First, Tech Startup

After 100 days at Giant Swarm, I have learned first and foremost about the processes and tech we have adopted but also heavily about working in a remote-first company, the kinds of people who thrive in them and the amazing channels that foster seamless communication across team members. But it doesn’t stop here, we are moving into what I call Phase II of our marketing strategy. This is going to include your big rock projects like website redesign, expansion of our reach through press contacts and the overbearing GDPR.

In the meantime, I still need to clean out my closet and get rid of my leather shoes, ties, sport coats and jackets, button-down dress shirts, and pleated khakis – just kidding, I’ve never owned a pair of those. Remote-first working doesn’t need these things and I feel right at home, literally.

Source

Feb 13, 2018  By Adam Sandor

I recently encountered an interesting problem while terraforming Kubernetes clusters on Google Cloud. Our Terraform configurations have some important information which is needed by applications running on the cluster. These are either values for which our Terraform resources are the primary source, or values which are outputs from them.

Here are some examples of the types of values we want to propagate from Terraform resources to application Pods running on Kubernetes:

• Name of the GKE (Google Kubernetes Engine) cluster – specified by Terraform.
• Endpoint of the Cloud SQL database – generated by Terraform
• Static Client IP address – generated by GCP (Google Cloud Platform)

These values are all available while Terraform is running but hard or impossible to access from a Pod running on the Kubernetes cluster. Most surprisingly the cluster name is not available in any kind of metadata on Google cloud but it has to be used when submitting custom container metrics to Google Stackdriver.

To propagate these values conveniently to our application Pods we can use the Terraform Kubernetes provider to create ConfigMaps or Secrets (depending on the sensitivity of the information) in Kubernetes. Applications running on the cluster can then have the values injected as environment variables or files.

The Kubernetes provider is quite new and in a general sense it’s questionable whether you should be provisioning Kubernetes objects using Terraform. You can read more on this from Harshal Shah in this post: http://blog.infracloud.io/terraform-helm-kubernetes/. For our use case however using the Kubernetes provider is perfect.

I was worried about the fact that we have to provision the Kubernetes cluster itself (using the Google Cloud provider container cluster resource), and use the credentials this resource outputs to set up the Kubernetes provider. This kind of inter-provider dependency is somewhat undefined in Terraform but it turns out to work perfectly.

Here are snippets from the code used to do all this.

Google Cloud provider setup and the Kubernetes cluster resource definition:

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

provider “google” { region = “europe-west3” credentials = “itsasecret” project = “adamsproject” } resource “google_container_cluster” “mycluster” { project = “adamsproject” name = “mycluster” zone = “europe-west3-a” initial_node_count = 1 node_version = “$” min_master_version = “$” node_config { machine_type = “g1-small” disk_size_gb = 50 } }

 

The Kubernetes provider is configured using outputs from the google_container_cluster resource – namely the CA certificate, the client certificate and the client secret key. We have to base64 decode these.

 

 

provider “kubernetes” { host = “$” client_certificate = “$” client_key = “$” cluster_ca_certificate = “$” }

 

Once the Kubernetes provider can access our new cluster, this resource definition will create a Secret with our database secrets:

 

 

resource “kubernetes_secret” “cloudsql-db-credentials” { “metadata” { name = “cloudsql-db-credentials” } data { username = “$” password = “$” } }

 

We can also create some ConfigMaps to hold other, less sensitive information. Here I’m creating a database connection string to be used by Google CloudSQL Proxy:

 

 

resource “kubernetes_config_map” “dbconfig” { “metadata” { name = “dbconfig” } data = { dbconnection = “$:$:$” } }

 

I’m not going into the details of how to get this configuration all the way to your application components as that is straightforward once you have ConfigMaps or Secrets created. Take a look at the Kubernetes documentation if you are not familiar with working with ConfigMaps or Secrets.

Summary

Terraform will inevitably become your primary source of truth for many variables in your environment. It will also know about variables that are set by the cloud provider during resource creation, like a random IP address. It’s a very elegant way to use Kubernetes Secrets and ConfigMaps to pass these values to your application components which need them, while the Kubernetes provider in Terraform offers a perfect way to create these objects.

Source

How to Manage Workloads on Kubernetes with Rancher 2.0 Online Meetup

Learn how to deploy Kubernetes applications in Rancher 2.0, and use the monitoring, logging and pipeline features.

Watch the video

Introduction

One of the biggest challenges with Kubernetes is bringing up a cluster for the
first time. There have been several projects that attempt to address this gap
including kmachine and
minikube. However, both assume
you are starting from a blank slate. What if you have your instances provisioned,
or have another means of provisioning hosts with configuration management?

This is the problem that the people at Rancher are solving with the
Rancher Kubernetes Engine or RKE.
At the core is a YAML based configuration file that is used to define the hosts
and make-up for your Kubernetes cluster. RKE will do the heavy lifting of
installing the required Kubernetes components, and configuring them for your
cluster.

What this tutorial will show you is how to quickly set up a few virtual systems
that can host our Kubernetes cluster, bring up Kubernetes
using RKE, and lastly setup a sample application hosted inside Kubernetes.

NOTE: This example will bring up a bare minimum cluster with a sample
application for experimentation, and in no way should be considered
a production ready deployment.

Prerequisites and Setup

For our virtual cluster, we will be using Exoscale
as they will allow us to get up and running with minimal effort. There are three
binaries that you will need to install to fully utilize this guide. While this
guide is written assuming Linux, the binaries are available for Linux, Windows,
and MacOS.

1. Exoscale CLI – Required to setup the environment and manage our systems
2. RKE CLI – Required to provision Kubernetes
3. kubectl – Required to manage our new Kubernetes cluster

In addition, you will need an Exoscale account
since it will be used to setup the Exoscale CLI.

Configure Exoscale CLI

Once you have your Exoscale account set-up, you need to configure the Exoscale
client. Assuming you are in the same directory containing the program you will run:

$ ./exo config

Hi happy Exoscalian, some configuration is required to use exo.

We now need some very important information, find them there.
<https://portal.exoscale.com/account/profile/api>

[+] API Key [none]: EXO························
[+] Secret Key [none]: ······································
[…]

NOTE: When you go to the API profile page
you will see the API Key and Secret. Be sure to copy and paste both at the prompts.

Provisioning the Kubernetes Environment with the Exoscale CLI

Now that we have configured the Exoscale CLI, we need to prepare the Exoscale
cloud environment. This will require setting up a firewall rule that will be
inherited by the instances that will become the Kubernetes cluster, and an optional
step of creating and adding your ssh public key.

Defining the firewall rules

The firewall or security group we will create must have at least three
ports exposed: 22 for ssh access, 6443 and 10240 for kubectl and rke to bring up
and manage the cluster. Lastly, we need to grant access to the security group so
the instances can interact amongst itself.

The first step to this is to create the firewall or security group:

$ ./exo firewall create rke-k8s -d “RKE k8s SG”
┼──────────┼─────────────────┼──────────────────────────────────────┼
│ NAME │ DESCRIPTION │ ID │
┼──────────┼─────────────────┼──────────────────────────────────────┼
│ rke-k8s │ RKE K8S SG │ 01a3b13f-a312-449c-a4ce-4c0c68bda457 │
┼──────────┼─────────────────┼──────────────────────────────────────┼

The next step is to add the rules (command output omitted):

$ ./exo firewall add rke-k8s -p ALL -s rke-k8s
$ ./exo firewall add rke-k8s -p tcp -P 6443 -c 0.0.0.0/0
$ ./exo firewall add rke-k8s -p tcp -P 10240 -c 0.0.0.0/0
$ ./exo firewall add rke-k8s ssh

You can confirm the results by invoking exo firewall show:

$ ./exo firewall show rke-k8s
┼─────────┼────────────────┼──────────┼──────────┼─────────────┼──────────────────────────────────────┼
│ TYPE │ SOURCE │ PROTOCOL │ PORT │ DESCRIPTION │ ID │
┼─────────┼────────────────┼──────────┼──────────┼─────────────┼──────────────────────────────────────┼
│ INGRESS │ CIDR 0.0.0.0/0 │ tcp │ 22 (ssh) │ │ 40d82512-2196-4d94-bc3e-69b259438c57 │
│ │ CIDR 0.0.0.0/0 │ tcp │ 10240 │ │ 12ceea53-3a0f-44af-8d28-3672307029a5 │
│ │ CIDR 0.0.0.0/0 │ tcp │ 6443 │ │ 18aa83f3-f996-4032-87ef-6a06220ce850 │
│ │ SG │ all │ 0 │ │ 7de233ad-e900-42fb-8d93-05631bcf2a70 │
┼─────────┼────────────────┼──────────┼──────────┼─────────────┼──────────────────────────────────────┼

Optional: Creating and adding an ssh key

One of the nice things about the Exoscale CLI is that you can use it to create
an ssh key for each instance you bring up. However there are times where you will
want a single administrative ssh key for the cluster. You can have the Exoscale
CLI create it or use the CLI to import your own key. To do that, you will use
Exoscale’s sshkey subcommand.

If you have a key you want to use:

$ ./exo sshkey upload [keyname] [ssh-public-key-path]

Or if you’d like to create a unique for this cluster:

$ ./exo sshkey create rke-k8s-key
┼─────────────┼─────────────────────────────────────────────────┼
│ NAME │ FINGERPRINT │
┼─────────────┼─────────────────────────────────────────────────┼
│ rke-k8s-key │ 0d:03:46:c6:b2:72:43:dd:dd:04:bc:8c:df:84:f4:d1 │
┼─────────────┼─────────────────────────────────────────────────┼
—–BEGIN RSA PRIVATE KEY—–
MIIC…
—–END RSA PRIVATE KEY—–

$

Save the contents of the RSA PRIVATE KEY component into a file as
that will be your sole means of accessing the cluster using that key name. In
both cases, we will need to make sure that the ssh-agent daemon is running, and
our key is added to it. If you haven’t done so already, run:

$ ssh-add [path-to-private-ssh-key]

Creating your Exoscale instances

At this point we are ready to create the instances. We will utilize the medium
sized templates as that will provide enough RAM for both Kubernetes and our
sample application to run. The OS Image we will use is Ubuntu-16.04 due to the
version of Docker required by RKE to bring up our cluster. Lastly, we use 10g of
disk space which will be enough to experiment with.

NOTE: If you go with a smaller instance size than medium, you will not
have enough RAM to bootstrap the Kubernetes cluster.

Step 1: Create instance configuration script

To automate the instance configuration, we will use cloud-init.
This is as easy as creating a YAML file to describe our actions, and specifying
the file on the Exoscale command line:

#cloud-config

manage_etc_hosts: true

package_update: true
package_upgrade: true

packages:
– curl

runcmd:
– “curl https://releases.rancher.com/install-docker/17.03.sh| bash”
– “usermod -aG docker ubuntu”
– “mkdir /data”

power_state:
mode: reboot

Copy and paste the block of text above into a new file called cloud-init.yml.

Step 2: Create the instances

Next, we are going to create 4 instances:

$ for i in 1 2 3 4; do
./exo vm create rancher-$i
–cloud-init-file cloud-init.yml
–service-offering medium
–template “Ubuntu 16.04 LTS”
–security-group rke-k8s
–disk 10
> done
Creating private SSH key
Deploying “rancher-1” …………. success!

What to do now?

1. Connect to the machine

> exo ssh rancher-1
ssh -i “/home/cab/.exoscale/instances/85fc654f-5761-4a02-b501-664ae53c671d/id_rsa” [email protected]

2. Put the SSH configuration into “.ssh/config”

> exo ssh rancher-1 –info
Host rancher-1
HostName 185.19.29.207
User ubuntu
IdentityFile /home/cab/.exoscale/instances/85fc654f-5761-4a02-b501-664ae53c671d/id_rsa

Tip of the day:
You’re the sole owner of the private key.
Be cautious with it.

NOTE: If you created or uploaded an SSH Keypair, then you can add the
–keypair <common key> argument where common key is the key name you chose
to upload.

NOTE 2: Save the hostname and IP address. You will need these for the RKE set-up

After waiting several minutes (about 5 to be safe) you will have four brand new
instances configured with docker and ready to go. A sample configuration will
resemble the following when you run ./exo vm list:

┼───────────┼────────────────┼─────────────────┼─────────┼──────────┼──────────────────────────────────────┼
│ NAME │ SECURITY GROUP │ IP ADDRESS │ STATUS │ ZONE │ ID │
┼───────────┼────────────────┼─────────────────┼─────────┼──────────┼──────────────────────────────────────┼
│ rancher-4 │ rke-k8s │ 159.100.240.102 │ Running │ ch-gva-2 │ acb53efb-95d1-48e7-ac26-aaa9b35c305f │
│ rancher-3 │ rke-k8s │ 159.100.240.9 │ Running │ ch-gva-2 │ 6b7707bd-9905-4547-a7d4-3fd3fdd83ac0 │
│ rancher-2 │ rke-k8s │ 185.19.30.203 │ Running │ ch-gva-2 │ c99168a0-46db-4f75-bd0b-68704d1c7f79 │
│ rancher-1 │ rke-k8s │ 185.19.30.83 │ Running │ ch-gva-2 │ 50605a5d-b5b6-481c-bb34-1f7ee9e1bde8 │
┼───────────┼────────────────┼─────────────────┼─────────┼──────────┼──────────────────────────────────────┼

RKE and Kubernetes

The Rancher Kubernetes Engine command is used to bring up, tear down, and
backup the configuration for a Kubernetes cluster. The core consists of
a configuration file that has the name of cluster.yml. While RKE
supports the creation of this configuration file with the command rke config,
it can be tedious to go through the prompts. Instead, we will pre-create
the config file.

The file below is a sample file that can be saved and modified as cluster.yml:

—

ssh_key_path: [path to ssh private key]
ssh_agent_auth: true

cluster_name: rke-k8s

nodes:
– address: [ip address of rancher-1]
name: rancher-1
user: ubuntu
role:
– controlplane
– etcd
– worker
– address: [ip address of rancher-2]
name: rancher-2
user: ubuntu
role:
– worker
– address: [ip address of rancher-3]
name: rancher-3
user: ubuntu
role:
– worker
– address: [ip address of rancher-4]
name: rancher-4
user: ubuntu
role:
– worker

Things you will need to modify:

• ssh_key_path: If you uploaded or created a public ssh key, then the path
  should be changed to reflect your private key location. Otherwise, you will
  need to move the ssh_key_path line to be inside each node entry and change
  the path to match the key generated for each instance that was created.
• address These should be changed to the IP addresses you saved from the
  previous step.
• cluster_name This should match your firewall/security group name.

Once you have saved your updated cluster.yml, $ ./rke up is all you need to
bring up the Kubernetes cluster. There will be a flurry of status updates as the
docker containers for the various Kubernetes components are downloaded into each
node, installed, and configured.

If everything goes well, then you will see the following when RKE finishes:

$ rke up

…

INFO[0099] Finished building Kubernetes cluster successfully

Congratulations, you have just brought up a Kubernetes cluster!

Configuring and using kubectl

One thing you will see that RKE created is the Kubernetes configuration file
kube_config_cluster.yml which is used by kubectl to communicate with the cluster.
To make running kubctl easier going forward, you will want to set the environment
variable KUBECONFIG so you don’t need to pass the config parameter each time:

export KUBECONFIG=/path/to/kube_config_cluster.yml

Here are a few sample status commands. The first command will give
you a listing of all registered nodes, their roles as defined from the
cluster.yml file above, and the Kubernetes version each node is running running.

$ ./kubectl get nodes
NAME STATUS ROLES AGE VERSION
159.100.240.102 Ready worker 3m v1.11.1
159.100.240.9 Ready worker 3m v1.11.1
185.19.30.203 Ready worker 3m v1.11.1
185.19.30.83 Ready controlplane,etcd,worker 3m v1.11.1

The second command is used to give you the cluster status.

$ ./kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {“health”: “true”}

NOTE: For more information about RKE and the cluster configuration file, you can visit
Rancher’s documentation page.

Optional: Installing the Kubernetes Dashboard

To make it easier to collect status information on the Kubernetes cluster, we
will install the Kubernetes dashboard. To install the dashboard, you will run:

$ ./kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

At this point, the dashboard is installed and running, but the only way to access
it is from inside the Kubernetes cluster. To expose the dashboard port onto your
workstation so that you can interact with it, you will need to proxy the port by
running:

Now you can now visit the dashboard at:
http://localhost:8001.

However, to be able to make full use of the dashboard, you will need to
authenticate your session. This will require a token from a specific
subsystem utilizing a set of secrets that were generated at the time we ran
rke up. The following command will extract the correct token that you can use
to authenticate against for the dashboard:

$ ./kubectl -n kube-system describe secrets
`./kubectl -n kube-system get secrets |awk ‘/clusterrole-aggregation-controller/ ‘`
|awk ‘/token:/ ‘

Copy and paste the long string that is returned into the authentication prompt
on the dashboard webpage, and explore the details of your cluster.

Adding Cassandra to Kubernetes

Now for the fun part. We are going to bring up Cassandra.
This will be a simple cluster that will use the local disks for storage. This
will give us something to play with when it comes to installing a service, and
seeing what happens inside Kubernetes.

To install Cassandra, we need to specify a service configuration that will be
exposed by Kubernetes, and an application definition file that specifies things
like networking, storage configuration, number of replicas, etc.

Step 1: Cassandra Service File

First, we will start with the services file:

apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
– port: 9042
selector:
app: cassandra

Copy and save the services file as cassandra-services.yaml, and load it:

./kubectl create -f ./cassandra-service.yml

You should see it load successfully, and you can verify using kubectl:

$ ./kubectl get svc cassandra
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None <none> 9042/TCP 46s

NOTE: For more details on Service configurations, you can read more about it in the
Kubernetes Service Networking Guide.

Cassandra StatefulSet

The StatefulSet
is a type of Kubernetes workload where the application is expected to persist
some kind of state such as our Cassandra example.

We will download the configuration from XXXXX/cassandra-statefulset.yaml,
and apply it:

$ ./kubectl create -f https://XXXX/cassandra-statefulset.yaml
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created

You can check the state of the StatefulSet we are loading:

$ ./kubectl get statefulset
NAME DESIRED CURRENT AGE
cassandra 3 3 4m

You can even interact with Cassandra inside its pod such as verifying that
Cassandra is up:

$ ./kubectl exec -it cassandra-0 — nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns (effective) Host ID Rack
UN 10.42.2.2 104.55 KiB 32 59.1% ac30ba66-bd59-4c8d-ab7b-525daeb85904 Rack1-K8Demo
UN 10.42.1.3 84.81 KiB 32 75.0% 92469b85-eeae-434f-a27d-aa003531cff7 Rack1-K8Demo
UN 10.42.3.3 70.88 KiB 32 65.9% 218a69d8-52f2-4086-892d-f2c3c56b05ae Rack1-K8Demo

Now suppose we want to scale up the number of replicas from 3, to 4? To do that, you
will run:

$ ./kubectl edit statefulset cassandra

This will open up your default text editor. Scroll down to the replica line,
change the value 3 to 4, save and exit. You should see the following with your
next invocation of kubectl:

$ ./kubectl get statefulset
NAME DESIRED CURRENT AGE
cassandra 4 4 10m

$ ./kubectl exec -it cassandra-0 — nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns (effective) Host ID Rack
UN 10.42.2.2 104.55 KiB 32 51.0% ac30ba66-bd59-4c8d-ab7b-525daeb85904 Rack1-K8Demo
UN 10.42.1.3 84.81 KiB 32 56.7% 92469b85-eeae-434f-a27d-aa003531cff7 Rack1-K8Demo
UN 10.42.3.3 70.88 KiB 32 47.2% 218a69d8-52f2-4086-892d-f2c3c56b05ae Rack1-K8Demo
UN 10.42.0.6 65.86 KiB 32 45.2% 275a5bca-94f4-439d-900f-4d614ba331ee Rack1-K8Demo

Final Note

There is one final note about cluster scaling and using the StatefulSet workload.
Kubernetes makes it easy to scale your cluster up to account for load, however to
ensure data gets preserved, Kubernetes will keep all data in place when you scale
the number of nodes down. What this means is that you will be responsible for
ensuring proper backups are made and everything is deleted before you can
consider the application cleaned up.

And here are gists with the yaml configurations from this tutorial that might be helpful as well:

Chris Baumbauer

Software Engineer

Chris Baumbauer is a freelance engineer whom has dabbled every piece of the stack from operating systems to mobile and web development with recent projects focused on Kubernetes such as Kompose and Kmachine.

Source

To help make it easier to build Kubernetes applications, Red Hat and the Kubernetes open source community today share the Operator Framework – an open source toolkit designed to manage Kubernetes native applications, called Operators, in a more effective, automated, and scalable way.

Operators are Kubernetes applications

You may be familiar with Operators from the concept’s introduction in 2016. An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your applications that run on Kubernetes. You can think of Operators as the runtime that manages this type of application on Kubernetes.

Conceptually, an Operator takes human operational knowledge and encodes it into software that is more easily packaged and shared with consumers. Think of an Operator as an extension of the software vendor’s engineering team that watches over your Kubernetes environment and uses its current state to make decisions in milliseconds. Operators follow a maturity model that ranges from basic functionality to having specific logic for an application. Advanced Operators are designed to handle upgrades seamlessly, react to failures automatically, and not take shortcuts, like skipping a software backup process to save time.

The pieces that are now being launched as the Operator Framework are the culmination of the years of work and experience of our team in building Operators. We’ve seen that Operators’ capabilities differ in sophistication depending on how much intelligence has been added into the implementation logic of the Operator itself. We’ve also learned that the creation of an Operator typically starts by automating an application’s installation and self-service provisioning capabilities, and then evolves to take on more complex automation.

We believe that the new Operator Framework represents the next big step for Kubernetes by using a baseline of leading practices to help lower the application development barrier on Kubernetes. The project delivers a software development kit (SDK) and the ability to manage app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise Operator capabilities on any Kubernetes cluster.

The Operator Framework: Introducing the SDK, Lifecycle Management, and Metering

The Operator Framework is an open source project that provides developer and runtime Kubernetes tools, enabling you to accelerate the development of an Operator. The Operator Framework includes:

• Operator SDK: Enables developers to build Operators based on their expertise without requiring knowledge of Kubernetes API complexities.
• Operator Lifecycle Management: Oversees installation, updates, and management of the lifecycle of all of the Operators (and their associated services) running across a Kubernetes cluster.
• Operator Metering (joining in the coming months): Enables usage reporting for Operators that provide specialized services.

Operator SDK

The Operator SDK provides the tools to build, test and package Operators. Initially, the SDK facilitates the marriage of an application’s business logic (for example, how to scale, upgrade, or backup) with the Kubernetes API to execute those operations. Over time, the SDK can allow engineers to make applications smarter and have the user experience of cloud services. Leading practices and code patterns that are shared across Operators are included in the SDK to help prevent reinventing the wheel.

Build and test iteration loop with the Operator SDK

Operator Lifecycle Manager

Once built, Operators need to be deployed on a Kubernetes cluster. The Operator Lifecycle Manager is the backplane that facilitates management of operators on a Kubernetes cluster. With it, administrators can control what Operators are available in what namespaces and who can interact with running Operators. They can also manage the overall lifecycle of Operators and their resources, such as triggering updates to both an Operator and its resources or granting a team access to an Operator for their slice of the cluster.

The lifecycle of multiple applications is managed on a Kubernetes cluster

Simple, stateless applications can leverage the Lifecycle Management features of the Operator Framework without writing any code by using a generic Operator (for example, the Helm Operator). However, complex and stateful applications are where an Operator can shine. The cloud-like capabilities that are encoded into the Operator code can provide an advanced user experience, automating such features as updates, backups and scaling.

Operator Metering

In a future version, the Operator Framework will also include the ability to meter application usage – a Kubernetes first, which provides extensions for central IT teams to budget and for software vendors providing commercial software. Operator Metering is designed to tie into the cluster’s CPU and memory reporting, as well as calculate IaaS cost and customized metrics like licensing.

We are actively working on Metering and it will be open-sourced and join the Framework in the coming months.

Operator Framework benefits

If you are a community member, builder, consumer of applications, or a user of Kubernetes overall, the Operator Framework offers a number of benefits.

For builders and the community

Today, there is often a high barrier to entry when it comes to building Kubernetes applications. There are a substantial number of pre-existing dependencies and assumptions, many of which may require experience and technical knowledge. At the same time, application consumers often do not want their services to be siloed across IT footprints with disparate management capabilities (for example, departments with differing tools for auditing, notification, metering, and so on).

The Operator Framework aims to address these points by helping to bring the expertise and knowledge of the Kubernetes community together in a single project that, when used as a standard application package, can make it easier to build applications for Kubernetes. By sharing this Framework with the community, we hope to enable an ecosystem of builders to more easily create their applications on Kubernetes via a common method and also provide a standardized set of tools to build consistent applications.

We believe a proper extension mechanism to Kubernetes shouldn’t be built without the community. To this end, Red Hat has proposed a “platform-dev” Special Interest Group that aligns well with the existing Kubebuilder project from Google and we look forward to working with other industry leaders should this group come to fruition.

“We are working together with Red Hat and the broader Kubernetes community to help enable this ecosystem with an easier way to create and operate their applications on Kubernetes,” said Phillip Wittrock, Software Engineer at Google, Kubernetes community member, and member of the Kubernetes steering committee. “By working together on platform development tools, we strive to make Kubernetes the foundation of choice for container-native apps – no matter where they reside.”

For application consumers and Kubernetes users

For consumers of applications across the hybrid cloud, keeping those applications up to date as new versions become available is of supreme importance, both for security reasons and for managing the applications’ lifecycles and other needs. The Operator Framework helps address these user requirements, aiding in the creation of cloud-native applications that are easier to consume, to keep updated, and to secure.

Get started

Learn more about the Operator Framework at https://github.com/operator-framework. A special thanks to the Kubernetes community for working alongside us. Take a test drive with the code-to-cluster reference example.

If you are at KubeCon 2018 in Europe, join our morning keynote on Thursday, May 3 to learn more about the framework. Can’t attend live? We’ll host an OpenShift Commons briefing on May 23 at 9 AM PT for a deeper dive on all things Operators.

Source

 

KubeDirector: The easy way to run complex stateful applications on Kubernetes

Author: Thomas Phelan (BlueData)

KubeDirector is an open source project designed to make it easy to run complex stateful scale-out application clusters on Kubernetes. KubeDirector is built using the custom resource definition (CRD) framework and leverages the native Kubernetes API extensions and design philosophy. This enables transparent integration with Kubernetes user/resource management as well as existing clients and tools.

We recently introduced the KubeDirector project, as part of a broader open source Kubernetes initiative we call BlueK8s. I’m happy to announce that the pre-alpha
code for KubeDirector is now available. And in this blog post, I’ll show how it works.

KubeDirector provides the following capabilities:

• The ability to run non-cloud native stateful applications on Kubernetes without modifying the code. In other words, it’s not necessary to decompose these existing applications to fit a microservices design pattern.
• Native support for preserving application-specific configuration and state.
• An application-agnostic deployment pattern, minimizing the time to onboard new stateful applications to Kubernetes.

KubeDirector enables data scientists familiar with data-intensive distributed applications such as Hadoop, Spark, Cassandra, TensorFlow, Caffe2, etc. to run these applications on Kubernetes – with a minimal learning curve and no need to write GO code. The applications controlled by KubeDirector are defined by some basic metadata and an associated package of configuration artifacts. The application metadata is referred to as a KubeDirectorApp resource.

To understand the components of KubeDirector, clone the repository on GitHub using a command similar to:

git clone http://<userid>@github.com/bluek8s/kubedirector.

The KubeDirectorApp definition for the Spark 2.2.1 application is located
in the file kubedirector/deploy/example_catalog/cr-app-spark221e2.json.

~> cat kubedirector/deploy/example_catalog/cr-app-spark221e2.json
{
“apiVersion”: “kubedirector.bluedata.io/v1alpha1”,
“kind”: “KubeDirectorApp”,
“metadata”: {
“name” : “spark221e2”
},
“spec” : {
“systemctlMounts”: true,
“config”: {
“node_services”: [
{
“service_ids”: [
“ssh”,
“spark”,
“spark_master”,
“spark_worker”
],
…

The configuration of an application cluster is referred to as a KubeDirectorCluster resource. The
KubeDirectorCluster definition for a sample Spark 2.2.1 cluster is located in the file
kubedirector/deploy/example_clusters/cr-cluster-spark221.e1.yaml.

~> cat kubedirector/deploy/example_clusters/cr-cluster-spark221.e1.yaml
apiVersion: “kubedirector.bluedata.io/v1alpha1”
kind: “KubeDirectorCluster”
metadata:
name: “spark221e2”
spec:
app: spark221e2
roles:
– name: controller
replicas: 1
resources:
requests:
memory: “4Gi”
cpu: “2”
limits:
memory: “4Gi”
cpu: “2”
– name: worker
replicas: 2
resources:
requests:
memory: “4Gi”
cpu: “2”
limits:
memory: “4Gi”
cpu: “2”
– name: jupyter
…

Running Spark on Kubernetes with KubeDirector

With KubeDirector, it’s easy to run Spark clusters on Kubernetes.

First, verify that Kubernetes (version 1.9 or later) is running, using the command kubectl version

~> kubectl version
Client Version: version.Info
Server Version: version.Info

Deploy the KubeDirector service and the example KubeDirectorApp resource definitions with the commands:

cd kubedirector
make deploy

These will start the KubeDirector pod:

~> kubectl get pods
NAME READY STATUS RESTARTS AGE
kubedirector-58cf59869-qd9hb 1/1 Running 0 1m

List the installed KubeDirector applications with kubectl get KubeDirectorApp

~> kubectl get KubeDirectorApp
NAME AGE
cassandra311 30m
spark211up 30m
spark221e2 30m

Now you can launch a Spark 2.2.1 cluster using the example KubeDirectorCluster file and the
kubectl create -f deploy/example_clusters/cr-cluster-spark211up.yaml command.
Verify that the Spark cluster has been started:

~> kubectl get pods
NAME READY STATUS RESTARTS AGE
kubedirector-58cf59869-djdwl 1/1 Running 0 19m
spark221e2-controller-zbg4d-0 1/1 Running 0 23m
spark221e2-jupyter-2km7q-0 1/1 Running 0 23m
spark221e2-worker-4gzbz-0 1/1 Running 0 23m
spark221e2-worker-4gzbz-1 1/1 Running 0 23m

The running services now include the Spark services:

~> kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubedirector ClusterIP 10.98.234.194 <none> 60000/TCP 1d
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
svc-spark221e2-5tg48 ClusterIP None <none> 8888/TCP 21s
svc-spark221e2-controller-tq8d6-0 NodePort 10.104.181.123 <none> 22:30534/TCP,8080:31533/TCP,7077:32506/TCP,8081:32099/TCP 20s
svc-spark221e2-jupyter-6989v-0 NodePort 10.105.227.249 <none> 22:30632/TCP,8888:30355/TCP 20s
svc-spark221e2-worker-d9892-0 NodePort 10.107.131.165 <none> 22:30358/TCP,8081:32144/TCP 20s
svc-spark221e2-worker-d9892-1 NodePort 10.110.88.221 <none> 22:30294/TCP,8081:31436/TCP 20s

Pointing the browser at port 31533 connects to the Spark Master UI:

That’s all there is to it!
In fact, in the example above we also deployed a Jupyter notebook along with the Spark cluster.

To start another application (e.g. Cassandra), just specify another KubeDirectorApp file:

kubectl create -f deploy/example_clusters/cr-cluster-cassandra311.yaml

See the running Cassandra cluster:

~> kubectl get pods
NAME READY STATUS RESTARTS AGE
cassandra311-seed-v24r6-0 1/1 Running 0 1m
cassandra311-seed-v24r6-1 1/1 Running 0 1m
cassandra311-worker-rqrhl-0 1/1 Running 0 1m
cassandra311-worker-rqrhl-1 1/1 Running 0 1m
kubedirector-58cf59869-djdwl 1/1 Running 0 1d
spark221e2-controller-tq8d6-0 1/1 Running 0 22m
spark221e2-jupyter-6989v-0 1/1 Running 0 22m
spark221e2-worker-d9892-0 1/1 Running 0 22m
spark221e2-worker-d9892-1 1/1 Running 0 22m

Now you have a Spark cluster (with a Jupyter notebook) and a Cassandra cluster running on Kubernetes.
Use kubectl get service to see the set of services.

~> kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubedirector ClusterIP 10.98.234.194 <none> 60000/TCP 1d
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
svc-cassandra311-seed-v24r6-0 NodePort 10.96.94.204 <none> 22:31131/TCP,9042:30739/TCP 3m
svc-cassandra311-seed-v24r6-1 NodePort 10.106.144.52 <none> 22:30373/TCP,9042:32662/TCP 3m
svc-cassandra311-vhh29 ClusterIP None <none> 8888/TCP 3m
svc-cassandra311-worker-rqrhl-0 NodePort 10.109.61.194 <none> 22:31832/TCP,9042:31962/TCP 3m
svc-cassandra311-worker-rqrhl-1 NodePort 10.97.147.131 <none> 22:31454/TCP,9042:31170/TCP 3m
svc-spark221e2-5tg48 ClusterIP None <none> 8888/TCP 24m
svc-spark221e2-controller-tq8d6-0 NodePort 10.104.181.123 <none> 22:30534/TCP,8080:31533/TCP,7077:32506/TCP,8081:32099/TCP 24m
svc-spark221e2-jupyter-6989v-0 NodePort 10.105.227.249 <none> 22:30632/TCP,8888:30355/TCP 24m
svc-spark221e2-worker-d9892-0 NodePort 10.107.131.165 <none> 22:30358/TCP,8081:32144/TCP 24m
svc-spark221e2-worker-d9892-1 NodePort 10.110.88.221 <none> 22:30294/TCP,8081:31436/TCP 24m

Get Involved

KubeDirector is a fully open source, Apache v2 licensed, project – the first of multiple open source projects within a broader initiative we call BlueK8s.
The pre-alpha code for KubeDirector has just been released and we would love for you to join the growing community of developers, contributors, and adopters.
Follow @BlueK8s on Twitter and get involved through these channels:

• KubeDirector chat room on Slack
• KubeDirector GitHub repo

Source

2/May 2018

By James Munnelly

Those of you who closely follow Jetstack’s open source projects may have already noticed that our
new certificate management tool, cert-manager, has been available for some time now.
In fact, we now have over 1,000 stars on GitHub!

Cert-manager is a general purpose x509 certificate management tool for Kubernetes.
In today’s modern web, securing application traffic is critical.
cert-manager aims to simplify management, issuance and renewal of certificates within your
organisation.

kube-lego, our original Let’s Encrypt certificate provisioning
tool for Kubernetes Ingress resources, has been a great success.
It makes securing traffic between your users and your cluster ingress point simple.
Over time however, the limitations of building a controller solely around Kubernetes Ingresses
became apparent.

By building cert-manager around Kubernetes CustomResourceDefinitions, we have been able to
greatly increase the flexibility of configuration, debugging capabilities and also support a wider
range of CAs than Let’s Encrypt alone.

This post is a high-level overview of how cert-manager works, and will highlight
some of the new features and recent developments in the project.

In the real world of x509 certificates, CAs (Certificate Authorities) are a point of trust, responsible
for issuing identities to various clients in the form of signed and trusted x509 certificates.

Let’s Encrypt introduced the ACME protocol, however, not all CAs support this protocol.

In order to support many different types of certificate authority, we have introduced the concept of
a CA to the Kubernetes API, in the form of ‘Issuers’.

A cert-manager ‘Issuer’ resource represents an entity that is able to sign x509 Certificate
requests.

Today, we support ACME, CA (i.e. a simple signing keypair stored in a Secret resource) and as of
the v0.3.0 release, Hashicorp Vault!

Here’s an example of an issuer resource:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: user@example.com
privateKeySecretRef:
name: letsencrypt-private-key
http01: {}

In order to request certificates from these Issuers, we also introduce Certificate resources.
These resources reference a corresponding Issuer resource in order to denote which CA they should
be issued by.

Here’s an example of a certificate resource, using the issuer shown previously:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: example-com
spec:
secretName: example-com-tls
dnsNames:
– example.com
– www.example.com
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
acme:
config:
– http01:
ingressClass: nginx
domains:
– example.com
– www.example.com

More info on Issuers and Certificates can be found in our documentation.

Over the last few weeks, we have been trialling an alpha release candidate of v0.3,
our upcoming new release.

This release is packed with new features and bug fixes alike, and this section describes
the headline features.

ACMEv2 and wildcard certificates

Let’s Encrypt recently announced v2 of the ACME protocol, which amongst other improvements,
now supports wildcard certificates. This has been a long-awaited and requested feature,
and one that hasn’t been possible until recently.

In order to allow cert-manager users to request and consume wildcard certificates, we have
switched exclusively to use ACMEv2 as part of the v0.3 release.

This allows users to request wildcard certificates just like any other – including full support
for doing so via annotated Ingress resources (just like kube-lego!).

This is a great example of how we can add new and more complex features to cert-manager, whilst
also continuing to support long-standing, legacy behaviour adopted from kube-lego.

If you’d like to get started and give this a try, check out the latest v0.3 alpha release
on our GitHub page. You can request a wildcard certificate just like any other, by specifying
*.domain.com as one of your DNS names. See the ACME DNS validation tutorial for more information
on how to get started with the DNS-01 challenge.

Vault Issuer support

After a lot of hard work, initial support for Hashicorp Vault has been merged into cert-manager!

This is another long requested feature, and a large addition to our set of supported Issuer types.

A special thanks to @vdesjardins, who single handedly and unprompted undertook this work, driving
it through the review process to the very end.

This allows end users to create a ‘Vault’ Issuer, which is paired with a single Vault PKI backend role.

Initially we support both token authentication and Vault’s ‘AppRole’ authentication mechanism
to securely authorize cert-manager against the Vault server.

The addition of the Vault Issuer bolsters our position as a general purpose certificate
manager, and shows the value in providing a single clean abstraction on top of the multitude
of different types of CAs out there.

Read more about creating and using the Vault Issuer at the docs.

New documentation site

Within any project, documentation is tough. It needs to provide a clear onboarding experience
for brand new users, as well as in-depth details for more advanced users. All the while accounting
for the varying skill levels of these users (for example, some may also be brand new to Kubernetes
itself!).

We’ve had some brilliant community contributions with user guides and tutorials explaining how
to get started with cert-manager.

Up until now, we have used markdown stored in GitHub to host our documentation.
This began to cause confusion when we started versioning cert-manager more strictly, and
releasing alpha/beta release candidates.
In order to handle this, and also to make it easier for users to navigate and discover the
various tutorials we do have – we have moved our documentation over to a hosted readthedocs
website.

You can check out the new docs on readthedocs – take note
that we have a ‘version switcher’ in the bottom left as well, if you are looking for info on the
latest 0.3 release.

Each page in the docs also has an “Edit in GitHub” link on the top right, so if you spot a mistake
or if you’ve got an idea for a new tutorial please dive in and submit PRs!

Cert-manager is still young, with much planned in the future! Here are a couple of highlights
from our roadmap:

Defining Policy on Certificates

Right now, any user that is able to create a Certificate resource can request certificates
of any sort from any ClusterIssuer, or Issuer within the same namespace.

We intend to provide mechanisms for administrators to define ‘policies’ for who can obtain
Certificates, and/or how those Certificates must be structured. This might include things such
as minimum and maximum durations, or a limited set of allowed DNS names.

By defining this policy within Kubernetes itself, we benefit from a common level of policy
control between all the different CAs within an organisation.
This will help your organisation audit and manage who can do what.

Advanced resource validation

Kubernetes has recently added support for ValidatingAdmissionWebhooks (as well as their ‘Mutating’ counterparts).

These can be used to provide resource validation (e.g. ensuring that all fields are set to
acceptable values), as well as advanced ‘mutation’ of resources (e.g. applying ‘default values’).

One common problem when configuring these webhooks, is that they require x509 Certificates in order
to be set up and run. This process can be cumbersome, and is exactly the sort of problem cert-manager
has been designed to solve!

In future releases of cert-manager, we will introduce our own Validating webhook in
order to provide fore-warning to developers of invalid configurations. This will involve
a novel ‘bootstrapping’ process in order to allow for ‘self hosted webhooks’ (i.e. webhooks that
power cert-manager, hosted by cert-manager).

Along with this, we will be creating tutorials that explain our ‘recommended deployment practice’
for these kinds of webhooks, and how you can utilise cert-manager to handle all aspects of securing them!

Pluggable/out-of-process Issuers

Some enterprise users of cert-manager have their own CA process, which is novel and bespoke
to their organisation.

It is not always feasible to switch a whole organisation over to a new protocol in a short period,
given so many different business units rely on a x509 Certificate platform.

In order to ease the transition period for these companies, we are exploring the addition
of a ‘gRPC based Issuer’.
This is in a similar vein to CNI (Container Networking Interface) and CRI (Container Runtime Interface).
The goal would be to provide a general purpose gRPC interface that anyone can implement
in order to sign and renew certificates.

Users that implement this interface will then immediately benefit from the additional
policy and renewal features that cert-manager already implements.

Cert-manager is growing quickly, and the community around it bolsters every day.

We’d love to see new members join the community, and believe it’s imperative if we
want the project to survive.

If you want to get involved, take a look over our Issue board on GitHub, or drop into
#cert-manager on the Kubernetes Slack and say hello!

Want to work on community projects like cert-manager for your day job?
Jetstack are hiring Software Engineers, including remote (EU) roles. Check out our careers page
for more info on the roles and how to apply.

Source