{"id":391,"date":"2018-10-16T18:34:45","date_gmt":"2018-10-16T18:34:45","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=391"},"modified":"2018-10-17T09:10:11","modified_gmt":"2018-10-17T09:10:11","slug":"kubernetes-1-8-hidden-gems-the-resource-metrics-api-the-custom-metrics-api-and-hpa-v2","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2018\/10\/16\/kubernetes-1-8-hidden-gems-the-resource-metrics-api-the-custom-metrics-api-and-hpa-v2\/","title":{"rendered":"Kubernetes 1.8: Hidden Gems – The Resource Metrics API, the Custom Metrics API and HPA v2 \/"},"content":{"rendered":"

By Luke Addison<\/a><\/p>\n

In the coming weeks we will be releasing a series of blog posts called Kubernetes 1.8: Hidden Gems, accenting some of the less obvious but wonderful features in the latest Kubernetes release. In this week\u2019s gem, Luke looks at some of the main components in the core metrics and monitoring pipelines<\/a> and in particular how they can be used to scale Kubernetes workloads.<\/p>\n

One of the features that makes Kubernetes so powerful is its extensibility. In particular, Kubernetes allows developers to easily extend the core API Server with their own API servers, which we will refer to as \u2018add-on\u2019 API servers. The resource metrics API<\/a> (also known as the master metrics API or just the metrics API) introduced in 1.8 and the custom metrics API<\/a>, introduced in 1.6, are implemented in exactly this way. The resource metrics API is designed to be consumed by core system components, such as the scheduler and kubectl top, whilst the custom metrics API has a wider use case.<\/p>\n

One Kubernetes component that makes use of both the resource metrics API and the custom metrics API is the HorizontalPodAutoscaler (HPA) controller which manages HPA resources. HPA resources are used to automatically scale the number of Pods in a ReplicationController, Deployment or ReplicaSet based on observed metrics (note that StatefulSet is not supported<\/a>).<\/p>\n

The first version of HPA (v1) was only able to scale based on observed CPU utilisation. Although useful for some cases, CPU is not always the most suitable or relevant metric to autoscale an application. HPA v2, introduced in 1.6, is able to scale based on custom<\/em> metrics and has been moved from alpha to beta in 1.8. This allows users to scale on any number of application-specific metrics; for example, metrics might include the length of a queue and ingress requests per second.<\/p>\n

The purpose of the resource metrics API is to provide a stable, versioned API that core Kubernetes components can rely on. Implementations of the API provide resource usage metrics for pods and nodes through the API Server and form part of the core metrics pipeline<\/a>.<\/p>\n

In order to get a resource metrics add-on API server up and running we first need to configure the aggregation layer<\/a>. The aggregation layer<\/a> is a new feature in Kubernetes 1.7 that allows add-on API servers to register themselves with kube-aggregator<\/a>. The aggregator will then proxy relevant requests to these add-on API servers so that they can serve custom API resources.<\/p>\n

\"kube-aggregator<\/p>\n

Configuring the aggregation layer involves setting a number of flags on the API Server. The exact flags can be found here<\/a> and more information about these flags can be found in the kube-apiserver reference documentation<\/a>. In order to set these flags you will need to obtain a CA certificate if your cluster provider has not taken care of that already. For more details on the various CAs used by the API Server, take a look at the excellent apiserver-builder documentation<\/a>.<\/p>\n

We now need to deploy the add-on API Server itself to serve these metrics. We can use Heapster\u2019s implementation of the resource metrics API by running it with the –api-server flag set to true, however the recommended way is to deploy metrics-server<\/a>, which is a slimmed-down version of Heapster specifically designed to serve resource usage metrics. You can do this using the deployment manifests<\/a> provided in the metrics-server repository. Pay special attention to the APIService resource included with the manifests. This resource claims a URL path in the Kubernetes API (\/apis\/metrics.k8s.io\/v1beta1 in this case) and tells the aggregator to proxy anything sent to that path to the registered service. For more information about metrics-server check out the metrics-server repository<\/a>.<\/p>\n

To test our resource metrics API we can use kubectl get –raw. The following command should return a list of resource usage metrics for all the nodes in our cluster.<\/p>\n

$ kubectl get –raw “\/apis\/metrics.k8s.io\/v1beta1\/nodes” | jq
\n{
\n“kind”: “NodeMetricsList”,
\n“apiVersion”: “metrics.k8s.io\/v1beta1”,
\n“metadata”: {
\n“selfLink”: “\/apis\/metrics.k8s.io\/v1beta1\/nodes”
\n},
\n“items”: [
\n{
\n“metadata”: {
\n“name”: “node1.lukeaddison.co.uk”,
\n“selfLink”: “\/apis\/metrics.k8s.io\/v1beta1\/nodes\/node1.lukeaddison.co.uk”,
\n“creationTimestamp”: “2017-10-09T14:21:06Z”
\n},
\n“timestamp”: “2017-10-09T14:21:00Z”,
\n“window”: “1m0s”,
\n“usage”: {
\n“cpu”: “247m”,
\n“memory”: “1846432Ki”
\n}
\n},
\n{
\n“metadata”: {
\n“name”: “node2.lukeaddison.co.uk”,
\n“selfLink”: “\/apis\/metrics.k8s.io\/v1beta1\/nodes\/node2.lukeaddison.co.uk”,
\n“creationTimestamp”: “2017-10-09T14:21:06Z”
\n},
\n“timestamp”: “2017-10-09T14:21:00Z”,
\n“window”: “1m0s”,
\n“usage”: {
\n“cpu”: “511m”,
\n“memory”: “3589560Ki”
\n}
\n},
\n{
\n“metadata”: {
\n“name”: “node3.lukeaddison.co.uk”,
\n“selfLink”: “\/apis\/metrics.k8s.io\/v1beta1\/nodes\/node3.lukeaddison.co.uk”,
\n“creationTimestamp”: “2017-10-09T14:21:06Z”
\n},
\n“timestamp”: “2017-10-09T14:21:00Z”,
\n“window”: “1m0s”,
\n“usage”: {
\n“cpu”: “301m”,
\n“memory”: “2453620Ki”
\n}
\n}
\n]
\n}<\/p>\n

The resource metrics API allows HPA v2 to scale on resource metrics such as CPU and memory usage, however this API does not allow us to consume application specific metrics – we need something extra.<\/p>\n

The purpose of the custom metrics API is to provide a stable, versioned API that end-users and Kubernetes components can rely on. Implementations of the custom metrics API provide custom metrics to the HPA controller and form part of the monitoring pipeline.<\/p>\n

The steps required to configure your cluster to use a custom metrics API implementation can be found in the Kubernetes HPA docs<\/a>. There are not a huge number of implementations<\/a> yet and none that are officially part of Kubernetes, but a good one to try at the moment is the Prometheus adapter<\/a>. This adapter translates queries for custom metrics into PromQL<\/a>, the Prometheus query language, in order to query Prometheus itself and pass the results back to the caller.<\/p>\n

There is a nice walk-through<\/a> by DirectXMan12<\/a> that covers cluster prerequisites, how to deploy Prometheus and how to inject a Prometheus adapter container into your Prometheus deployment. This adapter can then be registered with the API Server using an APIService resource to tell the aggregator where to forward requests for custom metrics. Note that the walk-through uses the API Server path \/apis\/custom-metrics.metrics.k8s.io but for 1.8 a decision<\/a> was made to use \/apis\/custom.metrics.k8s.io so you will need to change your APIService resource appropriately. luxas<\/a> has a nice example of everything<\/a> – thanks!<\/p>\n

As before, we can test our new add-on API server using kubectl get –raw. The following command should return a list of all custom metrics from the Prometheus adapter.<\/p>\n

$ kubectl get –raw “\/apis\/custom.metrics.k8s.io\/v1beta1” | jq
\n{
\n“kind”: “APIResourceList”,
\n“apiVersion”: “v1”,
\n“groupVersion”: “custom.metrics.k8s.io\/v1beta1”,
\n“resources”: [
\n{
\n“name”: “pods\/tasks_state”,
\n“singularName”: “”,
\n“namespaced”: true,
\n“kind”: “MetricValueList”,
\n“verbs”: [
\n“get”
\n]
\n},
\n{
\n“name”: “pods\/memory_failcnt”,
\n“singularName”: “”,
\n“namespaced”: true,
\n“kind”: “MetricValueList”,
\n“verbs”: [
\n“get”
\n]
\n},
\n…
\n{
\n“name”: “pods\/memory_swap”,
\n“singularName”: “”,
\n“namespaced”: true,
\n“kind”: “MetricValueList”,
\n“verbs”: [
\n“get”
\n]
\n}
\n]
\n}<\/p>\n

With the Prometheus adapter deployed, we can use the power of Prometheus to scale our workloads on custom metrics specifically tailored to our applications.<\/p>\n

The following example shows a HPA that scales an nginx deployment using a single resource metric (CPU) and two custom metrics (packets-per-second and requests-per-second)<\/p>\n

apiVersion: autoscaling\/v2beta1
\nkind: HorizontalPodAutoscaler
\nmetadata:
\nname: nginx
\nspec:
\nscaleTargetRef:
\napiVersion: apps\/v1beta1
\nkind: Deployment
\nname: nginx
\nminReplicas: 1
\nmaxReplicas: 10
\nmetrics:
\n– type: Resource
\nresource:
\nname: cpu
\ntargetAverageUtilization: 50
\n– type: Pods
\npods:
\nmetricName: packets-per-second
\ntargetAverageValue: 1k
\n– type: Object
\nobject:
\nmetricName: requests-per-second
\ntarget:
\napiVersion: extensions\/v1beta1
\nkind: Ingress
\nname: main-route
\ntargetValue: 2k<\/p>\n