If we draw analogs between the components of a traditional relational database and those of Elasticsearch, they look like this:<\/p>\n
- \n
- Database or Table -> Index<\/li>\n
- Row\/Column -> Document with properties<\/li>\n<\/ul>\n
Elasticsearch Advantages<\/h3>\n
- \n
- It originates from Apache Lucene, which provides the most robust full-text search capabilities of any open source product.<\/li>\n
- It uses a document-oriented architecture to store complex real-world entities as structured JSON documents. By default, it indexes all fields, which provides tremendous performance when searching.<\/li>\n
- It doesn\u2019t use a schema with its indices. Documents add new fields by including them, which gives the freedom to add, remove, or change relevant fields without the downtime associated with a traditional database schema upgrade.<\/li>\n
- It performs linguistic searches against documents, returning those that match the search condition. It scores the results using the TFIDF algorithm<\/a>, bringing more relevant documents higher up in the list of results.<\/li>\n
- It allows fuzzy searching<\/em>, which helps find results even with misspelled search terms.<\/li>\n
- It supports real-time search autocompletion, returning results while the user types their search query.<\/li>\n
- It uses a RESTful API, exposing its power via a simple, lightweight interface.<\/li>\n
- Elasticsearch executes complex queries with tremendous speed. It also caches queries, returning cached results for other requests that match a cached filter.<\/li>\n
- It scales horizontally, making it possible to extend resources and balance the load between cluster nodes.<\/li>\n
- It breaks indices into shards, and each shard has any number of replicas. Each node knows the location of every document in the cluster and routes requests internally as necessary to retrieve the data.<\/li>\n<\/ul>\n
Terminology<\/h3>\n
Elasticsearch uses specific terms to define its components.<\/p>\n
- \n
- Cluster: A collection of nodes that work together.<\/li>\n
- Node: A single server that acts as part of the cluster, stores the data, and participates in the cluster\u2019s indexing and search capabilities.<\/li>\n
- Index: A collection of documents with similar characteristics.<\/li>\n
- Document: The basic unit of information that can be indexed.<\/li>\n
- Shards: Indexes are divided into multiple pieces called shards, which allows the index to scale horizontally.<\/li>\n
- Replicas: Copies of index shards<\/li>\n<\/ul>\n
Prerequisites<\/h2>\n
To perform this demo, you need one of the following:<\/p>\n
- \n
- An existing Rancher deployment and Kubernetes cluster, or<\/li>\n
- Two nodes in which to deploy Rancher and Kubernetes, or<\/li>\n
- A node in which to deploy Rancher and a Kubernetes cluster running in a hosted provider such as GKE.<\/li>\n<\/ul>\n
This article uses the Google Cloud Platform, but you may use any other provider or infrastructure.<\/p>\n
Launch Rancher<\/h2>\n
If you don\u2019t already have a Rancher deployment, begin by launching one. The quick start guide<\/a> covers the steps for doing so.<\/p>\n
Launch a Cluster<\/h2>\n
Use Rancher to set up and configure your cluster according to the guide most suited to your environment<\/a>.<\/p>\n
Deploy Elasticsearch<\/h2>\n
If you are already comfortable with kubectl, you can apply the manifests directly. If you prefer to use the Rancher user interface, scroll down for those instructions.<\/p>\n
We will deploy Elasticsearch as a StatefulSet<\/a> with two Services: a headless service for communicating with the pods and another for interacting with Elasticsearch from outside of the Kubernetes cluster.<\/p>\n
svc-cluster.yaml<\/h3>\n
apiVersion: v1
\nkind: Service
\nmetadata:
\nname: elasticsearch-cluster
\nspec:
\nclusterIP: None
\nselector:
\napp: es-cluster
\nports:
\n– name: transport
\nport: 9300$ kubectl apply -f svc-cluster.yaml
\nservice\/elasticsearch-cluster created<\/p>\nsvc-loadbalancer.yaml<\/h3>\n
apiVersion: v1
\nkind: Service
\nmetadata:
\nname: elasticsearch-loadbalancer
\nspec:
\nselector:
\napp: es-cluster
\nports:
\n– name: http
\nport: 80
\ntargetPort: 9200
\ntype: LoadBalancer$ kubectl apply -f svc-loadbalancer.yaml
\nservice\/elasticsearch-loadbalancer created<\/p>\nes-sts-deployment.yaml<\/h3>\n
apiVersion: v1
\nkind: ConfigMap
\nmetadata:
\nname: es-config
\ndata:
\nelasticsearch.yml: |
\ncluster.name: my-elastic-cluster
\nnetwork.host: “0.0.0.0”
\nbootstrap.memory_lock: false
\ndiscovery.zen.ping.unicast.hosts: elasticsearch-cluster
\ndiscovery.zen.minimum_master_nodes: 1
\nxpack.security.enabled: false
\nxpack.monitoring.enabled: false
\nES_JAVA_OPTS: -Xms512m -Xmx512m
\n—
\napiVersion: apps\/v1beta1
\nkind: StatefulSet
\nmetadata:
\nname: esnode
\nspec:
\nserviceName: elasticsearch
\nreplicas: 2
\nupdateStrategy:
\ntype: RollingUpdate
\ntemplate:
\nmetadata:
\nlabels:
\napp: es-cluster
\nspec:
\nsecurityContext:
\nfsGroup: 1000
\ninitContainers:
\n– name: init-sysctl
\nimage: busybox
\nimagePullPolicy: IfNotPresent
\nsecurityContext:
\nprivileged: true
\ncommand: [“sysctl”, “-w”, “vm.max_map_count=262144”]
\ncontainers:
\n– name: elasticsearch
\nresources:
\nrequests:
\nmemory: 1Gi
\nsecurityContext:
\nprivileged: true
\nrunAsUser: 1000
\ncapabilities:
\nadd:
\n– IPC_LOCK
\n– SYS_RESOURCE
\nimage: docker.elastic.co\/elasticsearch\/elasticsearch:6.5.0
\nenv:
\n– name: ES_JAVA_OPTS
\nvalueFrom:
\nconfigMapKeyRef:
\nname: es-config
\nkey: ES_JAVA_OPTS
\nreadinessProbe:
\nhttpGet:
\nscheme: HTTP
\npath: \/_cluster\/health?local=true
\nport: 9200
\ninitialDelaySeconds: 5
\nports:
\n– containerPort: 9200
\nname: es-http
\n– containerPort: 9300
\nname: es-transport
\nvolumeMounts:
\n– name: es-data
\nmountPath: \/usr\/share\/elasticsearch\/data
\n– name: elasticsearch-config
\nmountPath: \/usr\/share\/elasticsearch\/config\/elasticsearch.yml
\nsubPath: elasticsearch.yml
\nvolumes:
\n– name: elasticsearch-config
\nconfigMap:
\nname: es-config
\nitems:
\n– key: elasticsearch.yml
\npath: elasticsearch.yml
\nvolumeClaimTemplates:
\n– metadata:
\nname: es-data
\nspec:
\naccessModes: [ “ReadWriteOnce” ]
\nresources:
\nrequests:
\nstorage: 5Gi$ kubectl apply -f es-sts-deployment.yaml
\nconfigmap\/es-config created
\nstatefulset.apps\/esnode created<\/p>\nDeploy Elasticsearch via the Rancher UI<\/h2>\n
If you prefer, import each of the manifests above into your cluster via the Rancher UI. The screenshots below shows the process for each of them.<\/p>\n
Import svc-cluster.yaml<\/h3>\n
![\"01\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/01-rancher-import-svc-cluster.png\")
<\/p>\n![\"02\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/02-rancher-select-yaml.png\")
<\/p>\n![\"03\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/03-rancher-hit-import1.png\")
<\/p>\n![\"04\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/04-rancher-service-discovery1.png\")
<\/p>\nImport svc-loadbalancer.yaml<\/h3>\n
![\"05\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/05-rancher-hit-import2.png\")
<\/p>\n![\"06\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/06-rancher-service-discovery2.png\")
<\/p>\nImport es-sts-deployment.yaml<\/h3>\n
![\"07\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/07-rancher-import-workload.png\")
<\/p>\n![\"08\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/08-rancher-hit-import3.png\")
<\/p>\n![\"09\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/09-rancher-workload.png\")
<\/p>\n![\"10\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/10-rancher-volumes.png\")
<\/p>\nRetrieve the Load Balancer IP<\/h2>\n
You\u2019ll need the address of the load balancer that we deployed. You can retrieve this via kubectl or the UI.<\/p>\n
Use the CLI<\/h3>\n
$ kubectl get svc elasticsearch-loadbalancer
\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
\nelasticsearch-loadbalancer LoadBalancer 10.59.246.186 35.204.239.246 80:30604\/TCP 33m<\/p>\nUse the UI<\/h3>\n
![\"11\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/11-rancher-loadbalancer-IP.png\")
<\/p>\nTest the Cluster<\/h2>\n
Use the address we retrieved in the previous step to query the cluster for basic information.<\/p>\n
$ curl 35.204.239.246
\n{
\n“name” : “d7bDQcH”,
\n“cluster_name” : “my-elastic-cluster”,
\n“cluster_uuid” : “e3JVAkPQTCWxg2vA3Xywgg”,
\n“version” : {
\n“number” : “6.5.0”,
\n“build_flavor” : “default”,
\n“build_type” : “tar”,
\n“build_hash” : “816e6f6”,
\n“build_date” : “2018-11-09T18:58:36.352602Z”,
\n“build_snapshot” : false,
\n“lucene_version” : “7.5.0”,
\n“minimum_wire_compatibility_version” : “5.6.0”,
\n“minimum_index_compatibility_version” : “5.0.0”
\n},
\n“tagline” : “You Know, for Search”
\n}<\/p>\nQuery the cluster for information about its nodes. The asterisk in the master column highlights the current master node.<\/p>\n
$ curl 35.204.239.246\/_cat\/nodes?v
\nip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
\n10.56.2.8 24 97 5 0.05 0.12 0.13 mdi – d7bDQcH
\n10.56.0.6 28 96 4 0.01 0.05 0.04 mdi * WEOeEqC<\/p>\nCheck the available indices:<\/p>\n
$ curl 35.204.239.246\/_cat\/indices?v
\nhealth status index uuid pri rep docs.count docs.deleted store.size pri.store.size<\/p>\nBecause this is a fresh install, it doesn\u2019t have any indices or data. To continue this tutorial, we\u2019ll inject some sample data that we can use later. The files that we\u2019ll use are available from the Elastic website<\/a>. Download them and then load them with the following commands:<\/p>\n
$ curl -H ‘Content-Type: application\/x-ndjson’ -XPOST
\n‘http:\/\/35.204.239.246\/shakespeare\/doc\/_bulk?pretty’ –data-binary @shakespeare_6.0.json
\n$ curl -H ‘Content-Type: application\/x-ndjson’ -XPOST
\n‘http:\/\/35.204.239.246\/bank\/account\/_bulk?pretty’ –data-binary @accounts.json
\n$ curl -H ‘Content-Type: application\/x-ndjson’ -XPOST
\n‘http:\/\/35.204.239.246\/_bulk?pretty’ –data-binary @logs.json<\/p>\nWhen we recheck the indices, we see that we have five new indices with data.<\/p>\n
$ curl 35.204.239.246\/_cat\/indices?v
\nhealth status index uuid pri rep docs.count docs.deleted store.size pri.store.size
\ngreen open logstash-2015.05.20 MFdWJxnsTISH0Z9Vr0aT3g 5 1 4750 0 49.9mb 25.2mb
\ngreen open logstash-2015.05.18 lLHV2nzvTOG9mzlpKaG9sg 5 1 4631 0 46.5mb 23.5mb
\ngreen open logstash-2015.05.19 PqNnVUgXTyaDSfmCQZwbLQ 5 1 4624 0 48.2mb 24.2mb
\ngreen open shakespeare rwl3xBgmQtm8B3V7GFeTZQ 5 1 111396 0 46mb 23.1mb
\ngreen open bank z0wVGsbrSiG2cQwRXwaCOg 5 1 1000 0 949.2kb 474.6kb<\/p>\nEach of these contains a different type of document. For the shakespeare index, we can search for the name of a play. For the logstash-2015.05.19 index we can query and filter data based on an IP address, and for the bank index we can search for information about a particular account.<\/p>\n
![\"12\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/12-rancher-query1.png\")
<\/p>\n![\"13\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/13-rancher-query2.png\")
<\/p>\n![\"14\"](\"https:\/\/rancher.com\/img\/blog\/2018\/deploying-elasticsearch\/14-rancher-query3.png\")
<\/p>\nConclusion<\/h2>\n
Elasticsearch is extremely powerful. It is both simple and complex \u2013 simple to deploy and use, and complex in the way that it interacts with its data.<\/p>\n
This article has shown you the basics of how to deploy it with Rancher<\/a> and Kubernetes and how to query it via the RESTful API.<\/p>\n
If you wish to explore ways to use Elasticsearch in everyday situations, we encourage you to explore the other parts of the ELK stack: Kibana<\/a>, Logstash<\/a>, and Beats<\/a>. These tools round out an Elasticsearch deployment and make it useful for storing, retrieving, and visualizing a broad range of data from systems and applications.<\/p>\n
![\"Calin](\"https:\/\/rancher.com\/img\/bio\/calin-rus.jpg\")
<\/p>\nCalin Rus<\/p>\n
![\"github\"](\"https:\/\/rancher.com\/img\/icon-github.svg\")
<\/a><\/p>\n
- It allows fuzzy searching<\/em>, which helps find results even with misspelled search terms.<\/li>\n