Deploying an Elasticsearch Cluster using Rancher Catalog

Elasticsearch is a Lucene-based search engine developed by the
open-source vendor, elastic. With
principal features like scalability, resiliency, and top-notch
performance, it has overtaken Apache
Solr
, one of its closest competitors.
Nowadays, Elasticsearch is almost everywhere where a search engine is
involved: it’s the E of the well-known ELK
stack
, which
makes it straightforward for your project to process analytics (the L
stands for Logstash
which is used to process data like logs, streams, metrics; K stands for
Kibana, a data
visualization platform – projects also managed by elastic).
Installing Elasticsearch from the Rancher Catalog Before we get
started, let me tell you a bit about the Rancher
catalog
. The Rancher
catalog uses rancher-compose and docker-compose to ease the installation
of whatever tool you need. Using the Rancher catalog, you can deploy
everything from a simple app like ghost
(blogging platform) to a full CI/CD stack like
GoCD. I’ll assume here that you have a fully
working Rancher platform (a server and several nodes). If not, then
head over to the Rancher documentation
here, before going any
further in this article and set up your environment. My environment
looks like this (Figure 1, built using docker-machine on my laptop):
rachid
1-1

Figure *1: Elasticsearch Environment***

Accessing the Rancher catalog is simple:

  • On the top menu bar of your Rancher UI, click on Catalog, then
    All.
  • Using the search box on the upper right, search for
    *Elasticsearch***.
  • You’ll see two versions of Elasticsearch are available (Figure 2).
    Both work fine, but for this article, we’ll stick to version on the
    left.
  • Click on View Details. You will need to fill in some simple
    information (Figure 3).
  • To fire up the installation, click Launch.

Rachid
1-2

Figure *2: Elasticsearch Options in the Rancher Catalog***

rachid
1-3

Figure *3: Elasticsearch Data Form***

You should now see something similar to the image below (Figure 4).
You can find more details about what Rancher is doing by clicking on the
name of your stack (in my case, I’ve installed Elasticsearch, and named
my stack LocalEs). rachid
1-4

Figure *4: LocalEs app Naming Convention***

Expanding our view of the stack (Figure 5), we can see that deploying
an Elasticsearch cluster using the Rancher catalog template has
included:

  • a Master node
  • a Data node
  • a Client node
  • kopf, an
    Elasticsearch management web app

rachid
1-5

Figure *5: Elasticsearch Cluster Stack View***

Each of these nodes (except for kopf) comes with sidekick containers,
which in this case are configuration and data volume containers. Your
Elasticsearch cluster will be fully functional when all the entries are
“active”. If you want to see how they are all connected to each other,
take a look at the graph view (available from the drop-down menu
on the right hand corner in Figure 6). rachid
1-6

Figure *6: Elasticsearch Cluster Graph View***

Now, we can visualize how all these containers as they are mapped
within the Rancher platform (Figure 7). rachid
1-7

Figure *7: Elasticsearch Visual Map***

That’s it, our Elasticsearch cluster is now up and running. Let’s
see how our cluster behaves! Cluster Management Depending on your
Rancher setup, kopf is deployed on one of your Rancher nodes. You
can access the application using http://[your kopf rancher host].
Here’s an example (Figure 8): rachid
1-8

Figure *8: kopf node identification***

As you can see, everything seems to be fine, as
kopf shows a green
top bar. Indeed, our cluster is running without any data stored, so
there’s no need for resiliency at this point. Let’s see how it goes if
we manually create an index called ‘ranchercatalog’, with three shards
and two replicas. Using curl, your query would be something like this:

curl -XPUT ‘http://[your kopf rancher host]/es/ranchercatalog/’ -d ‘{
“settings” : {
“index” : {
“number_of_shards” : 3,
“number_of_replicas” : 2
}
}
}’

Elasticsearch should reply {“acknowledged”:true}% Shards are
related to data storage, and replicas to resiliency. This means our
index will have its data stored using three shards but needs two more
replicas of these shards. Now that our index has been successfully
created, let’s talk a look at
kopf. rachid
1-9

Figure *9: kopf Status View***

As you can see in Figure 9, the top bar is now yellow, which indicates
there may be something wrong with our Elasticsearch cluster. We can also
see in the middle left of the page a warning sign (red triangle in Fig.
9) saying “six unassigned shards.” Remember when we created the
ranchercatalog index, we specified:

  • Three shards
  • Two replicas

By default, the Elasticsearch Rancher catalog item deploys only 1 data
node, so we need two more data nodes. Adding nodes can be easily done
using the Rancher scale option. The results are shown in Figure 10.
rachid
1-10

Figure *10: Adding Nodes using Rancher Scale Option***

To scale your data nodes, let’s go again to Applications, then
to Stack. Click on your stack, and then on
elasticsearch-datanodes. You should have something like what is
shown in Figure 10. Click 2 times on the + of the scale option and
let Rancher do the work. You should see data nodes popping up one after
another until you finally have something like what you see in Figure 11.
rachid
1-11

Figure *11: Node View to Verify Additions***

Let’s check if this is enough to bring back the beautiful green bar
to kopf. Figure 12
provides the proof. Rachid
1-12

Figure *12: Corrected Nodes Verification***

Voilà! We now have a perfect and fully functional Elasticsearch
Cluster. In my next post, we’ll see how to populate this index and do
some cool queries! rachid
photo
*Rachid is a former virtualization consultant and Instructor. * After a
successful experience building and training the ops team of the French
registry AFNIC, he is now the CIO of a worldwide recognized CRM and
ecommerce agency.

Source

Leave a Reply

Your email address will not be published. Required fields are marked *