{"id":1109,"date":"2019-01-24T03:48:23","date_gmt":"2019-01-24T03:48:23","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=1109"},"modified":"2019-01-24T03:53:26","modified_gmt":"2019-01-24T03:53:26","slug":"deploying-an-elasticsearch-cluster-using-rancher-catalog","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2019\/01\/24\/deploying-an-elasticsearch-cluster-using-rancher-catalog\/","title":{"rendered":"Deploying an Elasticsearch Cluster using Rancher Catalog"},"content":{"rendered":"<p>Elasticsearch is a Lucene-based search engine developed by the<br \/>\nopen-source vendor, <a href=\"https:\/\/www.elastic.co\/\">elastic<\/a>. With<br \/>\nprincipal features like scalability, resiliency, and top-notch<br \/>\nperformance, it has overtaken <a href=\"http:\/\/lucene.apache.org\/solr\/\">Apache<br \/>\nSolr<\/a>, one of its closest competitors.<br \/>\nNowadays, Elasticsearch is almost everywhere where a search engine is<br \/>\ninvolved: it\u2019s the E of the well-known <a href=\"https:\/\/www.elastic.co\/webinars\/introduction-elk-stack\">ELK<br \/>\nstack<\/a>, which<br \/>\nmakes it straightforward for your project to process analytics (the L<br \/>\nstands for <a href=\"https:\/\/www.elastic.co\/products\/logstash\">Logstash<\/a><br \/>\nwhich is used to process data like logs, streams, metrics; K stands for<br \/>\n<a href=\"https:\/\/www.elastic.co\/products\/kibana\">Kibana<\/a>, a data<br \/>\nvisualization platform \u2013 projects also managed by elastic).<br \/>\nInstalling Elasticsearch from the Rancher Catalog Before we get<br \/>\nstarted, let me tell you a bit about the <a href=\"https:\/\/github.com\/rancher\/community-catalog\">Rancher<br \/>\ncatalog<\/a>. The Rancher<br \/>\ncatalog uses rancher-compose and docker-compose to ease the installation<br \/>\nof whatever tool you need. Using the Rancher catalog, you can deploy<br \/>\neverything from a simple app like <a href=\"https:\/\/ghost.org\/\">ghost<\/a><br \/>\n(blogging platform) to a full CI\/CD stack like<br \/>\n<a href=\"https:\/\/www.go.cd\/\">GoCD<\/a>. I\u2019ll assume here that you have a fully<br \/>\nworking Rancher platform (a server and several nodes). If not, then<br \/>\nhead over to the Rancher documentation<br \/>\n<a href=\"http:\/\/docs.rancher.com\/rancher\/latest\/en\/\">here<\/a>, before going any<br \/>\nfurther in this article and set up your environment. My environment<br \/>\nlooks like this (Figure 1, built using docker-machine on my laptop):<br \/>\n<a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170452\/rachid-1-1.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170452\/rachid-1-1-1024x438.png\" alt=\"rachid\n1-1\" \/><\/a><\/p>\n<p><em>Figure<\/em> *1: Elasticsearch Environment***<\/p>\n<p>Accessing the Rancher catalog is simple:<\/p>\n<ul>\n<li>On the top menu bar of your Rancher UI, click on <em>Catalog<\/em>, then<br \/>\n<em>All<\/em>.<\/li>\n<li>Using the search box on the upper right, search for<br \/>\n*Elasticsearch***.<\/li>\n<li>You\u2019ll see two versions of Elasticsearch are available (Figure 2).<br \/>\nBoth work fine, but for this article, we\u2019ll stick to version on the<br \/>\nleft.<\/li>\n<li>Click on <em>View Details<\/em>. You will need to fill in some simple<br \/>\ninformation (Figure 3).<\/li>\n<li>To fire up the installation, click <em>Launch.<\/em><\/li>\n<\/ul>\n<p><a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170521\/Rachid-1-2.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170521\/Rachid-1-2-1024x467.png\" alt=\"Rachid\n1-2\" \/><\/a><\/p>\n<p><em>Figure<\/em> *2: Elasticsearch Options in the Rancher Catalog***<\/p>\n<p><a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170552\/rachid-1-3.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170552\/rachid-1-3-1024x890.png\" alt=\"rachid\n1-3\" \/><\/a><\/p>\n<p><em>Figure<\/em> *3: Elasticsearch Data Form***<\/p>\n<p>You should now see something similar to the image below (Figure 4).<br \/>\nYou can find more details about what Rancher is doing by clicking on the<br \/>\nname of your stack (in my case, I\u2019ve installed Elasticsearch, and named<br \/>\nmy stack <em>LocalEs<\/em>). <a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170610\/rachid-1-4.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170610\/rachid-1-4-1024x291.png\" alt=\"rachid\n1-4\" \/><\/a><\/p>\n<p><em>Figure<\/em> *4: LocalEs app Naming Convention***<\/p>\n<p>Expanding our view of the stack (Figure 5), we can see that deploying<br \/>\nan Elasticsearch cluster using the Rancher catalog template has<br \/>\nincluded:<\/p>\n<ul>\n<li>a Master node<\/li>\n<li>a Data node<\/li>\n<li>a Client node<\/li>\n<li><a href=\"https:\/\/github.com\/lmenezes\/elasticsearch-kopf\">kopf<\/a>, an<br \/>\nElasticsearch management web app<\/li>\n<\/ul>\n<p><a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170626\/rachid-1-5.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170626\/rachid-1-5-1024x428.png\" alt=\"rachid\n1-5\" \/><\/a><\/p>\n<p><em>Figure<\/em> *5: Elasticsearch Cluster Stack View***<\/p>\n<p>Each of these nodes (except for kopf) comes with sidekick containers,<br \/>\nwhich in this case are configuration and data volume containers. Your<br \/>\nElasticsearch cluster will be fully functional when all the entries are<br \/>\n<em>\u201cactive\u201d.<\/em> If you want to see how they are all connected to each other,<br \/>\ntake a look at the <em>graph view<\/em> (available from the drop-down menu<br \/>\non the right hand corner in Figure 6)<em>.<\/em> <a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170641\/rachid-1-6.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170641\/rachid-1-6-1024x431.png\" alt=\"rachid\n1-6\" \/><\/a><\/p>\n<p><em>Figure<\/em> *6: Elasticsearch Cluster Graph View***<\/p>\n<p>Now, we can visualize how all these containers as they are mapped<br \/>\nwithin the Rancher platform (Figure 7). <a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170705\/rachid-1-7.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170705\/rachid-1-7-1024x846.png\" alt=\"rachid\n1-7\" \/><\/a><\/p>\n<p><em>Figure<\/em> *7: Elasticsearch Visual Map***<\/p>\n<p>That\u2019s it, our Elasticsearch cluster is now up and running. Let\u2019s<br \/>\nsee how our cluster behaves! Cluster Management Depending on your<br \/>\nRancher setup, kopf is deployed on one of your Rancher nodes. You<br \/>\ncan access the application using http:\/\/[your kopf rancher host].<br \/>\nHere\u2019s an example (Figure 8): <a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170721\/rachid-1-8.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170721\/rachid-1-8-1024x346.png\" alt=\"rachid\n1-8\" \/><\/a><\/p>\n<p><em>Figure<\/em> *8: kopf node identification***<\/p>\n<p>As you can see, everything seems to be fine, as<br \/>\n<a href=\"https:\/\/github.com\/lmenezes\/elasticsearch-kopf\">kopf<\/a> shows a green<br \/>\ntop bar. Indeed, our cluster is running without any data stored, so<br \/>\nthere\u2019s no need for resiliency at this point. Let\u2019s see how it goes if<br \/>\nwe manually create an index called \u2018ranchercatalog\u2019, with three shards<br \/>\nand two replicas. Using curl, your query would be something like this:<\/p>\n<p>curl -XPUT &#8216;http:\/\/[your kopf rancher host]\/es\/ranchercatalog\/&#8217; -d &#8216;{<br \/>\n&#8220;settings&#8221; : {<br \/>\n&#8220;index&#8221; : {<br \/>\n&#8220;number_of_shards&#8221; : 3,<br \/>\n&#8220;number_of_replicas&#8221; : 2<br \/>\n}<br \/>\n}<br \/>\n}&#8217;<\/p>\n<p>Elasticsearch should reply {\u201cacknowledged\u201d:true}% Shards are<br \/>\nrelated to data storage, and replicas to resiliency. This means our<br \/>\nindex will have its data stored using three shards but needs two more<br \/>\nreplicas of these shards. Now that our index has been successfully<br \/>\ncreated, let\u2019s talk a look at<br \/>\n<a href=\"https:\/\/github.com\/lmenezes\/elasticsearch-kopf\">kopf<\/a>. <a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170736\/rachid-1-9.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170736\/rachid-1-9-1024x377.png\" alt=\"rachid\n1-9\" \/><\/a><\/p>\n<p><em>Figure<\/em> *9: kopf Status View***<\/p>\n<p>As you can see in Figure 9, the top bar is now yellow, which indicates<br \/>\nthere may be something wrong with our Elasticsearch cluster. We can also<br \/>\nsee in the middle left of the page a warning sign (red triangle in Fig.<br \/>\n9) saying \u201csix unassigned shards.\u201d Remember when we created the<br \/>\nranchercatalog index, we specified:<\/p>\n<ul>\n<li>Three shards<\/li>\n<li>Two replicas<\/li>\n<\/ul>\n<p>By default, the Elasticsearch Rancher catalog item deploys only 1 data<br \/>\nnode, so we need two more data nodes. Adding nodes can be easily done<br \/>\nusing the Rancher scale option. The results are shown in Figure 10.<br \/>\n<a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170752\/rachid-1-10.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170752\/rachid-1-10-1024x630.png\" alt=\"rachid\n1-10\" \/><\/a><\/p>\n<p><em>Figure<\/em> *10: Adding Nodes using Rancher Scale Option***<\/p>\n<p>To scale your data nodes, let\u2019s go again to <em>Applications<\/em>, then<br \/>\nto <em>Stack.<\/em> Click on your stack, and then on<br \/>\n<em>elasticsearch-datanodes<\/em>. You should have something like what is<br \/>\nshown in Figure 10. Click 2 times on the + of the scale option and<br \/>\nlet Rancher do the work. You should see data nodes popping up one after<br \/>\nanother until you finally have something like what you see in Figure 11.<br \/>\n<a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170808\/rachid-1-11.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170808\/rachid-1-11-1024x876.png\" alt=\"rachid\n1-11\" \/><\/a><\/p>\n<p><em>Figure<\/em> *11: Node View to Verify Additions***<\/p>\n<p>Let\u2019s check if this is enough to bring back the beautiful green bar<br \/>\nto <a href=\"https:\/\/github.com\/lmenezes\/elasticsearch-kopf\">kopf<\/a>. Figure 12<br \/>\nprovides the proof. <a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170823\/Rachid-1-12.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08170823\/Rachid-1-12-1024x509.png\" alt=\"Rachid\n1-12\" \/><\/a><\/p>\n<p><em>Figure<\/em> *12: Corrected Nodes Verification***<\/p>\n<p>Voil\u00e0! We now have a perfect and fully functional Elasticsearch<br \/>\nCluster. In my next post, we\u2019ll see how to populate this index and do<br \/>\nsome cool queries! <em><a href=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08180221\/rachid-photo.png\"><img decoding=\"async\" src=\"http:\/\/cdn.rancher.com\/wp-content\/uploads\/2016\/06\/08180221\/rachid-photo.png\" alt=\"rachid\nphoto\" \/><\/a><\/em><br \/>\n*Rachid is a former virtualization consultant and Instructor. * <em>After a<br \/>\nsuccessful experience building and training the ops team of the French<br \/>\nregistry AFNIC, he is now the CIO of a worldwide recognized CRM and<br \/>\necommerce agency.<\/em><\/p>\n<p><a href=\"https:\/\/rancher.com\/deploying-an-elasticsearch-cluster-using-rancher-catalog\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Elasticsearch is a Lucene-based search engine developed by the open-source vendor, elastic. With principal features like scalability, resiliency, and top-notch performance, it has overtaken Apache Solr, one of its closest competitors. Nowadays, Elasticsearch is almost everywhere where a search engine is involved: it\u2019s the E of the well-known ELK stack, which makes it straightforward for &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.appservgrid.com\/paw93\/index.php\/2019\/01\/24\/deploying-an-elasticsearch-cluster-using-rancher-catalog\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Deploying an Elasticsearch Cluster using Rancher Catalog&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1109","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1109","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/comments?post=1109"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1109\/revisions"}],"predecessor-version":[{"id":1121,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1109\/revisions\/1121"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/media?parent=1109"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/categories?post=1109"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/tags?post=1109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}