{"id":1368,"date":"2019-02-26T00:06:19","date_gmt":"2019-02-26T00:06:19","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=1368"},"modified":"2019-03-07T20:29:53","modified_gmt":"2019-03-07T20:29:53","slug":"aws-and-magento-creating-a-magento-cluster-on-aws","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2019\/02\/26\/aws-and-magento-creating-a-magento-cluster-on-aws\/","title":{"rendered":"AWS and Magento | Creating a Magento Cluster on AWS"},"content":{"rendered":"

\"magento-logo2\"<\/a><\/p>\n

[Usman is a server and infrastructure engineer, with experience in
\nbuilding large scale distributed services on top of various cloud
\nplatforms. You can read more of his work at
\ntechtraits.com<\/a>, or follow him on twitter
\n@usman_ismail<\/a>or on
\nGitHub<\/a>.]<\/em><\/p>\n

Magento<\/a> is an open-source content management
\nsystem (CMS) offering a powerful tool-set for managing eCommerce
\nweb-sites. Magento is used by thousands of companies including Nike and
\nOffice Max. Today we are going to walk through the process of setting up
\na Magento cluster using Docker and Rancher on the Amazon Elastic Compute
\nCloud (EC2). We use Docker because it makes deployments simple,
\nrepeatable, and portable across cloud service providers and in-house
\nhardware. Using Rancher we can extend the Docker primitives across
\nmultiple servers even if those servers are on different cloud providers.<\/p>\n

We will be using the Official MySQL
\nimage<\/a>, Official Memcached
\nImage<\/a> and
\na Magento<\/a> image I
\ncreated. Despite its many benefits, one area where Docker is still
\nevolving is managing entire clusters. It is possible to have multi-node
\ndeployments with Docker installed on each node, but you have to manage
\nthe containers on each node independently by connecting through ssh.
\nFurthermore, we lose the ability to connect containers using Docker
\nlinking. This is where Rancher comes in. Rancher creates a Virtual
\nPrivate Network (VPN) using IPSec between all Docker containers and
\nallows us to communicate between containers using standard Docker
\nprimitives. Additionally, Rancher gives us a nice UI to launch and
\nmanage containers without having to ssh into individual servers. We are
\nlaunching all nodes on EC2, however you could use any combination of
\nservers–some from EC2 or from in-house hardware or from Digital Ocean
\nif you choose.<\/p>\n

The next sections walk through setting up the Amazon environment,
\nRancher server, and a array of Rancher compute nodes. We will then use
\nthose nodes to launch our Magneto deployment.<\/p>\n

Amazon Environment Setup<\/h3>\n

We are going to bring up our cluster on top of Amazon EC2, and for this
\nwe need an Amazon Web Services (AWS)<\/a> account
\nand some familiarity with the AWS
\nConsole<\/a>.
\nIf you need a refresher you can peruse the AWS Console
\nDocumentation<\/a>.
\nOnce you are signed into the console, click through to the EC2<\/em> service
\nand from there select the Key Pairs<\/em> menu item in the Network and
\nSecurity<\/em> section of the side menu. Click Create Key Pair<\/em> and specify
\na name in the pop-up screen to create a key. When you click Create,<\/em> a
\npem file will be downloaded to your computer. Keep the key file
\nsomewhere safe as it is needed to login to your servers and also allows
\nsomeone to access your servers if they gain access to it. We will also
\nneed to setup a security group by selecting the Security Groups<\/em> option
\nin the side menu and clicking the Create Security Group<\/em> button. Select
\nthe default Virtual Private Cloud (VPC) to create the security group in
\nand open; port 8080, 9345 and 9346 to the internet. You will also need
\nto expose port 22 so that you can ssh into your server if necessary. For
\nthis port, you can select the *My IP *option instead of Anywhere if you
\nwould like to keep login access to your nodes limited.<\/p>\n

\"racher-security-group\"<\/a><\/p>\n

Rancher Server Launch<\/h3>\n

We are now ready to launch our Rancher server; for this we will Select
\nthe Instances<\/em> option from the side menu and then click Launch
\nInstance<\/em>. This is a 7-step process beginning with Selecting an Amazon
\nMachine Image (AMI). We will just use the default Amazon Linux AMI. Step
\ntwo requires us to choose an instance size for the Rancher server. We
\nwill use the *t2.micro, *the least expensive instance type but recommend
\nyou use a larger instance type for a real deployment. In step three, we
\nhave to specify instance details; you can leave most fields set to their
\ndefaults values but make sure the Auto-assign Public IP<\/em> setting is set
\nto enabled otherwise you will not be able to access your server.<\/p>\n

\"instance-details\"<\/p>\n

Scroll to the bottom of the page and expand the Advanced
\nDetails<\/em> section and add the following code into the User data<\/em> text
\nbox. Amazon uses this as an initializer for your instance and will make
\nsure that Docker and the Rancher Server is installed and running. Note
\nthat we are using the package manager yum to install Docker and then
\noverwriting the binary with the latest one from Docker.com. We do this
\nbecause Rancher requires Docker version 1.5 or higher but the
\nrepositories have not been updated past version 1.3 as yet.<\/p>\n

#!\/bin\/bash
\nyum install docker -y
\nwget https:\/\/get.docker.com\/builds\/Linux\/x86_64\/docker-latest -O docker
\nchmod +x docker
\nmv -f .\/docker $(which docker)
\nservice docker restart
\ndocker run -d -p 8080:8080 rancher\/server<\/p>\n

You may select the defaults for all subsequent options other than
\nSecurity Group for which you should select the one we created earlier.
\nSimilarly when asked to select ssh key select the one we created
\nearlier. A few minutes after you create your instance the Rancher server
\nshould be up and running. You can find your servers Public IP by
\nselecting the server in the console and then browse to
\nhttp:\/\/RANCHER_SERVER_IP:8080\/<\/a> in a browser of your choice and you
\nwill see the Rancher web console. Also note the Private IP of Rancher
\nfrom the details section of the Amazon Console; we will need this later
\non.<\/p>\n

\"rancher-server\"<\/a><\/p>\n

Rancher Compute Node Setup\"rancher-ui\"<\/a><\/h3>\n

The first step in creating your Rancher compute nodes is to get the
\nagent launch command from the Rancher server. To do this, open up your
\nRancher server in a browser and click Register a new host<\/em>. You will be
\npresented with a pop window with a Docker run command that you can use
\nto bring up a Docker agent. Copy that command, and replace the Public IP
\nof the Rancher Server with its private IP. This is so required because
\nRancher uses the private network for inter-server communication and
\nhence is not blocked by the Security Groups. You will also have to
\nremove the \u2018-it\u2019 switch and replace it with \u2018-d\u2019 to reflect the fact
\nthat we are running this container in a non-interactive shell. Also note
\nthat the IP Address (52.1.151.186) and secret (6C97B49FE39413B\u2026) shown
\nbelow are unique to each setup and will be different for you.<\/p>\n

docker run –rm -it –privileged -v \/var\/run\/docker.sock:\/var\/run\/docker.sock
\nrancher\/agent http:\/\/52.1.151.186:8080\/v1\/scripts\/6C97B49FE39413B2B76B:
\n1424538000000:6UL0o28EXZIkjZbmPOYMGxmM9RU<\/p>\n

With this command we can now create our array launch configuration. We
\ndo this by selecting the Launch Configurations<\/em> item from the side menu
\nand clicking *Create Launch Configuration. You will then be asked to
\nfollow the same 7-step form that you followed for the Rancher Server
\ninstance launch<\/em>. As before<\/em>,* select the Amazon Linux AMI, an instance
\ntype of your choice, storage size, and the security group and ssh key we
\ncreated earlier. The only difference for the instance setup form is on
\nStep 3, Configure Launch Configuration<\/em>. In the Advanced details
\nsection you must select \u201cAssign a public IP to every instance\u201d<\/em> to
\nevery instance if you wish for your Docker containers to be publicly
\naccessible. In addition add the following lines into the User data text
\nbox. This script is identical to the one we used to launch Rancher
\nserver other than the last line. We replaced the Docker run for the
\nRancher server command with the modified Docker run command for the
\nRancher agent which we copied from the UI.<\/p>\n

#!\/bin\/bash
\nyum install docker -y
\nwget https:\/\/get.docker.com\/builds\/Linux\/x86_64\/docker-latest -O docker
\nchmod +x docker
\nmv -f .\/docker $(which docker)
\nservice docker restart
\ndocker run –rm -d –privileged -v \/var\/run\/docker.sock:\/var\/run\/docker.sock Rancher\/agent http:\/\/172.30.2.200:8080\/<\/p>\n

Now that we have created the launch configuration we can create an
\nautoscaling array with our configuration. We select the Auto Scaling
\nGroups<\/em> option of the side menu and click *Create Auto Scaling
\nGroup. *In the first step we will be asked to specify the launch
\nconfiguration, select the launch configuration we just created in the
\nprevious paragraph. In the second step specify a name for this auto
\nscaling group, set the group size to however many compute nodes you
\nrequire. Note that this value can be changed later if your needs
\nincrease or decrease. We will be using two agents to illustrate the
\nnetwork connectivity features of Rancher.
\nFor Network<\/em> and Subnet<\/em> choose the default VPC and the same
\navailability zone you used for the Rancher server. The rest of the steps
\ncan be left at default settings. Once you have created your auto scaling
\ngroup wait a few minutes for the instances to be launched and then
\nbrowse to the Rancher Server URL to see that the new agents have been
\nregistered. Note that you can always add more agents to your cluster by
\nediting the Desired Nodes setting in the Auto scaling group
\nconfiguration. With our launch configuration setup all new nodes will
\nregister themselves with the Rancher server automatically.<\/p>\n

Magento Setup<\/h3>\n

Now that we have our two node Rancher cluster launched we can setup our
\nMagento containers. However before we launch our Magento Cluster we must
\nfirst launch a MySQL container to serve as our database and a Memcached
\ncluster for caching. Let\u2019s launch our MySQL container first on one of
\nthe compute nodes. We do this by mouse hovering over the server in the
\nRancher Server UI and clicking the plus icon. In the pop up menu we need
\nto specify a name for our container and mysql<\/em> as the source image.
\nSelect the *Command *tab in the menu to the left and add 4 Environment
\nVariables: mysql root password, mysql user, mysql password, and mysql
\ndatabase. You may choose any values for these variables except mysql
\ndatabase which must be set to Magento. After adding all of these
\nenvironment variables, hit create to create the container. Note that
\nmysql is official Docker mysql image and details of what is inside this
\ncontainer can be found on
\nits dockerhub<\/a> page.<\/p>\n

\"mysql_env\"<\/a><\/p>\n

Next we will create the Memcached container on the empty compute node by
\nhitting the plus icon. We again give the container a name and specify
\nits source image as memcached.<\/em> The Memcached container does not
\nrequire any further configuration and therefore we can just click create
\nto setup the container. Details of the memcached official container we
\nuse can be found on
\nits dockerhub<\/a> page.
\nLastly we will create the Magento container from an image I created
\ncalled usman\/magento<\/a>.
\nCreate the Magento container on the same compute node as the cache (so
\nthat cache is accessible faster). Specify usman\/magento as the source
\nimage. In the ports section add a mapping from 8080 public to 80 in host
\nports. Make sure that you select theManaged Network on docker0<\/em> option
\nfor both mysql and memcached containers so that we can connect to them
\nfrom our Magento container.<\/p>\n

\"links\"<\/a><\/p>\n

In the links section, add links to the mysql and memcached containers
\nthat we just setup. It is important the mysql container be named db and
\nthe memcached container be named cache as shown in the image to the
\nright. We need to choose these names because Rancher creates environment
\nvariables in the Magento container telling it where to connect to MySQL
\nand Memcached. These variables are based on the name of the linked
\ncontainer and hence we the need the values to remain the same. By
\nlinking containers through Rancher the containers are able to
\ncommunicate even though we did not expose their ports. Rancher extends
\nDocker\u2019s linking concept to support multiple nodes by providing
\na virtual private network between the hosts using IPsec tunneling. We
\nwill need to know the IP and Port at which MySQL is available to the
\nMagento container so that we can use it later when we configure Magento
\nthrough the web interface. So go head and hit create and wait for the
\ncontainer to get into the Running state.<\/p>\n

\"exec-shell\"<\/a><\/p>\n

Once the container is running we can use another Rancher feature to open
\na shell into containers directly from the UI to retrieve the network
\ninformation we require. Click the inverted chevron next to the Magento
\ncontainer and select Execute Shell<\/em>. You will be presented with pop up
\nwindow and an embedded shell connected to the container. In the shell
\nuse the env command to list the environment variables and grep
\nfor DB_PORT_3306_TCP_. This will list the IP, Port and Protocol at
\nwhich the DB is available. As an aside the ADDR will be the IP of the
\nNetwork Agent on the server that Magento is running. This is because the
\nnetwork agent will be proxying traffic to the remote container.<\/p>\n

env | grep DB_PORT_3306_TCP_
\nDB_PORT_3306_TCP_PORT=28428
\nDB_PORT_3306_TCP_PROTO=tcp
\nDB_PORT_3306_TCP_ADDR=10.42.81.155<\/p>\n

The Magento container should be up by now and you can browse to port
\n8080 on the public interface of the Amazon server running Magento. Note
\nthat the public IP is not shown inside Rancher as we used the private
\ninterfaces to setup agents–you will have to lookup the IP from the
\nAmazon console. If everything is working correctly you should see the
\nMagento installation wizard that will guide you through the rest of the
\nprocess. The initial page will ask you to accept the Terms of Service,
\nand then subsequent pages will as ask for locale information and
\ndatabase configuration. For database configuration enter the value of
\nthe $DB_PORT_3306_TCP_ADDR<\/em> that we looked up earlier followed by
\na colon and $DB_PORT_3306_TCP_PORT<\/em> in to the Host field. Also
\nspecify the database name as *magento and username<\/em> *and password to
\nmatch values you selected for MYSQL_USER<\/em> and MYSQL_PASSWORD<\/em>
\nrespectively. Next you will be asked to create an administrator user and
\npassword for Magento and with that you are done. You should have a fully
\nfunctional, albeit empty Magento website.<\/p>\n

\"magento-configruation\"<\/a><\/p>\n

If you got this far, you have successfully launched a Rancher cluster on
\nthe Amazon Compute Cloud and created a distributed Magento deployment
\nusing Docker containers. We have also used two salient features of
\nRancher–the VPN that allows private communication between containers
\nand the ability to manage containers on the entire cluster from a
\nweb-UI. During this entire tutorial we never had to ssh into any of our
\nservers. Although the Magento cluster we setup was just meant to be an
\nexample, with a few modifications such as using a replicated MySQL
\ndatabase and a larger array of Rancher compute nodes, we could very
\neasily use Rancher for a production Magento deployment.<\/p>\n

A word of caution if you do plan on using Rancher for large-scale
\ndeployments: Rancher is still in its infancy, and there are some
\nfeatures that have not been released yet. For example if we were to run
\nmultiple Magento containers, we would need to setup our own
\nload-balancing solution to distribute traffic to the containers. Another
\nissue is that container configuration is stored on the agents where the
\ncontainers are running and hence losing an agent means not only losing
\nthe containers running on it but also the configuration used to launch
\nthem. The containers we launched today are fairly simple but this is not
\nalways the case. Both ELBs and support for Docker Compose with some form
\nof templating are on the Rancher roadmap, so hopefully these items will
\nbe addressed soon. I am not using Rancher on user-facing productions
\nsystems just yet but it is going to be a larger and larger part of the
\nDocker based systems I manage.<\/p>\n

If you would like to learn more about Rancher, please request a
\none-on-one demo to get a better understanding of how the platform works.<\/p>\n

[Usman is a server and infrastructure engineer, with experience in
\nbuilding large scale distributed services on top of various cloud
\nplatforms. You can read more of his work at
\ntechtraits.com<\/a>, or follow him on twitter
\n@usman_ismail<\/a>or on
\nGitHub<\/a>.]<\/em><\/p>\n

Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

[Usman is a server and infrastructure engineer, with experience in building large scale distributed services on top of various cloud platforms. You can read more of his work at techtraits.com, or follow him on twitter @usman_ismailor on GitHub.] Magento is an open-source content management system (CMS) offering a powerful tool-set for managing eCommerce web-sites. Magento … <\/p>\n

Continue reading “AWS and Magento | Creating a Magento Cluster on AWS”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1368","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/comments?post=1368"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1368\/revisions"}],"predecessor-version":[{"id":1443,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/1368\/revisions\/1443"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/media?parent=1368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/categories?post=1368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/tags?post=1368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}