GlusterFS
is a scalable, highly available, and distributed network file system
widely used for applications that need shared storage including cloud
computing, media streaming, content delivery networks, and web cluster
solutions. High availability is ensured by the fact that storage data is
redundant, so in case one node fails another will cover it without
service interruption. In this post I’ll show you how to create a
GlusterFS cluster for Docker that you can use to store your containers
data. The storage volume where data resides is replicated twice, so data
will be accessible if at least one Gluster container is working. We’ll
use Rancher for Docker management and orchestration. In order to test
storage availability and reliability I’ll be deploying an Asteroids
game.
Prerequisites
Preparing AWS environment Before deploying the GlusterFS cluster you
need to satisfy the following requirements in AWS:
- Create an Access Key to use Rancher AWS provisioning feature.
You can get an Access Ke
-
- Allow 22/tcp, 2376/tcp and 8080/tcp ports from any source,
needed for Docker machine to provision hosts
- Allow 500/udp and 4500/udp ports from any source, needed for
Rancher network
- Allow 9345/tcp and 9346/tcp ports from any source, needed for UI
features like graphs, view logs, and execute shell
- Allow 80/tcp and 443/tcp ports from any source, needed to
publish the Asteroids game
- Create a RancherOS instance (look for RancherOS AMI in
Community AMIs). Configure it to run Rancher Server by
defining the following user data and associate it to the Gluster
Security Group. Once the instance is running you can browse to
Rancher UI: http://RANCHER_INSTANCE_PUBLIC_IP:8080/
#!/bin/bash
docker run -d -p 8080:8080 rancher/server:v0.17.1
Preparing Docker images
I have prepared two Docker images that we are using later. This is how I
built them. The GlusterFS server image This is the Dockerfile:
FROM ubuntu:14.04
MAINTAINER Manel Martinez <manel@nixelsolutions.com>
RUN apt-get update &&
apt-get install -y python-software-properties software-properties-common
RUN add-apt-repository -y ppa:gluster/glusterfs-3.5 &&
apt-get update &&
apt-get install -y glusterfs-server supervisor
RUN mkdir -p /var/log/supervisor
ENV GLUSTER_VOL ranchervol
ENV GLUSTER_REPLICA 2
ENV GLUSTER_BRICK_PATH /gluster_volume
ENV GLUSTER_PEER **ChangeMe**
ENV DEBUG 0
VOLUME [“/gluster_volume”]
RUN mkdir -p /usr/local/bin
ADD ./bin /usr/local/bin
RUN chmod +x /usr/local/bin/*.sh
ADD ./etc/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
CMD [“/usr/local/bin/run.sh”]
As you can see, we are using 2 replicas for distributing the Gluster
volume ranchervol. All its data will be persisted in Docker volume
/gluster_volume. Note that we are not exposing any port because
GlusterFS containers are connecting through Rancher network. The
run.sh script is as follows:
#!/bin/bash
[ “$DEBUG” == “1” ] && set -x
prepare-gluster.sh &
/usr/bin/supervisord
It will invoke another script to prepare GlusterFS cluster in
background. This is required because Gluster commands need to be
executed when its daemon is running. This is the content for
prepare-gluster.sh script:
#!/bin/bash
set -e
[ “$DEBUG” == “1” ] && set -x
if [ “$” == “**ChangeMe**” ]; then
# This node is not connecting to the cluster yet
exit 0
fi
echo “=> Waiting for glusterd to start…”
sleep 10
if gluster peer status | grep $ >/dev/null; then
echo “=> This peer is already part of Gluster Cluster, nothing to do…”
exit 0
fi
echo “=> Probing peer $…”
gluster peer probe $
echo “=> Creating GlusterFS volume $…”
my_rancher_ip=`echo $ | awk -F/ ”`
gluster volume create $ replica $ $:$ $:$ force
echo “=> Starting GlusterFS volume $…”
gluster volume start $
As we can see, if we don’t provide GLUSTER_PEER environment variable
the container will only start GlusterFS daemon and wait for a second
peer container to join the cluster. The second container needs to know
about GLUSTER_PEER address in order to contact it (peer probe) and
create the shared storage volume. This is the supervisor configuration
file, needed to start GlusterFS daemon:
[supervisord]
nodaemon=true
[program:glusterd]
command=/usr/sbin/glusterd -p /var/run/glusterd.pid
The following commands are required to publish the Docker image:
docker build -t nixel/rancher-glusterfs-server .
docker push nixel/rancher-glusterfs-server .
The Asteroids game image This is the image we are using to publish
the Asteroids HTML5 game for testing Gluster HA capabilities. This
container acts as a GlusterFS client that will mount the shared volume
where the following game content is being stored:
- static files (HTML, JS, CSS) needed to open the client-side game in
your browser. A Nginx server will publish this to the Internet.
- A WebSocket server application used to handle user connections and
control game logics. A Node.js service will publish this application
to the Internet.
This is the Dockerfile which defines the image:
FROM ubuntu:14.04
MAINTAINER Manel Martinez <manel@nixelsolutions.com>
RUN apt-get update &&
apt-get install -y python-software-properties software-properties-common
RUN add-apt-repository -y ppa:gluster/glusterfs-3.5 &&
apt-get update &&
apt-get install -y git nodejs nginx supervisor glusterfs-client dnsutils
ENV GLUSTER_VOL ranchervol
ENV GLUSTER_VOL_PATH /mnt/$
ENV GLUSTER_PEER **ChangeMe**
ENV DEBUG 0
ENV HTTP_CLIENT_PORT 80
ENV GAME_SERVER_PORT 443
ENV HTTP_DOCUMENTROOT $/asteroids/documentroot
EXPOSE $
EXPOSE $
RUN mkdir -p /var/log/supervisor $
WORKDIR $
RUN mkdir -p /usr/local/bin
ADD ./bin /usr/local/bin
RUN chmod +x /usr/local/bin/*.sh
ADD ./etc/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ADD ./etc/nginx/sites-available/asteroids /etc/nginx/sites-available/asteroids
RUN echo “daemon off;” >> /etc/nginx/nginx.conf
RUN rm -f /etc/nginx/sites-enabled/default
RUN ln -fs /etc/nginx/sites-available/asteroids /etc/nginx/sites-enabled/asteroids
RUN perl -p -i -e “s/HTTP_CLIENT_PORT/$/g” /etc/nginx/sites-enabled/asteroids
RUN HTTP_ESCAPED_DOCROOT=`echo $ | sed “s///\\\\//g”` && perl -p -i -e “s/HTTP_DOCUMENTROOT/$/g” /etc/nginx/sites-enabled/asteroids
RUN perl -p -i -e “s/GAME_SERVER_PORT/$/g” /etc/supervisor/conf.d/supervisord.conf
RUN HTTP_ESCAPED_DOCROOT=`echo $ | sed “s///\\\\//g”` && perl -p -i -e “s/HTTP_DOCUMENTROOT/$/g” /etc/supervisor/conf.d/supervisord.conf
CMD [“/usr/local/bin/run.sh”]
And this is the run.sh script:
#!/bin/bash
set -e
[ “$DEBUG” == “1” ] && set -x && set +e
if [ “$” == “**ChangeMe**” ]; then
echo “ERROR: You did not specify “GLUSTER_PEER” environment variable – Exiting…”
exit 0
fi
ALIVE=0
for PEER in `echo “$” | sed “s/,/ /g”`; do
echo “=> Checking if I can reach GlusterFS node $ …”
if ping -c 10 $ >/dev/null 2>&1; then
echo “=> GlusterFS node $ is alive”
ALIVE=1
break
else
echo “*** Could not reach server $ …”
fi
done
if [ “$ALIVE” == 0 ]; then
echo “ERROR: could not contact any GlusterFS node from this list: $ – Exiting…”
exit 1
fi
echo “=> Mounting GlusterFS volume $ from GlusterFS node $ …”
mount -t glusterfs $:/$ $
echo “=> Setting up asteroids game…”
if [ ! -d $ ]; then
git clone https://github.com/BonsaiDen/NodeGame-Shooter.git $
fi
my_public_ip=`dig -4 @ns1.google.com -t txt o-o.myaddr.l.google.com +short | sed “s/”//g”`
perl -p -i -e “s/HOST = ‘.*’/HOST = ‘$’/g” $/client/config.js
perl -p -i -e “s/PORT = .*;/PORT = $;/g” $/client/config.js
/usr/bin/supervisord
As you can see we need to inform about GlusterFS containers where
ranchervol storage is being served using GLUSTER_PEER environment
variable. Although GlusterFS client does not need to know about all
cluster nodes, this is useful for Asteroids container to be able to
mount the volume if at least one GlusterFS container is alive. We are
proving this HA feature later. In this case we are exposing 80 (Nginx)
and 443 (Node.js Websocket server) ports so we can open the game in our
browser. This is the Nginx configuration file:
server {
listen HTTP_CLIENT_PORT;
location / {
root HTTP_DOCUMENTROOT/client/;
}
}
And the following supervisord configuration is required to run Nginx and
Node.js:
[supervisord]
nodaemon=true
[program:nginx]
command=/usr/sbin/nginx
[program:nodejs]
command=/usr/bin/nodejs HTTP_DOCUMENTROOT/server/server.js GAME_SERVER_PORT
Finally, the run.sh script will download the Asteroids source code and
save it on GlusterFS shared volume. The last step is to replace the
required parameters on configuration files to run Nginx and Node.js
server application. The following commands are needed to publish the
Docker image:
docker build -t nixel/rancher-glusterfs-client .
docker push nixel/rancher-glusterfs-client .
Creating Docker hosts
Now we need to create three Docker hosts, two of them used to run
GlusterFS server containers, and the third to publish the Asteroids
game.
In Rancher UI, click + Add Host button and choose Amazon EC2
provider. You need to specify, at least, the following information:
- Container names
- Amazon Access Key and Secret Key that you got before.
- EC2 Region, Zone and VPC/Subnet ID. Be sure to choose the same
region, zone and VPC/subnet ID where Rancher Server is deployed.
- Type the Security Group name that we created before: Gluster.
Repeat this step three times to create gluster01, gluster02, and
asteroids hosts. 
Adding GlusterFS server containers
Now you are ready to deploy your GlusterFS cluster. First, click +
Add Container button on gluster01 host and enter the following
information:
- Name: gluster01
- Image: nixel/rancher-glusterfs-server:latest
Expand Advanced Options and follow these steps:
- Volumes section – Add this volume:
/gluster_volume:/gluster_volume
- Networking section – Choose Managed Network on Docker0
- Security/Host section – Enable Give the container full access to
the host checkbox
Now wait for gluster01 container to be created and copy its Rancher
IP address, you are needing it now. Then click + Add Container button
on gluster02 host to create the second GlusterFS server container
with the following configuration:
- Name: gluster02
- Image: nixel/rancher-glusterfs-server:latest
Expand Advanced Options and follow these steps:
- Command section – Add an Environment Variable named GLUSTER_PEER
which value is the gluster01 container IP. In my case it is
10.42.46.31
- Volumes section – Add this volume:
/gluster_volume:/gluster_volume
- Networking section – Choose Managed Network on Docker0
- Security/Host section – Enable Give the container full access to
the host checkbox
Now wait for gluster02 container to be created and open its menu,
then click View Logs option.
You will see the following messages at the bottom of log screen
confirming that shared volume was successfully created.
Adding Asteroids container
Now it is time to create our GlusterFS client container, which is
publishing an Asteroids game to the Internet. Click + Add Container on
asteroids host and enter the following container information:
- Name: asteroids
- Image: nixel/rancher-glusterfs-client:latest
- Port Map: map 80 (public) port to 80 (container) TCP
port
- Port Map: map 443 (public) port to 443 (container) TCP
port
Expand Advanced Options and follow these steps:
- Command section – Add an Environment Variable named
GLUSTER_PEER which value is a comma separated list of gluster01
and gluster02 containers IPs. In my case I’m typing this:
10.42.46.31,10.42.235.105
- Networking section – Choose Managed Network on Docker0
- Security/Host section – Enable Give the container full access to
the host checkbox
Note that we are not configuring any container volume, because all data
is stored in GlusterFS cluster.
Wait for asteroids container to be created and show its logs. You will
find something like this at the top:
You will also see how Nginx server and Node.js application are started
at the bottom:
At this point your Rancher environment is up and running.
Testing GlusterFS HA capabilities
It is time to play and test GLusterFS HA capabilities. What you are
doing now is to stop one GlusterFS container and check that game will
not suffer downtimes. Browse to http://ASTEROIDS_HOST_PUBLIC_IP and
you will access Asteroids game, enter your name and try to explode some
asteroids. Go to Rancher UI and stop gluster02 container, then open
a new browser tab and navigate to the game again. The game is
accessible. You can start gluster02 container, then stop
gluster01 container, and try again. You are still able to play.
Finally, keep gluster01 container stopped, restart asteroids
container and wait for it to start. As you can see, if at least one
GlusterFS server container is running you are able to play. Finally you
may want to stop gluster01 and gluster02 containers to check how
game becomes unavailable because its public content is not reachable
now. To recover the service start gluster01 and/or gluster02
containers again.
Conclusion
Shared storage is a required feature when you have to deploy software
that needs to share information across all nodes. In this post you have
seen how to easily deploy a Highly Available shared storage solution for
Rancher based on GlusterFS Docker images. By using an Asteroids game you
have checked that storage is available when, at least, one GlusterFS
container is running. In future posts we are combining the shared
storage solution with Rancher Load Balancing feature, added in 0.16
version, so you will see how to build scalable, distributable, and
Highly Available Web server solutions ready for production use. To learn
more about Rancher, please join us for our next online meetup, where
we’ll be demonstrating some of these features and answering your
questions. Manel Martinez is a Linux systems
engineer with experience in the design and management of scalable,
distributable and highly available open source web infrastructures based
on products like KVM, Docker, Apache, Nginx, Tomcat, Jboss, RabbitMQ,
HAProxy, MySQL and XtraDB. He lives in spain, and you can find him on
Twitter @manel_martinezg.
Source
* At the end the
Renowned computer
