Handling Signals and exit codes<\/h3>\n
When we pass a signal to container using Docker CLI, Docker passes the signal to the main process running inside container(PID-1). This link <\/a>has the list of all Linux signals. Docker exit codes<\/a> follow the chroot exit standard<\/a> for Docker defined exit codes. Other standard exit codes can come from the program running inside container. Container exit code can be seen from container events coming from Docker daemon when the container exits. For containers that have not been cleaned up, exit code can be found from \u201cdocker ps -a\u201d. $ docker ps -a Following is a sample \u201cdocker ps -a\u201d output where nginx container exited with exit code 137. Here, I used \u201cdocker kill\u201d to stop the container.<\/p>\n $ docker ps -a Following is the list of standard and Docker defined exit codes:<\/p>\n 0: Success Following is a simple Python program that handles Signals. This program will be run as Docker container to illustrate Docker signals and exit codes.<\/p>\n #!\/usr\/bin\/python<\/p>\n import sys def signal_handler_int(sigid, frame): def signal_handler_term(sigid, frame): def signal_handler_usr(sigid, frame): def main(): while True: # This is the standard boilerplate that calls the main() function. Following is the Dockerfile to convert this to container:<\/p>\n FROM python:2.7 Lets build the container:<\/p>\n docker build –no-cache -t smakam\/signaltest:v1 .<\/p>\n Lets start the container:<\/p>\n docker run -d –name signaltest smakam\/signaltest:v1<\/p>\n We can watch the logs from container using docker logs:<\/p>\n docker logs -f signaltest<\/p>\n The Python program above handles SIGINT, SIGTERM and SIGUSR1. We can pass these signals to the container using Docker CLI. docker kill –signal=SIGINT signaltest<\/p>\n In the Docker logs, we can see the following to show that this signal is handled:<\/p>\n signal 2 , Handling Ctrl+C\/SIGINT!<\/p>\n Following output shows the container exit status:<\/p>\n $ docker ps -a Following command sends SIGTERM to the container:<\/p>\n docker kill –signal=SIGTERM signaltest<\/p>\n In the Docker logs, we can see the following to show that this signal is handled:<\/p>\n signal 15 , Handling SIGTERM!<\/p>\n Following output shows the container exit status:<\/p>\n $ docker ps -a Following command sends SIGUSR1 to the container:<\/p>\n docker kill –signal=SIGUSR1 signaltest<\/p>\n In the Docker logs, we can see the following to show that this signal is handled:<\/p>\n signal 15 , Handling SIGUSR1!<\/p>\n Following output shows the container exit status:<\/p>\n $ docker ps -a When we execute \u201cdocker stop \u201c, Docker first sends SIGTERM signal to the container, waits for some time and then sends SIGKILL. This is done so that the program executing inside the container can use the SIGTERM signal to do the graceful shutdown of the program.<\/p>\n In the above example, the python program runs as PID 1 inside container since we used the EXEC form of ENTRYPOINT in Dockerfile. If we use the background method of ENTRYPOINT, shell process runs as PID 1 and the python program runs as another process. Following is a sample Dockerfile for starting the program as background process.<\/p>\n FROM python:2.7 In this example, Docker passes the signal to the shell process instead of to the Python program. This causes the python program to not see the signal sent to the container. If there are multiple processes running inside the container and we need to pass the signal, 1 possible approach is to run the ENTRYPOINT as a script, handle the signal in the script and pass it to the correct process. 1 example using this approach is mentioned here<\/a>.<\/p>\n \u201cdocker stop\u201d \u2013 Sends SIGTERM to container, waits some time for process to handle it and then sends SIGKILL. Container filesystem remains intact. When container exits without the container filesystem getting removed, we can still restart the container.<\/p>\n Container restart policy controls the restart actions when Container exits. Following are the supported restart options:<\/p>\n Following is an example of starting \u201csignaltest\u201d container with restart policy of \u201con-failure\u201d and retry count of 3. Retry count 3 is the number of restarts that will be done by Docker before giving up.<\/p>\n docker run -d –name=signaltest –restart=on-failure:3 smakam\/signaltest:v1<\/p>\n To show the restart happening, we can manually try to send signals to the container. In the \u201csignaltest\u201d example, signals SIGTERM, SIGINT and SIGKILL will cause non-zero exit code and SIGUSR1 will cause zero exit code. 1 thing to remember is that restart does not work if we stop the container or send signals using \u201cdocker kill\u201d. I think this is because there must be an explicit check added by Docker to prevent restart in these cases since the action is triggered by user. kill -s SIGINT<\/p>\n Lets check the \u201cdocker ps\u201d output. We can see that the \u201ccreated\u201d time is 50 seconds. Uptime is less than a second because container restarted.<\/p>\n $ docker ps Following command shows the restart policy and the restart count for the running container. In this example, container restart happened once.<\/p>\n $ docker inspect signaltest | grep -i -A 2 -B 2 restart To illustrate that restart does not work on exit code 0, lets send SIGUSR1 to the container that will cause exit code 0.<\/p>\n sudo kill -s SIGUSR1<\/p>\n In this case, container exits, but it does not get restarted.<\/p>\n Container restart does not work with \u201c\u2013rm\u201d option. This is because \u201c\u2013rm\u201d option causes container to be removed as soon as the container exit happens.<\/p>\n It is possible that container does not exit but it not performing as per the requirement. Health check probes can be used to identify misbehaving containers and take action rather than waiting till the end when container dies. Health check probes are used to accomplish the specific task of checking container health. For a container like webserver, health check probe can be as simple as sending curl request to webserver port. By using container\u2019s health, we can restart the container if health check fails. docker run -p 8080:8080 -d –rm –name health-check –health-interval=1s –health-timeout=3s –health-retries=3 –health-cmd “curl -f http:\/\/localhost:8080\/health || exit 1” effectivetrainings\/docker-health<\/p>\n Following are all parameters related to healthcheck:<\/p>\n Following \u201cdocker ps\u201d output shows container health status:<\/p>\n $ docker ps This container has a backdoor approach to mark container health as unhealthy. Lets use the backdoor approach to mark container as unhealthy like below:<\/p>\n curl “http:\/\/localhost:8080\/environment\/health?status=false”<\/p>\n Now, lets check the \u201cdocker ps\u201d output. The container\u2019s health has now become unhealthy.<\/p>\n $ docker ps Docker Swarm mode introduces a higher level of abstraction called Service and containers are part of the service. When we create a service, we specify the number of containers that needs to be part of the service using \u201creplicas\u201d parameter. Docker swarm will monitor the number of replicas and if any container dies, Swarm will create new container to keep the replica count as requested by the user. docker service create –name signaltest –replicas=2 smakam\/signaltest:v1<\/p>\n Following command output shows the 2 containers that are part of \u201csignaltest\u201d service:<\/p>\n $ docker service ps signaltest Following parameters control the container restart policy in a service:<\/p>\n docker service create –name signaltest –replicas=2 –restart-condition=on-failure –restart-delay=3s smakam\/signaltest:v1<\/p>\n Remember that sending signal \u201cSIGTERM\u201d, \u201cSIGINT\u201d, \u201cSIGKILL\u201d causes non-zero container exit codes and sending \u201cSIGUSR1\u201d causes zero container exit code. docker kill –signal=SIGTERM<\/p>\n Following is the \u201csignaltest\u201d service output that shows the 3 containers including the one that has exited with non-zero status:<\/p>\n $ docker service ps signaltest Following command sends SIGUSR1 signal to 1 of the containers which causes container to exit with status 0.<\/p>\n docker kill –signal=SIGUSR1<\/p>\n Following command shows that the container did not restart since the container exit code is 0.<\/p>\n $ docker service ps signaltest $ docker service ls I don\u2019t see a real need to change the default Swarm service restart policy from \u201cany\u201d.<\/p>\n In the previous sections, we saw how to use container health check with \u201ceffectivetrainings\/docker-health\u201d container. Even though we could detect the container as unhealthy, we could not restart the container automatically. For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.<\/p>\n \u201cdocker service\u201d command provides following options for health check and associated behavior.<\/p>\n Lets create \u201cswarmhealth\u201d service with 2 replicas of \u201cdocker-health\u201d containers.<\/p>\n docker service create –name swarmhealth –replicas 2 -p 8080:8080 –health-interval=2s –health-timeout=10s –health-retries=10 –health-cmd “curl -f http:\/\/localhost:8080\/health || exit 1” effectivetrainings\/docker-health<\/p>\n Following output shows the \u201cswarmhealth\u201d service output and the 2 healthy containers:<\/p>\n $ docker service ps swarmhealth $ docker ps Lets mark 1 of the container unhealthy using backdoor command:<\/p>\n curl “http:\/\/:8080\/environment\/health?status=false”<\/p>\n Following output shows that 1 of the containers that has been shutdown which is the unhealthy container and 2 more running replicas. 1 of the replicas got restarted after the other container became unhealthy.<\/p>\n $ docker service ps swarmhealth When we have new versions of service to be updated without taking service downtime, Docker provides many controls to do the upgrade and rollback. For example, we can control parameters like number of tasks to upgrade at a single time, actions on upgrade failure, delay between task upgrades etc. This helps us achieve release patterns like Blue green and Canary deployment patterns.<\/p>\n Following options are provided by Docker in \u201cdocker service\u201d command to control rolling upgrade and rollback.<\/p>\n Rolling upgrade:<\/p>\n Rollback:<\/p>\n To illustrate service upgrade, I have a simple python webserver program running as container. #!\/usr\/bin\/python<\/p>\n import sys class GetHandler(BaseHTTPRequestHandler):<\/p>\n def do_GET(self): def main(): # This is the standard boilerplate that calls the main() function. This is the Dockerfile to create the Container:<\/p>\n FROM python:2.7 I have 2 versions of Container, smakam\/webserver:v1 and smakam\/webserver:v2. The only difference is the message output that either shows \u201cYou are using version 1\u201d or \u201cYou are using version 2\u201d.<\/p>\n Lets create version 1 of the service with 2 replicas:<\/p>\n docker service create –name webserver –replicas=2 -p 8000:8000 smakam\/webserver:v1<\/p>\n We can access the service using script. The service request will get load balanced between the 2 replicas.<\/p>\n while true; do curl -s “localhost:8000”;sleep 1;done<\/p>\n Following is the service request output that shows we are using version 1 of the service:<\/p>\n You are using version 1 Lets upgrade to version 2 of the web service. Since we have specified update-delay of 3 seconds, there will be a 3 second gap between upgrades of 2 replicas. Since the \u201cupdate-parallelism\u201d default is 1, only 1 task will be upgraded at 1 time.<\/p>\n docker service update –update-delay=3s –image=smakam\/webserver:v2 webserver<\/p>\n Following is the service request output output that shows the request slowly getting migrated to version 2 as the upgrade happens 1 replica at a time.<\/p>\n You are using version 1 Now, lets rollback to version 1 of the webserver:<\/p>\n docker service update –rollback webserver<\/p>\n Following is the service request output output that shows the request slowly getting downgraded from version 2 to version 1.<\/p>\n You are using version 2 Please let me know your feedback and if you want to see more details on any specific topic related to this. I have put the code associated with this blog here<\/a>. The containers used in this blog(smakam\/signaltest, smakam\/webserver) are in Docker hub.<\/p>\n
\nFollowing is a sample \u201cdocker ps -a\u201d output where nginx container exited with exit code 0. Here, I used \u201cdocker stop\u201d to stop the container.<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\n32d675260384 nginx “nginx -g ‘daemon …” 18 seconds ago Exited (0) 7 seconds ago web<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\n9b5d8348cb89 nginx “nginx -g ‘daemon …” 11 seconds ago Exited (137) 2 seconds ago web<\/p>\n
\n125: Docker run itself fails
\n126: Contained command cannot be invoked
\n127: Containerd command cannot be found
\n128 + n: Fatal error signal n:
\n130: (128+2) Container terminated by Control-C
\n137: (128+9) Container received a SIGKILL
\n143: (128+15) Container received a SIGTERM
\n255: Exit status out of range(-1)<\/p>\n
\nimport signal
\nimport time<\/p>\n
\nprint “signal”, sigid, “,”, “Handling Ctrl+C\/SIGINT!”
\nsys.exit(signal.SIGINT)<\/p>\n
\nprint “signal”, sigid, “,”, “Handling SIGTERM!”
\nsys.exit(signal.SIGTERM)<\/p>\n
\nprint “signal”, sigid, “,”, “Handling SIGUSR1!”
\nsys.exit(0)<\/p>\n
\n# Register signal handler
\nsignal.signal(signal.SIGINT, signal_handler_int)
\nsignal.signal(signal.SIGTERM, signal_handler_term)
\nsignal.signal(signal.SIGUSR1, signal_handler_usr)<\/p>\n
\nprint “I am alive”
\nsys.stdout.flush()
\ntime.sleep(1)<\/p>\n
\nif __name__ == ‘__main__’:
\nmain()<\/p>\n
\nCOPY .\/signalexample.py .\/signalexample.py
\nENTRYPOINT [“python”, “signalexample.py”]<\/p>\n
\nFollowing command sends SIGINT to the container:<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\nc06266e79a43 smakam\/signaltest:v1 “python signalexam…” 36 seconds ago Exited (2) 3 seconds ago signaltest<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\n0149708f42b2 smakam\/signaltest:v1 “python signalexam…” 10 seconds ago Exited (15) 2 seconds ago signaltest<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\nc92f7b4dd45b smakam\/signaltest:v1 “python signalexam…” 12 seconds ago Exited (0) 2 seconds ago signaltest<\/p>\nCommon mistake in Docker signal handling<\/h4>\n
\nCOPY .\/signalexample.py .\/signalexample.py
\nENTRYPOINT python signalexample.py<\/p>\nDifference between \u201cdocker stop\u201d, \u201cdocker rm\u201d and \u201cdocker kill\u201d<\/h4>\n
\n\u201cdocker kill\u201d \u2013 Sends SIGKILL directly. Container filesystem remains intact.
\n\u201cdocker rm\u201d \u2013 Removes container filesystem. \u201cdocker rm -f\u201d will send SIGKILL and then remove container filesystem.
\nUsing \u201cdocker run\u201d with \u201c\u2013rm\u201d option will automatically remove containers including container filesystem when the container exits.<\/p>\nContainer restart policy<\/h3>\n
\n
\nLets send SIGINT to the container by passing the signal to the process. We can find the process id by doing \u201cps -eaf | grep signalexample\u201d in host machine.<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\nb867543b110c smakam\/signaltest:v1 “python signalexam…” 50 seconds ago Up Less than a second<\/p>\n
\n“Name”: “\/signaltest”,
\n“RestartCount”: 1,
\n“RestartPolicy”: {
\n“Name”: “on-failure”,
\n“MaximumRetryCount”: 3<\/p>\nContainer health check<\/h3>\n
\nTo illustrate health check feature, I have used the container described here<\/a>.
\nFollowing command starts the webserver container with health check capability enabled.<\/p>\n![\"container_healthcheck\"](\"https:\/\/sreeninet.files.wordpress.com\/2017\/08\/container_healthcheck.png?w=604\")
<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\n947dad1c1412 effectivetrainings\/docker-health “java -jar \/app.jar” 28 seconds ago Up 26 seconds (healthy) 0.0.0.0:8080->8080\/tcp health-check<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\n947dad1c1412 effectivetrainings\/docker-health “java -jar \/app.jar” 3 minutes ago Up 3 minutes (unhealthy) 0.0.0.0:8080->8080\/tcp health-check<\/p>\nService restart with Swarm<\/h3>\n
\nBelow command can be used to create signal service with 2 container replicas:<\/p>\n
\nID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
\nvsgtopkkxi55 signaltest.1 smakam\/signaltest:v1 ubuntu Running Running 36 seconds ago
\ndbbm05w91wv7 signaltest.2 smakam\/signaltest:v1 ubuntu Running Running 36 seconds ago<\/p>\n![\"service_restart\"](\"https:\/\/sreeninet.files.wordpress.com\/2017\/08\/service_restart.png?w=637&h=192\")
\nLets start the \u201csignaltest\u201d service with restart-condition of \u201con-failure\u201d:<\/p>\n
\nLets first send SIGTERM to 1 of the 2 containers:<\/p>\n
\nID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
\n35ndmu3jbpdb signaltest.1 smakam\/signaltest:v1 ubuntu Running Running 4 seconds ago
\nullnsqio5151 _ signaltest.1 smakam\/signaltest:v1 ubuntu Shutdown Failed 11 seconds ago “task: non-zero exit (15)”
\n2rfwgq0388mt signaltest.2 smakam\/signaltest:v1 ubuntu Running Running 49 seconds ago<\/p>\n
\nID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
\n35ndmu3jbpdb signaltest.1 smakam\/signaltest:v1 ubuntu Running Running 52 seconds ago
\nullnsqio5151 _ signaltest.1 smakam\/signaltest:v1 ubuntu Shutdown Failed 59 seconds ago “task: non-zero exit (15)”
\n2rfwgq0388mt signaltest.2 smakam\/signaltest:v1 ubuntu Shutdown Complete 3 seconds ago<\/p>\n
\nID NAME MODE REPLICAS IMAGE PORTS
\nxs8lzbqlr69n signaltest replicated 1\/2 smakam\/signaltest:v1<\/p>\nService health check<\/h3>\n
![\"service_health_check\"](\"https:\/\/sreeninet.files.wordpress.com\/2017\/08\/service_health_check.png?w=637&h=194\")
<\/p>\n
\nID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
\njg8d78inw97n swarmhealth.1 effectivetrainings\/docker-health:latest ubuntu Running Running 21 seconds ago
\nl3fdz5awv4u0 swarmhealth.2 effectivetrainings\/docker-health:latest ubuntu Running Running 19 seconds ago<\/p>\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
\nd9b1f1b0a9b0 effectivetrainings\/docker-health:latest “java -jar \/app.jar” About a minute ago Up About a minute (healthy) swarmhealth.1.jg8d78inw97nmmbdtjzrscg1q
\nbb15bfc6e588 effectivetrainings\/docker-health:latest “java -jar \/app.jar” About a minute ago Up About a minute (healthy) swarmhealth.2.l3fdz5awv4u045g2xiyrbpe2u<\/p>\n
\nID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
\nixxvzyuyqmcq swarmhealth.1 effectivetrainings\/docker-health:latest ubuntu Running Running 4 seconds ago
\njg8d78inw97n _ swarmhealth.1 effectivetrainings\/docker-health:latest ubuntu Shutdown Failed 23 seconds ago “task: non-zero exit (143): do\u2026”
\nl3fdz5awv4u0 swarmhealth.2 effectivetrainings\/docker-health:latest ubuntu Running Running 5 minutes ago<\/p>\nService upgrade and rollback<\/h3>\n
![\"service_upgrade\"](\"https:\/\/sreeninet.files.wordpress.com\/2017\/08\/service_upgrade.png?w=474&h=180\")
<\/p>\n![\"service_rollback\"](\"https:\/\/sreeninet.files.wordpress.com\/2017\/08\/service_rollback.png?w=604\")
<\/p>\n
\nFollowing is the Python program:<\/p>\n
\nfrom BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
\nimport urlparse
\nimport json<\/p>\n
\nmessage = “You are using version 1n”
\nself.send_response(200)
\nself.end_headers()
\nself.wfile.write(message)
\nreturn<\/p>\n
\nserver = HTTPServer((”, 8000), GetHandler)
\nprint ‘Starting server at http:\/\/localhost:8000’
\nserver.serve_forever()<\/p>\n
\nif __name__ == ‘__main__’:
\nmain()<\/p>\n
\nCOPY .\/webserver.py .\/webserver.py
\nENTRYPOINT [“python”, “webserver.py”]<\/p>\n
\nYou are using version 1
\nYou are using version 1<\/p>\n
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 2
\nYou are using version 2<\/p>\n
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 2
\nYou are using version 1
\nYou are using version 1<\/p>\nReferences<\/h3>\n
\n