{"id":633,"date":"2018-10-17T23:41:37","date_gmt":"2018-10-17T23:41:37","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw92\/?p=633"},"modified":"2018-10-18T15:04:22","modified_gmt":"2018-10-18T15:04:22","slug":"the-professional-approach-to-upgrading-linux-servers","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2018\/10\/17\/the-professional-approach-to-upgrading-linux-servers\/","title":{"rendered":"The Professional Approach to Upgrading Linux Servers"},"content":{"rendered":"<p>With the release of Ubuntu 18.04, I thought it would be the perfect time to talk about server upgrades. Specifically, I\u2019m going to share with you the process that I\u2019m using to perform upgrades.<\/p>\n<p>I don\u2019t shy away from work, but I hate doing work that really isn\u2019t needed. That\u2019s why my first question when it comes to upgrades is:<\/p>\n<h2>Is this upgrade even necessary?<\/h2>\n<p>The first thing to know is the EOL (End of Life) for support for the OS you\u2019re using. Here are the current EOLs for Ubuntu:<\/p>\n<p>Ubuntu 14.04 LTS: April 2019<br \/>\nUbuntu 16.04 LTS: April 2021<br \/>\nUbuntu 18.04 LTS: April 2023<\/p>\n<p>(By the way, Red Hat Enterprise Linux delivers at least 10 years of support for their major releases. This is just one example why you\u2019ll find RHEL being used in large organizations.)<\/p>\n<p>So, if you are thinking of upgrading from Ubuntu 16.04 to 18.04, consider if the service that server provides is even needed beyond April 2021. If the server is going away in the next couple of years, then it probably isn\u2019t worth your time to upgrade it.<\/p>\n<p>If you do decide to go ahead with the upgrade, then\u2026<\/p>\n<h2>Determine What Software Is Being Used<\/h2>\n<p>Hopefully, you have a script or used some sort of documented process to build the existing server. If so, then you have a good idea of what\u2019s already on the server.<\/p>\n<p>If you don\u2019t, it\u2019s time to start researching.<\/p>\n<p>Look at the running processes with the \u201cps\u201d command. I like using \u201cps -ef\u201d because it shows every process (-e) with a full-format listing (-f).<\/p>\n<p>ps -ef<\/p>\n<p>Look at any non-default users in \/etc\/passwd. What processes are they running? You can show the processes of a given user by using the \u201c-u\u201d option to \u201cps.\u201d<\/p>\n<p>ps -fu www-data<br \/>\nps -fu haproxy<\/p>\n<p>Determine what ports are open and what processes have those ports open:<\/p>\n<p>sudo netstat -nutlp<br \/>\nsudo lsof -nPi<\/p>\n<p>Look for any cron jobs being used.<\/p>\n<p>sudo ls -lR \/etc\/cron*<br \/>\nsudo ls -lR \/var\/spool\/cron<\/p>\n<p>Look for other miscellaneous clues such as disk usage and sudo configurations.<\/p>\n<p>df -h<br \/>\nsudo du -h \/home | sort -h<br \/>\nsudo cat \/etc\/sudoers<br \/>\nsudo ls -l \/etc\/sudoers.d<\/p>\n<h2>Determine the Current Software Versions<\/h2>\n<p>Now that you have a list of software that is running on your server, determine what versions are being used. Here\u2019s an example list for an Ubuntu 16.04 system:<\/p>\n<ul>\n<li>HAProxy 1.6.3<\/li>\n<li>Nginx 1.10.3<\/li>\n<li>MariaDB 10.0.34<\/li>\n<\/ul>\n<p>One way to get the versions is to look at the packages like so:<\/p>\n<p>dpkg -l haproxy nginx mariadb-server<\/p>\n<h2>Determine the New Software Versions<\/h2>\n<p>Now it\u2019s time to see what version of each piece of software ships with the new distro version. For ubuntu 18.04 you can use \u201capt show PKG_NAME\u201d:<\/p>\n<p>apt show HAProxy<\/p>\n<p>To display just the version, grep it out like so:<\/p>\n<p>apt show HAProxy | grep -i version<\/p>\n<p>Here\u2019s our list for Ubuntu 18.04:<\/p>\n<ul>\n<li>HAProxy 1.8.81<\/li>\n<li>Nginx 1.14.0<\/li>\n<li>MariaDB 10.1.29<\/li>\n<\/ul>\n<h2>Read the Release Notes<\/h2>\n<p>Now, find the release notes for each version of each piece of software. In this example, we are upgrading HAProxy from 1.6.3 to 1.8.81. Most software these days conform to Semantic Versioning guidelines. In short, given a version number MAJOR.MINOR.PATCH, increment the:<\/p>\n<p>MAJOR version when you make incompatible API changes,<br \/>\nMINOR version when you add functionality in a backwards-compatible manner, and<br \/>\nPATCH version when you make backwards-compatible bug fixes.<\/p>\n<p>This means we\u2019re the most concerned with major versions and somewhat concerned with minor versions, and we can pretty much ignore the patch version. This means we can think of this upgrade as being from 1.6 to 1.8.<\/p>\n<p>Because it\u2019s the same major version (1), we should be fine to just perform the upgrade. That\u2019s the theory, anyway. It doesn\u2019t always work in practice.<\/p>\n<p>In this case, read the release notes for HAProxy versions 1.7 and 1.8. Look for any signs of backward compatibility issues such as configuration syntax changes. Also look for new default values and then consider how those new default values could affect the environment.<\/p>\n<p>Repeat this process for the other major pieces of software. In this example that would be going from Nginx 1.10 to 1.14 and MariaDB 10.0 to 10.1.<\/p>\n<h2>Make Changes Based on the Release Notes<\/h2>\n<p>Based on the information from the release notes, make any required or desired adjustments to the configuration files.<\/p>\n<p>If you have your configuration files stored in version control, make your changes there. If you have configuration files or modifications performed by your build scripts, make your changes there. If you aren\u2019t doing either one of those, DO IT FOR THIS DEPLOYMENT\/UPGRADE. \ud83d\ude09 Seriously, just make a copy of the configuration files and make your changes to them. That way you can push them to your new server when it\u2019s time to test.<\/p>\n<p>If you\u2019re not sure what configuration file or files a given service uses, refer to its documentation. You can read the man page or visit the website for the software.<\/p>\n<p>Also, you can list the contents of its package and look for \u201cetc\u201d, \u201cconf\u201d, and \u201ccfg\u201d. Here\u2019s an example from an Ubuntu 16.04 system:<\/p>\n<p>dpkg -L haproxy | grep -E &#8216;etc|cfg|conf&#8217;<\/p>\n<p>The \u201cdpkg -L\u201d command lists the files in the package while the grep command matches \u201cetc\u201d, \u201ccfg\u201d, or \u201cconf\u201d. The \u201c-E\u201d option is for extended regular expressions. The pipe (|) acts as an \u201cor\u201d in regular expressions.<\/p>\n<p>You can also use the locate command.<\/p>\n<p>locate haproxy | grep -E &#8216;etc|cfg|conf&#8217;<\/p>\n<p>In case you\u2019re wondering, the main configuration file for haproxy is haproxy.cfg.<\/p>\n<h2>Install the Software on the Upgraded Server<\/h2>\n<p>Now install the major pieces of software on a new server running the new release of the distro.<\/p>\n<p>Of course, use a brand new server installation. You want to test your changes before you put them into production.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.linuxtrainingacademy.com\/wp-content\/uploads\/2018\/05\/test-upgrades-in-production-239x300.jpg\" alt=\"\" width=\"239\" height=\"300\" \/><\/p>\n<p>By the way, if you have a dedicated non-production (test\/dev) network, use it for this test. If you have a service on the server you are upgrading that connects to other servers\/services, it\u2019s a good idea to isolate it from production. You don\u2019t want to accidentally perform a production action when you\u2019re testing. This means you may need to replicate those other servers in your non-production environment before you can fully test the particular upgrade that you\u2019re working on.<\/p>\n<p>If you have deployment scripts you can use them to perform the installs. If you use Ansible or the like, use it against the new server. Or you can manually perform the install, making notes of all the commands you run so that you can put them in a script later on. For example, to manually install HAProxy on Ubuntu 18.04, run:<\/p>\n<p>apt install -y haproxy<\/p>\n<p>Next, put the configuration files in place.<\/p>\n<h2>Start the Services<\/h2>\n<p>If the software that you are installing is a service, make sure it starts at boot time.<\/p>\n<p>sudo systemctl enable haproxy<\/p>\n<p>Start the service:<\/p>\n<p>sudo systemctl start haproxy<\/p>\n<p>If your existing deployment script starts the service automatically, perform a restart to make sure that any of the new configuration file changes are being used.<\/p>\n<p>sudo systemctl restart haproxy<\/p>\n<p>See if the service is running.<\/p>\n<p>sudo systemctl status<\/p>\n<p>If it failed, read the error message and make the required corrections. Perhaps there is a configuration option that worked with the previous version that isn\u2019t valid with the new version, for example.<\/p>\n<h2>Import the Data<\/h2>\n<p>If you have services that store data, such as a database service, then import test data into the system.<\/p>\n<p>If you don\u2019t have test data, then copy over your production data to the new server.<\/p>\n<p>If you are using production data, you need to be very careful at this point.<\/p>\n<p>1) You don\u2019t want to accidentally destroy or alter any production data and\u2026<\/p>\n<p>2) You don\u2019t want your new system taking any unwanted actions based on production data.<\/p>\n<p>On point #1, you don\u2019t want to make a costly mistake such as getting your source and destinations mixed up and end up overwriting (or deleting) production data. Pro tip: make sure you have good production backups that you can restore.<\/p>\n<p>One point #2, you don\u2019t want to do something like double charge the business\u2019s customers or send out duplicate emails, etc. To this end, stop all the software and services that are not required for the import before you do it. For example, disable cron jobs and stop any in-house written software running on the test system that might kick off an action.<\/p>\n<p>It\u2019s a good idea to have TEST data. If you don\u2019t have test data, perhaps you can use this upgrade as an opportunity to create some. Take a copy of the production data and anonymize it. Change real email addresses to fake ones, etc.<\/p>\n<p>As previously mentioned, do your tests on a non-production network that cannot directly touch production.<\/p>\n<h2>Perform Service Checks<\/h2>\n<p>If you have a service availability monitoring tool (and why wouldn\u2019t you???), then point it at the new server. Let it do its job and tell you if something isn\u2019t working correctly. For example, you may have installed and started HAProxy, but perhaps it didn\u2019t open up the proper port because you accidentally forgot to copy over the configuration.<\/p>\n<p>Whether or not you have a service availability monitoring tool, use what you know about the service to see if it\u2019s working properly. For example, did it open up the proper port or ports? (Use the \u201cnetstat\u201d and \u201clsof\u201d commands from above). Are there any error messages you should be concerned about?<\/p>\n<p>If you\u2019re at all familiar with the service, test it. If it\u2019s a web server, does it serve up the proper web pages? If it\u2019s a database server, can you run queries against it?<\/p>\n<p>If you\u2019re not the familiar with the service or a normal user of the service, it\u2019s time to enlist help. If you have a team that is responsible for testing, hand it over to them. Maybe it\u2019s time to for someone in the business who uses the service to check it out and see if it works as expected.<\/p>\n<p>If you don\u2019t have a regression testing process in place, now would be a good time to create one. The goal is to make changes and know that those changes haven\u2019t broken the service. Upgrading the OS is a major change that has the potential to break things in a major way.<\/p>\n<h2>Prepare for Production<\/h2>\n<p>Once you\u2019ve completed this entire process and tested your work, put all your notes into a production implementation plan. Use that plan as a checklist when you\u2019re ready to go into production. It\u2019s probably worth it to test your plan on another newly installed system to make sure everything goes smoothly. This is especially true when you are working on a really important system.<\/p>\n<p>By the way, don\u2019t think less of yourself for having a detailed plan and checklist. It actually shows your professionalism and commitment to doing good work.<\/p>\n<p>For example, would you rather fly on a plane with a pilot who uses a checklist or one who just \u201cwings it.\u201d I don\u2019t care how smart or talented that pilot is, I want them to double check their work when it comes to my life.<\/p>\n<h2>Yes, It\u2019s a Lot of Work<\/h2>\n<p>You might be thinking to yourself, \u201cWow, this is a very tedious and time-consuming process.\u201d And you\u2019d be right.<\/p>\n<p>If you want to be a good\/great Linux professional, this is exactly what it takes. Attention to detail and hard work are part of the job.<\/p>\n<p>The good news is that you get compensated in proportion to your professionalism and level of responsibility.<\/p>\n<p>If it was fast and easy, everyone would be doing it, right?<\/p>\n<p>Hopefully, this post gave you some ideas beyond just blindly upgrading and hoping for the best. \ud83d\ude09<\/p>\n<p>Speaking of the best\u2026. I wish you the best!<\/p>\n<p>Jason<\/p>\n<p>P.S. If you\u2019re ready to level-up your Linux skills, <a href=\"https:\/\/courses.linuxtrainingacademy.com\/\">check out the courses I created for you here<\/a>.<\/p>\n<p><a href=\"https:\/\/www.linuxtrainingacademy.com\/professional-approach-to-upgrading-linux-servers\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>With the release of Ubuntu 18.04, I thought it would be the perfect time to talk about server upgrades. Specifically, I\u2019m going to share with you the process that I\u2019m using to perform upgrades. I don\u2019t shy away from work, but I hate doing work that really isn\u2019t needed. That\u2019s why my first question when &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.appservgrid.com\/paw92\/index.php\/2018\/10\/17\/the-professional-approach-to-upgrading-linux-servers\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;The Professional Approach to Upgrading Linux Servers&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-633","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/633","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=633"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/633\/revisions"}],"predecessor-version":[{"id":760,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/633\/revisions\/760"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=633"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=633"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=633"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}