{"id":635,"date":"2018-10-20T07:48:45","date_gmt":"2018-10-20T07:48:45","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw93\/?p=635"},"modified":"2018-10-21T00:20:23","modified_gmt":"2018-10-21T00:20:23","slug":"nextcloudpi-backup-strategies-own-your-bits","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw93\/index.php\/2018\/10\/20\/nextcloudpi-backup-strategies-own-your-bits\/","title":{"rendered":"NextCloudPi backup strategies \u2013 Own your bits"},"content":{"rendered":"<p>This post is a review of the different options that we have when deciding on our backup strategy. As more features have appeared over time, this information has become scattered around this blog, so it is nice to have a review in a single article.<\/p>\n<p>It goes without saying that we <em>need<\/em> to backup our sensitive data in case a data loss event occurs, such as server failure, hard drive failure or a house fire. Many people think about this only too late, and it can be devastating.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2017\/08\/nc-backup-new.jpg\" alt=\"\" width=\"865\" height=\"450\" \/><br \/>\nIdeally we would have at least two copies, better three, of all our data. This will cover us from hardware failure. If possible, one of them should be in a different location, in case an event such as somebody breaking in our home, or a house fire, and it also would ideally be encrypted.<\/p>\n<p>Now if we are running NCP on a low end hardware, options such as encryption, or RAID can have a prohibitive computational cost. With this in mind, the following are the backup features that have been developed for this scenario.<\/p>\n<h4>Periodic backups<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2017\/08\/nc-backup-auto.jpg\" alt=\"\" width=\"861\" height=\"418\" \/><\/p>\n<p>This one is the most basic one. Backup our Nextcloud files and database at regular intervals, in case our system breaks or our database becomes corrupted. As we add more and more Gigabytes of data, this can take really long and take up a lot of space.<\/p>\n<p>The data is better off at an external USB drive than at the SD card, not only because it is cheaper per Megabyte, but also because they are more reliable.<\/p>\n<p>Pros:<\/p>\n<ul>\n<li>Doesn\u2019t require any advanced filesystem such as BTRFS.<\/li>\n<li>Self contained. We can copy the backup to another NCP instance and restore easily.<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Can take up a lot of space, even if compressed.<\/li>\n<li>They can be slow.<\/li>\n<li>Our cloud will be inaccessible in maintenance mode during the process.<\/li>\n<\/ul>\n<p>It is recommended to use a second hard drive to save the backups to, in case the first one fails (<a href=\"https:\/\/github.com\/nextcloud\/nextcloudpi\/wiki\/How-to-periodically-backup-to-a-second-USB-drive-with-NextCloudPi\">instructions<\/a>). In order to do this just plug in a second hard drive and set it as the destination of backups in <em>nc-backup-auto<\/em>. Remember that if we are using more than one drive, we shoud reference each one by label in the NCP options as explained <a href=\"https:\/\/github.com\/nextcloud\/nextcloudpi\/wiki\/How-to-configure-an-external-USB-drive-with-NextCloudPi\">here<\/a>.<\/p>\n<h4>Periodic dataless backups<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2018\/10\/pexels-photo-270529.jpeg\" alt=\"\" width=\"1280\" height=\"960\" \/><\/p>\n<p>To alleviate some of these issues, we can do <em>dataless backups<\/em>. These are typically a couple hundred megabytes big and the downtime will be small. The problem with this is that we need to find another way of duplicating the data itself.<\/p>\n<p>Pros:<\/p>\n<ul>\n<li>Doesn\u2019t require any advanced filesystem such as BTRFS<\/li>\n<li>Almost self contained. We can copy the backup to another NCP instance and restore our accounts, apps and so on easily.<\/li>\n<li>The downtime quite small<\/li>\n<li>Don\u2019t take much additional space<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>We need to backup our data separatedly<\/li>\n<li>Restoring can be more complicated to include data<\/li>\n<\/ul>\n<p>After restoring, we have to edit <em>config.php<\/em> and point our instance to the path where our data is, then we need to run <em>nc-scan<\/em> to make Nextcloud aware of the new files.<\/p>\n<h4>Periodic BTRFS snapshots<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2018\/02\/snapshot-auto.png\" alt=\"\" width=\"928\" height=\"540\" \/><\/p>\n<p>We have dedicated <a href=\"https:\/\/ownyourbits.com\/category\/btrfs\/\">several posts<\/a> already to BTRFS and its features. NCP will default to format our USB drive to BTRFS, and our datadir will automatically be in a subvolume. Just activate <em>nc-snapshots<\/em> to get hourly snapshots of your data with <a href=\"https:\/\/ownyourbits.com\/2017\/12\/27\/schedule-btrfs-snapshots-with-btrfs-snp\/\"><em>btrfs-snp<\/em><\/a><\/p>\n<table>\n<tbody>\n<tr>\n<td>&nbsp;<\/p>\n<p>1<\/p>\n<p>2<\/p>\n<p>3<\/p>\n<p>4<\/p>\n<p>5<\/p>\n<p>6<\/p>\n<p>7<\/p>\n<p>8<\/p>\n<p>9<\/p>\n<p>10<\/p>\n<p>11<\/p>\n<p>12<\/p>\n<p>13<\/p>\n<p>14<\/p>\n<p>15<\/p>\n<p>16<\/p>\n<p>17<\/p>\n<p>18<\/p>\n<p>19<\/p>\n<p>20<\/p>\n<p>21<\/p>\n<p>22<\/p>\n<p>23<\/p>\n<p>24<\/p>\n<p>25<\/p>\n<p>26<\/p>\n<p>27<\/p>\n<p>28<\/p>\n<p>29<\/p>\n<p>30<\/p>\n<p>31<\/p>\n<p>32<\/p>\n<p>33<\/p>\n<p>34<\/p>\n<p>35<\/p>\n<p>36<\/p>\n<p>37<\/p>\n<p>38<\/p>\n<p>39<\/p>\n<p>&nbsp;<\/td>\n<td>$ ls -1 \/media\/myCloudDrive\/ncp-snapshots\/<\/p>\n<p>daily_2018-01-30_211703<\/p>\n<p>daily_2018-01-31_221703<\/p>\n<p>daily_2018-02-01_231702<\/p>\n<p>daily_2018-02-02_231703<\/p>\n<p>daily_2018-02-03_231702<\/p>\n<p>daily_2018-02-04_231702<\/p>\n<p>daily_2018-02-05_231702<\/p>\n<p>hourly_2018-02-05_221701<\/p>\n<p>hourly_2018-02-05_231701<\/p>\n<p>hourly_2018-02-06_001701<\/p>\n<p>hourly_2018-02-06_011701<\/p>\n<p>hourly_2018-02-06_021701<\/p>\n<p>hourly_2018-02-06_031701<\/p>\n<p>hourly_2018-02-06_041701<\/p>\n<p>hourly_2018-02-06_051701<\/p>\n<p>hourly_2018-02-06_061701<\/p>\n<p>hourly_2018-02-06_071701<\/p>\n<p>hourly_2018-02-06_081701<\/p>\n<p>hourly_2018-02-06_091701<\/p>\n<p>hourly_2018-02-06_101702<\/p>\n<p>hourly_2018-02-06_111701<\/p>\n<p>hourly_2018-02-06_121701<\/p>\n<p>hourly_2018-02-06_131701<\/p>\n<p>hourly_2018-02-06_141701<\/p>\n<p>hourly_2018-02-06_151701<\/p>\n<p>hourly_2018-02-06_161701<\/p>\n<p>hourly_2018-02-06_171701<\/p>\n<p>hourly_2018-02-06_181702<\/p>\n<p>hourly_2018-02-06_201701<\/p>\n<p>hourly_2018-02-06_211701<\/p>\n<p>hourly_2018-02-06_221701<\/p>\n<p>manual_2017-12-28_113633<\/p>\n<p>monthly_2017-12-28_101702<\/p>\n<p>monthly_2018-01-27_101703<\/p>\n<p>weekly_2018-01-11_111701<\/p>\n<p>weekly_2018-01-18_111702<\/p>\n<p>weekly_2018-01-25_111702<\/p>\n<p>weekly_2018-02-01_111702<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p>By virtue of BTRFS being a COW system, the new clones will only take as much additional space as new files we have added since the last snapshot, being space efficient.<\/p>\n<p>Also, the snapshots can be sent incrementally very efficiently to another BTRFS filesystem using <em><a href=\"https:\/\/ownyourbits.com\/2018\/03\/09\/easy-sync-of-btrfs-snapshots-with-btrfs-sync\/\">btrfs-sync<\/a><\/em>. We can sync to another hard drive on the same machine<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2018\/03\/nc-snapshot-sync_1.png\" alt=\"\" width=\"1007\" height=\"545\" \/><\/p>\n<p>, or a BTRFS filesystem on another machine, in which case the transfer is encrypted through SSH.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2018\/03\/nc-snapshot-sync_.png\" alt=\"\" width=\"1007\" height=\"545\" \/><br \/>\nIn a low bandwidth situation, <em>btrfs-sync<\/em> can send the deltas compressed, but that will take a big toll on the CPU so it is not recommended on low end boards.<\/p>\n<p>Pros:<\/p>\n<ul>\n<li>Virtually instant, no matter how many Terabytes we have<\/li>\n<li>No downtime<\/li>\n<li>Space efficient<\/li>\n<li>Can be synced efficiently to another hard drive or another machine through SSH by using btrfs-sync<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Need to run on BTRFS, which is the default filesystem that <em>nc-format-USB<\/em> uses<\/li>\n<li>If we want to sync it to another USB drive it also needs to use BTRFS<\/li>\n<li>If we want to sync it remotely we neend a BTRFS subvolume on the other end and setting up SSH credentials<\/li>\n<\/ul>\n<p>If we only care about our data this can mean zero downtime and an efficient means of prevent accidental deletion. In fact, I have the Trash and Versions apps disabled. I recommend combining <em>nc-snapshots<\/em> and <em>nc-snapshot-sync<\/em> with <em>dataless backups<\/em> to get the best of both worlds.<\/p>\n<h4>Rsync<\/h4>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2018\/03\/nc-rsync-auto.png\" alt=\"\" width=\"1007\" height=\"545\" \/><br \/>\nAnother option to save our data remotely is to sync it through <a href=\"https:\/\/ownyourbits.com\/2018\/03\/19\/nextcloudpi-brings-nc-13-0-1-remote-backups-zram-translations-web-improvements-and-more\/\">rsync<\/a>. This is also quite efficient but compared to BTRFS snapshots we won\u2019t retain a history of the datadir, just the latest version.<\/p>\n<p>Pros:<\/p>\n<ul>\n<li>Doesn\u2019t require a particular filesystem on either end<\/li>\n<li>Efficient delta sync, only copies the new files<\/li>\n<\/ul>\n<p>Cons:<\/p>\n<ul>\n<li>Need to setup SSH credentials<\/li>\n<li>Not able to sync snapshots, it will only mirror the latest version of our datadir.<\/li>\n<\/ul>\n<p>This option also allows us to keep our data safe if something happens in our house, and is more flexible as it doesn\u2019t require a BTRFS filesystem in the other end.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/ownyourbits.com\/wp-content\/uploads\/2018\/10\/719px-Sega-Saturn-Backup.jpg\" alt=\"\" width=\"719\" height=\"600\" \/><\/p>\n<p>Make sure you keep your data with you, we can never be too safe!<br \/>\n<a href=\"https:\/\/ownyourbits.com\/2018\/10\/19\/nextcloudpi-backup-strategies\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This post is a review of the different options that we have when deciding on our backup strategy. As more features have appeared over time, this information has become scattered around this blog, so it is nice to have a review in a single article. It goes without saying that we need to backup our &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.appservgrid.com\/paw93\/index.php\/2018\/10\/20\/nextcloudpi-backup-strategies-own-your-bits\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;NextCloudPi backup strategies \u2013 Own your bits&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-635","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/635","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/comments?post=635"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/635\/revisions"}],"predecessor-version":[{"id":663,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/posts\/635\/revisions\/663"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/media?parent=635"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/categories?post=635"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw93\/index.php\/wp-json\/wp\/v2\/tags?post=635"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}