{"id":13364,"date":"2019-04-01T22:19:34","date_gmt":"2019-04-01T22:19:34","guid":{"rendered":"http:\/\/www.appservgrid.com\/paw92\/?p=13364"},"modified":"2019-04-01T22:19:34","modified_gmt":"2019-04-01T22:19:34","slug":"rclone-sync-files-directories-from-different-cloud-storage","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/04\/01\/rclone-sync-files-directories-from-different-cloud-storage\/","title":{"rendered":"Rclone \u2013 Sync Files Directories from Different Cloud Storage"},"content":{"rendered":"<p><strong>Rclone<\/strong>\u00a0is a command line program written in\u00a0<a href=\"https:\/\/www.tecmint.com\/install-go-in-linux\/\" target=\"_blank\" rel=\"noopener\">Go language<\/a>, used to sync files and directories from\u00a0<a href=\"https:\/\/www.tecmint.com\/free-open-source-cloud-storage-tools-for-linux\/\" target=\"_blank\" rel=\"noopener\">different cloud storage providers<\/a>\u00a0such as: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dropbox, FTP, Google Cloud Storage, Google Drive, etc.<\/p>\n<p>As you see, it supports multiple platforms, which makes it a useful tool to\u00a0<a href=\"https:\/\/www.tecmint.com\/sync-two-apache-websites-using-rsync\/\" target=\"_blank\" rel=\"noopener\">sync your data between servers<\/a>\u00a0or to a private storage.<\/p>\n<p>Rclone comes with the following features<\/p>\n<ul>\n<li>MD5\/SHA1 hash checks at all times for ensuring file integrity integrity.<\/li>\n<li>Timestamps are preserved on files.<\/li>\n<li>Partial syncs supported on a whole file basis.<\/li>\n<li>Copy mode for new or changed files.<\/li>\n<li>One way sync to make a directory identical.<\/li>\n<li>Check mode \u2013 hash equality check.<\/li>\n<li>Can sync to and from network, eg two different cloud accounts.<\/li>\n<li>(Encryption) backend.<\/li>\n<li>(Cache) backend.<\/li>\n<li>(Union) backend.<\/li>\n<li>Optional FUSE mount (rclone mount).<\/li>\n<\/ul>\n<h3>How to Install rclone in Linux Systems<\/h3>\n<p>The installation of\u00a0<strong>rclone<\/strong>\u00a0can be completed in two different ways. The easier one is using their installation script, by issuing the following command.<\/p>\n<pre># curl https:\/\/rclone.org\/install.sh | sudo bash\r\n<\/pre>\n<p>What this script does is to check the OS type on which it is ran and download the archive related to that OS. Then it extracts the archive and copies\u00a0<strong>rclone<\/strong>\u00a0binary to\u00a0<strong>\/usr\/bin\/rclone<\/strong>\u00a0and gives\u00a0<strong>755<\/strong>\u00a0permissions on the file.<\/p>\n<p>In the end, when the installation is complete, you should see the following line:<\/p>\n<pre>Rclone v1.44 has successfully installed.\r\nNow run \u201crclone config\u201d for setup, Check https:\/\/rclone.org\/docs\/ for  more details.\r\n<\/pre>\n<p>The second way to install\u00a0<strong>rclone<\/strong>\u00a0is by issuing the following commands.<\/p>\n<pre># curl -O https:\/\/downloads.rclone.org\/rclone-current-linux-amd64.zip\r\n# unzip rclone-current-linux-amd64.zip\r\n# cd rclone-*-linux-amd64\r\n<\/pre>\n<p>Now copy the binary file and give it executable permissions.<\/p>\n<pre># cp rclone \/usr\/bin\/\r\n# chown root:root \/usr\/bin\/rclone\r\n# chmod 755 \/usr\/bin\/rclone\r\n<\/pre>\n<p>Install\u00a0<strong>rclone<\/strong>\u00a0manpage.<\/p>\n<pre># mkdir -p \/usr\/local\/share\/man\/man1\r\n# cp rclone.1 \/usr\/local\/share\/man\/man1\/\r\n# mandb \r\n<\/pre>\n<h3>How to Configure rclone in Linux Systems<\/h3>\n<p>Next what you will need to do is run the\u00a0<strong>rclone config<\/strong>\u00a0to create your config file. It will be used for authentication for future usage of\u00a0<strong>rclone<\/strong>. To run the configuration setup run the following command.<\/p>\n<pre># rclone config\r\n<\/pre>\n<p>You will see the following prompt:<\/p>\n<pre>2018\/11\/13 11:39:58 NOTICE: Config file \u201c\/home\/user\/.config\/rclone\/rclone.conf\u201d not found - using defaults\r\nNo remotes found - make a new one\r\nn) New remote\r\ns) Set configuration password\r\nq) Quit config\r\nn\/s\/q&gt;\r\n<\/pre>\n<p>The options are as follows:<\/p>\n<ul>\n<li><strong>n)<\/strong>\u00a0\u2013 Create new remote connection<\/li>\n<li><strong>s)<\/strong>\u00a0\u2013 set password protection for your configuration<\/li>\n<li><strong>q)<\/strong>\u00a0\u2013 exit the config<\/li>\n<\/ul>\n<p>For the purpose of this tutorial lets press\u00a0<code>\"n\"<\/code>\u00a0and create new connection. You will be asked to give the new connection a name. After that you will be prompted to select the type of storage to be configured:<\/p>\n<div id=\"attachment_31016\" class=\"wp-caption aligncenter\">\n<p><a href=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-new-remote-connection.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-31016\" src=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-new-remote-connection.png\" alt=\"rclone - New Remote Connection\" width=\"622\" height=\"606\" aria-describedby=\"caption-attachment-31016\" data-lazy-loaded=\"true\" \/><\/a><\/p>\n<p id=\"caption-attachment-31016\" class=\"wp-caption-text\">rclone \u2013 New Remote Connection<\/p>\n<\/div>\n<p>I have named by connection\u00a0<strong>\u201cGoogle\u201d<\/strong>\u00a0and selected\u00a0<strong>\u201cGoogle Drive\u201d<\/strong>, which is under the number\u00a0<strong>12<\/strong>. The rest of the questions you can answer by simply leaving the default answer, which is an empty\u00a0<strong>\u201c\u201d<\/strong>.<\/p>\n<p>When asked to, you may select\u00a0<strong>\u201cautoconfig\u201d<\/strong>, which will generate all the required info to connect to your\u00a0<strong>Google Drive<\/strong>\u00a0and give\u00a0<strong>rclone<\/strong>\u00a0permissions to use data from\u00a0<strong>Google Drive<\/strong>.<\/p>\n<p>The process looks something like this:<\/p>\n<pre>Google Application Client Secret - leave blank normally.\r\nclient_secret&gt;\r\nScope that rclone should use when requesting access from drive.\r\nChoose a number from below, or type in your own value\r\n 1 \/ Full access all files, excluding Application Data Folder.\r\n   \\ \"drive\"\r\n 2 \/ Read-only access to file metadata and file contents.\r\n   \\ \"drive.readonly\"\r\n   \/ Access to files created by rclone only.\r\n 3 | These are visible in the drive website.\r\n   | File authorization is revoked when the user deauthorizes the app.\r\n   \\ \"drive.file\"\r\n   \/ Allows read and write access to the Application Data folder.\r\n 4 | This is not visible in the drive website.\r\n   \\ \"drive.appfolder\"\r\n   \/ Allows read-only access to file metadata but\r\n 5 | does not allow any access to read or download file content.\r\n   \\ \"drive.metadata.readonly\"\r\nscope&gt; <strong>1<\/strong>\r\nID of the root folder - leave blank normally.  Fill in to access \"Computers\" folders. (see docs).\r\nroot_folder_id&gt; \r\nService Account Credentials JSON file path - needed only if you want use SA instead of interactive login.\r\nservice_account_file&gt;\r\nRemote config\r\nUse auto config?\r\n * Say Y if not sure\r\n * Say N if you are working on a remote or headless machine or Y didn't work\r\ny) <strong>Yes<\/strong>\r\nn) <strong>No<\/strong>\r\ny\/n&gt; <strong>y<\/strong>\r\nIf your browser doesn't open automatically go to the following link: http:\/\/127.0.0.1:53682\/auth\r\nLog in and authorize rclone for access\r\nWaiting for code...\r\nGot code\r\nConfigure this as a team drive?\r\ny) <strong>Yes<\/strong>\r\nn) <strong>No<\/strong>\r\ny\/n&gt; <strong>n<\/strong>\r\n--------------------\r\n[remote]\r\nclient_id = \r\nclient_secret = \r\nscope = drive\r\nroot_folder_id = \r\nservice_account_file =\r\ntoken = {\"access_token\":\"XXX\",\"token_type\":\"Bearer\",\"refresh_token\":\"XXX\",\"expiry\":\"2018-11-13T11:57:58.955387075Z\"}\r\n--------------------\r\ny) Yes this is OK\r\ne) Edit this remote\r\nd) Delete this remote\r\ny\/e\/d&gt; <strong>y<\/strong>\r\n<\/pre>\n<h3>How to Use rclone in Linux Systems<\/h3>\n<p>Rclone has quite a long list of available options and commands to be used with. We will try to cover some of the more important ones:<\/p>\n<h4>List Remote Directory<\/h4>\n<pre># rclone lsd &lt;remote-dir-name&gt;:\r\n<\/pre>\n<div id=\"attachment_31017\" class=\"wp-caption aligncenter\">\n<p><a href=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-List-Remote-Directory.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-31017\" src=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-List-Remote-Directory.png\" alt=\"rclone - List Remote Directory\" width=\"404\" height=\"91\" aria-describedby=\"caption-attachment-31017\" data-lazy-loaded=\"true\" \/><\/a><\/p>\n<p id=\"caption-attachment-31017\" class=\"wp-caption-text\">rclone \u2013 List Remote Directory<\/p>\n<\/div>\n<h4>Copy Data with rclone<\/h4>\n<pre># rclone copy source:sourcepath dest:destpath\r\n<\/pre>\n<p>Note that if\u00a0<strong>rclone<\/strong>\u00a0finds duplicates, those will be ignored:<\/p>\n<div id=\"attachment_31018\" class=\"wp-caption aligncenter\">\n<p><a href=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-Copy-Data.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-31018\" src=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-Copy-Data.png\" alt=\"rclone - Copy Data\" width=\"618\" height=\"241\" aria-describedby=\"caption-attachment-31018\" data-lazy-loaded=\"true\" \/><\/a><\/p>\n<p id=\"caption-attachment-31018\" class=\"wp-caption-text\">rclone \u2013 Copy Data<\/p>\n<\/div>\n<h4>Sync data with rclone<\/h4>\n<p>If you want to sync some data between directories, you should use\u00a0<strong>rclone<\/strong>\u00a0with\u00a0<strong>sync<\/strong>\u00a0command.<\/p>\n<p>The command should look like this:<\/p>\n<pre># rclone sync source:path dest:path [flags]\r\n<\/pre>\n<p>In this case the source is synced to destination, changing the destination only! This method skips unchanged files. Since the command can cause data loss, you can use it with\u00a0<strong>\u201c\u2013dry-run\u201d<\/strong>\u00a0to see what exactly will be copied and deleted.<\/p>\n<div id=\"attachment_31020\" class=\"wp-caption aligncenter\">\n<p><a href=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-Sync-Data.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-31020\" src=\"https:\/\/www.tecmint.com\/wp-content\/uploads\/2018\/11\/rclone-Sync-Data.png\" alt=\"rclone Sync Data\" width=\"606\" height=\"240\" aria-describedby=\"caption-attachment-31020\" data-lazy-loaded=\"true\" \/><\/a><\/p>\n<p id=\"caption-attachment-31020\" class=\"wp-caption-text\">rclone Sync Data<\/p>\n<\/div>\n<h4>Move Data with rclone<\/h4>\n<p>To move data, you can use\u00a0<strong>rclone<\/strong>\u00a0with\u00a0<strong>move<\/strong>\u00a0command. The command should look like this:<\/p>\n<pre># rclone move source:path dest:path [flags]\r\n<\/pre>\n<p>The content from the source, will be moved (deleted) and placed on the selected destination.<\/p>\n<h4>Other useful rclone Commands<\/h4>\n<p>To create a directory on destination.<\/p>\n<pre># rclone mkdir remote:path\r\n<\/pre>\n<p>To remove a directory.<\/p>\n<pre># rclone rmdir remote:path\r\n<\/pre>\n<p>Check if files on source and destination match:<\/p>\n<pre># rclone check source:path dest:path\r\n<\/pre>\n<p>Delete files:<\/p>\n<pre># rclone delete remote:path\r\n<\/pre>\n<p>Each of rclone commands can be used with different flags and includes its own help menu. For example, you can do a selective delete using the delete option. Lets say you want to delete files larger than\u00a0<strong>100M<\/strong>, the command would look like this.<\/p>\n<pre># rclone --min-size 100M delete remote:path\r\n<\/pre>\n<p>It is highly recommend to review the manual and help for each command to get the most of\u00a0<strong>rclone<\/strong>. The full documentation of rclone is available at:\u00a0<a href=\"https:\/\/rclone.org\/\" target=\"_blank\" rel=\"nofollow noopener\">https:\/\/rclone.org\/<\/a><\/p>\n<h5>Conclusion<\/h5>\n<p><strong>rclone<\/strong>\u00a0is a powerful command line utility to help you manage data between different Cloud storage providers. While in this article we scratched just the surface of\u00a0<strong>rclone<\/strong>\u00a0capabilities, there is much more to be achieved with it especially when used in combination with\u00a0<a href=\"https:\/\/www.tecmint.com\/11-cron-scheduling-task-examples-in-linux\/\" target=\"_blank\" rel=\"noopener\">cron service<\/a>\u00a0(for example).<\/p>\n<p><a href=\"https:\/\/www.tecmint.com\/rclone-sync-files-from-cloud-storage\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Rclone\u00a0is a command line program written in\u00a0Go language, used to sync files and directories from\u00a0different cloud storage providers\u00a0such as: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dropbox, FTP, Google Cloud Storage, Google Drive, etc. As you see, it supports multiple platforms, which makes it a useful tool to\u00a0sync your data between servers\u00a0or &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/04\/01\/rclone-sync-files-directories-from-different-cloud-storage\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Rclone \u2013 Sync Files Directories from Different Cloud Storage&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-13364","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/13364","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=13364"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/13364\/revisions"}],"predecessor-version":[{"id":13365,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/13364\/revisions\/13365"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=13364"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=13364"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=13364"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}