{"id":12129,"date":"2019-03-22T11:24:52","date_gmt":"2019-03-22T11:24:52","guid":{"rendered":"http:\/\/www.appservgrid.com\/paw92\/?p=12129"},"modified":"2019-03-22T11:24:52","modified_gmt":"2019-03-22T11:24:52","slug":"fdupes-a-command-line-tool-to-find-and-delete-duplicate-files-in-linux","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/03\/22\/fdupes-a-command-line-tool-to-find-and-delete-duplicate-files-in-linux\/","title":{"rendered":"fdupes \u2013 A Command Line Tool to Find and Delete Duplicate Files in Linux"},"content":{"rendered":"

It is a common requirement to find and replace duplicate files for most of the computer users. Finding and removing duplicate files is a tiresome job that demands time and patience. Finding duplicate files can be very easy if your machine is powered by GNU\/Linux, thanks to \u2018fdupes<\/strong>\u2018 utility.<\/p>\n

\n

\"Find<\/a><\/p>\n

Fdupes \u2013 Find and Delete Duplicate Files in Linux<\/p>\n<\/div>\n

What is fdupes?<\/h4>\n

Fdupes<\/strong>\u00a0is a Linux utility written by\u00a0Adrian Lopez<\/strong>\u00a0in\u00a0C<\/strong>\u00a0programming Language released under MIT License. The application is able to find duplicate files in the given set of directories and sub-directories. Fdupes recognize duplicates by comparing MD5 signature of files followed by a byte-to-byte comparison. A lots of options can be passed with Fdupes to list, delete and replace the files with hardlinks to duplicates.<\/p>\n

The comparison starts in the order:<\/p>\n

size comparison<\/strong>\u00a0>\u00a0Partial MD5 Signature Comparison<\/strong>\u00a0>\u00a0Full MD5 Signature Comparison<\/strong>\u00a0>\u00a0Byte-to-Byte Comparison<\/strong>.<\/p>\n

Install fdupes on a Linux<\/h3>\n

Installation of latest version of fdupes (fdupes version 1.51) as easy as running following command on\u00a0Debian<\/strong>based systems such as\u00a0Ubuntu<\/strong>\u00a0and\u00a0Linux Mint<\/strong>.<\/p>\n

$ sudo apt-get install fdupes\r\n<\/pre>\n
On\u00a0CentOS<\/strong>\/RHEL<\/strong>\u00a0and\u00a0Fedora<\/strong>\u00a0based systems, you need to turn on\u00a0epel repository<\/a>\u00a0to install fdupes package.<\/p>\n
# yum install fdupes\r\n# dnf install fdupes    [On Fedora 22 onwards]\r\n<\/pre>\n
Note<\/strong>: The default package manager\u00a0yum<\/strong>\u00a0is replaced by\u00a0dnf<\/strong>\u00a0from\u00a0Fedora 22<\/strong>\u00a0onwards\u2026<\/p>\n
How to use fdupes command?<\/h3>\n
1.<\/strong>\u00a0For demonstration purpose, let\u2019s a create few duplicate files under a directory (say\u00a0tecmint<\/strong>) simply as:<\/p>\n
$ mkdir \/home\/\"$USER\"\/Desktop\/tecmint && cd \/home\/\"$USER\"\/Desktop\/tecmint && for i in {1..15}; do echo \"I Love Tecmint. Tecmint is a very nice community of Linux Users.\" > tecmint${i}.txt ; done\r\n<\/pre>\n
After running above command, let\u2019s verify the duplicates files are created or not using\u00a0ls command<\/a>.<\/p>\n
$ ls -l<\/strong>\r\n\r\ntotal 60\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint10.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint11.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint12.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint13.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint14.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint15.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint1.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint2.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint3.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint4.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint5.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint6.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint7.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint8.txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9.txt\r\n<\/pre>\n
The above script create\u00a015<\/strong>\u00a0files namely tecmint1.txt, tecmint2.txt\u2026tecmint15.txt and every files contains the same data i.e.,<\/p>\n
\"I Love Tecmint. Tecmint is a very nice community of Linux Users.\"\r\n<\/pre>\n
2.<\/strong>\u00a0Now search for duplicate files within the folder\u00a0tecmint<\/strong>.<\/p>\n
$ fdupes \/home\/$USER\/Desktop\/tecmint<\/strong> \r\n\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint13.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint8.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint11.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint3.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint4.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint6.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint7.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint9.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint10.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint2.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint5.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint14.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint1.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint15.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint12.txt\r\n<\/pre>\n
3.<\/strong>\u00a0Search for duplicates recursively under every directory including it\u2019s sub-directories using the\u00a0-r<\/strong>\u00a0option.<\/p>\n
It search across all the files and folder recursively, depending upon the number of files and folders it will take some time to scan duplicates. In that mean time, you will be presented with the total progress in terminal, something like this.<\/p>\n
$ fdupes -r \/home<\/strong>\r\n\r\nProgress [37780\/54747] 69%\r\n<\/pre>\n
4.<\/strong>\u00a0See the size of duplicates found within a folder using the\u00a0-S<\/strong>\u00a0option.<\/p>\n
$ fdupes -S \/home\/$USER\/Desktop\/tecmint<\/strong>\r\n\r\n65 bytes each<\/strong>:                          \r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint13.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint8.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint11.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint3.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint4.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint6.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint7.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint9.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint10.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint2.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint5.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint14.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint1.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint15.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint12.txt\r\n<\/pre>\n
5.<\/strong>\u00a0You can see the size of duplicate files for every directory and subdirectories encountered within using the\u00a0-S<\/strong>and\u00a0-r<\/strong>\u00a0options at the same time, as:<\/p>\n
$ fdupes -Sr \/home\/avi\/Desktop\/<\/strong>\r\n\r\n65 bytes each:<\/strong>                          \r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint13.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint8.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint11.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint3.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint4.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint6.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint7.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint9.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint10.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint2.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint5.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint14.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint1.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint15.txt\r\n\/home\/tecmint\/Desktop\/tecmint\/tecmint12.txt\r\n\r\n107 bytes each:<\/strong>\r\n\/home\/tecmint\/Desktop\/resume_files\/r-csc.html\r\n\/home\/tecmint\/Desktop\/resume_files\/fc.html\r\n<\/pre>\n
6.<\/strong>\u00a0Other than searching in one folder or all the folders recursively, you may choose to choose in two folders or three folders as required. Not to mention you can use option\u00a0-S<\/strong>\u00a0and\/or\u00a0-r<\/strong>\u00a0if required.<\/p>\n
$ fdupes \/home\/avi\/Desktop\/ \/home\/avi\/Templates\/\r\n<\/pre>\n
7.<\/strong>\u00a0To delete the duplicate files while preserving a copy you can use the option\u00a0\u2018-d\u2019<\/strong>. Extra care should be taken while using this option else you might end up loosing necessary files\/data and mind it the process is unrecoverable.<\/p>\n
$ fdupes -d \/home\/$USER\/Desktop\/tecmint<\/strong>\r\n\r\n[1] \/home\/tecmint\/Desktop\/tecmint\/tecmint13.txt\r\n[2] \/home\/tecmint\/Desktop\/tecmint\/tecmint8.txt\r\n[3] \/home\/tecmint\/Desktop\/tecmint\/tecmint11.txt\r\n[4] \/home\/tecmint\/Desktop\/tecmint\/tecmint3.txt\r\n[5] \/home\/tecmint\/Desktop\/tecmint\/tecmint4.txt\r\n[6] \/home\/tecmint\/Desktop\/tecmint\/tecmint6.txt\r\n[7] \/home\/tecmint\/Desktop\/tecmint\/tecmint7.txt\r\n[8] \/home\/tecmint\/Desktop\/tecmint\/tecmint9.txt\r\n[9] \/home\/tecmint\/Desktop\/tecmint\/tecmint10.txt\r\n[10] \/home\/tecmint\/Desktop\/tecmint\/tecmint2.txt\r\n[11] \/home\/tecmint\/Desktop\/tecmint\/tecmint5.txt\r\n[12] \/home\/tecmint\/Desktop\/tecmint\/tecmint14.txt\r\n[13] \/home\/tecmint\/Desktop\/tecmint\/tecmint1.txt\r\n[14] \/home\/tecmint\/Desktop\/tecmint\/tecmint15.txt\r\n[15] \/home\/tecmint\/Desktop\/tecmint\/tecmint12.txt\r\n\r\nSet 1 of 1, preserve files [1 - 15, all]:<\/pre>\n
You may notice that all the duplicates are listed and you are prompted to delete, either one by one or certain range or all in one go. You may select a range something like below to delete files files of specific range.<\/p>\n
Set 1 of 1, preserve files [1 - 15, all]: 2-15<\/strong>\r\n\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint13.txt\r\n   [+] \/home\/tecmint\/Desktop\/tecmint\/tecmint8.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint11.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint3.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint4.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint6.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint7.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint9.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint10.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint2.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint5.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint14.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint1.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint15.txt\r\n   [-] \/home\/tecmint\/Desktop\/tecmint\/tecmint12.txt\r\n<\/pre>\n
8.<\/strong>\u00a0From safety point of view, you may like to print the output of\u00a0\u2018fdupes\u2019<\/strong>\u00a0to file and then check text file to decide what file to delete. This decrease chances of getting your file deleted accidentally. You may do:<\/p>\n
$ fdupes -Sr \/home > \/home\/fdupes.txt\r\n<\/pre>\n
Note<\/strong>: You may replace\u00a0\u2018\/home\u2019<\/strong>\u00a0with the your desired folder. Also use option\u00a0\u2018-r\u2019<\/strong>\u00a0and\u00a0\u2018-S\u2019<\/strong>\u00a0if you want to search recursively and Print Size, respectively.<\/p>\n
9.<\/strong>\u00a0You may omit the first file from each set of matches by using option\u00a0\u2018-f\u2019<\/strong>.<\/p>\n
First List files of the directory.<\/p>\n
$ ls -l \/home\/$USER\/Desktop\/tecmint<\/strong>\r\n\r\ntotal 20\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (3rd copy).txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (4th copy).txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (another copy).txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9 (copy).txt\r\n-rw-r--r-- 1 tecmint tecmint 65 Aug  8 11:22 tecmint9.txt\r\n<\/pre>\n
and then omit the first file from each set of matches.<\/p>\n
$ fdupes -f \/home\/$USER\/Desktop\/tecmint<\/strong>\r\n\r\n\/home\/tecmint\/Desktop\/tecmint9 (copy).txt\r\n\/home\/tecmint\/Desktop\/tecmint9 (3rd copy).txt\r\n\/home\/tecmint\/Desktop\/tecmint9 (another copy).txt\r\n\/home\/tecmint\/Desktop\/tecmint9 (4th copy).txt\r\n<\/pre>\n
10.<\/strong>\u00a0Check installed version of fdupes.<\/p>\n
$ fdupes --version<\/strong>\r\n\r\nfdupes 1.51\r\n<\/pre>\n
11.<\/strong>\u00a0If you need any help on fdupes you may use switch\u00a0\u2018-h\u2019<\/strong>.<\/p>\n
$ fdupes -h<\/strong>\r\n\r\nUsage: fdupes [options] DIRECTORY...\r\n\r\n -r --recurse     \tfor every directory given follow subdirectories\r\n                  \tencountered within\r\n -R --recurse:    \tfor each directory given after this option follow\r\n                  \tsubdirectories encountered within (note the ':' at\r\n                  \tthe end of the option, manpage for more details)\r\n -s --symlinks    \tfollow symlinks\r\n -H --hardlinks   \tnormally, when two or more files point to the same\r\n                  \tdisk area they are treated as non-duplicates; this\r\n                  \toption will change this behavior\r\n -n --noempty     \texclude zero-length files from consideration\r\n -A --nohidden    \texclude hidden files from consideration\r\n -f --omitfirst   \tomit the first file in each set of matches\r\n -1 --sameline    \tlist each set of matches on a single line\r\n -S --size        \tshow size of duplicate files\r\n -m --summarize   \tsummarize dupe information\r\n -q --quiet       \thide progress indicator\r\n -d --delete      \tprompt user for files to preserve and delete all\r\n                  \tothers; important: under particular circumstances,\r\n                  \tdata may be lost when using this option together\r\n                  \twith -s or --symlinks, or when specifying a\r\n                  \tparticular directory more than once; refer to the\r\n                  \tfdupes documentation for additional information\r\n -N --noprompt    \ttogether with --delete, preserve the first file in\r\n                  \teach set of duplicates and delete the rest without\r\n                  \tprompting the user\r\n -v --version     \tdisplay fdupes version\r\n -h --help        \tdisplay this help message\r\n<\/pre>\n
That\u2019s for all now. Let me know how you were finding and deleting duplicates files till now in Linux? and also tell me your opinion about this utility. Put your valuable feedback in the comment section below and don\u2019t forget to like\/share us and help us get spread.<\/p>\n
I am working on another utility called\u00a0fslint to remove duplicate files<\/a>, will soon post and you people will love to read.<\/p>\n
Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
It is a common requirement to find and replace duplicate files for most of the computer users. Finding and removing duplicate files is a tiresome job that demands time and patience. Finding duplicate files can be very easy if your machine is powered by GNU\/Linux, thanks to \u2018fdupes\u2018 utility. Fdupes \u2013 Find and Delete Duplicate … <\/p>\n
Continue reading “fdupes \u2013 A Command Line Tool to Find and Delete Duplicate Files in Linux”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-12129","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12129","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=12129"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12129\/revisions"}],"predecessor-version":[{"id":12130,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12129\/revisions\/12130"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=12129"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=12129"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=12129"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}