rsync not deleting file (which was hard linked)
I have a situation that I think might be a bug, but would like someone else to have a look at before I create a report. Here's a list of the source: drwxr-xr-x. 8 root root4096 Feb 24 10:56 current -rw-r--r--. 2 root root 2427204 Feb 21 12:18 qmailadmin-toaster-1.2.16-1.4.1.src.rpm -rw-r--r--. 2 root root 24875 Mar 1 2012 qmailmrtg-toaster-4.2-1.3.7.src.rpm -rw-r--r--. 2 root root 331753 Mar 8 2012 qmail-toaster-1.03-1.3.22.src.rpm Each of these files is linked to an identically named file in the current directory. Then I run rsync as such: rsync --archive \ --compress \ --delay-updates \ --delete-after \ --hard-links \ $source/* \ $target Afterwards, at the target site: drwxr-xr-x 8 root root4096 Feb 24 10:56 current -rw-r--r-- 1 root root 2416365 Aug 2 2012 qmailadmin-toaster-1.2.16-1.4.0.src.rpm -rw-r--r-- 2 root root 2427204 Feb 21 12:18 qmailadmin-toaster-1.2.16-1.4.1.src.rpm -rw-r--r-- 2 root root 24875 Mar 1 2012 qmailmrtg-toaster-4.2-1.3.7.src.rpm -rw-r--r-- 2 root root 331753 Mar 8 2012 qmail-toaster-1.03-1.3.22.src.rpm The qmailadmin*1.4.0.src.rpm file was deleted from the current directory, but the other in the target root remain (link count was 2, is now 1). I can't figure why the 1.4.0 file isn't being removed. It's not removed on subsequent runs either. Anyone have some insight? TIA. -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, how can you tell how much space is actually used?
Thanks Fredrick. That certainly clarifies the matter. Now if I could only remember to use info once in a while. Oh yeah, the navigation. ;) (coming from a long time vi user) -- -Eric 'shubes' On 10/22/2011 10:41 PM, Frederick Grose wrote: On Sun, Oct 23, 2011 at 12:14 AM, Kevin Korb k...@sanitarium.net mailto:k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Well, I just looked at 'man du' and you are right about it. I could swear that it used to say something more useful about the subject but I couldn't quote it. info du If two or more hard links point to the same file, only one of the hard links is counted. The FILE argument order affects which links are counted, and changing the argument order may change the numbers that `du' outputs. In general the man pages for GNU tools suck. On 10/23/11 00:05, Eric Shubert wrote: I've read the man page: -s, --summarize display only a total for each argument Leaves a bit to interpretation, I think. Could go either way, no? After all, the name says: du - estimate file space usage I guess du estimates better than I assumed. Thanks for setting me straight Kevin. :) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On 10/21/2011 07:14 PM, Ido Magal wrote: Thanks! Unfortunately, it doesn't look like the perl for DroboFS supports the required libraries for your script, error while loading shared libraries: libperl.so.5.10: cannot open shared object file: No such file or directory but --itemize-changes should be sufficient. On Fri, Oct 21, 2011 at 09:10, Kevin Korbk...@sanitarium.net wrote: --itemize-changes # find /backup/dir -type f -links 1 ? -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On 10/22/2011 04:04 PM, Ido Magal wrote: 'find' on the droboFS doesn't support '-links' so I had to get creative. And apparently perl has issues on it as well. On Sat, Oct 22, 2011 at 08:02, Eric Shuberte...@shubes.net wrote: # find /backup/dir -type f -links 1 ? Sorry, I'm not familiar with droboFS. So I checked it out a bit. From the looks if things, droboFS may not (from their web site) deliver the best file sharing experience ever. I certainly wouldn't trust my data to BeyondRAID technology alone (give me software Raid-10 any day). You're wise to be doing backups of the thing. Good luck with it. -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On 10/22/2011 07:12 AM, Andrew Gideon wrote: On Fri, 21 Oct 2011 12:10:09 -0400, Kevin Korb wrote: If you want something you can run after the fact here is a tool I wrote a while back that does a sort of diff across 2 --link-dest based backups: http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt It will also tell you what files were not included in the newer backup which --itemize-changes will not since it doesn't actually --delete anything. I like how comprehensive this is, but wouldn't it - by definition of what it means to have a new file in the new directory tree - always report a modified file as having a different inode? That takes me a quick and dirty solution for this. Simply do a find looking for files with a -links (link count) of 1 to see what files are newly copied in the newer directory tree. - Andrew Sorry for re-posting your idea Andrew. Great minds think alike. ;) That sometimes happens, especially in threaded view. -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
I agree. Raid is no backup. I expect your experience is extensive to see that many dual drive RAID1 failures. Haven't seen one yet myself (knock wood). Several years ago, I recovered all but a handful of files from a 3-drive raid-5 array where 2 drives had failed. Wasn't pretty. Needless to say, there were no backups (wasn't my doing). I know where you're coming from. ;) -- -Eric 'shubes' On 10/22/2011 08:12 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I would say that not doing backups of RAID at any level is rather unwise. Of course I have seen dual drive failures of a RAID1 array on about 5 occasions. I have also seen a dual drive failure and a controller SNAFU of a RAID5 on 2 occasions. Simply put, RAID is not a backup. RAID is to keep you running when a drive dies. Backups are to get you running again when RAID fails or when something happens that RAID doesn't protect from (file deletion/corruption either accident or on purpose). On 10/22/11 23:03, Eric Shubert wrote: On 10/22/2011 04:04 PM, Ido Magal wrote: 'find' on the droboFS doesn't support '-links' so I had to get creative. And apparently perl has issues on it as well. On Sat, Oct 22, 2011 at 08:02, Eric Shuberte...@shubes.net wrote: # find /backup/dir -type f -links 1 ? Sorry, I'm not familiar with droboFS. So I checked it out a bit. From the looks if things, droboFS may not (from their web site) deliver the best file sharing experience ever. I certainly wouldn't trust my data to BeyondRAID technology alone (give me software Raid-10 any day). You're wise to be doing backups of the thing. Good luck with it. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6jhg4ACgkQVKC1jlbQAQdzxgCfdj3zH10QrDXQtEYJ1iT68mnx XkMAoLckQyItx+lWIK0PskJmcNL8bk1q =V9RQ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, how can you tell how much space is actually used?
On 10/22/2011 12:32 PM, Kevin Korb wrote: 2. I wanted to know what files were unique to a backup and therefore consuming disk space and why they were different. Which begs another question that I've struggled a bit with lately. I've considered making a series of snapshots by using --link-dest, and moving(/copying) the backup directory prior to (in between) each run. Once that's done, and there are several snapshots and thus files with link counts of up to the number of snapshots, how can I tell how much space is being used by all of the snapshots (and current backup) put together, counting each file only once. du doesn't appear to cut it. ;) I think I have a solution, but am curious to know what others may think of the problem. -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, how can you tell how much space is actually used?
How sure are you of that? I guess I assumed that -s would simply summarize the total for each directory entry. Should probably do a test to see if that's indeed true or not. -- -Eric 'shubes' On 10/22/2011 08:43 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 du -shc backup1 backup2 ... du -s will not count the same file twice even if it appears 50 times. On 10/22/11 23:32, Eric Shubert wrote: On 10/22/2011 12:32 PM, Kevin Korb wrote: 2. I wanted to know what files were unique to a backup and therefore consuming disk space and why they were different. Which begs another question that I've struggled a bit with lately. I've considered making a series of snapshots by using --link-dest, and moving(/copying) the backup directory prior to (in between) each run. Once that's done, and there are several snapshots and thus files with link counts of up to the number of snapshots, how can I tell how much space is being used by all of the snapshots (and current backup) put together, counting each file only once. du doesn't appear to cut it. ;) I think I have a solution, but am curious to know what others may think of the problem. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, how can you tell how much space is actually used?
On 10/22/2011 08:49 PM, Eric Shubert wrote: How sure are you of that? I guess I assumed that -s would simply summarize the total for each directory entry. Should probably do a test to see if that's indeed true or not. Thanks Kevin. You are the wise one: testdir$ ls -lr * -rw-r--r-- 3 shubes shubes 1048576 2011-10-22 20:52 testfile2 -rw-r--r-- 3 shubes shubes 1048576 2011-10-22 20:52 testfile1 testdir2: total 1024 -rw-r--r-- 3 shubes shubes 1048576 2011-10-22 20:52 testfile21 testdir$ du -shc * 1.1Mtestdir2 1.1Mtotal testdir$ I really need to quit assuming. You'd think at my age I'd've learned better. Long live du. :) -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, how can you tell how much space is actually used?
I've read the man page: -s, --summarize display only a total for each argument Leaves a bit to interpretation, I think. Could go either way, no? After all, the name says: du - estimate file space usage I guess du estimates better than I assumed. Thanks for setting me straight Kevin. :) -- -Eric 'shubes' On 10/22/2011 08:51 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The man page says so. My experience doesn't disagree. I am satisfied. On 10/22/11 23:49, Eric Shubert wrote: How sure are you of that? I guess I assumed that -s would simply summarize the total for each directory entry. Should probably do a test to see if that's indeed true or not. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6jj1UACgkQVKC1jlbQAQeEXgCg7444EN2dCXIoXiXeqoZQ7gkS w3sAn2EIO33yFHucDsdut1hcQx4ylbz3 =C7Tp -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync compares all files again and again
On 10/14/2011 02:13 AM, Frank Steiner wrote: Hi, we do a 1:1 backup from our main raid to a backup raid every night with rsync -a --delete /mnt/raid1/ /mnt/raid2 rsync is 3.09, filesystems are ext3, OS is SLES 11 SP1. The rsync process takes several hours, even if no file has changed at all. Using -vv I see that rsync compares all files every time and that takes long for some hundreds of millions of small files. Can I tell rsync it should not compare files in a directory if the directory has not changed, i.e., ctime is the same in source and target? I.e., when I have a dir x with a subdir y and a file y1 in it, I see: rsync -avv x z created directory z delta-transmission disabled for local transfer or --whole-file x/ x/y/ x/y/y1 on the first run and rsync -avv x z x/y/y1 is uptodate on every following run. Why does rsync compare y1 again? I couldn't find a solution in the mailing list or FAQ and when googling. I found a mail in this list telling that -a was the culprit and one should add -O to fix it, but it doesn't make any difference. cu, Frank Perhaps using the ext3 dirindex option would help? -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
