Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
Here is the missing attachment ;-)
On Fri, 2023-09-22 at 21:01 +0200, rsync--- via rsync wrote:
> On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> > So I decided to do a quick test using the Linux kernel source tree since
> > it has lots of files.
>
> Excellent idea using kernel sources! A lot of different files...
> I will use this to create indicative benchmarks for different scenarios...
>
> > I duplicated a tree, used 'find . -type f -exec
> > chmod 444 {} +' to make read only files for rsync to want to chmod, then
> > used cp -al to make several duplicate trees using hard linked files.
> > [...]
> > But also, I did not experience the problem you are describing. My
> > surviving
> > hard links in the duplicate trees were still 444.
>
> If attached a script that does the same (with one file instead of the kernel
> source tree)
> but in my case 444 became 644 again.
>
> - Q: Which rsync version, distro and file system do you use?
>
> - Q: Does my attached script reproduce the permission change?
>
> - Q: Did my script attached to the initial post here reproduce the permission
> change?
>
>
>
> PS: In my case the attached script shows (excerpt):
>
> ./setup_cp_al.sh
>
> Tested with
> - rsync version 3.2.7 protocol version 31
> - ext4 file system
> - Ubuntu 22.04
>
> File stats BEFORE rsync --delete:
>
> File: snapshot2/read_only.txt
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 10305h/66309d Inode: 17571021 Links: 3
> Access: (0444/-r--r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
> Birth: 2023-09-22 20:51:16.690961150 +0200
>
> File stats AFTER rsync --delete
>
> File: snapshot2/read_only.txt
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 10305h/66309d Inode: 17571021 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
> Birth: 2023-09-22 20:51:16.690961150 +0200
>
> Results (after deleting snapshot1 via rsync --delete):
> 1. The 'Change' time of the hardlinked file was updated
> 2. The hardlinked file has now different access rights (644 instead of 444 so
> it is writable again!)
> This does NOT happen if 'rm -f snapshot1/' is used!
>
>
setup_cp_al.sh
Description: application/shellscript
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> So I decided to do a quick test using the Linux kernel source tree since
> it has lots of files.
Excellent idea using kernel sources! A lot of different files...
I will use this to create indicative benchmarks for different scenarios...
> I duplicated a tree, used 'find . -type f -exec
> chmod 444 {} +' to make read only files for rsync to want to chmod, then
> used cp -al to make several duplicate trees using hard linked files.
> [...]
> But also, I did not experience the problem you are describing. My surviving
> hard links in the duplicate trees were still 444.
If attached a script that does the same (with one file instead of the kernel
source tree)
but in my case 444 became 644 again.
- Q: Which rsync version, distro and file system do you use?
- Q: Does my attached script reproduce the permission change?
- Q: Did my script attached to the initial post here reproduce the permission
change?
PS: In my case the attached script shows (excerpt):
./setup_cp_al.sh
Tested with
- rsync version 3.2.7 protocol version 31
- ext4 file system
- Ubuntu 22.04
File stats BEFORE rsync --delete:
File: snapshot2/read_only.txt
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 10305h/66309d Inode: 17571021Links: 3
Access: (0444/-r--r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
Birth: 2023-09-22 20:51:16.690961150 +0200
File stats AFTER rsync --delete
File: snapshot2/read_only.txt
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 10305h/66309d Inode: 17571021Links: 2
Access: (0644/-rw-r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
Birth: 2023-09-22 20:51:16.690961150 +0200
Results (after deleting snapshot1 via rsync --delete):
1. The 'Change' time of the hardlinked file was updated
2. The hardlinked file has now different access rights (644 instead of 444 so
it is writable again!)
This does NOT happen if 'rm -f snapshot1/' is used!
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Fri 22 Sep 2023, Kevin Korb via rsync wrote:
> 444 {} +' to make read only files for rsync to want to chmod, then used cp
> -al to make several duplicate trees using hard linked files. An rm -rf on
> one such tree took .97 seconds while an rsync deletion took 1.25 seconds.
Be sure to drop the caches before such tests every time:
echo 3 > /proc/sys/vm/drop_caches
Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
So I decided to do a quick test using the Linux kernel source tree since
it has lots of files. I duplicated a tree, used 'find . -type f -exec
chmod 444 {} +' to make read only files for rsync to want to chmod, then
used cp -al to make several duplicate trees using hard linked files. An
rm -rf on one such tree took .97 seconds while an rsync deletion took
1.25 seconds. Clearly I need a bigger test to play with as that margin
could easily change just by different outputs if either had a -v. But
also, I did not experience the problem you are describing. My surviving
hard links in the duplicate trees were still 444.
BTW, it seems that rsync uses unlink() while rm uses unlinkat(). No
idea if/why there is a performance diff there but I didn't look into it
as it is now time for me to go to work.
On 9/22/23 04:14, rs...@altfeld-im.de wrote:
On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote:
I have heard in the past that rsyncing an empty dir over a tree to
delete the tree is faster than an rm -rf but I can't say I have ever
benchmarked it to get any actual numbers.
This **may** indeed be a myth (for a long time now) re-cited again and again and
- could no longer be valid today
- could apply only when deleting explicitly named files but not deleting the
complete folder
(as we need to do in "Back in Time")
At least I could not find a holistic benchmark with many files and different
scenarios
(file systems, rsync'ing locally vs. over network, snapshot sizes, number of
files, file sizes, rsync and rm versions...)
Q: Does `rsync` provide a test case that I could use as basis to prepare such a
holistic benchmark?
But now that I am hearing
that rsync actually adds a bunch of pointless chmods to the process. Is
it still faster given this problem? If so maybe we should be trying to
investigate why rm is so slow.
Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not.
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote: > I have heard in the past that rsyncing an empty dir over a tree to > delete the tree is faster than an rm -rf but I can't say I have ever > benchmarked it to get any actual numbers. This **may** indeed be a myth (for a long time now) re-cited again and again and - could no longer be valid today - could apply only when deleting explicitly named files but not deleting the complete folder (as we need to do in "Back in Time") At least I could not find a holistic benchmark with many files and different scenarios (file systems, rsync'ing locally vs. over network, snapshot sizes, number of files, file sizes, rsync and rm versions...) Q: Does `rsync` provide a test case that I could use as basis to prepare such a holistic benchmark? > But now that I am hearing > that rsync actually adds a bunch of pointless chmods to the process. Is > it still faster given this problem? If so maybe we should be trying to > investigate why rm is so slow. Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
I had intended to come back to this but because I didn't really think I
had anything to add to the discussion I put it at a low enough priority
that I forgot about it. But I saw your bug report and was surprised to
see that I was already unhelpful on this topic but because that original
poster didn't have access to do an rm while you are trying to get a
performance boost.
I have heard in the past that rsyncing an empty dir over a tree to
delete the tree is faster than an rm -rf but I can't say I have ever
benchmarked it to get any actual numbers. But now that I am hearing
that rsync actually adds a bunch of pointless chmods to the process. Is
it still faster given this problem? If so maybe we should be trying to
investigate why rm is so slow. Otherwise, I would agree that an
optimization to not do a bunch of pointless chmods is a good idea. I
would suspect that they are only there to prevent permission denied
errors perhaps on some obscure filesystem (I tested on vfat since it is
so non-compatible but read only files unlink fine there).
On 9/18/23 19:42, rsync--- via rsync wrote:
Context
---
I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:
"Back in Time"-Bug:
https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507
"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.
Older snapshots are then deleted according to a schedule.
Issue
-
When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted
files
with existing hardlinks:
mkdir empty
rsync -a --delete -s empty/ snapshot1/
This distorts the backup history.
There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the
issue is still unfixed:
https://bugzilla.samba.org/show_bug.cgi?id=12806
Questions
-
1. Is there a reliable workaround ("--super" is proposed in the issue but may
probably not always work)?
2. If a rsync developer is reading this: Is there any chance to fix this?
Steps to reproduce
--
I have attached a bash script "setup.sh" as txt file which makes the issue
fully reproducible (on Ubuntu 22.04).
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
Context
---
I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:
"Back in Time"-Bug:
https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507
"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.
Older snapshots are then deleted according to a schedule.
Issue
-
When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted
files
with existing hardlinks:
mkdir empty
rsync -a --delete -s empty/ snapshot1/
This distorts the backup history.
There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the
issue is still unfixed:
https://bugzilla.samba.org/show_bug.cgi?id=12806
Questions
-
1. Is there a reliable workaround ("--super" is proposed in the issue but may
probably not always work)?
2. If a rsync developer is reading this: Is there any chance to fix this?
Steps to reproduce
--
I have attached a bash script "setup.sh" as txt file which makes the issue
fully reproducible (on Ubuntu 22.04).
#!/bin/bash
clear
echo
echo "Minimal reproducible example (MRE)"
echo "--"
echo
echo "Tested with"
echo "- rsync version 3.2.7 protocol version 31"
echo "- ext4 file system"
echo
rm -rf MRE
mkdir MRE
cd MRE
mkdir source
touch source/read_only.txt
chmod 444 source/read_only.txt
mkdir snapshot1
# rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --perms --executability --group --owner source/* snapshot1/
rsync --perms --executability --group --owner source/* snapshot1/
mkdir snapshot2
# rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --perms --executability --group --owner --link-dest=../snapshot1
source/* snapshot2/
rsync --perms --executability --group --owner --link-dest=../snapshot1 source/*
snapshot2/
# stat snapshot1/read_only.txt
echo "File stats BEFORE rsync --delete":
echo
stat snapshot2/read_only.txt
mkdir empty
# delete snapshot1 using an empty source directory (shall be the fastest way to
delete a subdir)
# --debug=ALL
rsync -a --delete -s empty/ snapshot1/
# rm -f snapshot1/ # works as expected (unlink does neither change 'Changed'
time nor file permissions)
# strace -c rm -rf snapshot1
echo
stat snapshot2/read_only.txt
echo
echo "Results (after deleting snapshot1 via rsync --delete):"
echo "1. The 'Change' time of the hardlinked file was updated"
echo "2. The hardlinked file has now different access rights (644 instead of
444 so it is writable again!)"
echo "This does NOT happen if 'rm -f snapshot1/' is used!"
echo
echo "Files in snapshot2:"
ls -l snapshot2
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13991] rsync --delete --one-file-system skips deletes after crossing filesystems on destination.
https://bugzilla.samba.org/show_bug.cgi?id=13991 Wayne Davison changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #2 from Wayne Davison --- Yeah, the behavior when recursing back up out of a mount dir into a dir that is a part of a prior top-dir's tree is not handled right at all. Your attached test case was a nice help for testing that (though I just used bind mounts from dirs under /dev/shm instead of creating LVM mounts). I added some code to the delete_in_dir() function to make it keep track of the device values properly and committed it to git. As for the initial report, the -x option is documented as halting deletions under a mounted dir on the receiving side, so that is how it is supposed to work. If you want a different behavior you have several options: You might be able to use --no-x -M-x (on a pull) or -x -M-no-x (on a push) if you want -x on just the sending side. However, I have not tried that (and a local copy would need some kind of remote-shell usage so that the 2 sides can a different options, which the support/lsh script is useful in supporting). A better approach might be to parse the contents of /proc/mounts and turn mount points into filter rules. You then have full control over which dirs you want to exclude, hide, or protect with no guessing. You can even put rules into per-dir filter files if the client and sender side need different rules (and one side may not know what the other side needs). -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13991] rsync --delete --one-file-system skips deletes after crossing filesystems on destination.
https://bugzilla.samba.org/show_bug.cgi?id=13991 --- Comment #1 from Bruce Arden --- Created attachment 15235 --> https://bugzilla.samba.org/attachment.cgi?id=15235=edit Commands to demonstrate further problem set VG to an LVM volumegroup with spare space On my machine this gives: ... [root@arden-lt tmp]# /bin/rsync -axv --del testsrc/. testsrc/test1 testdst/. sending incremental file list deleting deleteme deleting test1/deleteme1 ./ test1/ test1/test11/ test2/ sent 154 bytes received 59 bytes 426.00 bytes/sec total size is 0 speedup is 0.00 [root@arden-lt tmp]# find testsrc -type f -print [root@arden-lt tmp]# find testdst -type f -print testdst/test1/test11/deleteme11 testdst/test2/deleteme2 [root@arden-lt tmp]# test1/test11/deleteme11 and test2/deleteme2 are left in the destination Note that test2/deleteme2 is in the original file system -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13991] New: rsync --delete --one-file-system skips deletes after crossing filesystems on destination.
https://bugzilla.samba.org/show_bug.cgi?id=13991
Bug ID: 13991
Summary: rsync --delete --one-file-system skips deletes after
crossing filesystems on destination.
Product: rsync
Version: 3.1.3
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: wa...@opencoder.net
Reporter: ar...@metronet.co.uk
QA Contact: rsync...@samba.org
When using rsync with the -x (or --one-file-system) option to sync to a
destination that crosses filesystems: --delete won't delete files in the second
filesystem.
Example done on a Centos7 machine where deleteme1 is left in the destination
after deleting from the source:
--
root@arden-lt ~]# ## Setup environment in lvm
root@arden-lt ~]# cd /tmp
[root@arden-lt tmp]# VG=vg_ardenkvm
[root@arden-lt tmp]# mkdir testsrc testdst
[root@arden-lt tmp]# lvcreate -l 1 $VG/lvtest1
Volume group name expected (no slash)
Run `lvcreate --help' for more information.
[root@arden-lt tmp]# lvcreate -l 1 -n lvtest1 $VG
Logical volume "lvtest1" created.
[root@arden-lt tmp]# mkfs /dev/$VG/lvtest1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
1024 inodes, 4096 blocks
204 blocks (4.98%) reserved for the super user
First data block=1
Maximum filesystem blocks=4194304
1 block group
8192 blocks per group, 8192 fragments per group
1024 inodes per group
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
[root@arden-lt tmp]# mkdir testdst/test1
[root@arden-lt tmp]# mount /dev/$VG/lvtest1 testdst/test1
[root@arden-lt tmp]# mkdir testsrc/test1
[root@arden-lt tmp]# touch testsrc/deleteme testsrc/test1/deleteme1
[root@arden-lt tmp]# ## First sync with no problems
[root@arden-lt tmp]# rsync -axv --del testsrc/. testdst/.
sending incremental file list
./
deleteme
test1/
test1/deleteme1
sent 213 bytes received 69 bytes 564.00 bytes/sec
total size is 0 speedup is 0.00
[root@arden-lt tmp]# \rm testsrc/deleteme testsrc/test1/deleteme1
[root@arden-lt tmp]# ## Second sync that should delete files
[root@arden-lt tmp]# rsync -axv --del testsrc/. testdst/.
sending incremental file list
deleting deleteme
./
test1/
sent 72 bytes received 31 bytes 206.00 bytes/sec
total size is 0 speedup is 0.00
[root@arden-lt tmp]# ## Files on src
[root@arden-lt tmp]# find testsrc -print
testsrc
testsrc/test1
[root@arden-lt tmp]# ## Files on dest that should be the same as src,
[root@arden-lt tmp]# ## but deleteme1 is still there
[root@arden-lt tmp]# find testdst -print
testdst
testdst/test1
testdst/test1/lost+found
testdst/test1/deleteme1
[root@arden-lt tmp]# ## Without -x it works as expected
[root@arden-lt tmp]# rsync -av --del testsrc/. testdst/.
sending incremental file list
deleting test1/lost+found/
deleting test1/deleteme1
sent 66 bytes received 53 bytes 238.00 bytes/sec
total size is 0 speedup is 0.00
[root@arden-lt tmp]# find testdst -print
testdst
testdst/test1
[root@arden-lt tmp]# rsync --version
rsync version 3.1.3 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, no ACLs, xattrs, iconv, symtimes, prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
[root@arden-lt tmp]#
---
[OK it's not very good removing lost+found, but that's not the problem].
I think the problem is the code in delete_in_dir() in generator.c:
if (one_file_system) {
if (file->flags & FLAG_TOP_DIR)
filesystem_dev = *fs_dev;
else if (filesystem_dev != *fs_dev)
return;
}
As far as I can tell this is unnecessary. --one-file-system should only check
the source filesystems not the destination filesystems.
This code can cause worse problems too. I haven't figured out the simplest way
of demonstrating it. But if there are other filesystems mounted, deletes can
fail at the higher level as well - contact me if you want details.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11086] rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 Wayne Davison way...@samba.org changed: What|Removed |Added Resolution|--- |INVALID Status|NEW |RESOLVED -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete
Hi, Rsync. I want to help rsink delete a folder with a large number of files and folders. Tried this: rsync -a --no-D --delete /dev/null /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file null rsync -a --no-D --delete /dev/zero /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file zero That's how it turns out rsync -a --delete /empty_folder/ /home/rc-41/data/061/2015-04-01-07-04/ But this option is not satisfied as if the disk is 100% filled to create an empty folder does not work mkdir /empty folder/ Disk quota ekstseeded Got an error. find /home/rc-41/data/061/2015-04-01-07-04/ -delete I know not suitable rm -rf /home/rc-41/data/061/2015-04-01-07-04/ is also not suitable How to do it differently? -- Sincerely, Dugin Sergey mailto: d...@qwarta.ru QWARTA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete
Wow, it took me a few seconds to figure out what you were trying to do. What's wrong with rm? Also I think trying to leverage the side of disqualifying all source files just to get the delete effect (very clever but somewhat obtuse!) risks creating a temporary file of some kind in the target at the start of the operation, and if you cant even mkdir then that exceeds disk quota immediately and fails. /kc On Thu, Apr 16, 2015 at 12:20:52PM +0300, ? ?? said: Hi, Rsync. I want to help rsink delete a folder with a large number of files and folders. Tried this: rsync -a --no-D --delete /dev/null /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file null rsync -a --no-D --delete /dev/zero /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file zero That's how it turns out rsync -a --delete /empty_folder/ /home/rc-41/data/061/2015-04-01-07-04/ But this option is not satisfied as if the disk is 100% filled to create an empty folder does not work mkdir /empty folder/ Disk quota ekstseeded Got an error. find /home/rc-41/data/061/2015-04-01-07-04/ -delete I know not suitable rm -rf /home/rc-41/data/061/2015-04-01-07-04/ is also not suitable How to do it differently? -- Sincerely, Dugin Sergey mailto: d...@qwarta.ru QWARTA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I don't understand what is wrong with rm either. But if you must have an empty directory is there a tmpfs where you can make one? Is there already an empty one like /var/empty? On 04/16/2015 10:13 AM, Ken Chase wrote: Wow, it took me a few seconds to figure out what you were trying to do. What's wrong with rm? Also I think trying to leverage the side of disqualifying all source files just to get the delete effect (very clever but somewhat obtuse!) risks creating a temporary file of some kind in the target at the start of the operation, and if you cant even mkdir then that exceeds disk quota immediately and fails. /kc On Thu, Apr 16, 2015 at 12:20:52PM +0300, ? ?? said: Hi, Rsync. I want to help rsink delete a folder with a large number of files and folders. Tried this: rsync -a --no-D --delete /dev/null /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file null rsync -a --no-D --delete /dev/zero /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file zero That's how it turns out rsync -a --delete /empty_folder/ /home/rc-41/data/061/2015-04-01-07-04/ But this option is not satisfied as if the disk is 100% filled to create an empty folder does not work mkdir /empty folder/ Disk quota ekstseeded Got an error. find /home/rc-41/data/061/2015-04-01-07-04/ -delete I know not suitable rm -rf /home/rc-41/data/061/2015-04-01-07-04/ is also not suitable How to do it differently? -- Sincerely, Dugin Sergey mailto: d...@qwarta.ru QWARTA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlUv1A8ACgkQVKC1jlbQAQfaGACfR7g0t19aeY5KiUTcsxBJqEVy tjcAnR63Viq8B0NZ4p+GgwMO+ZENjdPZ =aHlw -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I thought of something else if it must be rsync that is used... rsync -a --delete --exclude='*' --delete-excluded /anywhere/ /stuff/to/delete/ That would empty the target dir regardless of what is in the source dir. I would still like to hear how that is any different than rm. On 04/16/2015 11:28 AM, Ken Chase wrote: problem is he's trying to rsync into the target dir and have the side effect of delete. so an empty dir would necessarily need to be in the target of course and thus created there, triggering the quota block. he tried to avoid this by using device files then 'blocking all device files' but i think rsync figures out first there's nothing to do, so it just stops and doesnt do the delete. wonder if --delete-first would help there perhaps. however, this is a REALLY obtuse way of running rm. unless of course he's trying to inject some kinda options into a script that can only run rsync or something wonky like that. /kc On Thu, Apr 16, 2015 at 11:23:59AM -0400, Kevin Korb said: I don't understand what is wrong with rm either. But if you must have an empty directory is there a tmpfs where you can make one? Is there already an empty one like /var/empty? On 04/16/2015 10:13 AM, Ken Chase wrote: Wow, it took me a few seconds to figure out what you were trying to do. What's wrong with rm? Also I think trying to leverage the side of disqualifying all source files just to get the delete effect (very clever but somewhat obtuse!) risks creating a temporary file of some kind in the target at the start of the operation, and if you cant even mkdir then that exceeds disk quota immediately and fails. /kc On Thu, Apr 16, 2015 at 12:20:52PM +0300, ? ?? said: Hi, Rsync. I want to help rsink delete a folder with a large number of files and folders. Tried this: rsync -a --no-D --delete /dev/null /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file null rsync -a --no-D --delete /dev/zero /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file zero That's how it turns out rsync -a --delete /empty_folder/ /home/rc-41/data/061/2015-04-01-07-04/ But this option is not satisfied as if the disk is 100% filled to create an empty folder does not work mkdir /empty folder/ Disk quota ekstseeded Got an error. find /home/rc-41/data/061/2015-04-01-07-04/ -delete I know not suitable rm -rf /home/rc-41/data/061/2015-04-01-07-04/ is also not suitable How to do it differently? -- Sincerely, Dugin Sergey mailto: d...@qwarta.ru QWARTA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlUv2JwACgkQVKC1jlbQAQdV8wCfZnxIDPGGj+OiAYb5RDc61OOx PwAAnj2WkSoEUhZsd6g1yMmFNCMrmhqs =oH1r -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete
problem is he's trying to rsync into the target dir and have the side effect of delete. so an empty dir would necessarily need to be in the target of course and thus created there, triggering the quota block. he tried to avoid this by using device files then 'blocking all device files' but i think rsync figures out first there's nothing to do, so it just stops and doesnt do the delete. wonder if --delete-first would help there perhaps. however, this is a REALLY obtuse way of running rm. unless of course he's trying to inject some kinda options into a script that can only run rsync or something wonky like that. /kc On Thu, Apr 16, 2015 at 11:23:59AM -0400, Kevin Korb said: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I don't understand what is wrong with rm either. But if you must have an empty directory is there a tmpfs where you can make one? Is there already an empty one like /var/empty? On 04/16/2015 10:13 AM, Ken Chase wrote: Wow, it took me a few seconds to figure out what you were trying to do. What's wrong with rm? Also I think trying to leverage the side of disqualifying all source files just to get the delete effect (very clever but somewhat obtuse!) risks creating a temporary file of some kind in the target at the start of the operation, and if you cant even mkdir then that exceeds disk quota immediately and fails. /kc On Thu, Apr 16, 2015 at 12:20:52PM +0300, ? ?? said: Hi, Rsync. I want to help rsink delete a folder with a large number of files and folders. Tried this: rsync -a --no-D --delete /dev/null /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file null rsync -a --no-D --delete /dev/zero /home/rc-41/data/061/2015-04-01-07-04/ skipping non-regular file zero That's how it turns out rsync -a --delete /empty_folder/ /home/rc-41/data/061/2015-04-01-07-04/ But this option is not satisfied as if the disk is 100% filled to create an empty folder does not work mkdir /empty folder/ Disk quota ekstseeded Got an error. find /home/rc-41/data/061/2015-04-01-07-04/ -delete I know not suitable rm -rf /home/rc-41/data/061/2015-04-01-07-04/ is also not suitable How to do it differently? -- Sincerely, Dugin Sergey mailto: d...@qwarta.ru QWARTA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlUv1A8ACgkQVKC1jlbQAQfaGACfR7g0t19aeY5KiUTcsxBJqEVy tjcAnR63Viq8B0NZ4p+GgwMO+ZENjdPZ =aHlw -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - k...@heavycomputing.ca -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete
On 16.04.2015 12:20, ? ?? wrote: Hi, Rsync. find /home/rc-41/data/061/2015-04-01-07-04/ -delete I know not suitable -mindepth 1 ? That deletes the whole contents, but doesn't delete the folder itself. -- Matthias -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11086] New: rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 Bug ID: 11086 Summary: rsync --delete has stopped to delete destination files that don't exist at the source Product: rsync Version: 3.1.1 Hardware: x64 OS: Linux Status: NEW Severity: major Priority: P5 Component: core Assignee: way...@samba.org Reporter: luminar...@gmail.com QA Contact: rsync...@samba.org I have a system Linux root partition wich I backup to another disk periodically. I use the following script for that action: #!/bin/bash NICE=15 PH=/mnt/b cd / nice -n $NICE rm -v /root/.xsession-errors nice -n $NICE rm -v /root/dead.letter nice -n $NICE /usr/bin/rsync -avhHxq --progress --stats --force --delete --delete-before --log-file='/var/log/backup2diskrsync.log' --exclude='/usr/portage/distfiles/' --exclude='/home/ftp/' --exclude='/lost+found/' --exclude='/var/tmp/ccache/' --exclude='/root/.ccache/' --exclude='/var/cache/edb/' --exclude='/home/myuser/.cache/chromium/' --exclude='/home/myuser/.opera/cache/' --exclude='/home/myuser/.opera/opcache/' --exclude='/home/mysearch/search/fsearch/swapfile' --exclude='/home/myuser/workspace/mysearch/swapfile' / $PH/ The problem is that I don't see any delete messages in log file anymore. For example, I have the following in /usr/src/ directory on backup disk: # du -sh /mnt/b/usr/src/* 0 /mnt/b/usr/src/linux 944M/mnt/b/usr/src/linux-3.14.14-gentoo 961M/mnt/b/usr/src/linux-3.16.5-gentoo 964M/mnt/b/usr/src/linux-3.17.7-gentoo 972M/mnt/b/usr/src/linux-3.18.3-gentoo 971M/mnt/b/usr/src/linux-3.18.5-gentoo But I don't have this outdated long time ago directories on my root partition: # du -sh /usr/src/* 0 /usr/src/linux 943M/usr/src/linux-3.18.5-gentoo And that's just a one of examples of outdated files and directories. My backup storage partition will become full soon. The rsync log (after # /bin/bash backup2diskwithrsync.sh under root): 2015/01/31 17:25:56 [12091] building file list 2015/01/31 17:25:58 [12091] rsync: readlink_stat(/home/myuser/.gvfs) failed: Permission denied (13) 2015/01/31 17:25:59 [12091] 877346 files to consider 2015/01/31 17:26:01 [12091] f..t.. etc/openvpn/openvpn-status.log 2015/01/31 17:26:01 [12091] f..t.. home/incoming/scripts/enp2s0-upd.err 2015/01/31 17:26:01 [12091] f..t.. home/incoming/scripts/enp2s0-upd.info 2015/01/31 17:26:01 [12091] f..t.. home/incoming/scripts/ppp0-upd.err 2015/01/31 17:26:01 [12091] f..t.. home/incoming/scripts/ppp0-upd.info 2015/01/31 17:26:03 [12091] .d..t.. var/log/ 2015/01/31 17:26:03 [12091] f.st.. var/log/backup2diskrsync.log 2015/01/31 17:26:03 [12091] f.st.. var/log/messages 2015/01/31 17:26:03 [12091] f.st.. var/log/net-snmpd.log 2015/01/31 17:26:04 [12091] f.st.. var/log/samba/log.nmbd 2015/01/31 17:26:05 [12091] f..t.. var/tmp/kdecache-root/icon-cache.kcache 2015/01/31 17:26:05 [12091] f..t.. var/tmp/kdecache-myuser/plasma_theme_G-Remix-T_v5.3.kcache 2015/01/31 17:26:06 [12091] f..t.. var/tmp/kdecache-myuser/plasma_theme_internal-system-colors.kcache 2015/01/31 17:26:06 [12091] .d..t.. var/www/localhost/htdocs/rutorrent/share/users/myuserutorrent/settings/erasedata/ 2015/01/31 17:26:07 [12091] Number of files: 877,346 (reg: 758,168, dir: 67,562, link: 45,877, dev: 5,678, special: 61) 2015/01/31 17:26:07 [12091] Number of created files: 0 2015/01/31 17:26:07 [12091] Number of deleted files: 0 2015/01/31 17:26:07 [12091] Number of regular files transferred: 12 2015/01/31 17:26:07 [12091] Total file size: 25.31G bytes 2015/01/31 17:26:07 [12091] Total transferred file size: 379.83M bytes 2015/01/31 17:26:07 [12091] Literal data: 379.83M bytes 2015/01/31 17:26:07 [12091] Matched data: 0 bytes 2015/01/31 17:26:07 [12091] File list size: 24.44M 2015/01/31 17:26:07 [12091] File list generation time: 3.099 seconds 2015/01/31 17:26:07 [12091] File list transfer time: 0.000 seconds 2015/01/31 17:26:07 [12091] Total bytes sent: 404.40M 2015/01/31 17:26:07 [12091] Total bytes received: 291 2015/01/31 17:26:07 [12091] sent 404.40M bytes received 291 bytes 35.17M bytes/sec 2015/01/31 17:26:07 [12091] total size is 25.31G speedup is 62.60 2015/01/31 17:26:07 [12091] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1] Error log in the console (some files it doesn't like probably) rsync: readlink_stat(/home/xor/.gvfs) failed: Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1] And still there are a lot of garbage on backup partition wich had to be deleted a looong time ago: # du -sh /mnt/b/usr/src/* 0 /mnt/b/usr/src/linux 944M/mnt/b/usr/src/linux-3.14.14-gentoo 961M/mnt/b/usr/src/linux-3.16.5-gentoo 964M/mnt/b/usr/src/linux-3.17.7-gentoo 972M
[Bug 11086] rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 --- Comment #1 from Kevin Korb rs...@sanitarium.net --- This is your problem: 2015/01/31 17:25:58 [12091] rsync: readlink_stat(/home/myuser/.gvfs) failed: Permission denied (13) If rsync hits any error it aborts the deletion process which it would have said without -q. You can override that with --ignore-errors but better to --exclude=.gvfs since it isn't a real file/dir/filesystem. Also, if you are making backups you should really look into rsync --link-dest or an rsync based backup system if you don't feel like scripting it. Also, --delete-before is much slower than --delete-during which is the default version of --delete. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11086] rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 --- Comment #3 from Kevin Korb rs...@sanitarium.net --- What you have is a mirror not a backup system. Sure, it can protect you from accidental deletions or disk failures but it wouldn't protect you from intentional data destruction (think virus or hacker). Yes, I was referring to date+time based snapshots. That is what rsync with --link-dest and some scripting can do and that is what most of the rsync based backup systems use. I prefer to have many backups in a system. That way I can restore data from the past if I need to (which I have). It is pretty trivial to boot from something like SystemRescueCD and restore your whole system from such a backup. In fact, if your backups are on a different system and you setup a network boot environment you can boot from the backup over the network. I did that once when my desktop became unstable and I was trying to figure out if it was a hardware problem or if it was caused by recent updates. When I could duplicate a several day old problem on your computer with the software it was running a month ago I knew it was the hardware that was bad. Turned out there was a leaking capacitor behind the heatsink on the video card. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11086] rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 --- Comment #4 from ddd luminar...@gmail.com --- Well, maybe I should look into those rsync-based snapshot systems when I'll buy another 3 TB HDD. ) Can you recommend any of those backup systems? -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11086] rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 --- Comment #2 from ddd luminar...@gmail.com --- Thank you a lot! I added --exclude='.gvfs' parameter and everything works now. Thumbs up! I was trying to use --delete-before because of low space on the partition. Now it's really enough space and I use default --delete option. Also, if you are making backups you should really look into rsync --link-dest or an rsync based backup system if you don't feel like scripting it. Hmm, what --link-dest stands for? Is it somehow related to hard-links on the same filesystem? I usually try to exclude hardlinks like --exclude='/home/ftp/' wich usually works for me. --link-dest=DIR hardlink to files in DIR when unchanged I don't understand what that option is doing. I use rsync for easy backups of the system partition from the primary disk to a backup one 2 times a week. It's very useful when I delete smth by accident like one time I did crontab -r instead of crontab -e. ) And in case 1st system drive fails I change SATA interface immediately and start OS from the backup one. Rsync based backup system? I've seen some of those. They usually don't do what I need (like they backup to files, snapshots with dates in them instead of simple backup to another disk wich consumes too much space and you'll have to use liveDVD to start from such snapshot wich is time-consuming) or too overburdened with options, configs etc. My scripts is very simple and does exactly what I need. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11086] rsync --delete has stopped to delete destination files that don't exist at the source
https://bugzilla.samba.org/show_bug.cgi?id=11086 --- Comment #5 from Kevin Korb rs...@sanitarium.net --- Not really. I wrote my own. IIRC BackupPC is the most up to date one. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete-hide
Hello!
I was wondering wether or not this would be a viable option to add to rsync:
--delete-hide
Instead of deleting the file, rsync would rename the file using leading “.” and
trailing “.deleted-{timestamp}”.
rsync -rv --delete-hide /src/ /dst/
For example if /src/readme.txt had already been rsynced to /dst/readme.txt and
then later was deleted from /src, rsync would - instead of deleting
/dst/readme.txt - rename /dst/readme.txt to
/dst/.readme.txt.deleted-20140604120445.
Rgrds,
Saku Vainikainen.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete-hide
Hi,
I think a very similar functionality can be achieved by using the existing
--backup option? Using the backup (-b) option in conjunction with the --del
option allows you to set a suffix (using --suffix) . But of course you
can't add a prefix (to hide it) currently I think
So I might use something like:
rsync -rv -b --del --suffix=.deleted-$(date +%Y%m%d%H%M%S) -f P
*.deleted-[[:digit:]]* src/ dst/
The -f filter rule would be required to protect existing deleted files from
getting deleted again.
On Sun, Apr 6, 2014 at 3:00 PM, Saku Vainikainen saku.vainikai...@welho.com
wrote:
Hello!
I was wondering wether or not this would be a viable option to add to
rsync:
--delete-hide
Instead of deleting the file, rsync would rename the file using leading
. and trailing .deleted-{timestamp}.
rsync -rv --delete-hide /src/ /dst/
For example if /src/readme.txt had already been rsynced to /dst/readme.txt
and then later was deleted from /src, rsync would - instead of deleting
/dst/readme.txt - rename /dst/readme.txt to
/dst/.readme.txt.deleted-20140604120445.
Rgrds,
Saku Vainikainen.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete not working (no wildcards)
On 01/28/13 17:02, Stewart Jeacocke wrote: I'm trying to copy some directories with rsync and would like to delete files from the destination if they no longer exist in the source. I'm using the following command: rsync -v --archive --hard-links --delete --force --filter=':/.rsync-filter' /opt/ /mnt/backup/opt/ /opt does not contain any files the destination contains one file stewart at attobeast:/etc$ ls /mnt/backup/opt/ test-file after running the rsync command I would expect test-file to have been deleted but it isn't. I'm sure I'm missing something obvious in the man page that explains this behaviour but I can't figure it out. Can anyone enlighten me? On 01/28/13 15:45:39 MST, Kevin Korb wrote: Is there any output? How about with an additional -v ? Is there a .rsync-filter file anywhere? Thanks for your help Kevin. Adding -v -v revealed it was a .rysnc-filter file that was causing the mischief attobeast:/# rsync -v -v --archive --hard-links --delete --force --filter=': /.rsync-filter' /opt/ /mnt/backup/opt/ sending incremental file list delta-transmission disabled for local transfer or --whole-file [generator] protecting file test-file because of pattern /mnt/*/** [per-dir .rsync-filter] total: matches=0 hash_hits=0 false_alarms=0 data=0 sent 29 bytes received 15 bytes 88.00 bytes/sec total size is 0 speedup is 0.00 changing to using --delete-excluded made rsync behave in the way I wanted -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete not working (no wildcards)
Hi, I'm trying to copy some directories with rsync and would like to delete files from the destination if they no longer exist in the source. I'm using the following command: rsync -v --archive --hard-links --delete --force --filter=':/.rsync-filter' /opt/ /mnt/backup/opt/ /opt does not contain any files the destination contains one file stewart@attobeast:/etc$ ls /mnt/backup/opt/ test-file after running the rsync command I would expect test-file to have been deleted but it isn't. I'm sure I'm missing something obvious in the man page that explains this behaviour but I can't figure it out. Can anyone enlighten me? Thanks Stewart -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete not working (no wildcards)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Is there any output? How about with an additional -v ? Is there a .rsync-filter file anywhere? On 01/28/13 17:02, Stewart Jeacocke wrote: Hi, I'm trying to copy some directories with rsync and would like to delete files from the destination if they no longer exist in the source. I'm using the following command: rsync -v --archive --hard-links --delete --force --filter=':/.rsync-filter' /opt/ /mnt/backup/opt/ /opt does not contain any files the destination contains one file stewart@attobeast:/etc$ ls /mnt/backup/opt/ test-file after running the rsync command I would expect test-file to have been deleted but it isn't. I'm sure I'm missing something obvious in the man page that explains this behaviour but I can't figure it out. Can anyone enlighten me? Thanks Stewart - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlEG/5MACgkQVKC1jlbQAQcJhgCgoi+YfzaiheVg9HUP9TNBUs+d RFAAn3Q0vI3Lm2Wb7lQz8A4P96eTzXfj =C/YF -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete not working (no wildcards)
At 22:02 28.01.2013 +, Stewart Jeacocke wrote: Hi, I'm trying to copy some directories with rsync and would like to delete files from the destination if they no longer exist in the source. I'm using the following command: rsync -v --archive --hard-links --delete --force --filter=':/.rsync-filter' /opt/ /mnt/backup/opt/ /opt does not contain any files the destination contains one file stewart@attobeast:/etc$ ls /mnt/backup/opt/ test-file after running the rsync command I would expect test-file to have been deleted but it isn't. I'm sure I'm missing something obvious in the man page that explains this behaviour but I can't figure it out. Can anyone enlighten me? I'd say you have to give the parent folder as source to detect missing files: rsync ... /opt /mnt/backup/ Note the missing backslash of /opt. bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Understanding rsync --delete behavior
Hi, When running an rsync command of the form: $ rsync -avn --delete remotehost:/mnt/volsw_dev/AS5/ /mnt/volsw_dev/AS5 rsync wants to delete a slew of files and directories. Partial output includes, for example: deleting AS5/zeromq/2.2.0/share/man/man7/zmq_tcp.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_pgm.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_ipc.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_inproc.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_epgm.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_cpp.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq.7 deleting AS5/zeromq/2.2.0/share/man/man7/ deleting AS5/zeromq/2.2.0/share/man/man3/zmq_version.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_term.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_strerror.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_socket.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_setsockopt.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_send.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_recv.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_poll.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_size.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_move.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_init_size.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_init_data.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_init.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_data.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_copy.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_close.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_init.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_getsockopt.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_errno.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_device.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_connect.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_close.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_bind.3 deleting AS5/zeromq/2.2.0/share/man/man3/ deleting AS5/zeromq/2.2.0/share/man/ deleting AS5/zeromq/2.2.0/share/ deleting AS5/zeromq/2.2.0/lib/pkgconfig/libzmq.pc deleting AS5/zeromq/2.2.0/lib/pkgconfig/ deleting AS5/zeromq/2.2.0/lib/libzmq.so.1.0.1 deleting AS5/zeromq/2.2.0/lib/libzmq.so.1 deleting AS5/zeromq/2.2.0/lib/libzmq.so deleting AS5/zeromq/2.2.0/lib/libzmq.la deleting AS5/zeromq/2.2.0/lib/libzmq.a deleting AS5/zeromq/2.2.0/lib/ deleting AS5/zeromq/2.2.0/include/zmq_utils.h deleting AS5/zeromq/2.2.0/include/zmq.hpp deleting AS5/zeromq/2.2.0/include/zmq.h deleting AS5/zeromq/2.2.0/include/ deleting AS5/zeromq/2.2.0/ deleting AS5/zeromq/ However, if I run command: $ rsync -avn --delete remotehost:/mnt/volsw_dev/AS5/zeromq/ /mnt/volsw_dev/AS5/zeromq There are no changes reported (which is what I expected). Why does doing the rsync one level up (without zeromq) cause rsync to indicate the local zeromq directory should be deleted, but adding the zeromq directory to the source and destination path make rsync work as I expect? I'm sure I'm missing something basic in the way --delete works, but if someone can explain, I'd appreciate it. This is rsync 3.0.6 (on both the local and remote server). Thanks, Richard -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Understanding rsync --delete behavior
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 My guess is that you did it wrong once in the past and have an extra /mnt/AS5/AS5 on the target that is being purged. Take a look for that possibility and also try again with an addition -v and with - --itemize-changes. On 06/17/12 20:55, Richard Zinar wrote: Hi, When running an rsync command of the form: $ rsync -avn --delete remotehost:/mnt/volsw_dev/AS5/ /mnt/volsw_dev/AS5 rsync wants to delete a slew of files and directories. Partial output includes, for example: deleting AS5/zeromq/2.2.0/share/man/man7/zmq_tcp.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_pgm.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_ipc.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_inproc.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_epgm.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq_cpp.7 deleting AS5/zeromq/2.2.0/share/man/man7/zmq.7 deleting AS5/zeromq/2.2.0/share/man/man7/ deleting AS5/zeromq/2.2.0/share/man/man3/zmq_version.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_term.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_strerror.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_socket.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_setsockopt.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_send.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_recv.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_poll.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_size.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_move.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_init_size.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_init_data.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_init.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_data.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_copy.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_msg_close.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_init.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_getsockopt.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_errno.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_device.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_connect.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_close.3 deleting AS5/zeromq/2.2.0/share/man/man3/zmq_bind.3 deleting AS5/zeromq/2.2.0/share/man/man3/ deleting AS5/zeromq/2.2.0/share/man/ deleting AS5/zeromq/2.2.0/share/ deleting AS5/zeromq/2.2.0/lib/pkgconfig/libzmq.pc deleting AS5/zeromq/2.2.0/lib/pkgconfig/ deleting AS5/zeromq/2.2.0/lib/libzmq.so.1.0.1 deleting AS5/zeromq/2.2.0/lib/libzmq.so.1 deleting AS5/zeromq/2.2.0/lib/libzmq.so deleting AS5/zeromq/2.2.0/lib/libzmq.la deleting AS5/zeromq/2.2.0/lib/libzmq.a deleting AS5/zeromq/2.2.0/lib/ deleting AS5/zeromq/2.2.0/include/zmq_utils.h deleting AS5/zeromq/2.2.0/include/zmq.hpp deleting AS5/zeromq/2.2.0/include/zmq.h deleting AS5/zeromq/2.2.0/include/ deleting AS5/zeromq/2.2.0/ deleting AS5/zeromq/ However, if I run command: $ rsync -avn --delete remotehost:/mnt/volsw_dev/AS5/zeromq/ /mnt/volsw_dev/AS5/zeromq There are no changes reported (which is what I expected). Why does doing the rsync one level up (without zeromq) cause rsync to indicate the local zeromq directory should be deleted, but adding the zeromq directory to the source and destination path make rsync work as I expect? I'm sure I'm missing something basic in the way --delete works, but if someone can explain, I'd appreciate it. This is rsync 3.0.6 (on both the local and remote server). Thanks, Richard - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/efaMACgkQVKC1jlbQAQcVawCfdXXb9j37EVqHegBbFxAsCquj 12IAn2V+jh1jEa7UAnMsrJnqU6oWAA22 =+ihv -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5939] rsync: delete of stat xattr failed for ... (in backup): Operation not permitted (1)
https://bugzilla.samba.org/show_bug.cgi?id=5939 way...@samba.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Comment #4 from way...@samba.org 2011-02-22 09:10 CST --- In the latest code, xattrs are turned off for symlinks on Linux. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5939] rsync: delete of stat xattr failed for ... (in backup): Operation not permitted (1)
https://bugzilla.samba.org/show_bug.cgi?id=5939 wthr...@mit.edu changed: What|Removed |Added CC||wthr...@mit.edu --- Comment #2 from wthr...@mit.edu 2009-07-14 21:59 CST --- I also see this bug (using --fake-super), including in a nightly build last week. I'm actually getting this error on all symlinks I tested (tried broken and working to both a file and a directory, didn't try different owners). I think this is because I'm using an ext3 file system, and it doesn't support xattrs on symlinks. I notice that in the code, usually when an xattr change is attempted, if it fails on a symlink with EPERM, the failure is ignored, but that this check is not performed for the delete of stat xattr change. Adding a check makes the warnings disappear. I'm not sure if there is a reason for the absence of the check, but it seems to cause errors even when there should not be any xattrs set on the link (because the get_stat_xattr function claims there is an xattr there when it can't read the xattrs). Anyway, I've written a simple patch adding the check for EPERM on a symlink. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5939] rsync: delete of stat xattr failed for ... (in backup): Operation not permitted (1)
https://bugzilla.samba.org/show_bug.cgi?id=5939 --- Comment #3 from wthr...@mit.edu 2009-07-14 22:03 CST --- Created an attachment (id=4418) -- (https://bugzilla.samba.org/attachment.cgi?id=4418action=view) Patch to ignore delete of stat xattr failures on symlinks -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync delete architecture
On Mon, 2009-05-18 at 02:28 -0700, tammy wrote: I have 3 load balancing servers. Each user may login to different servers to upload their data, lets say, a company logo. That's why i tried to use rsync to synchronize the uploaded logo and some other necessary modules. This mean, each server can be a source or destination server to the another 2 servers. However, problem occurs when user tried to delete the logo. Maybe he access to server 1, he deletes the logo, however, few minutes later the logo will appear again because server 2 and server 3 still having the logo, never been deleted, and sync back to server1. This is a two-way synchronization job, which rsync is not really meant to do. Try a real two-way synchronization tool like Unison (http://www.cis.upenn.edu/~bcpierce/unison/). -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync delete architecture
Dear all, I have 3 load balancing servers. Each user may login to different servers to upload their data, lets say, a company logo. That's why i tried to use rsync to synchronize the uploaded logo and some other necessary modules. This mean, each server can be a source or destination server to the another 2 servers. However, problem occurs when user tried to delete the logo. Maybe he access to server 1, he deletes the logo, however, few minutes later the logo will appear again because server 2 and server 3 still having the logo, never been deleted, and sync back to server1. The manual job i can do is delete the same logo on server 2 and server 3. But i know this of course not the solution. I wonder if rsync can check the time of files. I've create a cron job which run every 2 minutes. How if i only sync files which having last modified date less than 2.5 minutes? Not sure if rsync has this feature, but it still couldnt solve the delete problem. The deleted file may not be sync anymore, but it wont be deleted as well.. or i just delete the file if it doesnt appear in the destination server and the file creation date is more than 2 minutes?? But is it risky to do so..? Any comment or suggestion are mostly welcome.. -- View this message in context: http://www.nabble.com/rsync-delete-architecture-tp23594031p23594031.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Hi, I still couldn't find a really simple test case but at least I can provide some more details regarding this obscure issue: - The problem only occurs when there is more than 1 file system to sync. When I only backup /, everything is fine, when I add another source file system, nothing below /var will be deleted (I still don't have a clue, why this occurs on /var but not on other paths) - The issue has been around for quite a while: It has been introduced by some change between rsync 2.6.4 and 2.6.5 Regards, Peter Daum Peter Daum wrote: as I just discovered, in recent versions of rsync the --delete option seems to not work in certain cases. I couldn't come up with a simple scenario to reproduce the problem, so I'll have to tell the whole story: I use rsync to regularly backup a system onto other disks. The filesystems I want to save are mounted on /, /usr/ and /data, the target is below /backup (and has /backup/usr and /backup/data on separate file systems). the rsync command line looks like this: rsync [other options ...] -a --delete / /usr /data /backup/ For some reason, it seems like old data below /backup/var will never be deleted again I don't see any reason why /var would be treated somehow special, but I couldn't find another directory where this phenomenon also occurs; the other options don't include anything referring to /var. New data below /var will be synced, but when files below /var are deleted, they will stay on /backup/var. I don't know when this started (fortunately, I didn't need the backup data recently ;-), I just discovered it when the partition on /backup/ ran out of space. This happened with rsync 3.03; I also tried 3.05 with the same result. When I tried the same command line using rsync 2.6.3, the leftover data on /backup/var/ was finally deleted. However, I would rather use a recent version. Any idea what is going on? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Peter Daum wrote: - The problem only occurs when there is more than 1 file system to sync. When I only backup /, everything is fine, when I add another source file system, nothing below /var will be deleted (I still don't have a clue, why this occurs on /var but not on other paths) ... a little detail, that I forgot to explicitly mention: the other options used include of course --one-file-system; I need a full backup and find it very error-prone to specify all exceptions (like /proc /sys the directory where the backup disk is mounted ...) with --exclude -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Hello, Peter, Are there any errors while doing the rsync? I have recently discovered that to create identical copy on mac os x (in the presence of any errors), you need to additionally specify -- ignore-errors and --force-delete, otherwise nothing will be deleted (it seems that as soon as any error is encountered, delete is switched off). izidor On 22.3.2009, at 16:25, Peter Daum wrote: Hi, as I just discovered, in recent versions of rsync the --delete option seems to not work in certain cases. I couldn't come up with a simple scenario to reproduce the problem, so I'll have to tell the whole story: -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Hi, Izidor Jerebic wrote: Are there any errors while doing the rsync? I have recently discovered that to create identical copy on mac os x (in the presence of any errors), you need to additionally specify --ignore-errors and --force-delete, otherwise nothing will be deleted (it seems that as soon as any error is encountered, delete is switched off). There is no indication that anything might be wrong - until the last backup failed because the space was exhausted and I investigated, how this could happen, I was confident to have an identical copy of my data on the 2nd drive (A pretty unsettling thought ...) For some reason, rsync 3.x just does not seem to think it should delete those files ... Regards, Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete won't delete certain files
Hi, as I just discovered, in recent versions of rsync the --delete option seems to not work in certain cases. I couldn't come up with a simple scenario to reproduce the problem, so I'll have to tell the whole story: I use rsync to regularly backup a system onto other disks. The filesystems I want to save are mounted on /, /usr/ and /data, the target is below /backup (and has /backup/usr and /backup/data on separate file systems). the rsync command line looks like this: rsync [other options ...] -a --delete / /usr /data /backup/ For some reason, it seems like old data below /backup/var will never be deleted again I don't see any reason why /var would be treated somehow special, but I couldn't find another directory where this phenomenon also occurs; the other options don't include anything referring to /var. New data below /var will be synced, but when files below /var are deleted, they will stay on /backup/var. I don't know when this started (fortunately, I didn't need the backup data recently ;-), I just discovered it when the partition on /backup/ ran out of space. This happened with rsync 3.03; I also tried 3.05 with the same result. When I tried the same command line using rsync 2.6.3, the leftover data on /backup/var/ was finally deleted. However, I would rather use a recent version. Any idea what is going on? Regards, Peter Daum -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6041] rsync --delete from ext3 to hfs+ always deletes files with special characters in name.
https://bugzilla.samba.org/show_bug.cgi?id=6041 i...@retrosync.com changed: What|Removed |Added Status|NEW |RESOLVED Resolution||INVALID --- Comment #2 from i...@retrosync.com 2009-01-18 09:20 CST --- Man do I feel like an idiot. I searched all the bugzilla tickets extensively but never looked at the rsync faq. Sorry about this! -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6041] rsync --delete from ext3 to hfs+ always deletes files with special characters in name.
https://bugzilla.samba.org/show_bug.cgi?id=6041 --- Comment #1 from m...@mattmccutchen.net 2009-01-16 20:58 CST --- The FAQ discusses a problem copying to HFS+: http://rsync.samba.org/FAQ.html#2 Is this what you are seeing? -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6041] New: rsync --delete from ext3 to hfs+ always deletes files with special characters in name.
https://bugzilla.samba.org/show_bug.cgi?id=6041 Summary: rsync --delete from ext3 to hfs+ always deletes files with special characters in name. Product: rsync Version: 3.1.0 Platform: x86 OS/Version: Mac OS X Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: way...@samba.org ReportedBy: i...@retrosync.com QAContact: rsync...@samba.org I am using rsync to sync from my ubuntu machine (ext3) to my mac (HFS+ disk image). When I use the --delete flag, i get the following problem: deleting bart/Fotos/André.jpg bart/Fotos/André.jpg Any file containing a special character (éäëüê etcetera) gets deleted and then resynced. When I omit the --delete flag, rsync does everything normally. I tested with 2.6.9 and 3.0.5 and both have the exact same behavior. The files themselves are completely correct (no corruption or anything), so my guess is the delete checking has problems comparing the two filenames across the different filesystems and then falsely marks them as deleted. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5939] New: rsync: delete of stat xattr failed for ... (in backup): Operation not permitted (1)
https://bugzilla.samba.org/show_bug.cgi?id=5939 Summary: rsync: delete of stat xattr failed for ... (in backup): Operation not permitted (1) Product: rsync Version: 3.0.4 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] I've been having a problem rsync (3.0.3 and I also tried 3.0.5pre2). I'm rsyncing to an rsync daemon with fake super = yes in the rsyncd.conf file and I get many errors like this: This ONLY happens to symlinks. Most symlinks are fine. Most of these are either broken symlinks or symlinks that happen to have a different owner than what they link to. I thought this might be the root of the problem so I created a small test for new symlinks with these attributes but my test worked without a problem. Also one of these errors is for a valid symlink that is owned by the same owner as what it points to and I've been unable to do anything to fix the error. Is there something I can do to increase the logging for the daemon so that I can't figure out exactly what the problem is? And... yes the file system was mounted with user_xattr. fake super seems to work just fine for me with the exception of these errors. 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/jtest/public_html (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/IDrive (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/KDrive (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/test (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/daycare (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/hosts (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/house.example.com (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/public_html (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/.kde/cache-vex.example.com (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/.kde/socket-vex.example.com (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/.kde/tmp-vex.example.com (in backup): Operation not permitted (1) 2008/12/04 13:53:15 [1818] rsync: delete of stat xattr failed for home/testuser/amazon/startupscript.sh (in backup): Operation not permitted (1) 2008/12/04 13:53:18 [1818] rsync: delete of stat xattr failed for var/www/hosts/house.example.com (in backup): Operation not permitted (1) 2008/12/04 13:53:18 [1818] rsync: delete of stat xattr failed for var/www/hosts/mail.example.com (in backup): Operation not permitted (1) 2008/12/04 13:53:18 [1818] rsync: delete of stat xattr failed for var/www/hosts/example.com/docs (in backup): Operation not permitted (1) 2008/12/04 13:53:18 [1818] rsync: delete of stat xattr failed for var/www/hosts/house.example.com/docs/mandriva-one-2007-kde1.iso (in backup): Operation not permitted (1) 2008/12/04 13:53:18 [1818] rsync: delete of stat xattr failed for var/www/hosts/mail.example.com2/docs/images (in backup): Operation not permitted (1) 2008/12/04 13:53:18 [1818] rsync: delete of stat xattr failed for var/www/hosts/test2.example.com/docs/preschool (in backup): Operation not permitted (1) 2008/12/04 14:06:22 [1842] rsync: delete of stat xattr failed for home/testuser/IDrive (in backup): Operation not permitted (1) 2008/12/04 14:06:22 [1842] rsync: delete of stat xattr failed for home/testuser/KDrive (in backup): Operation not permitted (1) 2008/12/04 14:06:23 [1842] rsync: delete of stat xattr failed for home/testuser/amazon/startupscript.sh (in backup): Operation not permitted (1) 2008/12/04 14:06:26 [1842] rsync: delete of stat xattr failed for var/www/hosts/mail.example.com2/docs/images (in backup): Operation not permitted (1) 2008/12/04 14:10:14 [1847] rsync: delete of stat xattr failed for home/testuser/amazon/startupscript.sh (in backup): Operation not permitted (1) 2008/12/04 14:12:05 [1852] rsync: delete of stat xattr failed for home/testuser/amazon/startupscript.sh (in backup): Operation not permitted (1) 2008/12/04 15:43:36 [1891] rsync: delete of stat xattr failed for home/testuser/amazon/startupscript.sh (in backup): Operation not permitted (1) 2008/12/04 15:45:07 [1896] rsync: delete of stat xattr failed for home
DO NOT REPLY [Bug 5939] rsync: delete of stat xattr failed for ... (in backup): Operation not permitted (1)
https://bugzilla.samba.org/show_bug.cgi?id=5939 --- Comment #1 from [EMAIL PROTECTED] 2008-12-04 13:52 CST --- Well I've managed to increase verbosity a little bit and tried rsyncing only one of the symlinks that was having this problem: ... recv_file_name(home/testuser/amazon/startupscript.sh) received 1 names [generator] flist start=7, used=1, low=0, high=0 [generator] i=7 4 home/testuser/amazon/startupscript.sh mode=0120777 len=35 uid=1000 gid=1000 flags=0 recv_file_list done recv_generator(home/testuser/amazon,6) send_files(6, /home/testuser/amazon) recv_generator(home/testuser/amazon/startupscript.sh,7) rsync: delete of stat xattr failed for home/testuser/amazon/startupscript.sh (in backup): Operation not permitted (1) set modtime of home/testuser/amazon/startupscript.sh to (1228399898) Thu Dec 4 14:11:38 2008 set uid of home/testuser/amazon/startupscript.sh from 0 to 1000 set gid of home/testuser/amazon/startupscript.sh from 0 to 1000 home/testuser/amazon/startupscript.sh send_files(7, /home/testuser/amazon/startupscript.sh) touch_up_dirs: home/testuser (1) generate_files phase=1 send_files phase=1 -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete problem with accent characters on Mac OS X HFS
Hello, I would like to sync my Linux firewall to my Mac OS X. The Linux is a Debian 4.0 with ext3 filesystem (UTF-8) $ rsync --version rsync version 3.0.2 protocol version 30 The Mac OS X is a 10.5 Leopard with HFS+ (UTF-8) $ /sw/bin/rsync --version rsync version 3.0.3 protocol version 30 (fink version) I have many accents in my filenames: $ ls -l | tee filelist total 12 -rw-r- 1 tamas users 3 2008-08-19 14:19 árvíztűrő tükörfúrógép.rtf -rw-r- 1 tamas users 4 2008-08-19 14:19 élet.txt -rw-r- 1 tamas users 0 2008-08-20 00:00 filelist -rw-r- 1 tamas users 5 2008-08-19 14:19 körte.rtf $ file filelist filelist: UTF-8 Unicode text The first sync: $ /sw/bin/rsync --archive --update --verbose --delete firewall:teszt . receiving incremental file list teszt/ teszt/filelist teszt/körte.rtf teszt/árvíztűrő tükörfúrógép.rtf teszt/élet.txt sent 91 bytes received 601 bytes 461.33 bytes/sec total size is 263 speedup is 0.38 $ ls -l teszt/ | tee filelist-on-macosx total 32 -rw-r- 1 tamas staff3 Aug 19 14:19 árvíztűrő tükörfúrógép.rtf -rw-r- 1 tamas staff4 Aug 19 14:19 élet.txt -rw-r- 1 tamas staff 251 Aug 20 00:00 filelist -rw-r- 1 tamas staff5 Aug 19 14:19 körte.rtf urbanit-wlan:~/temp$ file filelist-on-macosx filelist-on-macosx: UTF-8 Unicode text The second sync: $ /sw/bin/rsync --archive --update --verbose --delete firewall:teszt . receiving incremental file list deleting teszt/körte.rtf deleting teszt/élet.txt deleting teszt/árvíztűrő tükörfúrógép.rtf teszt/körte.rtf teszt/árvíztűrő tükörfúrógép.rtf teszt/élet.txt sent 69 bytes received 304 bytes 248.67 bytes/sec total size is 263 speedup is 0.71 The rsync deletes my files with accents and download it again. (it doesn't delete the 'filelist' file) It's really annoying if you have many big files with accents. I try the same config with Mac OS X Tiger and Leopard with UFS (Unix Filesystem) and it works correctly. The main problem with the UFS is the 4 GB maximum file size limit, so I can't use UFS fs on my backup partition. It works over SMBFS protocol (aka local file sync) but Samba destroys my softlinks. I tried with ZFS (as a local fs) and it shows the same problem. I tried it over NFS and the problem remain. Please help me to solve this problem! Best regards, Tamas Komaromi from Hungary-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete problem with accent characters on Mac OS X HFS
On Wed, 2008-08-20 at 00:20 +0200, Komáromi Tamás wrote: I would like to sync my Linux firewall to my Mac OS X. The rsync deletes my files with accents and download it again. $ /sw/bin/rsync --archive --update --verbose --delete firewall:teszt . receiving incremental file list deleting teszt/körte.rtf deleting teszt/élet.txt deleting teszt/árvíztűrő tükörfúrógép.rtf teszt/körte.rtf teszt/árvíztűrő tükörfúrógép.rtf teszt/élet.txt This is probably a result of the Mac filesystem's infamous behavior of automatically decomposing UTF-8 characters in filenames, which is mentioned in the rsync FAQ: http://rsync.samba.org/FAQ.html#2 To fix it on a pull, pass --iconv=UTF8-MAC,UTF-8 . Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Sun, 2008-08-10 at 23:04 +0100, Tiago Marques wrote: It worked, --delete-excluded was all that was missing. I really think this should be the implied behavior and would like to see it as default, if there really isn't a good reason for it being this way. The current behavior is useful if files matching a given pattern simply should not be processed at all. E.g., one could use --exclude='*.o' to copy source files from one build tree to another without affecting any object files so that the destination can then do its own incremental rebuilding. I imagine that before the full --filter support was added in rsync 2.6.4, --exclude without --delete-excluded was the only way to protect destination files from deletion. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
I see, you're right :) Is there anything useful that can be donated to the rsync developers? Keep up the great work. Best regards, Tiago Marques On Sun, Aug 10, 2008 at 11:16 PM, Matt McCutchen [EMAIL PROTECTED] wrote: On Sun, 2008-08-10 at 23:04 +0100, Tiago Marques wrote: It worked, --delete-excluded was all that was missing. I really think this should be the implied behavior and would like to see it as default, if there really isn't a good reason for it being this way. The current behavior is useful if files matching a given pattern simply should not be processed at all. E.g., one could use --exclude='*.o' to copy source files from one build tree to another without affecting any object files so that the destination can then do its own incremental rebuilding. I imagine that before the full --filter support was added in rsync 2.6.4, --exclude without --delete-excluded was the only way to protect destination files from deletion. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete* Bug or feature-request
Hello, I'm trying to do a differential backup. Each commands/scripts are launch on backup server side. - rsync-ref: 1 script perform a complete backup to a directory ref. something like that: rsync --archives \ station:/path1/ /servbasepath/ref/servpath1/ - rsync-diff: 1 script perform a differential backup to a directory diff. something like that: rsync --archives \ --compare-dest=/servpath1/ \ --delete \ station:/path1/ /servbasepath/diff/servpath1/ See the following scenario: on serv: - rsync-ref is done Result: large number of files are synchronized. on station: - the user create a new file test on serv: - rsync-diff is done Result: the new file test was synchronized in diff directory. - do rsync-ref Result: the new file test was synchronized in ref directory. - do rsync-diff Result: nothing done, all file are synchronized. That is my problem: the file test in diff directory may/must be deleted, because it is already on the ref directory. I think --delete (or some other --delete-* options too) should remove the file because it is already found in ref directory. But Is it really a bug ? or may I request a --delete-if-found-in-compare-dir ? For workaround I run a script that find duplicated copies of files on the both directories. And run the rsync-diff after. But I work on more than 6 millions of files, the rsync and find-scrtipt took a very lot of time. Rsync already compare the both dir with --compare-dest then it will probably easily append duplicated files to the delete queue. What is your opinion ? Best Regards PS: Sorry for my not very well english. -- Gabriel CORRE [EMAIL PROTECTED] - Four J's Development Tools - www.4js.com signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete* Bug or feature-request
On Tue, 2008-07-29 at 17:19 +0200, Gabriel CORRE wrote: I'm trying to do a differential backup. Each commands/scripts are launch on backup server side. - rsync-ref: 1 script perform a complete backup to a directory ref. something like that: rsync --archives \ station:/path1/ /servbasepath/ref/servpath1/ - rsync-diff: 1 script perform a differential backup to a directory diff. something like that: rsync --archives \ --compare-dest=/servpath1/ \ --delete \ station:/path1/ /servbasepath/diff/servpath1/ See the following scenario: on serv: - rsync-ref is done Result: large number of files are synchronized. on station: - the user create a new file test on serv: - rsync-diff is done Result: the new file test was synchronized in diff directory. - do rsync-ref Result: the new file test was synchronized in ref directory. - do rsync-diff Result: nothing done, all file are synchronized. That is my problem: the file test in diff directory may/must be deleted, because it is already on the ref directory. I think --delete (or some other --delete-* options too) should remove the file because it is already found in ref directory. But Is it really a bug ? or may I request a --delete-if-found-in-compare-dir ? Rsync is behaving as expected. The man page description of --link-dest says, rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists); it would be good to add that the same is true of --compare-dest. You are welcome to request an option to make rsync delete destination files that are matched in --compare-dest dirs. I would actually suggest a more general option called --recheck-basis-dirs that also replaces a destination file matched in a --link-dest dir with a hard link of the basis file. (I don't think the analogous behavior for --copy-dest would be useful.) For workaround I run a script that find duplicated copies of files on the both directories. And run the rsync-diff after. But I work on more than 6 millions of files, the rsync and find-scrtipt took a very lot of time. Rsync already compare the both dir with --compare-dest then it will probably easily append duplicated files to the delete queue. Another approach would be to run rsync-diff to a new destination each time, but I imagine you want to reuse the data in the old destination instead of retransferring it all over the network. The command rsync --compare-dest=/servbasepath/ref/servpath1/ \ --copy-dest=/servbasepath/diff/servpath1.old/ \ station:/path1/ /servbaseepath/diff/servpath1/ would do that, except that rsync currently doesn't support mixing different types of --*-dest options on the same run; see: http://lists.samba.org/archive/rsync/2007-February/017210.html Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete* Bug or feature-request
On Tue, 2008-07-29 at 12:10 -0400, Matt McCutchen wrote: You are welcome to request an option to make rsync delete destination files that are matched in --compare-dest dirs. I would actually suggest a more general option called --recheck-basis-dirs that also replaces a destination file matched in a --link-dest dir with a hard link of the basis file. (I don't think the analogous behavior for --copy-dest would be useful.) I entered an enhancement request for this: https://bugzilla.samba.org/show_bug.cgi?id=5644 Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
Rsync 2.6.9 seems to think that the destination is already up to date. What do you expect it to do that it isn't doing? Update an existing file? (What are the size and mtime of that file and the corresponding source file?) Delete an extraneous file? (Note that you'll have to convert the exclude to a hide filter, --filter='H /*/', or use --delete-excluded if you want rsync to delete extraneous directories.) Matt On Sun, 2008-07-13 at 04:07 +0100, Tiago Marques wrote: opening tcp connection to rsync.sabayonlinux.org port 873 connection via http proxy proxy.ua.pt port 3128 opening connection using --server --sender -ltpr . SabayonLinux/ (Client) Protocol versions: remote=29, negotiated=29 note: iconv_open(UTF-8, UTF-8) succeeded. receiving file list ... recv_file_name(.) recv_file_name(Sabayon-Linux-x86_64-3.5-Pod.iso) recv_file_name(Sabayon-Linux-x86-3.5-Pod.iso.md5) recv_file_name(SabayonLinux-x86_64-3.4.miniEdition.iso.md5) recv_file_name(SabayonLinux-x86_64-3.4.miniEdition.iso) recv_file_name(Sabayon-Linux-x86-3.5.iso.md5) recv_file_name(Sabayon-Linux-x86_64-3.5.iso.md5) recv_file_name(Sabayon-Linux-x86-3.5-Pod.portage.db.tar.bz2) recv_file_name(Sabayon-Linux-x86-3.5.iso) recv_file_name(Sabayon-Linux-x86_64-3.5-Pod.portage.db.tar.bz2) recv_file_name(Sabayon-Linux-x86-3.5-Pod.iso) recv_file_name(SabayonLinux-x86-3.4.miniEdition.iso) recv_file_name(Sabayon-Linux-x86_64-3.5.iso) recv_file_name(Sabayon-Linux-x86_64-3.5-Pod.iso.md5) recv_file_name(SabayonLinux-x86-3.4.miniEdition.iso.md5) received 15 names done [receiver] i=0 0 ./ mode=040775 len=4096 flags=9 [receiver] i=1 1 Sabayon-Linux-x86-3.5-Pod.iso mode=0100644 len=727721984 flags=0 [receiver] i=2 1 Sabayon-Linux-x86-3.5-Pod.iso.md5 mode=0100644 len=64 flags=0 [receiver] i=3 1 Sabayon-Linux-x86-3.5-Pod.portage.db.tar.bz2 mode=0100644 len=27708897 flags=0 [receiver] i=4 1 Sabayon-Linux-x86-3.5.iso mode=0100644 len=4195756032 flags=0 [receiver] i=5 1 Sabayon-Linux-x86-3.5.iso.md5 mode=0100644 len=60 flags=0 [receiver] i=6 1 Sabayon-Linux-x86_64-3.5-Pod.iso mode=0100644 len=734363648 flags=0 [receiver] i=7 1 Sabayon-Linux-x86_64-3.5-Pod.iso.md5 mode=0100644 len=67 flags=0 [receiver] i=8 1 Sabayon-Linux-x86_64-3.5-Pod.portage.db.tar.bz2 mode=0100644 len=27708897 flags=0 [receiver] i=9 1 Sabayon-Linux-x86_64-3.5.iso mode=0100644 len=4418715648 flags=0 [receiver] i=10 1 Sabayon-Linux-x86_64-3.5.iso.md5 mode=0100644 len=63 flags=0 [receiver] i=11 1 SabayonLinux-x86-3.4.miniEdition.iso mode=0100644 len=731901952 flags=0 [receiver] i=12 1 SabayonLinux-x86-3.4.miniEdition.iso.md5 mode=0100644 len=71 flags=0 [receiver] i=13 1 SabayonLinux-x86_64-3.4.miniEdition.iso mode=0100644 len=728174592 flags=0 [receiver] i=14 1 SabayonLinux-x86_64-3.4.miniEdition.iso.md5 mode=0100644 len=74 flags=0 recv_file_list done get_local_name count=15 /storage/mirrors/sabayon recv_files(15) starting generator starting pid=16295 count=15 deleting in . delete_in_dir(.) [generator] protecting directory entropy because of pattern /*/ [generator] make_file(Sabayon-Linux-x86-3.5-Pod.iso.md5,*,2) [generator] make_file(Sabayon-Linux-x86-3.5-Pod.iso,*,2) [generator] make_file(Sabayon-Linux-x86_64-3.5-Pod.portage.db.tar.bz2,*,2) [generator] make_file(Sabayon-Linux-x86-3.5-Pod.portage.db.tar.bz2,*,2) [generator] make_file(Sabayon-Linux-x86_64-3.5-Pod.iso,*,2) [generator] make_file(Sabayon-Linux-x86_64-3.5-Pod.iso.md5,*,2) [generator] make_file(Sabayon-Linux-x86_64-3.5.iso,*,2) [generator] make_file(Sabayon-Linux-x86-3.5.iso,*,2) [generator] make_file(Sabayon-Linux-x86_64-3.5.iso.md5,*,2) [generator] make_file(SabayonLinux-x86_64-3.4.miniEdition.iso,*,2) [generator] make_file(SabayonLinux-x86_64-3.4.miniEdition.iso.md5,*,2) [generator] make_file(Sabayon-Linux-x86-3.5.iso.md5,*,2) [generator] make_file(SabayonLinux-x86-3.4.miniEdition.iso,*,2) [generator] make_file(SabayonLinux-x86-3.4.miniEdition.iso.md5,*,2) [generator] i=0 0 Sabayon-Linux-x86-3.5-Pod.iso mode=0100644 len=727721984 flags=0 [generator] i=1 0 Sabayon-Linux-x86-3.5-Pod.iso.md5 mode=0100644 len=64 flags=0 [generator] i=2 0 Sabayon-Linux-x86-3.5-Pod.portage.db.tar.bz2 mode=0100644 len=27708897 flags=0 [generator] i=3 0 Sabayon-Linux-x86-3.5.iso mode=0100644 len=4195756032 flags=0 [generator] i=4 0 Sabayon-Linux-x86-3.5.iso.md5 mode=0100644 len=60 flags=0 [generator] i=5 0 Sabayon-Linux-x86_64-3.5-Pod.iso mode=0100644 len=734363648 flags=0 [generator] i=6 0 Sabayon-Linux-x86_64-3.5-Pod.iso.md5 mode=0100644 len=67 flags=0 [generator] i=7 0 Sabayon-Linux-x86_64-3.5-Pod.portage.db.tar.bz2 mode=0100644 len=27708897 flags=0 [generator] i=8 0 Sabayon-Linux-x86_64-3.5.iso mode=0100644 len=4418715648 flags=0 [generator] i=9 0 Sabayon-Linux-x86_64-3.5.iso.md5 mode=0100644 len=63 flags=0 [generator] i=10 0 SabayonLinux-x86-3.4.miniEdition.iso mode=0100644 len=731901952 flags=0 [generator] i=11 0 SabayonLinux-x86-3.4.miniEdition.iso.md5
Re: rsync --delete isn't deleting files
On Sat, 2008-07-12 at 03:48 +0100, Tiago Marques wrote: Hmm, looks like I'm not done with this, yet. With rsync 3.02, it works fine: rsync --exclude='/*/' --delete-before -tvlpr rsync://rsync.sabayonlinux.org/SabayonLinux/ /storage/mirrors/ But not with the 2.6.9 version :| I just get this, with the mirror completely outdated: receiving file list ... done sent 92 bytes received 518 bytes 93.85 bytes/sec total size is 11592052049 speedup is 19003364.01 I ran your command with rsync 2.6.9 and -n (I didn't want to actually download the files) and didn't see any obvious problem. Please increase the verbosity to -vvv and post the output you get. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
I was already using the -d option, it download the subdirs also. The --exclude option worked fine though, and --delete-before is now working very well. Tks to all! Best regards, Tiago Marques On Fri, Jul 4, 2008 at 11:18 PM, Wayne Davison [EMAIL PROTECTED] wrote: On Fri, Jul 04, 2008 at 05:08:41PM -0400, Matt McCutchen wrote: Use --recursive and then exclude the stuff you don't want, e.g., --exclude='/*/' to exclude all subdirectories. You can also use -d (--dirs) instead of -r (--recursive). That makes rsync transfer just the contents of a specified dir/ (with trailing slash) without the contents of any subdirs. All modern rsyncs will allow --delete* to be used with -d w/o -r (though older rsyncs will ignore the --delete* option w/o -r). Matt's suggestion (the use of excludes) will work with all rsync versions. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Fri, Jul 11, 2008 at 04:22:04PM +0100, Tiago Marques wrote: I was already using the -d option, it download the subdirs also. You also need to not use -r (e.g. either expand -a and omit -r, or specify --no-r). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Fri, 2008-07-11 at 13:01 -0700, Wayne Davison wrote: On Fri, Jul 11, 2008 at 04:22:04PM +0100, Tiago Marques wrote: I was already using the -d option, it download the subdirs also. You also need to not use -r (e.g. either expand -a and omit -r, or specify --no-r). -d will create the subdirectories themselves regardless of -r, which I think Tiago doesn't want. That's why I suggested an exclude instead of -d. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
Indeed, that worked great for me! Big thanks to both for your time. Tiago Marques On 7/11/08, Matt McCutchen [EMAIL PROTECTED] wrote: On Fri, 2008-07-11 at 13:01 -0700, Wayne Davison wrote: On Fri, Jul 11, 2008 at 04:22:04PM +0100, Tiago Marques wrote: I was already using the -d option, it download the subdirs also. You also need to not use -r (e.g. either expand -a and omit -r, or specify --no-r). -d will create the subdirectories themselves regardless of -r, which I think Tiago doesn't want. That's why I suggested an exclude instead of -d. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
Hmm, looks like I'm not done with this, yet. With rsync 3.02, it works fine: rsync --exclude='/*/' --delete-before -tvlpr rsync://rsync.sabayonlinux.org/SabayonLinux/ /storage/mirrors/ But not with the 2.6.9 version :| I just get this, with the mirror completely outdated: receiving file list ... done sent 92 bytes received 518 bytes 93.85 bytes/sec total size is 11592052049 speedup is 19003364.01 I would rather like to keep this version, since it's the one that comes with the distro, if possible. Help! Best regards, Tiago Marques On 7/12/08, Tiago Marques [EMAIL PROTECTED] wrote: Indeed, that worked great for me! Big thanks to both for your time. Tiago Marques On 7/11/08, Matt McCutchen [EMAIL PROTECTED] wrote: On Fri, 2008-07-11 at 13:01 -0700, Wayne Davison wrote: On Fri, Jul 11, 2008 at 04:22:04PM +0100, Tiago Marques wrote: I was already using the -d option, it download the subdirs also. You also need to not use -r (e.g. either expand -a and omit -r, or specify --no-r). -d will create the subdirectories themselves regardless of -r, which I think Tiago doesn't want. That's why I suggested an exclude instead of -d. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete isn't deleting files
I had this working on an older version, since 2.6.9 that when I set --delete and --dry-run, I get no Deleting ** lines. Also, the --delete-before never worked for me. Working on a space constrained mirror, this has proven very handy, when it works. What's up with the new versions? I haven't changed a thing. Best regards, Tiago Marques -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Fri, 2008-07-04 at 16:13 +0100, Tiago Marques wrote: I had this working on an older version, since 2.6.9 that when I set --delete and --dry-run, I get no Deleting ** lines. Be sure to pass -v because, since rsync 2.6.7, --dry-run no longer implies it: http://gitweb.samba.org/?p=rsync.git;a=blob;f=OLDNEWS;h=af1e59e10eb012c4910237f6e62bfc99811d9a48;hb=HEAD#l972 If that isn't the problem, please post your rsync command line and the paths of the files that should be reported as deleting but aren't. Also, the --delete-before never worked for me. In what way did --delete-before not work? Please post your command line and any error messages you got. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
Inside the target dir, there is a file named asdasdasd which isn't deleted, nor mentioned to, since this is a dry-run. with: rsync --delete-before -v -tvlpr rsync://rsync.sabayonlinux.org/SabayonLinux/* ./ --dry-run i get: opening tcp connection to rsync.sabayonlinux.org port 873 sending daemon args: --server --sender -vvnltpre.L . SabayonLinux/* receiving file list ... done deleting in entropy delta-transmission enabled Sabayon-Linux-x86-3.5.iso Sabayon-Linux-x86-3.5.iso.md5 Sabayon-Linux-x86_64-3.5.iso Sabayon-Linux-x86_64-3.5.iso.md5 SabayonLinux-x86-3.4.miniEdition.iso SabayonLinux-x86-3.4.miniEdition.iso.md5 SabayonLinux-x86_64-3.4.miniEdition.iso SabayonLinux-x86_64-3.4.miniEdition.iso.md5 entropy/ entropy/standard/ entropy/standard/jenna/ entropy/standard/jenna/database/ entropy/standard/jenna/database/amd64/ entropy/standard/jenna/database/amd64/make.conf entropy/standard/jenna/database/amd64/package.keywords entropy/standard/jenna/database/amd64/package.mask entropy/standard/jenna/database/amd64/package.unmask entropy/standard/jenna/database/amd64/package.use entropy/standard/sabayonlinux.org/packages/x86/3.5/xwininfo-1.0.3.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/xwud-1.0.1.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/xwud-1.0.1.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/xxkb-1.10.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/xxkb-1.10.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/yacc-1.9.1-r3.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/yacc-1.9.1-r3.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/yasm-0.6.2.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/yasm-0.6.2.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/zd1211-firmware-1.4.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/zd1211-firmware-1.4.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/zopeinterface-3.0.1-r1.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/zopeinterface-3.0.1-r1.tbz2.md5 entropy/standard/sabayonlinux.org/packages/x86/3.5/zvbi-0.2.22.tbz2 entropy/standard/sabayonlinux.org/packages/x86/3.5/zvbi-0.2.22.tbz2.md5 sent 187147 bytes received 1564186 bytes 166793.62 bytes/sec total size is 50338148416 speedup is 28742.76 (DRY RUN) -- From my understanding, the --delete-before should delete the files that aren't present in the mirror anymore, before it starts to download, right? That has never happenned for me, when I began using Ubuntu 6.06, it's still the same as of now or, as I've said, worse. Due to the space constraints that I've already mentioned, I need to delete the old files before pulling the new ones. In the past, since --delete-before did nothing, I did a dry run and then parsed the output to have it rm the files that were to be deleted. That so also because the --delete needs the --recursive option, which for me is very impractical. For this Sabayon mirror, I only pretend to mirror the images in the root directory and needing the --recursive is not good for me. The packages would require me space that I cannot afford. Best regards, Tiago Marques On 7/4/08, Matt McCutchen [EMAIL PROTECTED] wrote: On Fri, 2008-07-04 at 16:13 +0100, Tiago Marques wrote: I had this working on an older version, since 2.6.9 that when I set --delete and --dry-run, I get no Deleting ** lines. Be sure to pass -v because, since rsync 2.6.7, --dry-run no longer implies it: http://gitweb.samba.org/?p=rsync.git;a=blob;f=OLDNEWS;h=af1e59e10eb012c4910237f6e62bfc99811d9a48;hb=HEAD#l972 If that isn't the problem, please post your rsync command line and the paths of the files that should be reported as deleting but aren't. Also, the --delete-before never worked for me. In what way did --delete-before not work? Please post your command line and any error messages you got. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Fri, Jul 04, 2008 at 08:11:36PM +0100, Tiago Marques wrote: rsync --delete-before -v -tvlpr rsync://rsync.sabayonlinux.org/SabayonLinux/* ./ --dry-run Check out the manpage on the --delete option. The very first paragraph tells you why you shouldn't be using a wildcard (*) there: http://rsync.samba.org/ftp/rsync/rsync.html ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Fri, 2008-07-04 at 20:11 +0100, Tiago Marques wrote: In the past, since --delete-before did nothing, I did a dry run and then parsed the output to have it rm the files that were to be deleted. That so also because the --delete needs the --recursive option, which for me is very impractical. For this Sabayon mirror, I only pretend to mirror the images in the root directory and needing the --recursive is not good for me. The packages would require me space that I cannot afford. Use --recursive and then exclude the stuff you don't want, e.g., --exclude='/*/' to exclude all subdirectories. You may wish to use hide filters or --delete-excluded. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete isn't deleting files
On Fri, Jul 04, 2008 at 05:08:41PM -0400, Matt McCutchen wrote: Use --recursive and then exclude the stuff you don't want, e.g., --exclude='/*/' to exclude all subdirectories. You can also use -d (--dirs) instead of -r (--recursive). That makes rsync transfer just the contents of a specified dir/ (with trailing slash) without the contents of any subdirs. All modern rsyncs will allow --delete* to be used with -d w/o -r (though older rsyncs will ignore the --delete* option w/o -r). Matt's suggestion (the use of excludes) will work with all rsync versions. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete (but not all files) question
Hi all, getting started with rsync. One of the best utilities I've used! My question is: I am syncing from my local web development directory (/var/www/mysite) to another drive shared by our dev team. The copy I am syncing to is a local subversion repository, and thus contains .svn directories. So I want to sync from the dev directory (which doesn't have .svn directories) to one that does. However, I want to use the --delete option for all files and directories *other* than these .svn directories. I want to keep the .svn directories as is, but otherwise sync the two directories exactly, erasing files that no longer appear in the source directory. I am also using an --exclude-from file to remove emacs (~) backup files and such things. I have something like: rsync -va --delete --exclude-from=$EXCLUDES_FILE --stats /var/www/mysite remoteDirectory I'm not sure how to use --delete and still keep certain files / dirs (those .svn dirs). Any help would be appreciate. Thanks, GMF. -- View this message in context: http://www.nabble.com/rsync---delete-%28but-not-all-files%29-question-tp17751838p17751838.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete (but not all files) question
On Tue, Jun 10, 2008 at 03:03:23AM -0700, GiveMF wrote: So I want to sync from the dev directory (which doesn't have .svn directories) to one that does. An exclude does two things: it prevents files that match from being transferred from the sender, and it protects files that match from being deleted on the receiver. You can also specify those two actions separately using a filter rule. In your case, the easiest thing to do is just to add - .svn/ into your excludes so that all .svn directories are excluded. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync du (was rsync delete)
Le dimanche 23 décembre 2007, Matt McCutchen a écrit : On Sat, 2007-12-22 at 18:47 -0800, Jesse Thompson wrote: Now I'm interested in a new possibility however. Using rsync (connecting to a remote rsync server via rsync protocol) is there a way to measure the size of a directory, kind of like du, without having to transfer it? Yes. Do a transfer of the directory in dry-run mode (so no data is actually copied) and pass --stats. Look at the Total file size statistic, which is the total size of all regular file data and symlink target paths: Tips, I manage a mirror (http://distrib-coffee.ipsl.jussieu.fr if you're intesrested) and when I want to mirror a new distribution but need to know the size before taking final decision (the common question: will I have the space for it) I simple run: rsync -avPHn rsync://server/share/ /tmp/a_non_existing_dir/. Which produce somethings like: [...] alpine/v1.7/usbdrive/alpine-1.7.9-i386.tar.gz.sha1 alpine/v1.7/usbdrive/syslinux.exe sent 36799 bytes received 437407 bytes 45162.48 bytes/sec total size is 23422648587 speedup is 49393.40 (DRY RUN) This simulate a real sync and give me size (about 23GB here). BTW: my test for this mail end by: rsync error: some files could not be transferred (code 23) at main.c(1535) [generator=3.0.0pre7] Yet another unclean mirror with stupid files permissions probably :\ I do think we should provide a good practice how to for people providing mirror via rsync. A lot of them sucks. signature.asc Description: This is a digitally signed message part. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync du (was rsync delete)
On Sat, 2007-12-22 at 18:47 -0800, Jesse Thompson wrote: Now I'm interested in a new possibility however. Using rsync (connecting to a remote rsync server via rsync protocol) is there a way to measure the size of a directory, kind of like du, without having to transfer it? Yes. Do a transfer of the directory in dry-run mode (so no data is actually copied) and pass --stats. Look at the Total file size statistic, which is the total size of all regular file data and symlink target paths: $ rsync -na --stats remote::volume/directory/ /nonexistent/ Number of files: 30 Number of files transferred: 12 == Total file size: 9148 bytes Total transferred file size: 9124 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 625 File list generation time: 0.003 seconds File list transfer time: 0.000 seconds Total bytes sent: 737 Total bytes received: 111 sent 737 bytes received 111 bytes 1696.00 bytes/sec total size is 9148 speedup is 10.79 (DRY RUN) If you like, you can account for inodes and directories (very approximately) by adding some multiple of the File list size. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete problems
Seams I have sorted out my problem until futher. By doing rsync over SSH to my NAS drive instead of mapping my NAS drive using smbmount, --delete works fine. /Torben For some time I have been using rsync to backup data between different machines and my NAS drive. But within the last 2 days I have found out that rsync is not removing old files from destination - directories renamed or removed from source. In this example I want to sync files on my Debian Linux server to a NAS drive. NAS drive is mounted using smbmount. Using -avhzn dry-run it indicate f3 and file3 to be removed. == Method SMBFS mount == smbmount diskstation /mnt/NAS.20071201.204045 -o username=root,password=xxx == rsync -avhzn --delete --progress /mnt/6/tst /mnt/NAS.20071201.204045/Archive building file list ... 4 files to consider tst/f3/file3*deleting tst/f3/*deleting tst/.d...p. tst/file.f...p. tst/f4/file3.f...p. sent 121 bytes received 38 bytes 318.00 bytes/sec total size is 7.48M speedup is 47059.14 == ls -l /mnt/NAS.20071201.204045/Archive total 0 drwx-- 1 root root 0 2007-12-01 19:27 tst == ls -lR /mnt/NAS.20071201.204045/Archive/tst /mnt/NAS.20071201.204045/Archive/tst: total 3660 drwx-- 1 root root 0 2007-12-01 19:06 f3 drwx-- 1 root root 0 2007-12-01 19:06 f4 -rwx-- 1 root root 3741202 2007-11-29 00:46 file /mnt/NAS.20071201.204045/Archive/tst/f3: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 /mnt/NAS.20071201.204045/Archive/tst/f4: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 But actual run does not remove any files on destination. == Method SMBFS mount == smbmount diskstation /mnt/NAS.20071201.204156 -o username=root,password=xxx == rsync -avhz --delete --progress /mnt/6/tst /mnt/NAS.20071201.204156/Archive building file list ... 4 files to consider tst/.d...p. tst/file.f...p. tst/f4/file3.f...p. sent 121 bytes received 38 bytes 318.00 bytes/sec total size is 7.48M speedup is 47059.14 == ls -l /mnt/NAS.20071201.204156/Archive total 0 drwx-- 1 root root 0 2007-12-01 19:27 tst == ls -lR /mnt/NAS.20071201.204156/Archive/tst /mnt/NAS.20071201.204156/Archive/tst: total 3660 drwx-- 1 root root 0 2007-12-01 19:06 f3 drwx-- 1 root root 0 2007-12-01 19:06 f4 -rwx-- 1 root root 3741202 2007-11-29 00:46 file /mnt/NAS.20071201.204156/Archive/tst/f3: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 /mnt/NAS.20071201.204156/Archive/tst/f4: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 If I use rsync to sync directories within my server everything works fine and old files are deleted without problems. rsync -avhz --delete --progress /mnt/6/tst/ /mnt/6/tstdest/ rsync version 2.6.9 protocol version 29 Have I found a bug, or what am I doing wrong??? Brgds Torben -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete problems
For some time I have been using rsync to backup data between different machines and my NAS drive. But within the last 2 days I have found out that rsync is not removing old files from destination - directories renamed or removed from source. In this example I want to sync files on my Debian Linux server to a NAS drive. NAS drive is mounted using smbmount. Using -avhzn dry-run it indicate f3 and file3 to be removed. == Method SMBFS mount == smbmount diskstation /mnt/NAS.20071201.204045 -o username=root,password=xxx == rsync -avhzn --delete --progress /mnt/6/tst /mnt/NAS.20071201.204045/Archive building file list ... 4 files to consider tst/f3/file3*deleting tst/f3/*deleting tst/.d...p. tst/file.f...p. tst/f4/file3.f...p. sent 121 bytes received 38 bytes 318.00 bytes/sec total size is 7.48M speedup is 47059.14 == ls -l /mnt/NAS.20071201.204045/Archive total 0 drwx-- 1 root root 0 2007-12-01 19:27 tst == ls -lR /mnt/NAS.20071201.204045/Archive/tst /mnt/NAS.20071201.204045/Archive/tst: total 3660 drwx-- 1 root root 0 2007-12-01 19:06 f3 drwx-- 1 root root 0 2007-12-01 19:06 f4 -rwx-- 1 root root 3741202 2007-11-29 00:46 file /mnt/NAS.20071201.204045/Archive/tst/f3: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 /mnt/NAS.20071201.204045/Archive/tst/f4: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 But actual run does not remove any files on destination. == Method SMBFS mount == smbmount diskstation /mnt/NAS.20071201.204156 -o username=root,password=xxx == rsync -avhz --delete --progress /mnt/6/tst /mnt/NAS.20071201.204156/Archive building file list ... 4 files to consider tst/.d...p. tst/file.f...p. tst/f4/file3.f...p. sent 121 bytes received 38 bytes 318.00 bytes/sec total size is 7.48M speedup is 47059.14 == ls -l /mnt/NAS.20071201.204156/Archive total 0 drwx-- 1 root root 0 2007-12-01 19:27 tst == ls -lR /mnt/NAS.20071201.204156/Archive/tst /mnt/NAS.20071201.204156/Archive/tst: total 3660 drwx-- 1 root root 0 2007-12-01 19:06 f3 drwx-- 1 root root 0 2007-12-01 19:06 f4 -rwx-- 1 root root 3741202 2007-11-29 00:46 file /mnt/NAS.20071201.204156/Archive/tst/f3: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 /mnt/NAS.20071201.204156/Archive/tst/f4: total 3660 -rwx-- 1 root root 3741202 2007-11-29 00:46 file3 If I use rsync to sync directories within my server everything works fine and old files are deleted without problems. rsync -avhz --delete --progress /mnt/6/tst/ /mnt/6/tstdest/ rsync version 2.6.9 protocol version 29 Have I found a bug, or what am I doing wrong??? Brgds Torben -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5091] Incremental-recursive, list-only rsync --delete tries to clean out working dir
https://bugzilla.samba.org/show_bug.cgi?id=5091 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Comment #1 from [EMAIL PROTECTED] 2007-11-21 10:16 CST --- This is now fixed in git. Thanks for the report! -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5091] New: Incremental-recursive, list-only rsync --delete tries to clean out working dir
https://bugzilla.samba.org/show_bug.cgi?id=5091 Summary: Incremental-recursive, list-only rsync --delete tries to clean out working dir Product: rsync Version: 3.0.0 Platform: x86 OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] In an empty directory, run: mkdir src rsync -r --delete src/ The result: drwx-- 48 2007/11/19 16:54:29 . rsync: delete_file: rmdir(src) failed: Read-only file system (30) rsync error: some files could not be transferred (code 23) at main.c(1059) [sender=3.0.0pre5] Rsync should not be trying to delete in list-only mode! Thank goodness for the check in syscall.c or I might have had some serious data loss. The error occurs only when incremental recursion is in use. The trouble is that the call to delete_in_dir at generator.c:2051 is not conditioned on !list_only . To prevent more mistakes like this, it might make sense to set dry_run to a large value in list-only mode. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync delete
Hello :) I am trying to build a backup system based upon Rsync, and I have a question about it's deleting features. I want to be able to use rsync to delete and remove a directory from the remote server, similar to rm -Rf directory The closest I seem to be able to get is the following progression: # rm -Rf directory (nuke local directory) # mkdir directory (recreat as empty directory) # rsync -mr --delete 'directory' remote::volume/directory ( woops, now remote::volume/directory continues to exist and contains an empty remote::volume/directory/directory :P ) # rsync -mr --delete 'directory' remote::volume (says no files to transfer, and nothing happens. @#$%) # touch directory/dummy_file # rsync -mr --delete 'directory' remote::volume/directory ( vroom! now the contents of remote::volume/directory are being cleaned out. unfortunately remote::volume/directory/dummy_file will remain for all eternity.. ) What would be the closest approximation to something like this? # rsync -r --just-nuke-the-target remote::volume/directory Is it something that can be done with rsync? Thanks! - - Jesse Thompson Webformix, Bend OR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync delete
On Sun, 2007-11-04 at 13:50 -0800, Jesse Thompson wrote:
What would be the closest approximation to something like this?
# rsync -r --just-nuke-the-target remote::volume/directory
Is it something that can be done with rsync?
Yes, but it is a bit awkward because rsync considers deletion of an
extraneous destination file to be part of the processing of its parent
directory. One simple approach is to push a directory to the parent of
directory (i.e., the module) and use filters to make the source
effectively empty and to prevent anything other than directory from
being deleted from the destination:
rsync -d --del --filter='R /directory' --filter='- /*' . remote::volume
Here is how you would write an rsyncrm script that can be invoked like
rsyncrm -ni remote::volume/directory. Note that this reveals the
attributes of the current directory to the server.
#!/bin/bash
victim=${@: -1:1}
rsync -d --del --filter=R /$(basename -- $victim) --filter='- /*' \
${@:1:$#-1} . $(dirname -- $victim)
Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete issue
Hi, I hope this is the right place to post this... I came across this old unsolved post that is describing the exact problem i am having, tried emailing the author but the email bounced. http://lists.samba.org/archive/rsync/2006-June/015697.html Basically files on the destination get deleted if I use any of the --delete options. As if it does not see that those file are indeed in the source. I haven't found any solution online or in the rsync manpage. If anyone knows what the problem could be then please let me know. Thanks -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete issue
On 8/25/07, Ameerov [EMAIL PROTECTED] wrote: Hi, I hope this is the right place to post this... I came across this old unsolved post that is describing the exact problem i am having, tried emailing the author but the email bounced. http://lists.samba.org/archive/rsync/2006-June/015697.html Basically files on the destination get deleted if I use any of the --delete options. As if it does not see that those file are indeed in the source. I haven't found any solution online or in the rsync manpage. If anyone knows what the problem could be then please let me know. Please re-run rsync at -vvv verbosity to get more information about what rsync thinks it is doing. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete-existing needed
On 5/5/07, Matthias Schniedermeyer [EMAIL PROTECTED] wrote:
If you have subdirectories to be killed:
find . -type d -empty -exec rmdir {}\;
You have to run that a few times (One time for each deepth-level), and don't
worry about the 'No such file or directory' warnings that it will spit for
every deleted directory.
Find just can't enter the directory it had deleted a nanosecond ago.
Using the -depth option makes find visit each directory after visiting
its contents, which solves both the warnings and the need to run find
multiple times.
Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete-existing needed
Hi, I made the mistake of doing an rsync of a directory into the wrong destination, so that the destination became a mix of two directories of unrelated files. Top unravel the mess, I could need something like a --delete-existing option. The semantics would be to delete at the destination ONLY those files that exists at the source. I tried to concoct a combination of options that would do this, but without success. I thought someone here might know the answer. Or would this be a handy feature to add? [One should of course use --dry-run and check the result before doing such an operation.] Erik _ Exercise your brain! Try Flexicon. http://games.msn.com/en/flexicon/default.htm?icid=flexicon_hmemailtaglineapril07 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete-existing needed
Erik Red wrote:
Hi,
I made the mistake of doing an rsync of a directory into the wrong
destination,
so that the destination became a mix of two directories of unrelated files.
Top unravel the mess, I could need something like a --delete-existing
option.
The semantics would be to delete at the destination ONLY those files
that exists at the
source.
I tried to concoct a combination of options that would do this, but
without success. I thought someone here might know the answer. Or would
this be a handy feature to add?
[One should of course use --dry-run and check the result before doing
such an operation.]
I would suggest a bit of good old shell magic here as it is, or at least should
be, some case of an one-time emergency.
Make a list of all the files on the source-side:
find . -type f -o -type l files.txt
or (If you have whitespaces)
find . -type f -o -type l | tr [\\n] [\\0] files0.txt
Then you copy over the list and and use it to kill all the files in it:
xargs rm files.txt
or (Whitespaces)
xargs -0 rm files0.txt
If you have subdirectories to be killed:
find . -type d -empty -exec rmdir {}\;
You have to run that a few times (One time for each deepth-level), and don't
worry about the 'No such file or directory' warnings that it will spit for
every deleted directory.
Find just can't enter the directory it had deleted a nanosecond ago.
Or, if you had empty directory on the target you don't want to be killed
find . -type d | sort -r dirs.txt
(Thorough in the tr from above if you have whitespaces)
And then
xargs rmdir dirs.txt
(Add the -0 if added the tr)
Bis denn
--
Real Programmers consider what you see is what you get to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a you asked for it, you got it text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync --delete is not deleting
I managed to solve this in the end. It was file permissions. I had originally copied all the data across by restoring from Backup tape and then started using Rsync to keep the two identical. After deleting the directories in question and rsyncing them from scratch, the problem has been solved. Thanks, Hedley Phillips Letterpart Ltd Email Notice --- This email and any files transmitted with it are confidential and may also be privileged. They are intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient please notify Letterpart Ltd immediately by return email or by calling 01737 223329. You should not copy this email or use it for any purpose nor disclose its contents to any other person. Although this email and any attachments are believed to be free of any virus, or any other defect, which might affect any computer or IT system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Letterpart Ltd for any loss or damage arising in any way from receipt or use thereof. ##Email Scanned by Trend Micro## Registered in England no 2607162 Registered Office as above End of Letterpart Ltd Email Notice - 12/4/2007 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete is not deleting
Hi, wondering if someone can help. I have googled for this and scoured the man page, but have got no further. I am running rsync 2.6.6 on windows and am finding it doesn't delete files from the target. For example my batch file has: rsync -av --delete /cygdrive/G/SAN_ARRAY/XPP_Data/alljobz/ /cygdrive/z/alljobz/ This copies everything over nicely, but doesn't delete files or folders from the target that have been deleted on the source. But, if I drill down the directory structure further: rsync -av --delete /cygdrive/G/SAN_ARRAY/XPP_Data/alljobz/test/alljobz/CLS_PaulTest/GRP_Jor dansFamily/ /cygdrive/z/alljobz/test/alljobz/CLS_PaulTest/GRP_JordansFamily/ Then rsync will delete sub folders from the GRP_JordansFamily/ directory as I would expect. Where am I going wrong and which blindingly obvious instruction have I missed. Thank you, Hedley Phillips -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete problem
On Mon, Jun 05, 2006 at 08:19:24AM -0500, Smemoe, Richard L. wrote: It looks like it's only copying the changes to the original filesystem since the last rsync. That's not very likely to be what is happening. I'd suggest using the -v option so that you can see what rsync is deleting -- this should show you if the problem is rsync deleting things that you weren't expecting, or rsync not copying things that you were expecting. Then, you can give us more information to help you figure out what is going wrong. Also, after upgrading to 2.6.8, I tried copying the whole filesystem at once with 'cd /backup/directory; rsync -a --delete-during /production/filesystem/ .', but this gave me nothing in my backup filesystem. Did rsync exit with an error or crash? Use one or more -v options to see what it is doing. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete problem
Hello, I need to create a local backup every night of a filesystem that has about 3.5 million files. Ive been using rsync and everything has been fine until I started to use the delete option to delete files on the backup that dont exist in the production data. When I added the delete option, my backup started taking about two days to finish. So I changed my backup script to rsync individual top-level directories instead of the whole filesystem at once. This finished must faster, except now rsync seems to be only copying files that have changed on the production data filesystem, and deleting files on my backup that havent changed. Here are the three different scripts that Ive tried and they all do the same thing: 1: cd /backup/directory rsync -au --delete /path/to/files/a* . rsync -au --delete /path/to/files/b* . rsync -au --delete /path/to/files/c* . 2: cd /backup/directory rsync -au --delete --include /path/to/files/a* --exclude /path/to/files/ /path/to/files/a* . rsync -au --delete --include /path/to/files/b* --exclude /path/to/files/ /path/to/files/b* . rsync -au --delete --include /path/to/files/c* --exclude /path/to/files/ /path/to/files/c* . 3: cd /backup/directory for file in /path/to/files/*; do rsync -au --delete $file .; done Does anyone know what Im doing wrong here? Thanks, Richard -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete problem
On Thu, Jun 01, 2006 at 09:31:49AM -0500, Smemoe, Richard L. wrote: When I added the -delete option, my backup started taking about two days to finish. You'd be better off using --del (--delete-during) than --delete, as it doesn't require a separate scan through the destination hierarchy. Also, older versions of rsync created an additional in-memory representation of the entire destination file tree to do the delete, so it is possible that such an older version caused rsync to start swapping, which could easily slow it down by an order of magnitude. So, make sure you're using a modern version (at least 2.6.4). As for any incorrect behavior of --delete, you'll need to be more explicit in what you think it is doing wrong. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
can rsync delete only one file in one command? is it a bug?
I want to call rsync from java webserver in Linux. situation as below: source directory: 001.doc 002.doc /D01/001.doc /D01/002.doc /D01/D11/001.doc /D02/001.doc destination directory: 001.doc 003.doc 004.doc /D01/001.doc /D01/003.doc /D01/004.doc /D01/D11/003.doc /D02/003.doc now, I want to only delete /003.doc when synchronization, no add, no update, This exclude works: rsync -avzu --delete --include="/003.doc" --exclude="*" /home/tomcat/webapps/filesync/synsrc2/ 172.16.100.19::hyhtest2 But, If I want to only delete /D01/003.doc when synchronization, no add, no update, this exclude NOT works: rsync -avzu --delete --include="/D01/003.doc" --exclude="*" /home/tomcat/webapps/filesync/synsrc2/ 172.16.100.19::hyhtest2 nothing was delete or add,update. why it NOT delte /D01/003.doc? is it a bug? how to delete any specify file in any directory and mean while no add or update? thank you. Best Regards - yhhuang 黄勇辉 Server Dept. TeleNav Shanghai Inc. Tel:(21)6337 2220 ext 8651 www.telenav.com www.telenav.cn -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
can rsync delete only one file in one command? is it a bug? 2.6.6
Rsync version 2.6.6 I want to call rsync from java webserver in Linux. situation as below: source directory: 001.doc 002.doc /D01/001.doc /D01/002.doc /D01/D11/001.doc /D02/001.doc destination directory: 001.doc 003.doc 004.doc /D01/001.doc /D01/003.doc /D01/004.doc /D01/D11/003.doc /D02/003.doc now, I want to only delete /003.doc when synchronization, no add, no update, This exclude works: rsync -avzu --delete --include="/003.doc" --exclude="*" /home/tomcat/webapps/filesync/synsrc2/ 172.16.100.19::hyhtest2 But, If I want to only delete /D01/003.doc when synchronization, no add, no update, this exclude NOT works: rsync -avzu --delete --include="/D01/003.doc" --exclude="*" /home/tomcat/webapps/filesync/synsrc2/ 172.16.100.19::hyhtest2 nothing was delete or add,update. why it NOT delte /D01/003.doc? is it a bug? how to delete any specify file in any directory and mean while no add or update? thank you. Best Regards - yhhuang 黄勇辉 Server Dept. TeleNav Shanghai Inc. Tel:(21)6337 2220 ext 8651 www.telenav.com www.telenav.cn -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: can rsync delete only one file in one command? is it a bug?
On Tue, Mar 28, 2006 at 09:39:07AM +0800, Huang, YongHui wrote: why it NOT delte /D01/003.doc? is it a bug? Nope. Your exclude affected directory D01, so the include has no chance to effect anything. Add a preceding --include=/D01/ and try again. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Using --files-from but also having rsync delete...
I am using rsync with --files-from to specify an exclusive list of files for rsync to back up to an rsync server. I give rsync the list using find to spit out a list of files that are 2 or more days old from a list of files that can change. The backup of the files works perfectly, however I realized that once the files are stored on the backup server, I have no way of informing rsync the next time around if there are files it should *delete* from the rsync server, as I want the rsync server to contain only the files folders that the source system has, and nothing more (ie, like you would using --delete). I thought I could use --include-from and then add --exclude='*' to see if rsync would backup what I needed and then delete anything not in my list from the rsync server side, however it did not, not to mention coming up quite short in the list of files it should have backed up (only backed up about 10% of the files I told it to). After messing with options for about 4-5 hours, I seem to have only 2 useable options: 1) wipe out all files from the rsync server prior to doing my backup to ensure the rsync server has only the current files, and no files that no longer exist on the source system 2) request a wishlist feature which allows rsync on the source system to tell the rsync server to delete all files NOT in the --files-from list. #2 is preferrable, and I was hoping something like that already existed, however I was unable to get anything to work for me that would do this. Does anyone have any suggestions for things maybe I haven't thought of? Failing that, is my wishlist item feasable? If so would someone here be willing to implement it for me (and how soon could that be done)? Eli. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using --files-from but also having rsync delete...
On Sun, Feb 27, 2005 at 04:31:15AM -0500, Eli wrote: I thought I could use --include-from and then add --exclude='*' to see if rsync would backup what I needed and then delete anything not in my list from the rsync server side, however it did not, not to mention coming up quite short in the list of files it should have backed up You probably neglected to add an --include='*/' to include all directories on the way down to the files. I.e.: rsync -av --include='*/' --include-from=files_from_file --exclude='*' \ --delete /src/ /dest/ That works as long as all the files in the files_from_file are relative to the same /src/ path. It also forces rsync to recurse through all the directories, which makes it run a little slower. Another potential solution is to generate the list of deleted files on the system as you generate the file-from list, and then use a command like this to delete them: ssh -l remote_user remote_host xargs rm /local/delete_list That assumes that you have remote-shell access. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Can rsync delete file without receive?
On Fri, Dec 17, 2004 at 05:00:04PM +0800, kevinwatt wrote: I want client could delete the files which are don't exist on sender. but without to receive the new files back. You could run rsync in --dry-run mode and pipe the output to a script that will notice all the deleting lines and remove the files that were mentioned. There is no way for rsync to do the deletions itself without also asking it to transfer files. If, however, you want rsync to update existing files, you could use the --existing option instead of --dry-run. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Can rsync delete file without receive?
Hello everyone, for some reason, I want client could delete the files which are don't exist on sender. but without to receive the new files back. how to do it? I have been try a few way. but just can't work currect. Sincerely, Yen dewei -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Bug in rsync? (--delete[-after])
Hello! I think I've found a bug in some (older) versions of rsync. The problem does not seem to have been reported at bugzilla. And of the 9 bugs found there by searching on --delete-after all have got the name [EMAIL PROTECTED] attached. I ask you therefore to check if this problem is found in the newest version of rsync as well! I've read the warnings at rsync.samba.org concering when and how to report bugs/postings to the rsync list. (It would have been a nice feature to be able to search the complete archive at http://lists.samba.org/archive/rsync/ for --delete-after) The problem is in short that rsync used with the option --delete-after does NOT delete older files rsynced, but --delete does. I choosed initially --delete-after to keep copies until all new files where copied so there should always be an online copy of the file. The versions I have used is, from [fra (no.)] hox /# rsync --version rsync version 2.5.5 protocol version 26 - and to [til (no.)]: hox /# ssh anvil rsync --version rsync version 2.6.2 protocol version 28 The newer command is on the machine hox: while read fra til; rsync -a -H -p -x --delete --exclude quota.user --exclude quota.group \ --rsync-path=/rsync $fra/ anvil:$til done /rsynke The old was the same, just with --delete-after and not --delete The datafile /rsynke: i# cat /rsynke /mn/hox/geofysikk /mn/anvil/geofysikk /mn/hox/u4 /mn/anvil/adm-u1 /mn/hox/u6 /mn/anvil/student-u2 /mn/hox/u1 /mn/anvil/ozon-u1 /mn/hox/u8 /mn/anvil/ozon-u2 /mn/hox/u11 /mn/anvil/dynmet-u3 /mn/hox/u10 /mn/anvil/ozon-u4 /mn/hox/student-u1 /mn/anvil/student-u1 /mn/hox/u5 /mn/anvil/hydro-u1 /mn/hox/u9 /mn/anvil/ozon-u3 /mn/hox/u7 /mn/anvil/dynmet-u2 /mn/hox/u2 /mn/anvil/osean-u1 /mn/hox/u3 /mn/anvil/dynmet-u1 /mn/hox/u12 /mn/anvil/slettes-u1 Here is a before and after status of the most troublesome disk; the first the from-disk and the last two the to-disk; of these the first by use of --delete-after and the last by use of --delete: hox /# df -h /mn/hox/u7 Filesystem SizeUsed Available Capacity Mounted enhalvtil#dynmet229G 23G 6003M80%/mn/hox/u7 hox /# ssh anvil df -h /mn/anvil/dynmet-u2 Filesystem Size Used Available Capacity tera3scsi#dynmet-u2 52G50G 2734M95%/mn/anvil/dynmet-u2 hox /# ssh anvil df -h /mn/anvil/dynmet-u2 FilesystemSize Used Available Capacity Mounted tera3scsi#dynmet-u252G 23G 30G44%/mn/anvil/dynmet-u2 As you can see --delte removed about 27G files on this filesystem alone. Regards, Kjell Andresen Systems administrator, University of Oslo, Norway Center for Information Technology Services and Department of Geosciences -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug in rsync? (--delete[-after])
Kjell Andresen wrote: The problem is in short that rsync used with the option --delete-after does NOT delete older files rsynced, but --delete does. The versions I have used is, from [fra (no.)] hox /# rsync --version rsync version 2.5.5 protocol version 26 - and to [til (no.)]: hox /# ssh anvil rsync --version rsync version 2.6.2 protocol version 28 Does using both parameters --delete and --delete-after solve the problem? rsync only implies --delete when using --delete-after if you have rsync-2.5.6 or above. Regards, Stefan -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Help on rsync delete option
I had a similar problem once. Try this instead:
rsync -avz --delete ${SOURCE_HOST}:/mlinkfam/extract/sccs/pa/.
/mlinkfam/extract/sccs/pa
The only change was * to ..
On Fri, 01 Nov 2002 11:36:56 -0500 Chaturvedi.Rajesh [EMAIL PROTECTED]
wrote:
Hello,
Any help on the following issue will be appreciated.
How do I delete files or directories in my first level of directory that
is
specified as host. Here is an example :
rsync -avz --delete ${SOURCE_HOST}:/mlinkfam/extract/sccs/pa/*
/mlinkfam/extract/sccs/pa
Giving above command I find redundant files or directory not getting
deleted
from the first level of target directory /mlinkfam/extract/sccs/pa.
However,
files or directories do get deleted from subdirectories
/mlinkfam/extract/sccs/pa/qa or /mlinkfam/extract/sccs/pa/field.
Thanx,
Raj
--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.tuxedo.org/~esr/faqs/smart-questions.html
Dan Ramaley
Digital Media Library Specialist
(515) 271-1934
Cowles Library 140, Drake University
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Help on rsync delete option
On Fri, Nov 01, 2002 at 11:36:56AM -0500, Chaturvedi.Rajesh wrote:
Hello,
Any help on the following issue will be appreciated.
How do I delete files or directories in my first level of directory that is
specified as host. Here is an example :
rsync -avz --delete ${SOURCE_HOST}:/mlinkfam/extract/sccs/pa/*
/mlinkfam/extract/sccs/pa
Giving above command I find redundant files or directory not getting deleted
from the first level of target directory /mlinkfam/extract/sccs/pa. However,
files or directories do get deleted from subdirectories
/mlinkfam/extract/sccs/pa/qa or /mlinkfam/extract/sccs/pa/field.
This isn't windows. Your shell is responsible for globbing.
To spell that out lets assume this the source host looks
something like so (i'm shortening the paths for readabilty) :
/sccs/pa
/sccs/pa/qa
/sccs/pa/qa/file1
.
. lots of other files and subdirs in pa/qa
.
/sccs/pa/field
.
. lots of other files and subdirs in pa/field
.
/sccs/pa/configdoodle
and the destination host has a file pa/oldanddeleted
/sccs/pa/oldanddeleted
in addition to the other stuff.
When you execute
rsync -avz --delete /sccs/pa/* ${DEST_HOST}:/sccs/pa
The shell will glob to produce
rsync -avz --delete /sccs/pa/qa /sccs/pa/field /sccs/pa/configdoodle
${DEST_HOST}:/sccs/pa
Which is the equivalent of
rsync -avz --delete /sccs/pa/qa ${DEST_HOST}:/sccs/pa
rsync -avz --delete /sccs/pa/field ${DEST_HOST}:/sccs/pa
rsync -avz --delete /sccs/pa/configdoodle ${DEST_HOST}:/sccs/pa
In order to make this more readable i changed this to a push
the behavior and shell stuff will still apply. The glob of
a remote source will be done by a shell on the remote host.
You haven't instructed rsync to operate on sccs/pa or on the
contents of sccs/pa but on a list of files and directories
that happen to be in sccs/pa. Also of note, no dot files in
sccs/pa will be synced because * doesn't match any filename
starting with dot.
rsync -avz --delete ${SOURCE_HOST}:/sccs/pa/ /sccs/pa
Will do what i think you want. All the contents of sccs/pa
will be synced.
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync: --delete fails with multiple source directories
On Mon, 22 Jul 2002, Edward Farrar wrote:
Rsync 2.5.5 is producing this error message and a core file when executing the
command /usr/local/bin/rsync -av --delete --force /net/OSCM/OS_ATLAS2/CONFIG/.
/net/OSCM/OS_TITAN1/2.6/CONFIG/. /OS/2.6/CONFIG
building file list ... done
rsync: connection unexpectedly closed (8 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(150)
I looked at the code in flist_find(), and I had the theory that the code
would fail if it found a duplicate name as the last item in the flist.
Sure enough, creating two directories with one duplicate between them
would crash in the same way if that duplicated item is the last one
alphabetically but would succeed otherwise. The problems stems from the
flist_up() function marching right off the top of the list if the last
item has its basename zeroed out (indicating it is a duplicate).
The easiest fix appears to be to simply trim the high value to ignore
removed items. Like so:
Index: flist.c
--- flist.c 11 Apr 2002 02:21:41 - 1.124
+++ flist.c 27 Jul 2002 17:40:10 -
-1151,7 +1151,9
{
int low = 0, high = flist-count - 1;
- if (flist-count = 0)
+ while (high = 0 !flist-files[high]-basename) high--;
+
+ if (high 0)
return -1;
while (low != high) {
I've tested it and it fixes my crashing testcase, so I'll commit this
to CVS.
..wayne..
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Rsync --delete does not work
Hi I just started to use rsync samba to backup some windows NT 98 shared directories to a linux machine linux: debian 2.2.4 rsync: 2.2.5 samba: 2.2.3a-6 I first mount the windows directory on the linux and then do a local rsync. ---command on linux umount /data/transfer mount -t smbfs -o username=user,password=passwd //ntserver/exp_dir /data/transfer cd /data/transfer rsync -a --delete * /data/exp_dir umount /data/transfer --- this works fine, except for deleted files. Files that got deleted on the windows remain on the linux side, even after repeated rsync's This is what the --delete option is supposed to do. I've tested this several times. the samba mount show the correct state, i.e. without the deleted files. Rsync does not notice they're gone. I've seen/read a lot of posts in the google archives concerning this --delete option. Most of them came from this mailing list. They all seem to suggest it really should work. So, what did I do wrong? best regards glenn -- __ Sign-up for your own FREE Personalized E-mail at Mail.com http://www.mail.com/?sr=signup Save up to $160 by signing up for NetZero Platinum Internet service. http://www.netzero.net/?refcd=N2P0602NEP8 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Rsync --delete does not work
Try adding -v to your rsync command to get more verbose output, perhaps giving you an idea why --delete is not working as expected. --delete works for me, but there are situations where rsync will ignore --delete for safety (such as if certain errors occur). Also, remember that * is expanded by your shell, before rsync sees the command line. So if your /data/transfer contained files a, b, c, and d, and you deleted c, rsync would see your command line as: rsync -a --delete a b d /data/exp_dir In other words, you've told rsync not to sync c, so it would have no reason to delete it at the destination directory. Try using . instead of * to get more reliable results. Doug -Original Message- From: g dm [mailto:[EMAIL PROTECTED]] Sent: Tuesday, July 23, 2002 11:55 AM To: [EMAIL PROTECTED] Subject: Rsync --delete does not work Hi I just started to use rsync samba to backup some windows NT 98 shared directories to a linux machine linux: debian 2.2.4 rsync: 2.2.5 samba: 2.2.3a-6 I first mount the windows directory on the linux and then do a local rsync. ---command on linux umount /data/transfer mount -t smbfs -o username=user,password=passwd //ntserver/exp_dir /data/transfer cd /data/transfer rsync -a --delete * /data/exp_dir umount /data/transfer --- this works fine, except for deleted files. Files that got deleted on the windows remain on the linux side, even after repeated rsync's This is what the --delete option is supposed to do. I've tested this several times. the samba mount show the correct state, i.e. without the deleted files. Rsync does not notice they're gone. I've seen/read a lot of posts in the google archives concerning this --delete option. Most of them came from this mailing list. They all seem to suggest it really should work. So, what did I do wrong? best regards glenn -- __ Sign-up for your own FREE Personalized E-mail at Mail.com http://www.mail.com/?sr=signup Save up to $160 by signing up for NetZero Platinum Internet service. http://www.netzero.net/?refcd=N2P0602NEP8 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
