Re: include/exclude: I miss the point
On 10-09-10 21:13, Hans Troost wrote:
So my exclude-files contains:
.*
the include file contains:
.kde4/share/apps/kabc/distlists
.kde4/share/apps/kabc/std.vcf
Result:
1. Success: 2 .kde4/share/apps/kabc-files
Are you sure about that part ? '.*' ('H, .*' as seen by sending rsync)
should short-circuit any .kde/ directory subtree and exclude it from
file list/transfer. You would need something more elaborate to get just
those two fles from .kde4 and nothing else.
Anyway, putting that detail aside and if I'm reading things right.
Your rules (combined) look like:
+, .mozilla/
+, .mozilla/firefox/
+, .mozilla/firefox/plp6wzjm.default/
+, .mozilla/firefox/plp6wzjm.default/bookmarkbackups/*
+, .mybackup
H, .*
H, .mybackup/backup.log.*
H, .mozilla/extensions
H, *.*~
H, Desktop
H, Downloads
H, Movies
H, Music
H, Picturestmp
H, Videos
+, * (implicit final rule)
H - because you added --delete-excluded, so receiving side will not
protect them from deletion.
For receiving rsync that means just:
R, * (as consider everything not part of the transfer a candidate to
delete)
For sending rsync:
S, .mozilla/
S, .mozilla/firefox/
S, .mozilla/firefox/plp6wzjm.default/
S, .mozilla/firefox/plp6wzjm.default/bookmarkbackups/*
S, .mybackup
H, .* - this makes following 2 rules redundant
H, .mybackup/backup.log.*
H, .mozilla/extensions
H, *.*~
H, Desktop
H, Downloads
H, Movies
H, Music
H, Picturestmp
H, Videos
S, * (implicit final rule)
Which does exactly what you experienced:
3. Failure: I have all other files, sub-dirs and their content as well in
.mozilla/firefox/plp6wzjm.default.
They were all caught by the final implicit include, and none of them
matched your H (exclude) rules.
4. when adding
.mozilla/firefox/plp6wzjm.default/Cache
.mozilla/firefox/plp6wzjm.default/chrome
.mozilla/firefox/plp6wzjm.default/extensions
.mozilla/firefox/plp6wzjm.default/firebug
.mozilla/firefox/plp6wzjm.default/OfflineCache
to the exclude file this folders are not in, but the normal
.mozilla/firefox/plp6wzjm.default/ files are.
Being part of exclude rules, they will be catched and excluded from
transfer, but the rest will not.
Note, that nothing from .mybackup that matches H (exclude) rules will be
backed up.
Question: it seems that I miss the point how to do this. can you please
give a hint about what I ḿ doing wrong?
rsync command )some date-time variable for files, but that is working):
rsync -axhhX --no-perms --omit-dir-times\
--backup --backup-dir=$NiceDate/updated --suffix=.$Now\
--delete --delete-excluded\
--backup-dir-dels=$NiceDate/deleted --suffix-dels=.`date +%H%M`\
--log-file=/home/hans/.mybackup/backup.log.$RawDate-$Now\
--include-from=/home/hans/.mybackup/include.txt\
--exclude-from=/home/hans/.mybackup/exclude.txt\
/home/hans /mnt/backup
===
2nd, similar question:
to play around with this stuff I wanted to test with the hidden files only
(.*), so I tried to add all non-hidden directories and normal files to this
exclude-test.txt, but did not get it working (it worked too much):
exclude-test.txt
.*
.mybackup/backup.log.*
[a-z]*
[A-Z]*
while my include-test is:
.kde4/share/apps/kabc/distlists
.kde4/share/apps/kabc/std.vcf
.mozilla/
.mozilla/firefox/
.mozilla/firefox/plp6wzjm.default/
.mozilla/firefox/plp6wzjm.default/bookmarkbackups/*
.mybackup
In this case I don't get any files backed up (dry-run, so checked in log-file
only).
Hmm, you should get
.mozilla/firefox/plp6wzjm.default/bookmarkbackups/*
.mybackup/
backed up. Otherwise the rest will be hidden from transfer (to be
precise, any file/directory starting with a letter or .).
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-during acts like --delete-before
On 09.09.2010 02:05, Michal Soltys wrote: On 10-09-09 01:07, Craig Bell wrote: 1) 1st delete pass, honoring global (on B or sent from A, if rsync was executed on A) and per directory (on B) rules of what to protect/delete (see --filter's options R P) 2) 2nd transfer pass To be a bit more specific - initially what to delete is of course based on what would be transferred / updated. Only elements that are not part of the file list are candidates to remove. Rsync won't do a thing such as delete everything before transfer, then retransfer deleted files that wouldn't be touched otherwise. Basically - 0th stage is generate file list, which is sent to the receiver before any further action takes place. If you increase verbosity, it will be visible at the begining in the output. It looks like --delete-during should operate in-fix, i.e. step into a directory, delete the old file, pull the new file, and then move on to the next directory. For my transfers, all deletes seem to take place up front. Hmmm, rsync wouldn't do that. As mentioned above, files being part of the transfer (whenever uptodate ones, or the ones to be updated) will not be preemptivly removed. Even the old files are not deleted first - they are transfered to a temporary file and moved over (unless --inplace is being used). -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-during acts like --delete-before
On 10-09-09 20:17, Craig Bell wrote:
Michal Soltys wrote:
As stated, I would prefer this, however I do not have enough space to keep
two editions of the files.
I must delete as I go along, however I don't want to delete everything up
front, and risk exposure.
My workaround was to run rsync once for each subdir, which worked but is far
from optimal.
Well, the stages - file list transfer / delete scan / actual
file transfer are not absolutely strict in context of dir-by-dir boundary.
Try for example something analogous to:
for i in `seq 1 1 10` ; do mkdir -p {src,dst}/$i ; for j in `seq 1 1 10` ; do
mkdir -p {src,dst}/$i/$j ; for k in `seq 1 1 10` ; do touch src/$i/$j/file-$k
; touch dst/$i/$j/filea-$k ; done ; done ; done
and do rsync -a --delete-during -i localhost::del dst/ | tee log
In the log file, you should see interleaved delete and transfer
groups-of-sorts.
You shouldn't be risking much, and even if transfer is interrupted, rsync
invocation with a few other options I mentioned should quickly fix things.
On a related note, if you call rsync locally, you will see strict
boundaries though, e.g. with:
and do rsync -a --delete-during -i src/ dst/ | tee log
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-during acts like --delete-before
On 10-09-09 01:07, Craig Bell wrote: Hello, I use rsync v3.0.7 with --delete-during, but it appears to act more like --delete-before. I've checked the docs and list archives, but I am still discovering some of the subtleties of rsync, and I can't see where I'm going wrong. There's an example of my issue below... can anybody set me straight? =-) Functionally yes. In more details, if you for example do rsync A - B with --delete-before, rsync will do: 1) 1st delete pass, honoring global (on B or sent from A, if rsync was executed on A) and per directory (on B) rules of what to protect/delete (see --filter's options R P) 2) 2nd transfer pass If you used --delete-after - the order of the above would be reversed - first transfer (which would have potentially useful effect of first transferring [updated] per directory rules from A - B, and then using them for deletion during 2nd pass (delete). If you used --delete-during - rsync would combine 1st and 2nd pass of --delete-before into single pass, where deletion and transfer would be done on directory per directory basis, recursively. --delete-delay combined with --delete during would alter that combined pass in the following way - on a dir-by-dir basis: 1) find what to delete 2) transfer files 3) delete This is almost like --delete-after, but it won't work if you expect updated per directory rules to be used for delete pass. It looks like --delete-during should operate in-fix, i.e. step into a directory, delete the old file, pull the new file, and then move on to the next directory. For my transfers, all deletes seem to take place up front. The destination directory is emptied at the beginning of the session, which is what I want to avoid in case the link is severed, and the transfer is interrupted. --delete-delay is probably what you're after. Also check options: --ignore-existing, --ignore-non-exisitng, --partial, --delay-updates -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backups Directory Timestamps Not Preserved
Backup options are concered with files from what I can see after some tests. Extra directory (--backup-dir) is an addition to keep things more tidy. I guess the idea was to have some sort of protection against careless rsync invocation, etc. - not as a solution for incremental backups. I don't really know for sure if it's intended or not though. As for incremental backup, --link-dest is generally used for that purpose, and saves a lot of space due to hardlink usage. You will likely need to make some shell script for automation's sake (or you can look for premade ones - e.g. rsnapshot is often quoted). -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Local disk rsync
listserv.traf...@sloop.net wrote: I've done quite a bit of looking, but I haven't found an answer that answers this question. Environment: cygwin on Windows rsync 3.0.4 Using a native windows sync tool, a sync of the source to the destination takes perhaps 5 minutes. (~24G, 80K files) However using rsync takes a very long time. (I've never let it finish, but it's still running an hour later or more...) That's weird. I routinely use rsync to backup all my harddrives under win64 (using cwrsync) and I haven't noticed any excessive times (at least not beyond common sense). Although my config is much more complicated, the core approach is the same. I never felt any real reason to compare to cp or other regular programs (drive example: 0.5TB, 137k files, although only ~100GB is copied due to hide/show/etc. rules). I'll do some comparisons later, when I have a bit of time. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: best rsync for use with windows (client) and long file names
Fabian Cenedese wrote: I also use cwrsync but the few cases where the path was too long I just shortened the source. Of course that may not be possible for you. bye Fabi Remember about cygwin1.dll with utf8 support (using unicode win32 api). Side effect of it is that the paths are limited to 32767 iirc. Either way - it works fine for me, and I have some very long weird japanese file names here and there. Check: http://www.okisoft.co.jp/esc/utf8-cygwin/ for details. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backup to spare drive (rsync / crontab)
drc...@yahoo.com wrote: But the scheduled backups never happen. I must be doing something wrong, but can't figure out what. All help would be gratefully appreciated. Not sure if this is an rsync issue or a FreeBSD / Crontab issue. Cron might be ignoring $PATH, so the scripts called don't know where rsync is. Just a guess. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Some help with sudo on remote side of rsync
Carney Mimms wrote: I am now able to ssh from remote to local and local to remote without passwords using RSA keys, but if I run the script without sudo, i.e. Sudo /usr/local/bin/rsync ... It can’t copy files that need root permissions. If I leave out the sudo and run the script after becoming root (sudo su), I get prompted for a password again. When you become root, ssh will try to read keys from the home directory proper for a new uid, regardless of the $HOME setting. The same happens when you specify sudo rsync remotely. If you setup the keys properly for root and allow it to login using ssh (eg. PermitRootLogin without-password), it should work fine. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest; I'm missing the boat
Harry Putnam wrote: I must be seriously misunderstanding the man page coverage of --compre-dest. My take was that if a file in compare-dest=dir matches a file in SOURCE/ then it won't be transferred to DEST/. I tried this test. (d1 has single files and 2 subdir with files) cp -a d1 d1a mkdir d2 rsync -avv --compare-dest=./d1a d1/ d2/ d1a is carbon copy of d1 but still every last file in d1 is copied to d2. If you specify DIR as a relative link, it will be taken as relative to destination dir. That's probably the culprit here. mkdir d2 cp -a d1 d2/d1a rsync -avv --compare-dest=./d1a d1/ d2/ ...should do the thing. Or just specify DIR as absolute. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird interaction between --delete-delay and --partial-dir when transfer is interrupted
Either way - as a workaround, you could drop delete-excluded, and
then just finalize the thing with something like: find /bar/ -depth
-name .rsync-partial -exec rm -rf {} \;
I meant more like: -name .rsync-partial -prune -exec .
Shivkumar Venkatasubrahmanyam wrote:
Makes sense. And this does seem like a corner case :) I did some more
testing along the lines you suggested ...
mkdir src ; touch src/a
mkdir dst ; mkdir dst/.rp
rsync -a - --delete-excluded --delete-delay --delay-updates
--partial-dir=.rp src/ dest/ log 21
It seems this bug/corner case requires all the following conditions:
(1) [...]
I've straced it a bit - it looks like the flow is as follows:
1) with delete-excluded + delay-updates - .rp is deleted before
transfers (delete-excluded implies --delete-during, which in turn
behaves like per-dir --delete-before), then recreated when there's a
need to rename temporary file to that dir `rename(.a.7e7C6w, .rp/a`
then .rp is deleted as it's no longer needed, /unless/ it has something
left in it.
2) with delete-excluded + delete-delay - due to lack of delay-updates,
.rp is not needed by rsync at all, /unless/ the transfer is interrupted
(then .rp will be created, and parital file moved there). Deletion
happens only once, at the end, due to delete-delay.
3) with all the 3 above - situation similar to 1) happens, but at the
very end, and there's funny conflict - .rp will be deleted due to not
being needed anymore, but then delete-delay will take the action. First
trying to delete .rp's contents, but ... the dir is not there anymore,
thus ENOENT (due to opendir failing) and rsync's exit code 23.
I use all three of these options: --delete-excluded because I have added
--filter options over time and I want the destination to reflect these
newly excluded files,
Any particular reason for that ? Sender/receiver rules can achieve the
same, with much nicer flexibility. What delete-exclude does, is just
turn exclude into sender-exclude globally. You can do that explicitly, e.g.
H, something
-,s something
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird interaction between --delete-delay and --partial-dir when transfer is interrupted
Shivkumar Venkatasubrahmanyam wrote: ... then without -f 'R .rp/', dst/.rp is not removed. With -f 'R .rp/', it is removed but if dst/a is updated then we have the same issue (exit with code 23). This is entirely consistent with your strace i.e. code 23 whenever there are two separate attempts to remove .rp/. Well, contents of nonexisitng dir, to be precise :) You can change --delete-delay into --delete-after, and it will work without problems then. Delete pass will be done separately after file transfers, and checking what files to delete will happen after .rp is already gone. Thanks for all your help :) Shiv Np. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird interaction between --delete-delay and --partial-dir when transfer is interrupted
Shivkumar Venkatasubrahmanyam wrote:
Hi,
I'm not sure if this is a bug, but after reading the manual, this does
not seem like expected behavior. I'm using the following rsync command
to approximate an atomic update (I can't use --link-dest as hard links
in hfsplus filesystems are fubar under linux as of 2.6.27):
rsync -a --delete-excluded --backup --backup-dir=../bar.archive
--backup-suffix=.2008-12-15-0115 --delete-delay --delay-updates
--partial-dir=.rsync-partial --verbose --human-readable
--log-file=/bar.rsync-log host:/foo/ /bar
If a transfer is interrupted, .rsync-partial/ directories are left over
in the backup directory, as expected. The next rsync run exits with
non-zero exit status even though all files are transferred (verified
this using md5sum). A third rsync run transfers nothing (going by the
log file) but exits with success. I'm guessing this is because: (1)
--delete-excluded causes the .rsync-partial/ directories to be marked
for deletion during file list generation time of the second rsync run
(there are no .rsync-partial/ directories in host:/foo/), (2) successful
transfer of all files during the second run results in rsync removing
the .rsync-partial/ directories as they become empty, and (3)
--delete-delay which I understand is necessary to allow rsync to use the
files in .rsync-partial/ to speed up the second transfer, fails to find
the .rsync-partial/ directories at the end of the transfer and
complains. The third transfer sees no .rsync-partial/ directories in
/bar and so returns success.
I would like the second run to return success if all files are
transferred. Is there some option I'm supposed to use (e.g. --filter)
or is this a bug?
Shiv
What is the exact exit value after the 2nd run - 23 perhaps ?
You can do a simple test:
touch /test/src/a
mkdir /test/dst/.rp
rsync -a - --delete-excluded --delay-updates --delete-delay
--partial-dir=.rp /test/src/ /test/dst/ | less
In this case, rsync will exit with value 23. It looks like rsync is
deleting .rp before any file transfer starts.
If you put any bogus file in .rp before rsync call, it will exit with 0
though. Looks like it prevents it from a deletion.
If you remove delete-excluded, it will work as expected with or w/o
bogus file (with bogus file, .rp dir will not get deleted, otherwise it
will), exit value is 0 in both cases.
It does look like sort of a tiny bug. Or maybe there's something subtle
we're missing.
Either way - as a workaround, you could drop delete-excluded, and then
just finalize the thing with something like: find /bar/ -depth -name
.rsync-partial -exec rm -rf {} \;
Rsync will usually clean up after itself - only bogus files (not being
part of the transfer) would prevent it from doing so, as far as I can see.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: character set for Windows?
Steve Lefevre wrote: I'm using cygwin to backup files from a windows machine to a unix host with rsync. I originally uploaded the bulk of the files using a windows ftp client, and then planned on doing updates nightly with rsync. This can also be important, especially regarding how the ftp server is configured - as it might have converted some of the filenames itself to the locale used on your unix - for example extended vsftpd builds have full iconv functionality themselves. From win32's side, and if you don't use utf8 modified cygwin1.dll ( http://www.okisoft.co.jp/esc/utf8-cygwin/ ), filenames will be converted according to the reginal and language options - advanced setting whenever non-unicode win32 api is used, as far as I know - so that would include standard cygwin and cli ftp client on windows. Modified cygwin1.dll certainly simplifies life (I've been using it for quite a while), as everything coming from win32 will be in utf8, and coming to win32 should be in utf8. Dll will take the care of the conversion to native unicode used on win32, and legacy regional options will be irrelevant then. I was looking for what switch to use to tell rsync to use a windows character set, and it seems like I need to use is --iconv, but i'm unsure what character set windows uses ( this is xp ), or in other words, what the argument should be. If you decide to use mentioned cygwin1.dll , and your linux operates in utf8 by default, you shouldn't need anything else. If you use some legacy locale on unix - --iconv=.,utf8 should do the thing - assuming win32 is remote machine, and locale is set properly on unix machine. Otherwise, it's up to you and --iconv / --no-iconv . The latter should work perfectly fine as well (ignoring filename adjustments completely - unix shouldn't have any problems with storing any filenames), although locale-aware programs (ls, etc.) might display them with a bit of weirdness. Even simple tab-completion (under e.g. bash) will depend on locale and inputrc settings as well. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync command exclude option
Kaushal Shriyan wrote: Hi again I have a directory logfiles and inside it it has lots of sub directories. Inside it there are lots of .gz, .txt and .sql files I want only .sql files to be rsynced to the destination host is there a way to achieve it using rsync utility Assuming you won't be deleting on the receiving side: file .rf: + *.sql + */ - * rsync -avm --filter='., .rf' source destination This is actually identical example to what you can find in rsync's manual at the description of -m option. Delete variant would probably be of use for you as well. And - depending on specific needs - there're other possibilities. ps. Please take a look at http://www.samba.org/ftp/rsync/rsync.html whenever you have time. Rsync grants /a lot/ of flexibility, but can be pretty subtle at times as well. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync command exclude option
Kaushal Shriyan wrote: I want only .sql files to be rsynced to the destination host. when i do rsync -avz oldUnicelFiles hostB:/home/kaushal --include-from=include.txt it rsync all the files under oldUnicelFiles, I mean even .gz and .txt files too. ~/oldUnicelFiles $ ls 01-07-2008_LOG.txt.gz 01-08-2008_LOG.txt 02-09-2008_LOG.txt.sql ~/oldUnicelFiles $ cat include.txt # /oldUnicelFiles/*.sql # am i missing something Include is rsync's default action, if no explicit pattern that would override it is specified. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync command exclude option
Kaushal Shriyan wrote: Hi I do rsync -avz oldUnicelFiles hostB:/home/kaushal --exclude-from=exclude.txt but it rsynced the excluded files too my exclude.txt file contains /home/kaushal/oldUnicelFiles/*.gz /home/kaushal/oldUnicelFiles/*.txt Any ideas as what am i doing wrong From rsync perspective, root of file transfers in your case would be /home/kaushal , not / . So ... /oldUnicelFiles/*.gz /oldUnicelFiles/*.txt ... should do the thing. If you used 'oldUnicelFiles/' instead of 'oldUnicelFiles' in rsync invocation, the root would be /home/kaushal/oldUnicelFiles/ . A note of warning if you plan on doing deletions in the future: rsync -avz --del oldUnicelFiles hostB:/home/kaushal ... would remove all files not existing on sending side, in hostB:/home/kaushal/oldUnicelFiles/ ...but: rsync -avz --del oldUnicelFiles/ hostB:/home/kaushal ...would do the same in hostB:/home/kaushal/ Both of the above would also protect already existing files on the receiving side, that match patterns in exclude.txt . Be sure to read carefully rsync's man. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Configuring Rsync to run without any user input or Cron job.
Etienne D wrote: Hi ! I would like to know if it's possible for Rsync to automatically mirror a folder and it's content from one deamon to another. (or one client to a deamon) Basically I want users to drag and drop files in a folder and Rsync to mirror these files as soon as it sees them. A cron or schedule task is not a viable solution since we can receive files any time of night or day and we need to send them in very timely manner. I was looking into Rsync list of parameters, but couldn't find that precise option... We were using Double-Take in the past, but it's only giving us troubles. Thanks in advance for the tip. Additionally to what Matthias wrote in another reply about inotify - you could experiment with http://search.cpan.org/~mlehmann/Linux-Inotify2-1.2/ if you're somewhat used to perl. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync for the brave
Ashley M. Kirchner wrote: It's not a folder that Windows will allow me to delete, so consequently rsync will attempt to do that and fail. Adding --exclude=Network Trash Folder has absolutely no effect what so ever. Any suggestions? You could use --ignore-errors - with this, rsync will continue deletions. It's useful when dealing with win32 world. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude vs hide
Matthew Monaco wrote: What is the difference between exclude and hide in the filter rules? Normally, 'hide' is interpreted only by the sender and the files will never show up in the file list sent to the receiver. 'Exclude' is both 'hide' (sender) and 'protect' (receiver) - so files that will not show up in the file list, will not be deleted by the receiver (if you use any of the available delete options). With separate 'P' and 'H', as well as 'S' and 'R' (respectively - sender and receiver's 'include') you can create quite clever rules. When 'hide' is used in rsyncd.conf filters with rsync in daemon mode, it may work a bit differently. When rsync daemon operates as a sender 'hide' will function as usual, but when it operates as a receiver - it will also be interpreted (normally receiver wouldn't) and will not allow any updates to files matching the wildcard used. The same goes for 'protect' and 'exclude'. So in short - functionality is extended from 'don't delete' to 'don't update or delete'. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is rsync copying over the entire file? Sent bytes matches actual file size.
Administrator wrote: Hi, [cut] Number of files: 2 Number of files transferred: 1 Total file size: 584742598 bytes Total transferred file size: 584742598 bytes Literal data: 584742598 bytes Matched data: 0 bytes File list size: 118 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 584814160 Total bytes received: 48 sent 584814160 bytes received 48 bytes 15595045.55 bytes/sec total size is 584742598 This looks to me as though it is transferring the entire file as the sent bytes is 584814160 and I am now confused as it should have only transferred the changes in the zip file, although it does say that it only received 48 bytes. Am I reading this wrong? Thanks for your help. Locally used rsync will default to -W which causes whole files to be transferred (if they differ - also check -c option if mtime and size are not reliable in your case). But it still distinguishes between client and server - the stats are for the client - which sent 584814160 bytes in one direction, but received only 48 from the server. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is rsync copying over the entire file? Sent bytes matches actual file size.
Hedley J. Phillips wrote: Question 2: The time taken to run the command is the same with or without the switch. Why is this? I would expect the command with the no whole file switch to run faster as it only has to transfer a portion of the entire file. Do you mean precisely the same, or maybe just felt similar ? Either way - to know which portion(s) to transfer, rsync has to read (and process) the file on both ends, and update the destination file accordingly. http://rsync.samba.org/tech_report/node2.html With --whole-file it's just read here, write there. If the bandwidth for the transfer is large, it will likely be faster. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question regarding --delete-during/after and backup file cases
Matt McCutchen wrote: (It would have made my life easier if you had replied directly to that message so I didn't have to search for it.) Ahh yes - it was an old thread back from June - I kinda did, but the question was a bit weird different and the thread ended. Either way, sorry. If I understand it correctly - those backup-file cases mean -b option and - accidental or deliberate - override of the protect rule added implicitly by rsync (as explained in the man page). But if this rule is overriden (effectivly allowing deletion of backuped files), then assuming no per-dir rules are changed, the outcome will be the same regardless if we use --delete-during + --delete-delay or --delete-after. I found one case where the outcome differs. If the protect filter is overridden and a destination file is backed up before being *updated*, then --delete-after will delete the backup file but --delete-delay won't (because it checks for deletions before the backup file is created in the first place). E.g., with this sequence of commands: mkdir src dest touch dest/foo echo NEWDATA src/foo rsync -r --delete-WHEN --filter='R *' -b src/ dest/ dest/foo~ will exist at the end if WHEN is delay but not if WHEN is after. I can't think of any other such cases when no per-dir rules are changed, but that doesn't mean there aren't any: an rsync run is a complex process with numerous steps that can interact in unexpected ways. Matt I could swear I tested such (basic) scenario - I must have had backup files sitting already in dest/ dir, or I didn't update source file. Oh well :o Big thanks for the example - that tiny detail kept bugging me all the time. Michal -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Question regarding --delete-during/after and backup file cases
A good while ago I asked about difference between --delete-during/delay and --delete-after, when per-directory files are updated (all is perfectly clear for me here), but during the discussion there was a hint made by Wayne, that the outcome can differ in more situations: It is useful for things such as --delay-updates --delete-delay (to have all updates happen more rapidly at the end), and the option avoids an extra dir-scan delete pass in such a case. And for folks that don't have per-dir filter files being updated, it works the same as --delete-after (if we disregard certain backup-file cases where the suffix is not excluded), just more optimally. If I understand it correctly - those backup-file cases mean -b option and - accidental or deliberate - override of the protect rule added implicitly by rsync (as explained in the man page). But if this rule is overriden (effectivly allowing deletion of backuped files), then assuming no per-dir rules are changed, the outcome will be the same regardless if we use --delete-during + --delete-delay or --delete-after. So there must be something I missed or misunderstood. Could someone provide me a simple example of such backup-file case ? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS/X Leopard Server and rsync backups
Few remarks: Greg Shenaut wrote: --delete-excluded: delete files no longer present any excluded files Yes. You can achieve the same with 'H' or 'R' instead of '-', without having to specify --delete-excluded. 'H' is sender-only exclude, 'R' is receiver-only include. In your case: H somefile will cause the extra file on the receiving side to disappear, unless it's matched by some other rule later, that would protect it otherwise. Similary: R somefile - some* would work as expected as well. Extra file on the receiver would be deleted, if it wasn't transferred - sender will not send it, and 'R' will match before '-' on the receiving side. But e.g. 'someotherfile' would match '-' on both sides, so it wouldn't be deleted, if it wasn't transferred. --link-dest=$OLD: copy only files that are different from those in directory $OLD and hardlink the unchanged ones. //: the base of the source tree is the filesystem root directory One slash is enough. A line of the form : .rsync-filter means incorporate filter rules from any .rsync-filter file found and apply them to everything in and beneath its directory. Yes. Careful about delete modes though - you will need --delete-after, if the per-dir rules are being transferred, and they differ from the ones already present on the receiving side (and you expect the new ones to influence deletions). The line - */.Trash means don't copy any .Trash file, even the one in the root directory. * alone will match any non-empty component, so .Trash (dir or file) in the root will not be excluded. - .Trash would do the thing - the rule is inherited in every subdirectory (unless you inhibit that with .n $FLTR ), so it will match anything .Trash it encounters. The line - .Spotlight-*/ means copy all directories whose names match .Spotlight-* but nothing beneath them. This one will exclude any directory that matches .Spotlight-* . So neither dir, nor its contents will be copied (as rsync won't visit the directory in the first place). If you have regular file matching .Spotlight-*, it will be copied though. Btw - try running some test case rsync with -vv / -vvv / -i and/or with custom logging options - you will see when and how rsync does things. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delete-delay vs. delete-after in 3.0.2 (and possible bug)
Wayne Davison wrote: Diffs are always welcomed. Please feel free. ..wayne.. Allright, tiny update to the rsync.yo file. I haven't included information about backup/suffix case you mentioned, as I'm not sure about the details yet. --- rsync.yo.org2008-05-17 18:45:28.0 +0200 +++ rsync.yo2008-06-18 22:38:48.0 +0200 @@ -1170,6 +1170,13 @@ using bf(--delete-after) (which it cannot do if bf(--recursive) is doing an incremental scan). +bf(--delete-delay) is particulary useful with bf(--delay-updates) option, as +it avoids an extra dir-scan delete pass in such case. + +Note, that bf(--delete-delay) is not equivalent to bf(--delete-after), when +per-directory rules on the receiving and sending side are different - in such +case, deletions are computed using old rules. + dit(bf(--delete-after)) Request that the file-deletions on the receiving side be done after the transfer has completed. This is useful if you are sending new per-directory merge files as a part of the transfer and -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delete-delay vs. delete-after in 3.0.2 (and possible bug)
Wayne Davison wrote: On Tue, Jun 10, 2008 at 09:58:34PM +0200, Michal Soltys wrote: have per-dir filter files being updated, its works the same as --delete-after (if we disregard certain backup-file cases where the suffix is not excluded), just more optimally. You mean - the difference will be in the moment where the non-excluded files get deleted ? As long as per-driectory filters are the same on sender and receiver, the final outcome will still be the same regardless of a delete algorithm used ? Anyway - would it be allright to submit a small manpage diff, to add some details about difference between delete-delay and delete-after ? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and running scripts
becca23 wrote: I'm using the rsync protocol to keep a windows and linux system mirrored. I have set up the rsync and ssh to run properly, but I would like to know if there is an option, whether in the rsync command or placed in the rsyncd.conf file where I can call a script that runs each time rsync executes? The reason i need to do this is to change the groups/permissions of files when they are transferred over. I could just set this script to run every minute or so, but I would rather strive for accuracy by just having it run every time rsync completes a run through. What about initiating the transfer on the receiving side ? This way - a script and a cron job (or some batch file and task scheduler, if it's on windows side) would do the thing nicely. Or if it must be sending side, just ssh remote command (script) to do any cleanup you might need after rsync transfer completes. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and running scripts
becca23 wrote: Would you do this using the ssh -s option? Just regular command, like ssh [EMAIL PROTECTED] 'ls -al /' also, would this run the script before or after rsync completes? That depends on your needs. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
delete-delay vs. delete-after in 3.0.2 (and possible bug)
Hello I've noticed today, that --delete-delay introduced in 3.0.2 is not functionally equivalent of --delete-after (I was under impression it should work this way). Looking at following example: /src /src/testfile /src/.rf /dst /dst/testfile .rf consists of following lines: S, .rf H, * P, testfile rsync -aHx --delete-after --filter=dir-merge, .rf /src/ /dst/ ...will execute properly - which means that already existing testfile in /dst will not be removed. But if I execute: rsync -aHx --delete-delay --filter=dir-merge, .rf /src/ /dst/ ...testfile will be removed, unless .rf (with above rules) already exists on the receiving side. Is this how it should be ? I was under impression, that --delete-delay gave functionality of --delete-after, when the chosen algorithm was that of --delete-during. Cheers -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delete-delay vs. delete-after in 3.0.2 (and possible bug)
Wayne Davison wrote: per-dir rules set right for both the on-going copying, and a late delete scan). I don't think this is likely to change any time soon, but it would be nice to have a way of doing something like early per-dir filter copying or something. ..wayne.. Oki, thanks for clarification. Still, in that case - what's the benefit of using --delete-delay option ? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
