Re: process --files-from filelist as given
Hi. On Sun, 31 Dec 2023 20:28:21 +0100 Roland via rsync wrote: > apparently, rsync sorts the list of files provided to "--files-from". > how can i avoid sorting of that list ? According to the man, this is not possible. See: SORTED TRANSFER ORDER that suggest also the --delay‐updates option. > I want to copy a list of files in specific order Why ? -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.2.7 hangs when --usermap is used and receiver is not a super-user
Hi. On Wed, 08 Mar 2023 22:21:28 +0100 Tomasz Chmielewski via rsync wrote: > After upgrading to rsync 3.2.7, the following command hangs forever > (using "--usermap" causes the hang; without "--usermap" it doesn't > hang): > rsync -v -p -e --usermap user:user /etc/services user@remote: This command is incorrect: the -e option needs a command. Without -e this command works for me on Fedora-37 with rsync-3.2.7-1.fc37.x86_64 -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
Hi On Sun, 05 Mar 2023 07:45:16 -0800 Robin Lee Powell via rsync wrote: > Oh, yeah, I missed that part. Yeah, don't do that; it's easy to add > a lock file to a shell script. Not so easy IMO to do that properly. Use the flock command if your system provides it. That said, using a systemd service (resp timer) for interactive (resp batch/cron) use: - gives you that locking for free - enforce using the same execution context in both cases (ex: the Unix environment, the locale ...) -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
Hi. On Sat, 04 Mar 2023 00:39:52 -0600 Albert Croft via rsync wrote: > The rsync commands may be launched from command-line or cron, but use > the same format and options in either case. As a result, there may be > multiple rsync processes pulling files from the same remote path to the > same local path. I think you should first prevent this to happen. If your receiving machine is using systemd: - define a X.service for doing the rsync - define a X.timer unit to replace using cron - launch from the command line with: systemctl start X - this will not start a new rsync if one runs already (if X.service is running) -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 15254] New: rsync performs full replication with option -I since last upgrade to version 3.2.3 protocol version 31
Hi. Note: re-sending to rsync@lists.samba.org On Sat, 26 Nov 2022 21:24:45 + just subscribed for rsync-qa from bugzilla via rsync wrote: > I can replicate the issue in a simple test scenario /tmp/source to /tmp/dest > via > rsync -rauvxHI --delete --stats --log-file=/tmp/rsync.log \ > /tmp/source/ /tmp/dest/ > Performs a full replication every time the command is being launched If you add verbosity (ex: -vvv) you will see: delta-transmission disabled for local transfer or --whole-file and thus full replication. But you are right, even with adding --no-whole-file or using SSH produces a full replication with -I rsync -i -rauxHI --no-whole-file --delete --stats --log-file=/tmp/rsync.log /tmp/source/ /tmp/dest/ -n >f. foo Number of files: 2 (reg: 1, dir: 1) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 4 bytes rsync -i -rauxHI --delete --stats --log-file=/tmp/rsync.log /tmp/source/ localhost:/tmp/dest/ -n rsync -i -rauxH --delete --stats --log-file=/tmp/rsync.log /tmp/source/ localhost:/tmp/dest/ -n Number of files: 2 (reg: 1, dir: 1) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 PS: the -a option implies -r: you can thus suppress -r -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 15254] New: rsync performs full replication with option -I since last upgrade to version 3.2.3 protocol version 31
Hi. On Sat, 26 Nov 2022 21:24:45 + just subscribed for rsync-qa from bugzilla via rsync wrote: > I can replicate the issue in a simple test scenario /tmp/source to /tmp/dest > via > rsync -rauvxHI --delete --stats --log-file=/tmp/rsync.log \ > /tmp/source/ /tmp/dest/ > Performs a full replication every time the command is being launched If you add verbosity (ex: -vvv) you will see: delta-transmission disabled for local transfer or --whole-file and thus full replication. But you are right, even with adding --no-whole-file or using SSH produces a full replication with -I rsync -i -rauxHI --no-whole-file --delete --stats --log-file=/tmp/rsync.log /tmp/source/ /tmp/dest/ -n >f. foo Number of files: 2 (reg: 1, dir: 1) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 4 bytes rsync -i -rauxHI --delete --stats --log-file=/tmp/rsync.log /tmp/source/ localhost:/tmp/dest/ -n rsync -i -rauxH --delete --stats --log-file=/tmp/rsync.log /tmp/source/ localhost:/tmp/dest/ -n Number of files: 2 (reg: 1, dir: 1) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 PS: the -a option implies -r: you can thus suppress -r -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync just hanging sometimes
Hi. On Wed, 27 Jul 2022 13:30:32 +0300 Alexander Gribanov via rsync wrote: > Rsync starts, goes as in the log below and just hangs like this for minutes > or maybe even hours and nothing changes... This may happen when windows is involved. Try to add the --whole-file option. -- francis.montag...@inria.fr, DSI-SP, Tel: (33) 04 92 38 79 11, Bur: C111 INRIA Sophia, 2004, route des Lucioles - BP 93 06902 Sophia Antipolis Cedex -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is there a default for 'pid file' for rsync in daemon mode?
Hi. On Fri, 01 Jan 2021 09:57:38 + Chris Green via rsync wrote: > My backup system crashed a couple of nights ago due to a power cut > (can't really blame it!) and I went and restarted it after the power > came back. However, as I note above, files in /home/chris/tmp/pid > aren't cleared out so rsync refused to run when systemd attempted to > start it up. > I guess the pid file should either be in /tmp or somewhere in > /run/user. However shouldn't there be some sort of sensible default > if it isn't set in rsyncd.conf? Since rsyncd is launched by systemd, I would say that no pid file is needed: systemd will prevent simultaneous launch of rsyncd. Can you try without spicifying "pid file" in rsyncd.conf ? What is the content of the rsyncd.service file on your system ? On Fedora it is a system (not user) service: cat /lib/systemd/system/rsyncd.service [Unit] Description=fast remote file copy program daemon ConditionPathExists=/etc/rsyncd.conf Wants=network-online.target After=network-online.target [Service] EnvironmentFile=/etc/sysconfig/rsyncd ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS" [Install] WantedBy=multi-user.target -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Using --inplace and --sparse to a NetApp NAS through NFS
Hi. I noticed, when using --inplace and --sparse to update a file of 36 G located in an NFS mounted directory from a NetApp NAS through NFS, that rsync is rewriting most of it: 27 G, while rsync (--stats) indicates a literal data of only 262,144 bytes. This do not happens with --inplace without --sparse: only ~204 K are written, according to the size of a snapshot made before updating with rsync. Is it a known issue when combining --inplace and --sparse or is it due to the file contents or NFS ? Thanks. I'm fine to do the initial copy with '--inplace --sparse' and the subsequent incremental ones with only --inplace. The NAS will anyway deduplicate zero-blocks after the copies. Context: OS: Fedora-32 running kernel-5.7.8-200.fc32.x86_64 NFS3 rsync-3.2.3-1.fc32.x86_64 -- Francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync with mode --delete
Hi. On Sat, 27 Jun 2020 00:23:12 +0700 Budi Janto wrote: > On 6/23/20 12:37 PM, francis.montag...@inria.fr wrote: >> If the rsync source path is /mnt/DATA yes, it will remove data on the >> client, but if the source path is a sub-directory of /mnt/DATA the >> rsync will fail without doing any deletion. > Thanks for first information. This is for more detail, which has already > been done. > Server side: > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > pool 4.98T 9.07T 4.98T /mnt/DATA > # cat /some/path/rsyncd.conf > [data] ... > path = /mnt/DATA ... > Client side: > # crontab -l > 0 */3 * * * /path/to/rsync.sh > # cat /path/to/rsync.sh > rsync -avPH --chmod=D755,F644 --delete rsync://xxx.xxx.xxx.xxx/data/ > Any idea or suggestion to prevent client's data loss? Thank you. The simplest way is probably to: - move any files/directories of /mnt/DATA in a subdir. Example: /mnt/DATA/data - change rsyncd.conf to reflect that I will do: 1. stop the rsync daemon on the server 2. do the move (assuming you have no files matching .?) cd /mnt/DATA/ mkdir .a mv * .??* .a mv .a data 3. change rsyncd.conf: path = /mnt/DATA/data 4. start the rsync daemon 5. check in dry-run on one client: - with /mnt/DATA mounted - with /mnt/DATA unmounted -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync with mode --delete
Hi. On Tue, 23 Jun 2020 11:14:41 +0700 Budi Janto via rsync wrote: > I need to know, I use rsync-3.1.3_1 under FreeBSD both of server & > client side. Server running ZFS with RAID system, stored data mount > point to /mnt/DATA. Under certain circumstances, ZFS failure to mounting > data in /mnt/DATA (Just empty directory), so client with rsync running > periodically with mode --delete it's safety or will remove data on > client too? If the rsync source path is /mnt/DATA yes, it will remove data on the client, but if the source path is a sub-directory of /mnt/DATA the rsync will fail without doing any deletion. To verify in dry-run ... of course. -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: implied-dirs tail component being chown'd
Hi. On Tue, 27 Aug 2019 14:56:25 -0700 Chris Roehrig via rsync wrote: > rsync -a --super --relative --no-implied-dirs "--filter=. HomeWin.rfilter" > /cygdrive/c/Users/me/./ myserver:/WinBACKUP/Users/me/ If you are connecting to myserver as you (not root), simply suppressing the --super option should work. Otherwise (with a recent version of rsync) add: --chown me:me -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Correct, resumable, large-file flags
Hi On Sat, 27 Jul 2019 11:45:58 +0100 Mark Raynsford wrote: > Same effect on ^C, unfortunately. The exact command I'm running (with > names changed to protect the innocent :): > $ /usr/local/bin/rsync \ > -a -L -i '--chmod=ugo-rwx,Dugo+x,ugo+r,u+w' \ > --delete-after \ > --delay-updates \ > --partial \ > /A/ > www2.example.com:/B/ > If I ^C this command, I see temporary files left in /B, and the next > time I run this same command, new temporary files are created and the > old ones are ignored. I'm unable to reproduce that (on Linux). With --delay-updates you should see temporary files only in temporary directories named: .~tmp~ Is it what you see? > This is on FreeBSD 12. > $ rsync --version > rsync version 3.1.3 protocol version 31 > Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others. > Web site: http://rsync.samba.org/ > Capabilities: > 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, > socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, > append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags Same version for me, except: prealloc instead of no prealloc, file-flags but that shouldn't count (I guess). -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Correct, resumable, large-file flags
Hi On Fri, 26 Jul 2019 20:15:27 +0100 Mark Raynsford via rsync wrote: > I want to mirror a directory A on server www1.example.com to a > directory B on www2.example.com. The directory A contains very > large but essentially read-only files. A server process on > www1.example.com periodically creates new large files and deletes old > ones. I want to synchronize the directories roughly every ten minutes. > It's important that a file in B only becomes "visible" (that is, has > the same name as a file in A) when the two files contain the exact > same content. Files that no longer exist in A should be deleted from B. > An initial pass: > www1$ rsync -avz --delete-after /A/ www2.example.com:/B/ Try: rsync -avz --delete-after --delay-updates --partial /A/ www2.example.com:/B/ The man indicate also for --delay-updates: See also the "atomic-rsync" perl script in the "support" subdir for an update algorithm that is even more atomic (it uses --link-dest and a parallel hierarchy of files). PS: You may prefer also: - using -i (--itemize-changes) instead of -v - do not compress (no -z) > This works, but the transfers are not resumable for some reason. If I > ^C the rsync process and then run it again, I see partial files left in > B, and rsync ignores these the next time I run the command. Weird: never seen that. -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-missing-args doesn't delete
Hi. On Tue, 23 Apr 2019 16:35:15 +0200 MI via rsync wrote: > If someone has a good suggestion on how to rsync a list of files and > delete from the destination any file that is not listed in --files-from, > that would be welcome. Assuming you are talking about files in a same source directory, you can do: - generate a file (KEEP_FILE) with the list of files you want - call: rsync ... --include-from KEEP_FILE \ --delete --delete-excluded \ SOURCE_DIR/ DESTINATION_DIR KEEP_FILE should have the format: + /keep1/*** + /keep2/*** ... - * leading / to match only at the first level /*** to match any level of sub-directory (if keep1 ... are directories) -- Francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 13587] New: Add a --dry-run way to show destination for each item
Hi (sorry to only answer to the list) On Tue, 21 Aug 2018 23:03:56 - just subscribed for rsync-qa from bugzilla via rsync wrote: > User very much wants to know where the files _will go_ before daring to try an > actual run. > He tries even > rsync -vv --dry-run --itemize-changes -avz x.y.z:dir1/dir2/backup/ backup/ > but still there is no way to make it say > File x.y.z:dir1/dir2/backup/F1 will be copied to backup/F1 > File x.y.z:dir1/dir2/backup/F2 will be copied to backup/F2 What about that: with this hierarchy: find dir1 dir3 dir1 dir1/dir2 dir1/dir2/backup dir1/dir2/backup/subdir dir1/dir2/backup/subdir/F3 dir1/dir2/backup/F2 dir1/dir2/backup/F1 dir3 dir3/backup dir3/backup/F4 one may say, for two source directories: rsync --dry-run --out-format='%i %f -> %n%L' -a dir1/dir2/backup/ dir3/backup/ backup/ cd+ dir1/dir2/backup/. -> ./ >f+ dir1/dir2/backup/F1 -> F1 >f+ dir1/dir2/backup/F2 -> F2 >f+ dir3/backup/F4 -> F4 cd+ dir1/dir2/backup/subdir -> subdir/ >f+ dir1/dir2/backup/subdir/F3 -> subdir/F3 Ref: man rsyncd.conf: log format %i an itemized list of what is being updated %f the filename (long form on sender; no trailing "/") %n the filename (short form; trailing "/" on dir) -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Getting --delete to work, simply
Hi. On Thu, 03 May 2018 15:06:14 -0500 Mike Brown via rsync wrote: > On Thu, May 03, 2018 at 02:59:11PM -0400, Kevin Korb via rsync wrote: >> Note that these are working because S* and s* are being applied to all >> the dirs and files within dirs. So if you had x/Something it wouldn't >> get copied and if you had S/else it wouldn't get copied. Maybe you want >> to add --prune-empty-dirs and an --include='*/' so that rsync will look >> inside of all dirs for matching files. > All not working so when when working with the next HDD backup: > rsync -vain --prune-empty-dirs --delete \ > --include='[T-Z]*' --include='*/' --exclude='*' \ > /share/Public/videos/ /share/Backup-HDD-05 > When run without the --prune-empty-dirs it picked up a directory starting > with iZ, but didn't list the sub-directory and its files. With the > --prune-empty-dirs, it was no longer listed. It should never have been listed > in the first place as I never included "i". But, worse than that, the > --include='*/' is causing ALL directories in /share/Public/videos/ to be > queued up for transfer to the backup HDD. > The filter rule is not working out in practice. For example, the following > directories do not get copied: > Victoria/Season-1 > Victoria/Season-2 > Why? Because the "S" in Season is getting excluded because it is not in > the initial include rule. Right. > How do you get the filter rule to only look at the first part of the path > and ignore the remaining part of the path? You can use simply /PATTERN that will match only at the first level. If you want also to suppress the "others" directories on the destination you could use --delete-excluded. Try: rsync -vain --delete --delete-excluded \ --include='/[T-Z]*' --exclude='/*' \ /share/Public/videos/ /share/Backup-HDD-05 > This has been a real brain teaser. Such a powerfull mechanism cannot be simple :-( -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
