Re: Why try to update (some) permissions which are the same?
Am 06.09.23 um 08:49 schrieb Paul Slootman via rsync: The current version is 3.2.7, especially 2.6.8 is quite ancient. You may want to upgrade before going bug hunting, chances are your problem has already been fixed. Oh yes, exactly the step to 3.* did a lot both to option and PERFORMANCE! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
ignore mtime - or any metadata
Hi, SHORT: rsync shall ignore differences in meta (inode) data, but does not even with --size-only. -- If in combination with --link-dest. Is there any flag I missed? EXPLANATION: I use rsync -avbuH --size-only --stats --delete-excluded --link-dest=/path/to/last "user@system:Shared/Data" /path/to/now to sync backup-style and use the --link-dest option to minimize traffic (and space) and still have a full backup each time. So far nothing special. Now it happens that at the source side data changes are moderate, but meta information (like atime, mtime) do change a lot. I didn't find a solution to make rsync ignore these differences to use a link. --size-only and -t (implied by -a) work ok with a simple syncs, but obviously (for me) it does not in combination with --link-dest. I can understand the motivation behind this: rsync is not allowed to change meta data of the existing inode, nor can it neglect the change - or can it? Hardy -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ignore mtime - or any metadata
Thanks Kevin, -a implies rlptgoD and I obviously used it by comfort and habit. Omitting -t (and perhaps also gop, not really needed) might solve my problem? Will give it a try. Am 25.09.22 um 20:30 schrob Kevin Korb via rsync: -a is telling rsync to copy the metadata. --size-only is telling rsync to skip the contents of files that are the same time. Without --link-dest rsync would just update the metadata on the target. However, with --link-dest rsync is being told to make the same file with 2 different metadatas. It can only do so by duplicating the file (effectively the same as --copy-dest). On 9/25/22 14:26, Hardy via rsync wrote: Hi, SHORT: rsync shall ignore differences in meta (inode) data, but does not even with --size-only. -- If in combination with --link-dest. Is there any flag I missed? EXPLANATION: I use rsync -avbuH --size-only --stats --delete-excluded --link-dest=/path/to/last "user@system:Shared/Data" /path/to/now to sync backup-style and use the --link-dest option to minimize traffic (and space) and still have a full backup each time. So far nothing special. Now it happens that at the source side data changes are moderate, but meta information (like atime, mtime) do change a lot. I didn't find a solution to make rsync ignore these differences to use a link. --size-only and -t (implied by -a) work ok with a simple syncs, but obviously (for me) it does not in combination with --link-dest. I can understand the motivation behind this: rsync is not allowed to change meta data of the existing inode, nor can it neglect the change - or can it? Hardy -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question abount pre-xfer exec
You only log you would like to to mount /backup, but the actual command is missing. You should also log errors, so something like /usr/bin/mount /backup >> /var/log/rsyncd.log 2>&1 would be adequate before your line to check what is mounted. Hope this helps Hardy Am 24.09.22 um 15:15 schrob dotdeb--- via rsync: I've been using rsync for years to backup my machines both at work and at home. These days I faced a new "challenge": at work I connect my laptop to a docking station with an external usb disk. I'd like to use this disk as a backup volume. I put my disk in /etc/fstab to be mounted at boot (with 'nofail' option to avoid errors when I'm at home). I have no problems if the laptop is booted after the connection to the docking station but, if I work at home, suspend the laptop and then go to work and connect it to the docking station and resume it, the /backup volume will not be mounted automatically. I found that rsyncd.conf can execute scripts before and I tried to create a script to be executed (as early stage or pre transfer? a bit confuser about it) to check if /backup is mounted and mount it if not. I verified that the script is executed (I put there some debugging "echo" sent to the log file) but the mount command within it does not mount anything. Here it is the rsyncd.conf ## read only = false write only = false usechroot = true uid = 0 gid = 0 early exec = /tmp/test-pre-exec [rsync-backup-xxx] comment Local rsync-backup of xxx path = /backup/xxx log file = /var/log/rsyncd.log ## and the script /tmp/test-pre-exec ## #!/bin/sh echo -n "executing pre-xfer script ..." >> /var/log/rsyncd.log if ! grep -qs '/backup ' /proc/mounts then echo -n "mounting /backup ..." >> /var/log/rsyncd.log /usr/bin/mount >> /var/log/rsyncd.log fi echo " done" >> /var/log/rsyncd.log exit 0 ## -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
I second Francis here. You don't need to diagnose incomplete file transfers as long as you have racing conditions as you described. This leads to strange result inevitably. NEVER start several rsync jobs manipulating the same data - especially if there are modifications to BOTH sides source and destination. You do not necessarily define a service like Francis suggests. A simple semaphore approach suffices. Perhaps even something like # ps fax | grep -v grep | grep $0 && exit to prevent this exact command "$0" to start concurrently. Hardy Am 04.03.23 um 08:38 schrieb Francis.Montagnac--- via rsync: Hi. On Sat, 04 Mar 2023 00:39:52 -0600 Albert Croft via rsync wrote: The rsync commands may be launched from command-line or cron, but use the same format and options in either case. As a result, there may be multiple rsync processes pulling files from the same remote path to the same local path. I think you should first prevent this to happen. If your receiving machine is using systemd: - define a X.service for doing the rsync - define a X.timer unit to replace using cron - launch from the command line with: systemctl start X - this will not start a new rsync if one runs already (if X.service is running) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync verify its writes?
Robin is right. Read back verification has lost its meaning since old days when we used extremely unreliable media or transports. Who writes to floppies anymore? In those old days you could/should use any copy program with a -v verify flag. But read-back wore down your floppy faster, so in the end... Transfer by TCP and like protocols sense errors and do repair "on the fly". (Re-sent packages happen, and you never even notice.) If not correctable (only if connection breaks) you really receive an error that is sensed and passed through all layers to the application. And if this is not really weird programming, the message will be caught by the user or calling program. The same is true for modern hardware. Your s.m.a.r.t HDD informs the system of problems. Since 1980s or so I NEVER had an error displayed by a direct (immediate) verification read. At some LATER time your aged CD might become unreadable... To make it short: Technical failure WILL be noticed by modern systems. It is admin's job to react. Nowadays 99% of data loss is neglect in programming or user attention. And the last may even occur with a verification message. Do you write your script so you REALLY receive an error if one happens? Even operating worn/aged HW is not remedied by read back, but in fact a human neglect. Still, flushing all volatile memory at the end of a write process is a good idea. A proper sync does that, a read back does not. Au contraire, it will happily read cached data. Am 17.02.23 um 17:51 schrieb Robin Lee Powell via rsync: That's not the same as a read-back write verification. I believe that in general, rsync assumes that the disk actually wrote whatever it was told to write. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Can rsync write to a FIFO?
If this helps, in old days I used to use cpio for a similar thing. I do not want to spam you with my whole script, but willing to share if you want. I think you will get the hang of it by the following snippet. (Get yourself man-knowledge about the -i -o -p mechanism of cpio and the use of dd.) This was in the good (?) old days when rsh worked as simple (and insecure) as this. In modern *n*x like systems rsh is a link to ssh, which is (besides being entirely wrong!) a pitfall to finding correct cli arguments. But it is manageable if you are aware of it. CPIOP = parameter arguments to cpio /tmp/$$.f = list of files Snippet: case $CPIOP in -i*) rsh -l $RUSER $RHOST dd if=$RDEV | cpio $CPIOP ;; -o*) cpio $CPIOP &2 exit ;; esac Hope this gives an idea Hardy Am 10.02.23 um 10:31 schrieb Chris Green via rsync: I have searched a little and read the man page but I can't really find a good definite answer to this. Can rsync write to a FIFO? Obviously one needs the --inplace to do this, does one also need --write-devices? It would be very handy if one can do this, to use as a simple message passing mechanism. Write something to a file on system A and rsync it to a FIFO on system B where there is a simple script reading the FIFO. The script gets the contents of the file every time it's written. (this is all within a LAN behind a reasonably secure firewall) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync sends again already existing files
Am 29.06.23 um 22:31 schrieb Stephane Ascoet via rsync: Kevin Korb le 29/06/2023 04:52: --itemize-changes will cause rsync to tell you what it thinks is Hi, thank you so much! Today I used a little different way of doing it, and another computer, and the behaviour is the same. It seems that the reason is a different timestamp. So the whole file is replaced just for this? I've always read that Rsync was using an advanced checksum mecanism instead of timestamps-only? Are you so sure rsync actually copies the file? It should correct the timestamp and tell you it did. smime.p7s Description: S/MIME Cryptographic Signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Per file Log output to understand hard-link decision
Don't know if this is enough for you, but it may help at least a bit to hunt down your problem. There is a flag -i From man rsync --itemize-changes, -ioutput a change-summary for all updates this gives either a "." for no change, or a letter code for each change in "data, time, size" etc., for any filename it logs. You will figure most flags. Others you may need to look up somewhere. Hope this helps (a bit). Hardy Am 14.05.23 um 14:38 schrieb c.buhtz--- via rsync: Hello, I know it is a often discussed topic how rsync decide about using hardlinks or copy a file. Even if content is unchanged problems are often file permissions and owner ships. I know that. Is it possible to configure rsync that way that it logs for each file its decision about using a hardlink and if not why exactly it doesn't? The background of my question: I'm part of maintainer team of "Back In Time" a desktop backup software using rsync in the back. For years we have users reporting about the hardlink problem. And we are sure that it isn't their fault but ours. We can't reproduce the problems for sure but we observe the behavior also on our own machines sometimes. It would help our investigation if we could better understand the hardlink-decision for each file and folder. Kind Christian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
