Re: Question About Rsync and Modification Times
That isn't how rsync should work with -a. Is something preventing it
from backdating the file? What is the filesystem? Can you try copying
your 2015 file with cp -a?
On 10/9/24 14:56, McDowell, Blake wrote:
Hi Kevin,
The -a flag in this instance is not back-dating the timestamp of the
copied file to the source file. It is modifying it to the time of
transfer and leaving it that way. Then any time I rerun that command it
always updates the timestamp since it is always making it the time of
transfer. I have a source file the has a modification date of 2015 and
when I rsync it to day with -a the copied file has a modification date
of today.
Again, this only happens on files that I use rsync to copy for the first
time over to empty storage. If I drag and drop the timestamp stays the
same as the source file and any rsync -a commands subsequently run do
not modify the file in any way. When I use rsync -a to copy over the
first time, the timestamp is modified to the current time and then
continues to update anytime I run rsync -a
-Blake
*From: *rsync on behalf of Kevin Korb
via rsync
*Date: *Wednesday, October 9, 2024 at 14:45
*To: *rsync@lists.samba.org
*Subject: *Re: Question About Rsync and Modification Times
External Email - Exercise Caution
You are using rsync -a which copies (preserves) the timestamp. Meaning
that rsync will copy the file then back-date it to the timestamp of the
source file. Most copying tools do not do this though cp's -a does it
too. Note that your itemized output says that the timestamp is
different meaning that the file has changed since rsync last ran.
Without the -a (or the included -t) the itemized output would have shown
a 'T' instead of a 't' meaning "The timestamp is different and I'm not
fixing it.".
If the file wasn't being modified the timestamp wouldn't be different
and rsync would have just skipped it.
On 10/9/24 14:06, McDowell, Blake via rsync wrote:
Hello,
I have a question about how/why rsync updates modification times, which
I haven’t been able to find an answer to.
I have two locally connected storage devices running TrueNAS Core: one
is new and empty, while the other is filled with files.
When I run the following rsync command:
/rsync -avPh --itemize-changes --stats "${@}"/
to transfer files from the full storage to the empty one, the
modification times of the transferred files are updated to the time of
transfer. I understand that using rsync to transfer to an empty storage
may not offer any real advantage, and that the modification times are
updated because rsync "touches" the files. However, I would prefer for
rsync not to update the modification times, though I gather this might
not be possible.
For context, the /--itemize-changes/ output during the initial transfer
looks like this:
/>f++/
Now, if I wait for a minute and then run a dry run with the same command
on the files I just transferred, the output changes to:
/>f..t.../
If I actually rerun the command, rsync updates the modification times to
the current time. This behavior repeats endlessly each time I run the
command.
Interestingly, if I manually "drag and drop" the files to the empty
storage, the modification times remain the same as the original files,
which is the outcome I prefer. Moreover, if I run the same rsync command
(/rsync -avPh --itemize-changes --stats "${@}"/) on the files that I
dragged and dropped, rsync doesn’t transfer or update any file, and the
/--itemize-changes/ output looks like this:
/.f/
I’m puzzled as to why rsync behaves differently for files initially
transferred using rsync versus those transferred via drag and drop.
Could this behavior be more related to TrueNAS rather than rsync?
Any clarification would be greatly appreciated.
Thanks,
Bleakley
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.samba.org%2Fmailman%2Flistinfo%2Frsync&data=05%7C02%7Cmcdowellh%40si.edu%7Cfccb6ec4d6ea49df31ed08dce8927c85%7C989b5e2a14e44efe93b78cdd5fc5d11c%7C0%7C0%7C638640963111753917%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Bl10Cbn3HhpoUOKIaeOnZOEYt4QXlHZWc5e1CnuSnO0%3D&reserved=0
<https://lists.samba.org/mailman/listinfo/rsync>
Before posting, read:
https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.catb.org%2F~esr%2Ffaqs%2Fsmart-questions.html&data=05%7C02%7Cmcdowellh%40si.edu%7Cfccb6ec4d6ea49df31ed08dce8927c85%7C989b5e2a14e44efe93b78cdd5fc5d11c%7C0%7C0%7C638640963111780341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=DHfdTKxDHHKQFEB1TQeQZpaKOUduFh%2Fx3SUQxBxEcI8%3D&reserved=0
<http://www.catb.org/~esr/faqs/s
Re: Question About Rsync and Modification Times
You are using rsync -a which copies (preserves) the timestamp. Meaning
that rsync will copy the file then back-date it to the timestamp of the
source file. Most copying tools do not do this though cp's -a does it
too. Note that your itemized output says that the timestamp is
different meaning that the file has changed since rsync last ran.
Without the -a (or the included -t) the itemized output would have shown
a 'T' instead of a 't' meaning "The timestamp is different and I'm not
fixing it.".
If the file wasn't being modified the timestamp wouldn't be different
and rsync would have just skipped it.
On 10/9/24 14:06, McDowell, Blake via rsync wrote:
Hello,
I have a question about how/why rsync updates modification times, which
I haven’t been able to find an answer to.
I have two locally connected storage devices running TrueNAS Core: one
is new and empty, while the other is filled with files.
When I run the following rsync command:
/rsync -avPh --itemize-changes --stats "${@}"/
to transfer files from the full storage to the empty one, the
modification times of the transferred files are updated to the time of
transfer. I understand that using rsync to transfer to an empty storage
may not offer any real advantage, and that the modification times are
updated because rsync "touches" the files. However, I would prefer for
rsync not to update the modification times, though I gather this might
not be possible.
For context, the /--itemize-changes/ output during the initial transfer
looks like this:
/>f++/
Now, if I wait for a minute and then run a dry run with the same command
on the files I just transferred, the output changes to:
/>f..t.../
If I actually rerun the command, rsync updates the modification times to
the current time. This behavior repeats endlessly each time I run the
command.
Interestingly, if I manually "drag and drop" the files to the empty
storage, the modification times remain the same as the original files,
which is the outcome I prefer. Moreover, if I run the same rsync command
(/rsync -avPh --itemize-changes --stats "${@}"/) on the files that I
dragged and dropped, rsync doesn’t transfer or update any file, and the
/--itemize-changes/ output looks like this:
/.f/
I’m puzzled as to why rsync behaves differently for files initially
transferred using rsync versus those transferred via drag and drop.
Could this behavior be more related to TrueNAS rather than rsync?
Any clarification would be greatly appreciated.
Thanks,
Bleakley
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delay-updates --temp-dir does it work?
Note that I don't have much experience with these options. My understanding is that --temp-dir doesn't make a tree because it is only designed to hold individual files until rsync is done making them then they aren't temporary anymore and are moved.. --delay-updates on the other hand keeps files out of place until all of the files are done then it expects to be able to move them all into place all at once meaning that they need to be on the same filesystem. You might want to look at the atomic-rsync script in rsync's contrib dir. It still needs there to be space on the filesystem to hold the files that are different but depending on the change it will use hard links to handle the unchanged files so that the new and old trees can simply be swapped at the end. On 9/11/24 07:26, Jiří Prchal via rsync wrote: Hi all, I can't get this work as man page mention: " --delay-updates put all updated files into place at end --temp-dir=DIR, -T create temporary files in directory DIR " I'm doing whole root update over slow and unstable connection and I understood that those options should solve my problem of corrupted fs on partial update. I thought this will transmit all changed files at first and then move them over old ones. Sice the free space on receiver partition is limited, I need to use other partition/device there, therefore DIR is absolute path. But while transmitting, only one file at time appears in temp dir. And in one place, no complete dir hierarchy. I tried -partial-dir also, but it fails too because there are more files with the same name, so this works as man page describes. Am I missing something? Any help, please? Thanks Jiri -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Diff between using "-i" and not using "-i"
It appears that xattr changes (that is what the x mens) never made it into the verbose output. I would call that a bug but I would rather it be fixed by making -v include -i. On 6/3/24 05:22, - via rsync wrote: Hi, Version: sync version 3.3.0 protocol version 31 If I do a I get: = sending incremental file list var/log/journal/ sent 263,31K bytes received 1,14K bytes 528,90K bytes/sec total size is 294,92M speedup is 1.115,21 (DRY RUN) = including option <-i>: rsync -avhHXASDin --del /mnt/ /foo/mnt/ I get: = sending incremental file list .fx usr/bin/ping .d...a. var/log/journal/ sent 263,31K bytes received 1,14K bytes 528,90K bytes/sec total size is 294,92M speedup is 1.115,21 (DRY RUN) = How is that, that "usr/bin/ping" is not included / included in results? That is no feature, is it? Yours lopiuh -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync as a de-duplication-only tool, using --link-dest
I don't believe that what you are asking for can be done with rsync. At
first thought you can't mix --ignore-existing with --ignore-non-existing
as that would ignore everything. Something would have to at least exist
and not be ignored for rsync to link to it.
Anyway, for a laugh, I asked chatgpt to make something to do this.
After I got my laugh I cleaned up some of the silly stuff it did and
came up with this:
#!/bin/bash
# Define the directories to compare
dir1="$1"
dir2="$2"
# Recursively list all files in both directories
files1=$(find "$dir1" -type f)
# Loop through files in first directory
for file1 in $files1; do
# Get relative path of file1
rel_path="${file1#$dir1}"
file2="$dir2$rel_path"
# Check if file exists in the second directory
if [ -f "$file2" ]; then
# Get metadata of both files
metadata1=$(stat -c "%Y%s" "$file1")
metadata2=$(stat -c "%Y%s" "$file2")
# Compare metadata
if [ "$metadata1" -eq "$metadata2" ]; then
# Delete file1 and create a hard link to file2
# rm "$file1"
# ln "$file2" "$file1"
echo "Hard linked: $file2 to $File1"
# else
# echo "Different: $file1"
fi
fi
done
Note that I only tested it a little bit which is why anything actually
destructive is commented.
On 5/1/24 19:34, B via rsync wrote:
Recently I was thinking about --link-dest= and if it was possible to use
rsync to de-duplicate two nearly-identical directory structures.
Normally I would use a tool like hardlink, jdupes, or rdfind, but in
this case the files are huge and numerous, so hashing them would take
forever. I did a test run and these tools mostly choked to death after a
few hours.
These directories were made using rsync in the first place, so I know
the files are duplicate and I would be willing to use rsync's
quick-check (path/filename, mtime, size) to assume uniqueness of the files.
My objective is to hard-link files with the same relative path/filename,
mtime, and size. Nothing more. Files which are different should not be
touched. Files which exist in the destination but not the source should
not be deleted. Files which exist in the source but not the destination
should not be transferred.
The problem is that I don't want to create any new files in the
destination. That's the sticking point.
I thought maybe I could do something wacky like 'rsync -a
--ignore-existing --ignore-non-existing --link-dest="../new/" old/ new',
but that doesn't work. The existing files get ignored and nothing is
linked.
Is there a way to do this with rsync?
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync replacing symlinks without warning (resend)
I'm not really blaming the user. If it were up to me, -v would include -i.
On 2/9/24 05:36, Andreas Gruenbacher wrote:
On Sun, Feb 4, 2024 at 7:20 PM Kevin Korb via rsync
wrote:
rsync's -v is fairly useless. Learn to use -i instead or in addition to.
Well, note that I didn't say anything about the lib/ directory in that
command; it's just that rsync decided to remove the symlink component
from the path lib/modules/.
Wouldn't it be more useful to make it obvious in the --verbose and
--dry-run output what rsync has actually decided to do instead of
blaming the user, which is what you're doing? Especially if the
behavior is all but self-explanatory?
Thanks,
Andreas
On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:
Hello,
when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.
In the following reproducer, rsync is called as follows:
rsync --verbose --recursive --relative --delete a/./lib/modules b/
Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.
In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.
I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...
it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?
Thanks,
Andreas
#! /bin/sh
tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"
umask 022
mkdir -p a/lib/modules
echo foo > a/lib/modules/foo
mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib
show() {
find "$@" | xargs stat -c "%F %N" | sort -k2
}
echo "from:"
show a
echo
echo "to:"
show b
echo
echo "rsync:"
rsync \
--verbose \
--recursive \
--relative \
--delete \
a/./lib/modules \
b/
echo
echo "to:"
show b
# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes received 47 bytes 414.00 bytes/sec
# total size is 4 speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: BUG? rsync ends without message by delete files
now it sounds like you have too many hard links for rsync to handle. On 2/7/24 08:05, Franke via rsync wrote: Am 06.02.24 um 23:20 schrieb Roland: and then, it stops totally quiet. you mean it simply exits without any message? Yes rsync ends totally quit. what's the return code ( echo $? ) After some more tests, the ERR-Code are: "22 - Error allocating core memory buffers" (with --delete-before) But, now there are only ~100.000 Files to delete, in some Tranches. IMHO: rsync read the Fileslist without problems, begins with deleting (and on thi can decrease the list too) and fail than with a Momory-Error? :-o Source- and Target-system have 64GB RAM, could be enough... This seem for me as a Bug. And sorry, but i cant make Issues on Github. Gruss Franke -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: BUG? rsync ends without message by delete files
Normally, when rsync isn't deleting things the problem is that there is some kind of error (possibly scrolled off screen unnoticed) but it sounds like you are getting no output at all which would eliminate that possibility. The other likely cause is your $SOURCE being something that contains a * or other wildcard. If there is a wildcard in the source parameter then the shell expands that wildcard giving rsync a list of sources. The --delete only operates within that list meaning that anything not currently in that list is immune to deletion. If that doesn't answer your question I would suggest adding -ii (doubled --itemize-changes) as it will show you the files it is analyzing and what it plans to do about them. On Tue, 6 Feb 2024, Franke via rsync wrote: Date: Tue, 6 Feb 2024 19:53:32 +0100 From: Franke via rsync To: just subscribed for rsync-qa from bugzilla via rsync Subject: BUG? rsync ends without message by delete files Hi! Is this a bug, or did I miss a setting? When synchronizing and deleting files, rsync (version 3.2.7 protocol version 31 AND before) simply stops whitout comments. No error, no message, nothing in the syslog. Rsync works normally, but after delete some on, it ends. I try'd --max-delete=1 to, but no effekt. Command are: rsync -vaxHSPAX --delete $SOURCE $TARGET The dirs containing in sync containing approximately 40 million files to delete. And I can't delete it by hand as we need to drive some recoveries after a server crash and relocation. Any Ideas? Gruss Franke -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync replacing symlinks without warning (resend)
rsync's -v is fairly useless. Learn to use -i instead or in addition to.
On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:
Hello,
when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.
In the following reproducer, rsync is called as follows:
rsync --verbose --recursive --relative --delete a/./lib/modules b/
Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.
In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.
I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...
it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?
Thanks,
Andreas
#! /bin/sh
tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"
umask 022
mkdir -p a/lib/modules
echo foo > a/lib/modules/foo
mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib
show() {
find "$@" | xargs stat -c "%F %N" | sort -k2
}
echo "from:"
show a
echo
echo "to:"
show b
echo
echo "rsync:"
rsync \
--verbose \
--recursive \
--relative \
--delete \
a/./lib/modules \
b/
echo
echo "to:"
show b
# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes received 47 bytes 414.00 bytes/sec
# total size is 4 speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Archiving to vfat
Also, instead of -a use -rt. Those are the only parts of -a that FAT even pretends to support. On 1/21/24 16:42, Roland via rsync wrote: it's most likely because of vfat timestamp limitation try --modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situa- tions. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second). roland Am 21.01.24 um 22:15 schrieb Ian Z via rsync: I am trying to use rsync between two local directories on Linux. The source directory is on a normal ext4 partition, under my home directory. The destination is an SD card that I insert into the card reader on the computer, formatted with a vfat filesystem. The command line is like rsync -avC --delete /home/itz/foo/ /media/itz/DEAD-BEEF/foo/ This does not work as I expected: all files are always transferred, even when I run this command 2 times with nothing in between. I expected only new or changed files to be transferred. Is this - a user error - a rsync bug - a limitation of the vfat filesystem? In any case, what can I do, using rsync or possibly something else, to do what I want? I note that the absolute timestamps on the files are unimportant; I can change them to whatever. All that matters is that new and changed files are copied every time I run this command, and only those files. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync over ssh fails with --files-from
The errors column is 0. The drop column is 18. The second bit number
is the number of packets which should grow. At least that is how I read
it. Column makes it more readable in a terminal but not so much in an
email.
On 12/21/23 14:18, Alex wrote:
Can someone help me determine if these errors are normal or if this
could somehow be the cause? I've removed the last three columns for
readability - they were all zeros.
# column -t /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed
multicast|bytes packets errs drop fifo
lo: 133093161 146045 0 0 0 0 0
0 133093161 146045 0 0 0
ens18: 166999724 256655 0 16 0 0 0
0 167638513 218267 0 0 0
The "errs" for ens18 are steadily increasing, but the "drop" column is
steady. It's also curious the errors on loopback would also be increasing?
Other ideas on how to troubleshoot this would be greatly appreciated.
Thanks,
Alex
On Wed, Dec 20, 2023 at 9:27 PM Alex <mailto:mysqlstud...@gmail.com>> wrote:
Hi,
On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync
mailto:rsync@lists.samba.org>> wrote:
What is the error? I assume you know that with that syntax the
filelist.txt is local rather than remote.
Yes, I do know it refers to the list of local files.
There is no error - it just hangs indefinitely until some timeout
period. This is what it looks like on the remote side:
$ ps ax|grep rsync
324075 ? Ss 0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/
324094 ? S 0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/
On the local side, with a few - added to rsync, I see this:
recv_file_list done
get_local_name count=9 /var/tmp/one/
generator starting pid=324075
delta-transmission enabled
recv_generator(config1.cf <http://config1.cf/>,0)
config1.cf <http://config1.cf/> is uptodate
send_files(0, config1.cf <http://config1.cf/>)
then it just stalls until it eventually times out. However, if I
remove any one of the nine files from the filelist, it completes
normally.
It actually also exhibits the same problem without a filelist at all
- as long as I'm transferring multiple files, it fails. I suppose if
I were syncing a directory with less than nine files, it might
succeed, but local directory to any remote directory on this one
server fails.
This also isn't a disk space or inode problem or corrupt filesystem
problem - the same command works from a different host to this one
problematic host without a problem.
I've also confirmed the open file limit is large enough on both sides:
# ulimit -n
5
I've now spent hours and hours trying to isolate and troubleshoot
this problem. It really feels like it's somehow related to the
number of files being transferred, not the size of the files. If I
break up a directory into multiple attempts, I can eventually
transfer all the files (maybe 40 in total), but trying to send all
40 files at once and none get transferred.
It almost certainly has something to do with building the initial
list of files. If that list is too large, the transfer fails, even
if the majority of the files in the source are already in the
destination.
It also happens when I'm pushing the files to the destination or
pulling them from the remote to local.
Thanks,
Alex
On 12/20/23 09:50, Alex via rsync wrote:
> Hi, I've been using rsync on fedora over ssh to sync
directories for
> decades, but suddenly having a problem with transferring
multiple files
> at a time to one specific host using --files-from. I can't
think of what
> might have changed to have caused this. Using rsync to
transfer a single
> file to this problematic host works successfully. It appears
to be
> related to the number of files in the --files-from filelist.
More than
> nine and it stalls; less than nine and it finishes
successfully. I'm
> using the same version of rsync on both sides.
>
> (Server) Protocol versions: remote=31, negotiated=31
> (Client) Protocol versions: remote=31, negotiated=31
>
> When running the following command, it appears to collect the
list of
> files to be transferred, successfully makes the connection
with the
> remote server, but then just stalls. Using strace on the
remote side
> shows rsy
Re: rsync over ssh fails with --files-from
What is the error? I assume you know that with that syntax the filelist.txt is local rather than remote. On 12/20/23 09:50, Alex via rsync wrote: Hi, I've been using rsync on fedora over ssh to sync directories for decades, but suddenly having a problem with transferring multiple files at a time to one specific host using --files-from. I can't think of what might have changed to have caused this. Using rsync to transfer a single file to this problematic host works successfully. It appears to be related to the number of files in the --files-from filelist. More than nine and it stalls; less than nine and it finishes successfully. I'm using the same version of rsync on both sides. (Server) Protocol versions: remote=31, negotiated=31 (Client) Protocol versions: remote=31, negotiated=31 When running the following command, it appears to collect the list of files to be transferred, successfully makes the connection with the remote server, but then just stalls. Using strace on the remote side shows rsync appears to be waiting for data. rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i /root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/ sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of other ed25519 keys. Could it be related to packet size or some kind of network disparity? It's not related to the size of the files, as I've tried large and small and it doesn't matter - if the number of files exceeds 9, it fails. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Unfortunately, exit 23 litterally just means something else went wrong and might have scrolled off of the screen if you have rsync listing files (--verbose or --itemize_changes). Essentially, it is anything that doesn't have its own exit code. I just ignore it. The most common reaosn is file vanished. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., On Thu, 14 Dec 2023, rsync--- via rsync wrote: Date: Thu, 14 Dec 2023 19:20:15 +0100 From: rsync--- via rsync To: rsync@lists.samba.org Subject: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some? I am trying to find a solution for the open source Linux software "Back In Time" (https://github.com/bit-team/backintime) where we evaluate the rsync exit code when taking a backup via rsync and inform the user that an error has occured. Questions: 1. Is there full list of possible reasons available that make rsync exit with the return value 23? 'rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]' I want to decide for each reason if we treat it as an error or warning. 2. I want to ignore some reasons for exit code 23 by only logging it, mainly: symlink has no referent: "/home/user/Documents/dead-link" Is there a way to - either prevent that this error leads to exit code 23 (if no other reason occurs) - or to prevent this specific check at all (but eg. copy the symlink "as is")? THX a lot! PS 1: The typcially used rsync command line looks like this: rsync --recursive --times --devices --specials --hard-links --human-readable -s --copy-links --acls --perms --executability --group --owner -- info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup -- chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 --exclude=/home/username/.local/share/backintime --exclude=.local/share/backintime/mnt -- include=/home/username/Documents/ --include=/home/username/ --include=/home/ --include=/home/username/temp/deleted_folder/ -- include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs --exclude=.cache/* --exclude=.thumbnails* -- exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ --exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* -- exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb --exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* -- exclude=/var/backups/* --exclude=.Private --exclude=lock --exclude=root_only_file.txt --include=/home/username/Documents/** -- include=/home/username/temp/deleted_folder/** --include=/home/username/.mozilla/** --exclude=* / /home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup PS 2: You can find more details or even answer in our Github issue too: https://github.com/bit-team/backintime/issues/1587 -BEGIN PGP SIGNATURE- iF0EARECAB0WIQSHERqysePm7S8yuR9UoLWOVtABBwUCZXtTzQAKCRBUoLWOVtAB B2CnAJ9YGQ/gXTPP2Ntg3arHHDC11cRnfgCeJVpDOnGhTH0OreZfj8pIbnsL3SI= =Eqei -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
So I decided to do a quick test using the Linux kernel source tree since
it has lots of files. I duplicated a tree, used 'find . -type f -exec
chmod 444 {} +' to make read only files for rsync to want to chmod, then
used cp -al to make several duplicate trees using hard linked files. An
rm -rf on one such tree took .97 seconds while an rsync deletion took
1.25 seconds. Clearly I need a bigger test to play with as that margin
could easily change just by different outputs if either had a -v. But
also, I did not experience the problem you are describing. My surviving
hard links in the duplicate trees were still 444.
BTW, it seems that rsync uses unlink() while rm uses unlinkat(). No
idea if/why there is a performance diff there but I didn't look into it
as it is now time for me to go to work.
On 9/22/23 04:14, rs...@altfeld-im.de wrote:
On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote:
I have heard in the past that rsyncing an empty dir over a tree to
delete the tree is faster than an rm -rf but I can't say I have ever
benchmarked it to get any actual numbers.
This **may** indeed be a myth (for a long time now) re-cited again and again and
- could no longer be valid today
- could apply only when deleting explicitly named files but not deleting the
complete folder
(as we need to do in "Back in Time")
At least I could not find a holistic benchmark with many files and different
scenarios
(file systems, rsync'ing locally vs. over network, snapshot sizes, number of
files, file sizes, rsync and rm versions...)
Q: Does `rsync` provide a test case that I could use as basis to prepare such a
holistic benchmark?
But now that I am hearing
that rsync actually adds a bunch of pointless chmods to the process. Is
it still faster given this problem? If so maybe we should be trying to
investigate why rm is so slow.
Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not.
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
I had intended to come back to this but because I didn't really think I
had anything to add to the discussion I put it at a low enough priority
that I forgot about it. But I saw your bug report and was surprised to
see that I was already unhelpful on this topic but because that original
poster didn't have access to do an rm while you are trying to get a
performance boost.
I have heard in the past that rsyncing an empty dir over a tree to
delete the tree is faster than an rm -rf but I can't say I have ever
benchmarked it to get any actual numbers. But now that I am hearing
that rsync actually adds a bunch of pointless chmods to the process. Is
it still faster given this problem? If so maybe we should be trying to
investigate why rm is so slow. Otherwise, I would agree that an
optimization to not do a bunch of pointless chmods is a good idea. I
would suspect that they are only there to prevent permission denied
errors perhaps on some obscure filesystem (I tested on vfat since it is
so non-compatible but read only files unlink fine there).
On 9/18/23 19:42, rsync--- via rsync wrote:
Context
---
I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:
"Back in Time"-Bug:
https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507
"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.
Older snapshots are then deleted according to a schedule.
Issue
-
When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted
files
with existing hardlinks:
mkdir empty
rsync -a --delete -s empty/ snapshot1/
This distorts the backup history.
There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the
issue is still unfixed:
https://bugzilla.samba.org/show_bug.cgi?id=12806
Questions
-
1. Is there a reliable workaround ("--super" is proposed in the issue but may
probably not always work)?
2. If a rsync developer is reading this: Is there any chance to fix this?
Steps to reproduce
--
I have attached a bash script "setup.sh" as txt file which makes the issue
fully reproducible (on Ubuntu 22.04).
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature Concept: enable iCloud Drive with rsync
Is this being accessed via a fuse mount? If so it seems like that is where this kind of feature should be implemented (like a mount option to decide how to handle such files). Rsync shouldn't need special features to deal with every kind of file storage. On 9/12/23 05:22, Brian "bex" Exelbierd via rsync wrote: Hi, I have also posted this on GitHub but it isn’t clear that was the right place: https://github.com/WayneD/rsync/issues/522 iCloud Drive will evict files that are unused or when additional space is needed on the local drive. The evicted files are replace by "bookmark" files that allow MacOS to continue to report the files in the file system as though they were actually present. The files are downloaded again either on request or when needed. rsync, like all similar tools I can find, doesn't have any way of handling these evicted files. I have been thinking about this and I think I know how to make it work. The short explanation is that iCloud Drive preserves access to the required metadata. Here is an abbreviated output from `mdls` of an evicted file: ``` % mdls .1998-tax-return.pdf.icloud kMDItemContentCreationDate = 2018-08-28 18:32:24 + kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 + kMDItemContentModificationDate = 2018-08-28 18:32:24 + kMDItemDateAdded = 2021-11-05 15:42:15 + kMDItemDisplayName = "1998-tax-return.pdf" kMDItemFSContentChangeDate = 2018-08-28 18:32:24 + kMDItemFSCreationDate = 2018-08-28 18:32:24 + kMDItemFSSize = 1929932 kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 + kMDItemLogicalSize = 1929932 kMDItemPhysicalSize = 1929932 ``` This metadata, I think, is enough to pass the rsync quick check as described in the man page. Therefore, I suspect what needs to happen, from a code perspective, is that rsync needs to be modified to do the following when it finds an evicted file. All evicted files are named consistently, `..iCloud` so they are easy to spot. 1. Perform the system call equivalent of the mdls above to obtain the appropriate dates and sizes. 2. If the file cannot be skipped because it either has changed or a checksum is required, perform the system call equivalent of `brctl download ` to get the file downloaded. 3. Rsync the file 4. If the file was downloaded by rsync, perform the system call equivalent of `brctl evict ` to remove the file to leave the system in the same state. This simplistic algorithm would leave some open issues/caveats: - It is likely that the rsync can only be run one-way from iCloud Drive to non-iCloud Drive data storage. While it is possible it could be run two-way research is needed on whether you’d have to download the old file before you replace it. - Timeouts may happen. iCloud Drive still gets stuck sometimes and there will be non-zero time during downloads that don’t get stuck. - There is no guarantee there is ever enough room on disk to hold a specific file from iCloud Drive. This is likely resolved by MacOS directly, however, this may take excessive time. - If you thought running with checksums was slow before, strap in, because those downloads are going to add up. All that said, I think this would accomplish a high percentage of what users are looking for. Is this kind of a patch welcome? Is the plan sound? I haven't written C since college (in the 90s!) and would be open to advice/mentoring. Thank you. regards, bex -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Why try to update (some) permissions which are the same?
You have --itemize-changes but either it didn't or you filtered it out. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., On Sun, 3 Sep 2023, Perry Hutchison via rsync wrote: Date: Sun, 03 Sep 2023 00:35:29 -0700 From: Perry Hutchison via rsync To: rsync@lists.samba.org Subject: Why try to update (some) permissions which are the same? On the source system: $ rsync --version rsync version 2.6.8 protocol version 29 Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others. <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, ACLs, symlinks, batchfiles, inplace, IPv6, file flags, 32-bit system inums, 64-bit internal inums ... $ ll -d fcst-200[89] fcst-201[01] dr-xr-xr-x 2 perryh perryh 7168 Nov 27 2009 fcst-2008 dr-xr-xr-x 2 perryh perryh 9216 Jul 21 2010 fcst-2009 drwxr-xr-x 2 perryh perryh 9216 Jul 7 2011 fcst-2010 drwxr-xr-x 2 perryh perryh 4608 Jul 7 2011 fcst-2011 $ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] fbsd81:/home/perryh building file list ... done rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation not permitted (1) rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation not permitted (1) sent 15794 bytes received 20 bytes 31628.00 bytes/sec total size is 278224 speedup is 17.59 rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8] On the destination system: $ ll -d fcst-200[89] fcst-201[01] dr-xr-xr-x 2 perryh perryh uchg,uunlnk 7168 Nov 27 2009 fcst-2008 dr-xr-xr-x 2 perryh perryh uchg,uunlnk 9216 Jul 21 2010 fcst-2009 drwxr-xr-x 2 perryh perryh uchg,uunlnk 9216 Jul 7 2011 fcst-2010 drwxr-xr-x 2 perryh perryh uchg,uunlnk 4608 Jul 7 2011 fcst-2011 $ rsync --version rsync version 3.0.7 protocol version 30 Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, no iconv, symtimes ... The question is: why does rsync attempt (and fail) to change the permissions of two destination directories, and not the other two, when the permissions of all four destination directories already match the corresponding source directories? (The change attempt fails because of the destinations' uchg,uunlnk flags, but as far as I can see the change should never have been attempted in the first place.) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: print only first level directory name when copying files
find /mnt/foo/* -maxdepth 0 -print -exec rsync -an `realpath {}`
/mnt/bar/ \;
(realpath eliminates the trailing slashes)
On 8/3/23 22:27, Fourhundred Thecat via rsync wrote:
Hello,
I am copying /mnt/foo to /mnt/bar/
rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/
/mnt/foo contains deep directory structure, ie:
/mnt/foo/aaa/
/mnt/foo/aaa/somestuff/
/mnt/foo/aaa/somestuff/file1
/mnt/foo/bbb/
/mnt/foo/bbb/someotherstuff/
/mnt/foo/bbb/someotherstuff/file2
I am not interested in details which individual files were copied, just
the main directory. Is it somehow possible for rsync to only report the
first level directory?
ie, to have output like this when copying:
/mnt/foo/aaa/
/mnt/foo/bbb/
thanks,
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to tune rsync to speed up?
NFS is slowing things down even more than your bandwidth measurements as it is also forcing --whole-file. On 8/2/23 05:03, Perry Hutchison via rsync wrote: Sebastian G??decke via rsync wrote: We're facing some flapping traffic when rsyncing atm 70T from one server to an DELL Isilon. Both systems are connected with 10G Fiber (not Channel). So we started with one simple "rsync -a /src /dest" to the DELL by using NFS3. ... I always thought (and had observed it so far with rsync) that it makes full use of the network card ... If you are using NFS, it is NFS -- not rsync -- which is managing the network traffic. In such a setup rsync will perform about as well as cp(1). -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync sends again already existing files
You should also read about --inplace. Without it --no-whole-file you are telling it to do all the extra data diffing only to write out an entire new file anyway (just using data from source and target to create it). On 6/30/23 21:29, Selva Nair via rsync wrote: So this disable a lot of interest in Rsync :-( Isn't there a way to disable "--whole-file"? "--no-whole-file" should do it though for local copies, forcing delta transfer is not going to speed up anything in most cases. Selva -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync sends again already existing files
On 6/29/23 16:31, Stephane Ascoet via rsync wrote: Kevin Korb le 29/06/2023 04:52: --itemize-changes will cause rsync to tell you what it thinks is Hi, thank you so much! Today I used a little different way of doing it, and another computer, and the behaviour is the same. It seems that the reason is a different timestamp. So the whole file is replaced just for this? I've always read that Rsync was using an advanced checksum mecanism instead of timestamps-only? Your rsync isn't networking. That means --whole-file is forced making rsync work pretty much like cp. If rsync was networking it would determine what is different about the files then use the existing version + changes to make a new file (this is just extra effort when rsync isn't networking). different. Also, -z is counter-productive when rsync isn't networking. Thanks again. It seems that "--progress" is redondant with "-P"(but it's because I use a list of rsync commands skeletons, and some doesn't have "-P" and I want to be sure to always have the progress indication). But, since I will now always use "-i", do "-v" and "--progress" add something to "-i"? -i, -v, and --progress all only affect the output. Once -i is in play -v adds a header and footer and --progress of course adds the per-file progress bar. -P is just --partial and --progress. I am not a fan of -P as those 2 options are very different and I don't really see why they should go together. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync sends again already existing files
--itemize-changes will cause rsync to tell you what it thinks is different. Also, -z is counter-productive when rsync isn't networking. On 6/28/23 22:28, Stephane Ascoet via rsync wrote: Hi, /media/clec1enextsanstampon/gigamopourcdas4/ contains an old copy of files of /media/mo. I'm updating with "cd /media/clec1enextsanstampon/ && rsync --backup --backup-dir=/media/clec1enextsanstampon/elementssupprimesdepourcdas4/ --del -hPrStvXz --progress /media/mo/ /media/clec1enextsanstampon/gigamopourcdas4/" But unmodified and already in place files in /media/clec1enextsanstampon/gigamopourcdas4/ are re-sended from /media/mo instead of being skipped, why? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [External] Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file
Rsync has no memory of what other instances of rsync have done in the past. An existing file with a different name is of no interest to it unless --delete in which case it would be deleted. Maybe what you really want is --partial-dir? On 6/7/23 16:17, Lacey, Nathan wrote: I'm suggesting a partial file that isn't hidden is worse. Because any other app looking at the file would think the transfer is complete and try to use it. It doesn't need to be a orphaned file, when called next time with --partial rsync could re-opened the incomplete/hidden file, than it could continue putting more bytes to the file until it was really completed. When running in --partial mode, the filename wouldn't have to be random, instead it could be based on the source ip address, so that rsync knows what file to continue processing on. *From:* rsync on behalf of Kevin Korb via rsync *Sent:* Wednesday, June 7, 2023 3:26 PM *To:* rsync@lists.samba.org *Subject:* [External] Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file [You don't often get email from rsync@lists.samba.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification <https://aka.ms/LearnAboutSenderIdentification> ] This message is from an EXTERNAL SENDER - be CAUTIOUS of links and attachments. THINK BEFORE YOU CLICK. That is what --partial does. It keeps the partial file. If it kept the temporary file with the random file name that would just be a useless orphaned file. On 6/7/23 15:02, Lacey, Nathan via rsync wrote: This is a clone from https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0 <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0 <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0>> run rsync with --progress --size-only --partial (version 3.1.2) also tested/built current master ( 3.3.0pre1-5-g6f3c5ecc ) everything works fine unless I interrupt a transfer ( ctrl c ) I would expect to see the incomplete, temporary, hidden file present on the remote machine. So that when I call rsync again, it could just pick up where it left off and continue transferring data. But instead there is no hidden file, and the partial/incomplete file exists, as the actual filename. This is bad, because any code that is expecting existing non-hidden files to be complete will have issues. I apologize if I've missed the command line options I'm looking for Thanks for maintaining a critical piece of software! -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fsanitarium.net%2F&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j42u1PbBcts8QAs7rqJTVuLXLW2
Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file
That is what --partial does. It keeps the partial file. If it kept the temporary file with the random file name that would just be a useless orphaned file. On 6/7/23 15:02, Lacey, Nathan via rsync wrote: This is a clone from https://github.com/WayneD/rsync/issues/484 <https://github.com/WayneD/rsync/issues/484> run rsync with --progress --size-only --partial (version 3.1.2) also tested/built current master ( 3.3.0pre1-5-g6f3c5ecc ) everything works fine unless I interrupt a transfer ( ctrl c ) I would expect to see the incomplete, temporary, hidden file present on the remote machine. So that when I call rsync again, it could just pick up where it left off and continue transferring data. But instead there is no hidden file, and the partial/incomplete file exists, as the actual filename. This is bad, because any code that is expecting existing non-hidden files to be complete will have issues. I apologize if I've missed the command line options I'm looking for Thanks for maintaining a critical piece of software! -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Per file Log output to understand hard-link decision
The only way I know of to determine this behavior is to use the --link-dest option in rsync OR use the much older cp -al then rsync over top of it method. With --link-dest a change in the file's metadata causes rsync to duplicate the file in order to store both versions of the metadata. With the old cp -al method rsync just makes the metadata change on the hard link already in the target which causes all instances of the file to be changed rather than the file being duplicated. On 5/14/23 08:38, c.buhtz--- via rsync wrote: Hello, I know it is a often discussed topic how rsync decide about using hardlinks or copy a file. Even if content is unchanged problems are often file permissions and owner ships. I know that. Is it possible to configure rsync that way that it logs for each file its decision about using a hardlink and if not why exactly it doesn't? The background of my question: I'm part of maintainer team of "Back In Time" a desktop backup software using rsync in the back. For years we have users reporting about the hardlink problem. And we are sure that it isn't their fault but ours. We can't reproduce the problems for sure but we observe the behavior also on our own machines sometimes. It would help our investigation if we could better understand the hardlink-decision for each file and folder. Kind Christian -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: prevent filter excluded files from being deleted
I am not 100% sure I am interpreting this correctly but I think you are
complaining that the file was being deleted in the first command? If
so, instead of -F try --include='*/' --exclude='*'. Otherwise, maybe
you want a second -F?
On 3/6/23 16:04, Heiko Schlittermann via rsync wrote:
Hello,
given are 2 directories:
a
├── a-file
└── .rsync-filter
b
└── a-file
I'd like to sync a/ -> b/, but I'd like to *exclude* all files. But I do
not want to delete the excluded files. (The real scenario is a way more
complex, the above is my reproducer.)
and the following rsync command: `rsync -F --del a/ b/`
Note, there is *no* `--delete-excluded`:
$ rsync -F --del -inav a/ b/
sending incremental file list
*deleting a-file
.d..t.. ./
sent 62 bytes received 29 bytes 182.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
Why? Because it is excluded? Using the `--exclude` option does what I
expect:
$ rsync --exclude 'a-file' --del -inav a/ b/
sending incremental file list
.d..t.. ./
>f+ .rsync-filter
sent 92 bytes received 22 bytes 228.00 bytes/sec
total size is 4 speedup is 0.04 (DRY RUN)
even does what I expect:
$ rsync --filter '- *' --del -inav a/ b/
sending incremental file list
.d..t.. ./
sent 58 bytes received 19 bytes 154.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
Version:
rsync version 3.2.3 protocol version 31
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Apparent synchronization bug in rsync
The included github test case failed for me on vanilla rsync-3.2.7... % ./test.sh new.bin test.bin differ: byte 28345, line 1 % md5sum -b *.bin 7e11df4130968b5e4859f58b9bd6b453 *initial.bin ff14ab080514d5681c99e4e1de01e29f *new.bin 151dd8b1c7bd628afcf38ac6c44155be *test.bin % rsync --version rsync version 3.2.7 protocol version 31 Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others. Web site: https://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, symlinks, symtimes, hardlinks, hardlink-specials, hardlink-symlinks, no IPv6, atimes, batchfiles, inplace, append, ACLs, xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes Optimizations: no SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5 Checksum list: xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none Compress list: zstd lz4 zlibx zlib none Daemon auth list: sha512 sha256 sha1 md5 md4 rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. On 12/9/22 00:02, Wayne Davison wrote: On Thu, Dec 8, 2022 at 12:49 PM Kevin Korb wrote: The first "Bug Fix" listed in https://rsync.samba.org/ftp/rsync/NEWS#3.2.4 <https://rsync.samba.org/ftp/rsync/NEWS#3.2.4> is almost certainly related. Apparently it was fixed via remote option but since your test doesn't include networking that fix wouldn't help. No, the bug was fixed in 3.2.4 and it included an extra feature that allows rsync to bypass the bug in an older remote rsync that still has the bug. On 12/8/22 15:03, Mark Raymond wrote: This looks like a bug to me; am I missing something? If it is a bug, I'm happy to create an issue in GitHub. I have managed to get a test case that fails. If you want to also supply one, feel free. ..wayne.. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Apparent synchronization bug in rsync
The first "Bug Fix" listed in https://rsync.samba.org/ftp/rsync/NEWS#3.2.4 is almost certainly related. Apparently it was fixed via remote option but since your test doesn't include networking that fix wouldn't help. On 12/8/22 15:03, Mark Raymond via rsync wrote: I believe I have found a synchronization bug in rsync. I have a pair of files, which when synced with|| || |rsync --no-whole-file --inplace -S||| || |are not| identical afterwards. All three options are required, if I remove any one of the the files sync correctly. I have put the two files, along with a minimal test script, in this GitHub repo: https://github.com/mark-raymond/rsync-test This looks like a bug to me; am I missing something? If it is a bug, I'm happy to create an issue in GitHub. || -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
Rsync does not have a --no-clobber. -n is --dry-run. It shows what rsync would do without actually doing it. But I didn't know you were using --backup. Anyway, I tried it again with --backup and --remove-source-files and it did in fact delete all the files (but not the dirs). It didn't have any output indicating such so it seems like a problem to me. BTW, it isn't the --backup either. Just the --remove-source-files is enough I just didn't know it because it isn't in the output so --dry-run gives no clue. On 10/17/22 23:51, Sridhar Sarnobat wrote: I think I must be missing something. If source and dest are the same place rsync shouldn't do anything unless it it responding to changes happening at the same time. For example, when I do 'rsync -vain --remove-source-files /tmp/ /tmp/' rsync does nothing. Thanks for the response. You are using "-n" / "--no-clobber". In my case I'm using "-b" / "--backup" (or overwrite). My use case is to get everything out of the source location, and "--no-clobber" will leave things behind. So I guess I should refine my issue to "when you're not using --no-clobber." Do I have a case here? Or is this erasing both the source and destination a valid behaviour in my case? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
I think I must be missing something. If source and dest are the same place rsync shouldn't do anything unless it it responding to changes happening at the same time. For example, when I do 'rsync -vain --remove-source-files /tmp/ /tmp/' rsync does nothing. On 10/17/22 23:12, Sridhar Sarnobat via rsync wrote: >> why not avoid using "--remove-source-files" and delete files manually/via extra step afterwards Multiple reasons. Among them: 1) my jobs run a long time and there are a lot of them. Trying to remember what I did over ssh (where history doesn't get saved) is an extra cognitive load and the system is in a transient state (I've ended up with duplicate files unless I do the extra work later - potentially much later depending on the size of the job). I may also have multiple rsync commands running in separate terminals. 2) I am managing a lot of large files, and my collection of disks doesn't have 2x free space. 3) 2 operations instead of 1 gives more room to make a mistake. This is a workaround that I can occasionally use, and you could even say "why not just use ext4magic?" to recover accidentally deleted files etc. Regardless, the semantics of deleting source files without leaving one copy of your files intact should never happen right? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to use --rsh with multiple sub-arguments?
You can put it all in quotes. Also, if you are scripting you might find setting $RSYNC_RSH easier. On 10/7/22 04:46, c.buhtz--- via rsync wrote: Hello, rsync does offer the --rsh argument. I would like to use it that way rsync --rsh=ssh -o ServerAliveInterval=240 -o LogLevel=Error -o IdentityFile=/home/user/.ssh/id_rsa -p 22 --delete --delete-excluded -v -i / user@localhost:"/tmp/tmp6x75lz10" You see there are several sub-arguments that need to be put to "ssh" via the "--rsh". How can I be sure that e.g. "-o ServerAliveInterval" is interpreted by ssh and not by rsync? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ignore mtime - or any metadata
I can't say I have tried it but -pog would all also result in duplicating files (assuming they are different). On 9/25/22 14:49, Hardy via rsync wrote: Thanks Kevin, -a implies rlptgoD and I obviously used it by comfort and habit. Omitting -t (and perhaps also gop, not really needed) might solve my problem? Will give it a try. Am 25.09.22 um 20:30 schrob Kevin Korb via rsync: -a is telling rsync to copy the metadata. --size-only is telling rsync to skip the contents of files that are the same time. Without --link-dest rsync would just update the metadata on the target. However, with --link-dest rsync is being told to make the same file with 2 different metadatas. It can only do so by duplicating the file (effectively the same as --copy-dest). On 9/25/22 14:26, Hardy via rsync wrote: Hi, SHORT: rsync shall ignore differences in meta (inode) data, but does not even with --size-only. -- If in combination with --link-dest. Is there any flag I missed? EXPLANATION: I use rsync -avbuH --size-only --stats --delete-excluded --link-dest=/path/to/last "user@system:Shared/Data" /path/to/now to sync backup-style and use the --link-dest option to minimize traffic (and space) and still have a full backup each time. So far nothing special. Now it happens that at the source side data changes are moderate, but meta information (like atime, mtime) do change a lot. I didn't find a solution to make rsync ignore these differences to use a link. --size-only and -t (implied by -a) work ok with a simple syncs, but obviously (for me) it does not in combination with --link-dest. I can understand the motivation behind this: rsync is not allowed to change meta data of the existing inode, nor can it neglect the change - or can it? Hardy -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ignore mtime - or any metadata
-a is telling rsync to copy the metadata. --size-only is telling rsync to skip the contents of files that are the same time. Without --link-dest rsync would just update the metadata on the target. However, with --link-dest rsync is being told to make the same file with 2 different metadatas. It can only do so by duplicating the file (effectively the same as --copy-dest). On 9/25/22 14:26, Hardy via rsync wrote: Hi, SHORT: rsync shall ignore differences in meta (inode) data, but does not even with --size-only. -- If in combination with --link-dest. Is there any flag I missed? EXPLANATION: I use rsync -avbuH --size-only --stats --delete-excluded --link-dest=/path/to/last "user@system:Shared/Data" /path/to/now to sync backup-style and use the --link-dest option to minimize traffic (and space) and still have a full backup each time. So far nothing special. Now it happens that at the source side data changes are moderate, but meta information (like atime, mtime) do change a lot. I didn't find a solution to make rsync ignore these differences to use a link. --size-only and -t (implied by -a) work ok with a simple syncs, but obviously (for me) it does not in combination with --link-dest. I can understand the motivation behind this: rsync is not allowed to change meta data of the existing inode, nor can it neglect the change - or can it? Hardy -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question abount pre-xfer exec
You aren't logging any stderr. That is where any error messages would go. Add some 2>&1. Also, mount has a -v On 9/24/22 09:15, dotdeb--- via rsync wrote: I've been using rsync for years to backup my machines both at work and at home. These days I faced a new "challenge": at work I connect my laptop to a docking station with an external usb disk. I'd like to use this disk as a backup volume. I put my disk in /etc/fstab to be mounted at boot (with 'nofail' option to avoid errors when I'm at home). I have no problems if the laptop is booted after the connection to the docking station but, if I work at home, suspend the laptop and then go to work and connect it to the docking station and resume it, the /backup volume will not be mounted automatically. I found that rsyncd.conf can execute scripts before and I tried to create a script to be executed (as early stage or pre transfer? a bit confuser about it) to check if /backup is mounted and mount it if not. I verified that the script is executed (I put there some debugging "echo" sent to the log file) but the mount command within it does not mount anything. Here it is the rsyncd.conf ## read only = false write only = false usechroot = true uid = 0 gid = 0 early exec = /tmp/test-pre-exec [rsync-backup-xxx] comment Local rsync-backup of xxx path = /backup/xxx log file = /var/log/rsyncd.log ## and the script /tmp/test-pre-exec ## #!/bin/sh echo -n "executing pre-xfer script ..." >> /var/log/rsyncd.log if ! grep -qs '/backup ' /proc/mounts then echo -n "mounting /backup ..." >> /var/log/rsyncd.log /usr/bin/mount >> /var/log/rsyncd.log fi echo " done" >> /var/log/rsyncd.log exit 0 ###### -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync timeout non-effective
You are using rsync over ssh. The connection timeout (and port) options don't matter if rsync isn't doing the networking. On 9/1/22 08:49, Roland via rsync wrote: hello, i do some backup via rsync/ssh and pull data from remote machine to local machine. whenever on remote machine there is some network attached subdir getting "stuck" (i.e. read/write is blocking, eg. by suspended zfs pool because of error), rsync backup also hangs forever. i wonder what's the purpose of "--timeout" then. local machine: 3.1.2 remote machine: 3.2.3 command rsync -avi --stats --delete --delete-excluded --inplace --numeric-ids --out-format=%-15i %-15b %-15l %-20M %n%L --timeout=900 --rsync-path=export PATH=/usr/local/bin:$PATH;rsync --exclude-from=/backup/rsync.exclude.all --exclude-from=/zfspool/backup/myhost/rsync.exclude root@myhost:/ /zfspool/backup/myhost/backup what can i do to make this more robust ? rsync should not hang, as it's getting started via script and the hang blocks backup of other hosts. roland -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: do I always have to use either --checksum or --times ?
Yeah, unfortunately rsync's --checksum doesn't even employ obvious optimizations that the man page says it does. There is no reason for it to checksum files that only exist on one end or are different in sizes yet it does. Just try doing an rsync of a huge tree to an empty dir. You won't run out of space. You will get tired of waiting for it to do something and abort it. On 7/15/22 02:57, Robin Lee Powell wrote: --checksum is only slower than re-copying if your network connection between the hosts is similar in speed (or faster than) each host's local disk access. If local disk access is 10x your network link, it is definitely not slower than re-copying. Having said that, it really is *very* slow, and before you use it you should have a clear situation in mind that makes it plausible that two files could have the same size and last mod time and still not have the same data. Or, alternately, be in a situation where a bit flip would be catastrophic. Such situations are not common. On Thu, Jul 14, 2022 at 04:26:48AM -0400, Kevin Korb via rsync wrote: You should almost never use --checksum. It is slower than just re-copying everything. You should almost always use --times (or --archive which includes --times). Without this rsync is almost as dumb as cp. Also, ssh has been the default --ssh for a long time. On 7/14/22 04:22, Fourhundred Thecat via rsync wrote: Hello, I want to sync local folder to remote server. When I run follwing command repeatedly, it always transfers everything each time again and again: rsync --rsh='ssh' foo/ server:/foo/ does it mean I have to always use either --checksum or --times, to prevent repeated transfer of files that have not changed ? thank you, -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: do I always have to use either --checksum or --times ?
You should almost never use --checksum. It is slower than just re-copying everything. You should almost always use --times (or --archive which includes --times). Without this rsync is almost as dumb as cp. Also, ssh has been the default --ssh for a long time. On 7/14/22 04:22, Fourhundred Thecat via rsync wrote: Hello, I want to sync local folder to remote server. When I run follwing command repeatedly, it always transfers everything each time again and again: rsync --rsh='ssh' foo/ server:/foo/ does it mean I have to always use either --checksum or --times, to prevent repeated transfer of files that have not changed ? thank you, -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync verify its writes?
Note that it won't actually tell you that the file was written wrongly just that it is different. In fact, you can only even know that it might be detecting a broken file with --itemize-changes and look for files with a checksum difference but not a timestamp difference. Though there are legitimate causes for that. On 7/12/22 17:26, Paul Slootman via rsync wrote: On Tue 12 Jul 2022, Kevin Korb via rsync wrote: Rsync does not verify writes. --checksum doesn't verify anything. Sounds like you want a file verification tool. The simplest would be md5sum. Running rsync --checksum directly after transferring your files will verify that the files are written correctly (if the source hasn't changed in the meantime). That might help to give some peace of mind. Paul On 7/12/22 02:31, Mark Filipak via rsync wrote: Hello. Does rsync verify its writes? Re, 'info rsync'. Maybe I just being stupid, but there's no mention of verification in the 'DESCRIPTION' section, so despite the words in the 'OPTIONS' section, '-c, --checksum' topic (which I may be misinterpreting), I assume rsync does not verify except for the checksum directive. I admit that I'm paranoid. ;-) ...Please clarify. Regards, and Thanks, Mark Filipak. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync verify its writes?
Rsync does not verify writes. --checksum doesn't verify anything. Sounds like you want a file verification tool. The simplest would be md5sum. On 7/12/22 02:31, Mark Filipak via rsync wrote: Hello. Does rsync verify its writes? Re, 'info rsync'. Maybe I just being stupid, but there's no mention of verification in the 'DESCRIPTION' section, so despite the words in the 'OPTIONS' section, '-c, --checksum' topic (which I may be misinterpreting), I assume rsync does not verify except for the checksum directive. I admit that I'm paranoid. ;-) ...Please clarify. Regards, and Thanks, Mark Filipak. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Customizing compression per file?
It actually does that by default. Though you might need to add to the list of file types with --skip-compress. On 6/24/22 19:35, Joseph Garvin via rsync wrote: I have an rsync cron job to backup folders that contain many different types of files. Most of these are uncompressed files, so the -z option would speed up their transfer. However there are a few files that are already compressed where -z will just needlessly burn CPU and slow down the transfer. Is there any way to express something like, "compress all files except those with extension XXX?" The files are in a proprietary format so it doesn't make sense to add them to a whitelist in rsync itself. Thanks, Joe G. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Users and Groups
Nope. Rsync groups are not groups of users they are just @users with
their own password. I believe the @ just designates that you intend
multiple people to have that password and use that username.
On 6/24/22 12:34, Alexander Gribanov via rsync wrote:
Hello, Kevin! Thank You very much for the reply.
пт, 24 июн. 2022 г. в 19:00, Kevin Korb via rsync <mailto:rsync@lists.samba.org>>:
I think you are thinking too much of rsync here. Rsync groups are the
same as users they just have an @ in front of the name. If you want
UNIX style users and groups then use rsync over ssh and get the
bonus of
ssh's authentication as well as not needing an rsyncd.conf file at all.
Well, actually, I wouldn't like to do anything with system users and groups,
but it would be nice to have virtual users and groups, which are
available for rsyncd only.
Is it possible?
Or maybe there is an opportunity to use external authentication
mechanisms or some kind of modules, etc?
On 6/24/22 11:13, Alexander Gribanov via rsync wrote:
> Hello everybody!
>
> I'm trying to configure rsync-server, but still I can't figure
out how
> to manage users and groups.
>
> I read in the documentation that I could use both, but there is
nothing
> about how to add a user into a particular group...
>
> Why am I doing this?
> 1. I have about 10 modules, which are different and must be
available
> for some users and not available for other users.
> 2. There are many users, new users are arriving, some old users are
> going out, so it would be easier to assign group permissions to the
> modules and then manage just user-group relations, but I couldn't
find
> that in documentation, nor examples on the internet...
>
> Could anybody please give me a hint or a clue about how to assign
a user
> to a particular group? I'm sure that this should work, but I
can't find
> it out myself :(
>
> Thank you very much.
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net
<mailto:k...@sanitarium.net> (personal)
Web page: https://sanitarium.net/ <https://sanitarium.net/>
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Users and Groups
I think you are thinking too much of rsync here. Rsync groups are the same as users they just have an @ in front of the name. If you want UNIX style users and groups then use rsync over ssh and get the bonus of ssh's authentication as well as not needing an rsyncd.conf file at all. On 6/24/22 11:13, Alexander Gribanov via rsync wrote: Hello everybody! I'm trying to configure rsync-server, but still I can't figure out how to manage users and groups. I read in the documentation that I could use both, but there is nothing about how to add a user into a particular group... Why am I doing this? 1. I have about 10 modules, which are different and must be available for some users and not available for other users. 2. There are many users, new users are arriving, some old users are going out, so it would be easier to assign group permissions to the modules and then manage just user-group relations, but I couldn't find that in documentation, nor examples on the internet... Could anybody please give me a hint or a clue about how to assign a user to a particular group? I'm sure that this should work, but I can't find it out myself :( Thank you very much. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: Problem with file exclusion pattern
Sure, that is the way it normally works. By default everything is included. Excludes exclude things. Includes are exceptions to includes that follow. Note that this can get very complicated. --exclude-from=file.txt makes things much easier as you can use the +/- syntax to put both includes and excludes into the same file. On 4/30/22 20:48, H via rsync wrote: On 04/30/2022 08:22 PM, Kevin Korb via rsync wrote: Includes override excludes that follow. So, your include of * meant that nothing was being excluded. An exclude before any includes isn't affected by the includes. On 4/30/22 20:04, H via rsync wrote: On 04/30/2022 07:56 PM, H via rsync wrote: Ah, I was under the impression that all inclusion patterns need to preceed the exclusion patterns? Forwarded Message Subject: Re: Problem with file exclusion pattern Date: Sat, 30 Apr 2022 18:49:22 -0400 From: Kevin Korb To: H Drop the include of * or move the exclude of *.~lock* before it. Also, -c is almost always a bad idea. On 4/30/22 18:18, H via rsync wrote: I am running rsync on Linux and have yet to find the appropriate pattern to exclude files containing '.~lock' as part of the name of files existing in any directory, ie ~/test and below. rsync -vHrltDium -c --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats --include='*' --include='*/' --exclude='*.~lock*' --exclude='*' -e "ssh -y -p 22" ~/test/ someone@1.2.3.4:~/test Can anyone point out what I have missed in the above? Thank you. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., When I dropped the '*' include it did not sync files in the topmost directory? Perhaps it always needs to be included among the options? Thank you. Would one ever put an exclude /after/ the includes? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: Problem with file exclusion pattern
Includes override excludes that follow. So, your include of * meant that nothing was being excluded. An exclude before any includes isn't affected by the includes. On 4/30/22 20:04, H via rsync wrote: On 04/30/2022 07:56 PM, H via rsync wrote: Ah, I was under the impression that all inclusion patterns need to preceed the exclusion patterns? Forwarded Message Subject:Re: Problem with file exclusion pattern Date: Sat, 30 Apr 2022 18:49:22 -0400 From: Kevin Korb To: H Drop the include of * or move the exclude of *.~lock* before it. Also, -c is almost always a bad idea. On 4/30/22 18:18, H via rsync wrote: > I am running rsync on Linux and have yet to find the appropriate pattern to exclude files containing '.~lock' as part of the name of files existing in any directory, ie ~/test and below. > > rsync -vHrltDium -c --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats --include='*' --include='*/' --exclude='*.~lock*' --exclude='*' -e "ssh -y -p 22" ~/test/ someone@1.2.3.4:~/test > > Can anyone point out what I have missed in the above? > > Thank you. > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., When I dropped the '*' include it did not sync files in the topmost directory? Perhaps it always needs to be included among the options? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to elevate rsync privileges when connecting over ssh without using NOPASSWD in sudoers
Rsync includes a script named rrsync that handles this perfectly. On 3/12/22 01:08, Richard Hector via rsync wrote: On 12/03/22 18:38, Richard Hector via rsync wrote: And I do my backups (using dirvish) as root, using a key with a forced command. FWIW, that forced command is here: https://github.com/rwhector/dirvish-forced-command It's rather unpolished and undocumented, but comments very welcome :-) I've also had an issue due to some server-side-only arguments to rsync being undocumented, which means I can't validate them, and basically have to accept anything ... I'd love to know why this is or has to be the case :-) I didn't get any particularly useful answers back in January 2019 ... Cheers, Richard -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Confused as to why rsync thinks time, owner and group of many files differ
Are you using the same source and target each time? I ask because the only discrepancy I see is the link count which shows that there are 11 more instances of that inode on the source than the target. Maybe instances in other snapshots are being updated/re-linked? The only other thing to mention is that when you abort rsync (with -P or --inplace) incomplete files are left. Rsync doesn't fix the owner+group until it is done with a directory and it doesn't fix the timestamp until it is done with a file. This would be why you shouldn't mix those options with --update since the truncated file will be newer than the source file. On 2/3/22 17:04, Andy Smith via rsync wrote: Hi, I am at the moment using rsync to move quite a big set of backups from one machine to another. The source filesystem is xfs; the target filesystem is btrfs. For various reasons I have been stopping the rsync part way through and re-starting. I have noticed that a large number of files are transferred over and over and I can't work out why. Example: sudo rsync -iPva \ --inplace \ --numeric-ids \ --delete \ /data/backup/rsnapshot/daily.0/cacti/ \ root@koff:/data/backup/rsnapshot/daily.0/cacti/ ... http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, prealloc Destination: $ rsync --version rsync version 3.2.3 protocol version 31 Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others. Web site: https://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv, symtimes, prealloc, stop-at, no crtimes Optimizations: SIMD, asm, openssl-crypto Checksum list: xxh128 xxh3 xxh64 (xxhash) md5 md4 none Compress list: zstd lz4 zlibx zlib none What am I missing? Thanks, Andy -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Copying a large file from an USB 3 disk (NTFS or EXT4) to another one (UFS/FreeBSD) is a very slow process....
Rsync is designed to reduce the amount of data transmitted over the network. If rsync isn't networking it can't do that. However, it still uses the same code so it is still using a sender and a receiver rather than simply reading and writing as cp does. Also, rsync forces --whole-file because using rsync's algorithm to delta-copy is slower than just re-copying a file (especially if one of the local paths is really a network mount and double especially if that is the writing end). On 11/28/21 16:44, Harry Mangalam wrote: Can you elaborate on why this is? I wrote a parallel rsync wrapper that works very well over networks but is similarly very slow over local disks. I thought it was a bug in my code but didn't get around to tracking it down since my use cases were all network/ parallel file systems. Harry On Sun, Nov 28, 2021, 11:43 AM Kevin Korb via rsync mailto:rsync@lists.samba.org>> wrote: rsync is terribly slow at local copies. Also, it doesn't do its normal optimizing (see --whole-file). Just use cp. On 11/28/21 13:38, Mario Marietto via rsync wrote: > Hello to everyone. > > I'm copying a large file from a NTFS formatted disk to another > one,UFS/FreeBSD disk,both are removable disks attached to the USB 3 > port. The file is 200 GB large and it is copied very slowly. Why it is > so slow ? I don't know where to store my virtual machines. I tried to > save them on the ext4 disk because I wanted to share them easily between > Linux and FreeBSD but I've realized that when I mount the disk in > FreeBSD after some time it corrupts. I tried to store it on the NTFS > disk but it happens the same. So,now I'm on FreeBSD and I'm copying them > to a dedicated UFS/FreeBSD style disk,but as I said,the speed is very > slow. How can I increase the speed ?. Actually I'm using this > command,because I want to resume the uploading if it breaks at some point : > > > root@marietto:/mnt/da3p2/bhyve/Ubuntu # rsync -avAXEWSlHh > /mnt/da0p1/Backups/OS/bhyve/Ubuntu/im* . --no-compress --info=progress2 > > sending incremental file list > impish-cuda-11-4-nvidia-470.img > > 2.13M 0% 9.49kB/s 6284:55:38 > > > and : where do you save large files ? what's the procedure that you use > to copy large files with a decent speed ? Unfortunately under Linux is > not safe to use a RW ufs disk access. So,I'm out of solutions. > > - > Mario. > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net <mailto:k...@sanitarium.net> (personal) Web page: https://sanitarium.net/ <https://sanitarium.net/> PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync <https://lists.samba.org/mailman/listinfo/rsync> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html <http://www.catb.org/~esr/faqs/smart-questions.html> -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Copying a large file from an USB 3 disk (NTFS or EXT4) to another one (UFS/FreeBSD) is a very slow process....
rsync is terribly slow at local copies. Also, it doesn't do its normal optimizing (see --whole-file). Just use cp. On 11/28/21 13:38, Mario Marietto via rsync wrote: Hello to everyone. I'm copying a large file from a NTFS formatted disk to another one,UFS/FreeBSD disk,both are removable disks attached to the USB 3 port. The file is 200 GB large and it is copied very slowly. Why it is so slow ? I don't know where to store my virtual machines. I tried to save them on the ext4 disk because I wanted to share them easily between Linux and FreeBSD but I've realized that when I mount the disk in FreeBSD after some time it corrupts. I tried to store it on the NTFS disk but it happens the same. So,now I'm on FreeBSD and I'm copying them to a dedicated UFS/FreeBSD style disk,but as I said,the speed is very slow. How can I increase the speed ?. Actually I'm using this command,because I want to resume the uploading if it breaks at some point : root@marietto:/mnt/da3p2/bhyve/Ubuntu # rsync -avAXEWSlHh /mnt/da0p1/Backups/OS/bhyve/Ubuntu/im* . --no-compress --info=progress2 sending incremental file list impish-cuda-11-4-nvidia-470.img 2.13M 0% 9.49kB/s 6284:55:38 and : where do you save large files ? what's the procedure that you use to copy large files with a decent speed ? Unfortunately under Linux is not safe to use a RW ufs disk access. So,I'm out of solutions. - Mario. -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Workaround for rsyncing when fileflag schg is set?
No, rsync normally handles such problems well. Unfortunately, --fileflags is an OS vendor added feature rather than an official rsync feature so it is less well thought out. On 10/31/21 3:51 AM, Perry Hutchison wrote: > That sort of snafu is why find(1) has the -depth directive. > Does rsync have anything similar? > > Kevin Korb via rsync wrote: > >> There maybe a proper solution but an obvious workaround would be to run >> rsync twice. The first time without the --fileflags option. >> >> --no-perms wouldn't help. That is only the standard unix permissions. >> >> On 10/30/21 8:04 PM, Fred Fugate via rsync wrote: >>> Hi, >>> >>> I have some subdirectories within a home directory that are chflagged >>> schg, aka system immutable. >>> >>> When I rsync the home directory to another machine, I get lots of errors >>> for these subdirectories and they fail to copy to the target. NOTE: The >>> target is empty; rsync is copying to an empty remote machine, not trying >>> to overwrite anything. >>> >>> I understand why this is happening; it's the schg flag. But is there a >>> workaround using some combination of rsync options or multiple passes? >>> >>> Here is an example. >>> >>> The source directory is /Users/redacted. >>> >>> Within /Users/redacted is a subdirectory foo that has flag schg set: >>> /Users/redacted/Documents/artwork/foo >>> >>> # ls -laO /Users/redacted/Documents/artwork >>> total 80 >>> drwxr-xr-x ?? 20 redacted ??staff ??- ?? ?? ?? ??680 Sep 21 21:06 . >>> drwx--@ 609 redacted ??staff ??- ?? ?? ??20706 Oct 29 16:07 .. >>> -rw-r--r--@ ?? 1 redacted ??staff ??- ?? ?? ??18436 Sep 21 20:55 somefile >>> drwxrwxrwx@ ??18 redacted ??staff ??schg ?? ?? 612 Apr 12 ??2006 foo >>> >>> >>> My rsync args are: >>> >>> --verbose --archive --one-file-system --acls --hard-links --xattrs >>> --protect-args --delete-after --numeric-ids --itemize-changes --crtimes >>> --fileflags --force-change >>> --rsync-path=/opt/rsync323/bin/rsync >>> >>> My rsync version is 3.2.3 >>> >>> My execution and output looks like this, run as root: >>> >>> # /opt/rsync323/bin/rsync [ARGS ABOVE] /Users/redacted >>> remotemachine.domain.com:/Users >>> . >>> . >>> rsync: [receiver] mkstemp >>> "/Users/redacted/Documents/artwork/foo/bar/.background.tiff.vYOAS2" >>> failed: Operation not permitted (1) >>> rsync: [receiver] mkstemp >>> "/Users/redacted/Documents/artwork/foo/bar/.dc4.orange.CMYK.tiff.rYCmAP" >>> failed: Operation not permitted (1) >>> rsync: [receiver] mkstemp >>> "/Users/redacted/Documents/artwork/foo/bar/.barcode.tiff.2E4mec" failed: >>> Operation not permitted (1) >>> . >>> . >>> >>> In a nutshell: >>> >>> Subdirectory foo gets created on the receiver, but it gets created with >>> the schg flag set. >>> >>> No further copying into foo can happen after that, because the schg flag >>> prevents that. >>> >>> Subdirectory bar cannot be created under foo, and >>> .background.tiff.vYOAS2 and other files cannot be created under bar, etc. >>> >>> How can I force the schg flags to be set on the receiver AFTER >>> everything has been copied from the source? >>> >>> I really don't want to remove all of my schg settings on the source >>> before rsyncing to the target. >>> >>> Would --no-perms allow this? But what would it break? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Workaround for rsyncing when fileflag schg is set?
There maybe a proper solution but an obvious workaround would be to run rsync twice. The first time without the --fileflags option. --no-perms wouldn't help. That is only the standard unix permissions. On 10/30/21 8:04 PM, Fred Fugate via rsync wrote: > Hi, > > I have some subdirectories within a home directory that are chflagged > schg, aka system immutable. > > When I rsync the home directory to another machine, I get lots of errors > for these subdirectories and they fail to copy to the target. NOTE: The > target is empty; rsync is copying to an empty remote machine, not trying > to overwrite anything. > > I understand why this is happening; it's the schg flag. But is there a > workaround using some combination of rsync options or multiple passes? > > > > Here is an example. > > The source directory is /Users/redacted. > > Within /Users/redacted is a subdirectory foo that has flag schg set: > /Users/redacted/Documents/artwork/foo > > # ls -laO /Users/redacted/Documents/artwork > total 80 > drwxr-xr-x 20 redacted staff - 680 Sep 21 21:06 . > drwx--@ 609 redacted staff - 20706 Oct 29 16:07 .. > -rw-r--r--@ 1 redacted staff - 18436 Sep 21 20:55 somefile > drwxrwxrwx@ 18 redacted staff schg 612 Apr 12 2006 foo > > > My rsync args are: > > --verbose --archive --one-file-system --acls --hard-links --xattrs > --protect-args --delete-after --numeric-ids --itemize-changes --crtimes > --fileflags --force-change > --rsync-path=/opt/rsync323/bin/rsync > > My rsync version is 3.2.3 > > My execution and output looks like this, run as root: > > > # /opt/rsync323/bin/rsync [ARGS ABOVE] /Users/redacted > remotemachine.domain.com:/Users > . > . > rsync: [receiver] mkstemp > "/Users/redacted/Documents/artwork/foo/bar/.background.tiff.vYOAS2" > failed: Operation not permitted (1) > rsync: [receiver] mkstemp > "/Users/redacted/Documents/artwork/foo/bar/.dc4.orange.CMYK.tiff.rYCmAP" > failed: Operation not permitted (1) > rsync: [receiver] mkstemp > "/Users/redacted/Documents/artwork/foo/bar/.barcode.tiff.2E4mec" failed: > Operation not permitted (1) > . > . > > > > > > In a nutshell: > > Subdirectory foo gets created on the receiver, but it gets created with > the schg flag set. > > No further copying into foo can happen after that, because the schg flag > prevents that. > > Subdirectory bar cannot be created under foo, and > .background.tiff.vYOAS2 and other files cannot be created under bar, etc. > > How can I force the schg flags to be set on the receiver AFTER > everything has been copied from the source? > > I really don't want to remove all of my schg settings on the source > before rsyncing to the target. > > Would --no-perms allow this? But what would it break? > > > > Thanks, > Fred > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: BackUp with "inverse" increments
See --link-dest. That is what makes rsync shine for backups. On 10/6/21 10:09 AM, Helmut Jarausch via rsync wrote: > Hi, > > I'd like to mirror my root file system (e.g.) to a different disk. The > mirror should always be most recent. > In addition, I'd like to be able to restore my file system to the state > of one or more backups before > without storing full size snapshots. > > The --backup-dir option does most of the work, but what happens if the > old backup doesn't contain a file > which is part of the current source. How can I restore the old version > including deletion of files > which didn't exist in the old version? > > Many thanks for some hints or pointers, > Helmut > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Passing result of shuf to rsync
Quoting from the man page
SORTED TRANSFER ORDER
Rsync always sorts the specified filenames into its internal
transfer list. This handles the merging together of the contents of identi‐
cally named directories, makes it easy to remove duplicate
filenames, and may confuse someone when the files are transferred in a
differ‐
ent order than what was given on the command-line.
If you need a particular file to be transferred prior to another,
either separate the files into different rsync calls, or consider using
--delay-updates (which doesn't affect the sorted transfer order,
but does make the final file-updating phase happen much more rapidly).
and...
NOTE: sorting the list of files in the --files-from input
helps rsync to be more efficient, as it will avoid re-visiting the path
elements that are shared between adjacent entries. If
the input is not sorted, some path elements (implied directories) may end
up being scanned multiple times, and rsync will eventually
unduplicate them after they get turned into file-list elements.
On 9/17/21 8:41 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Saturday, September 18, 2021 12:29 AM, Kevin Korb
> wrote:
>
>> Well, what you have should function. But rsync is going to sort the list.
>
> I want rsync to use the list as given to it by shuf.
>
>> On 9/17/21 8:26 PM, hancooper wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Saturday, September 18, 2021 12:06 AM, Kevin Korb via rsync
>>> rsync@lists.samba.org wrote:
>>>
>>>> Not really sure what you are trying to accomplish here. Seems like it
>>>> should work the way you have it. Note that many wonky rsync kludges are
>>>> due to people not realizing that rsync can have multiple source
>>>> arguments. Instead of the source simply being . it can be a list of
>>>> stuff. Also, note that -a includes --recursive except when --files-from
>>>> is in play.
>>>
>>> I am trying send via rsync a shuffled list of files.
>>>
>>>> On 9/17/21 7:11 PM, hancooper via rsync wrote:
>>>>
>>>>> I am trying to pass filename results that have been shuffled, then pass
>>>>> the file to rsync.
>>>>> Not quite sure whether to use --files-from=- or some other way.
>>>>> shuf -n "$nf" -e ${dpath}/${incl} |
>>>>> rsync -av --update --files-from=- . "$dst"
>>>>
>>>> --
>>>>
>>>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,
>>>>
>>>> Kevin Korb Phone: (407) 252-6853
>>>>
>>>> Systems Administrator Internet:
>>>>
>>>> FutureQuest, Inc. ke...@futurequest.net (work)
>>>>
>>>> Orlando, Florida k...@sanitarium.net (personal)
>>>>
>>>> Web page: https://sanitarium.net/
>>>>
>>>> PGP public key available on web site.
>>>>
>>>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,
>>>> -------
>>>>
>>>> Please use reply-all for most replies to avoid omitting the mailing list.
>>>> To unsubscribe or change options:
>>>> https://lists.samba.org/mailman/listinfo/rsync
>>>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> --
>>
>> ~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,
>> Kevin Korb Phone: (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. ke...@futurequest.net (work)
>> Orlando, Florida k...@sanitarium.net (personal)
>> Web page: https://sanitarium.net/
>> PGP public key available on web site.
>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,._.,
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Passing result of shuf to rsync
Well, what you have should function. But rsync is going to sort the list.
On 9/17/21 8:26 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Saturday, September 18, 2021 12:06 AM, Kevin Korb via rsync
> wrote:
>
>> Not really sure what you are trying to accomplish here. Seems like it
>> should work the way you have it. Note that many wonky rsync kludges are
>> due to people not realizing that rsync can have multiple source
>> arguments. Instead of the source simply being . it can be a list of
>> stuff. Also, note that -a includes --recursive except when --files-from
>> is in play.
>
> I am trying send via rsync a shuffled list of files.
>
>
>> On 9/17/21 7:11 PM, hancooper via rsync wrote:
>>
>>> I am trying to pass filename results that have been shuffled, then pass
>>> the file to rsync.
>>> Not quite sure whether to use --files-from=- or some other way.
>>> shuf -n "$nf" -e ${dpath}/${incl} |
>>> rsync -av --update --files-from=- . "$dst"
>>
>> --
>>
>> ~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,
>> Kevin Korb Phone: (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. ke...@futurequest.net (work)
>> Orlando, Florida k...@sanitarium.net (personal)
>> Web page: https://sanitarium.net/
>> PGP public key available on web site.
>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,._.,
>> --
>>
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Passing result of shuf to rsync
Not really sure what you are trying to accomplish here. Seems like it
should work the way you have it. Note that many wonky rsync kludges are
due to people not realizing that rsync can have multiple source
arguments. Instead of the source simply being . it can be a list of
stuff. Also, note that -a includes --recursive except when --files-from
is in play.
On 9/17/21 7:11 PM, hancooper via rsync wrote:
> I am trying to pass filename results that have been shuffled, then pass
> the file to rsync.
> Not quite sure whether to use --files-from=- or some other way.
>
>
> shuf -n "$nf" -e ${dpath}/${incl} |
> rsync -av --update --files-from=- . "$dst"
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync options –update and –append-verify
--inplace can prevent rsync from running you out of disk space when updating a large file. But you don't want to mix it with --update. Judging by your other message you think that --update means modiify files that are already on both ends. It doesn't. It only means that rsync is forbidden to change files on the target that have a newer timestamp than is on the source. If you don't need that then you don't want --update. Also, when in doubt, --dry-run. On 9/11/21 11:51 PM, hancooper wrote: > ‐‐‐ Original Message ‐‐‐ > On Sunday, September 12, 2021 2:34 AM, hancooper via rsync > wrote: > >> ‐‐‐ Original Message ‐‐‐ >> On Sunday, September 12, 2021 2:03 AM, Kevin Korb k...@sanitarium.net wrote: >> >>> I thought I did elaborate. If it is a problem for you then maybe you >>> shouldn't be using --update. Or you should let rsync delete incomplete >>> files upon abort as it does by default. >> >> I am using the following >> >> rsync -av --inplace --update --log-file=/media/hagbard/hc/snapshot/test1.log >> /media/hagbard/hc1/snapshot/test1 /media/hagbard/hc/snapshot/ >> >> I want that if I run the rsync command again, the files already transferred >> are not resent, thus I used --update. > > Can I re-run without the --update option? Am I right to say that the files > already successfully transferred will not be re-sent again using -av > --inplace ? > > What can I do for the files which have been sent but got rsync aborted during > transfer? > > >>> On 9/11/21 9:29 PM, hancooper wrote: >>> >>>> ‐‐‐ Original Message ‐‐‐ >>>> On Saturday, September 11, 2021 11:20 PM, Kevin Korb via rsync >>>> rsync@lists.samba.org wrote: >>>> >>>>> --archive is all you really need. I actually wish --archive was the >>>>> default because it is all most people need and with the exception of >>>>> writing to a FAT filesystem it is almost always needed. >>>>> --append is for very special cases and should only be used if you really >>>>> know you need it and why. --append-verify only exists because people >>>>> seem to think they need --append and get annoyed that it corrupts their >>>>> files. >>>>> --update is sometimes helpful however it interacts badly with --partial >>>>> and --inplace. >>>> >>>> That's right. Can you elaborate how --update interacts badly with --partial >>>> and --inplace? >>>> >>>>> Since rsync doesn't set the timestamp of a file until it >>>>> is finished then any file left incomplete by an aborted rsync will have >>>>> the timestamp of the abort not the source file. Therefore it will be >>>>> newer than the source file and unless the source is updated rsync >>>>> --update will never complete the file. >>>> >>>> Is there a solution to this problem and should we consider it a bug? >>>> I am encountering this, and it's a real problem for me if files are >>>> not completed. >>>> >>>>> On 9/11/21 6:50 PM, hancooper via rsync wrote: >>>>> >>>>>> I am struggling to understand exactly what the rsync options --update >>>>>> and --append-verify do. >>>>>> Doing info rsync gives >>>>>> -u, --update >>>>>> This forces rsync to skip any files which exist on the destina‐ >>>>>> tion and have a modified time that is newer than the source >>>>>> file. (If an existing destination file has a modification time >>>>>> equal to the source file’s, it will be updated if the sizes are >>>>>> different.) >>>>>> --append-verify >>>>>> This works just like the --append option, but the existing data >>>>>> on the receiving side is included in the full-file checksum >>>>>> verification step, which will cause a file to be resent if the >>>>>> final verification step fails (rsync uses a normal, non-append‐ >>>>>> ing --inplace transfer for the resend). >>>>>> I am using rsync to transfer directories recursively. There are times >>>>>> where I have to stop the rsync transfer, and resume the transfer a few >>>>>> hours or days later, without affecting the already transferred files at >>>>>> destination. >>>>>> I also got some files that return errors by rsyn
Re: Rsync options –update and –append-verify
I thought I did elaborate. If it is a problem for you then maybe you shouldn't be using --update. Or you should let rsync delete incomplete files upon abort as it does by default. On 9/11/21 9:29 PM, hancooper wrote: > ‐‐‐ Original Message ‐‐‐ > On Saturday, September 11, 2021 11:20 PM, Kevin Korb via rsync > wrote: > >> --archive is all you really need. I actually wish --archive was the >> default because it is all most people need and with the exception of >> writing to a FAT filesystem it is almost always needed. >> >> --append is for very special cases and should only be used if you really >> know you need it and why. --append-verify only exists because people >> seem to think they need --append and get annoyed that it corrupts their >> files. >> >> --update is sometimes helpful however it interacts badly with --partial >> and --inplace. > > That's right. Can you elaborate how --update interacts badly with --partial > and --inplace? > >> Since rsync doesn't set the timestamp of a file until it >> is finished then any file left incomplete by an aborted rsync will have >> the timestamp of the abort not the source file. Therefore it will be >> newer than the source file and unless the source is updated rsync >> --update will never complete the file. > > Is there a solution to this problem and should we consider it a bug? > I am encountering this, and it's a real problem for me if files are > not completed. > >> On 9/11/21 6:50 PM, hancooper via rsync wrote: >> >>> I am struggling to understand exactly what the rsync options --update and >>> --append-verify do. >>> Doing info rsync gives >>> -u, --update >>> This forces rsync to skip any files which exist on the destina‐ >>> tion and have a modified time that is newer than the source >>> file. (If an existing destination file has a modification time >>> equal to the source file’s, it will be updated if the sizes are >>> different.) >>> --append-verify >>> This works just like the --append option, but the existing data >>> on the receiving side is included in the full-file checksum >>> verification step, which will cause a file to be resent if the >>> final verification step fails (rsync uses a normal, non-append‐ >>> ing --inplace transfer for the resend). >>> I am using rsync to transfer directories recursively. There are times where >>> I have to stop the rsync transfer, and resume the transfer a few hours or >>> days later, without affecting the already transferred files at destination. >>> I also got some files that return errors by rsync, such as >>> rsync: read errors mapping >>> "/media/hc1/a1-chaos/amvib/IACL-2017-07-19T00:00:00-2017-07-19T23:59:59.mseed": >>> Input/output error (5) >>> I would like to retry the transfer of these files, at a later time too, >>> without affecting the files that had been transferred successfully. >> -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync options –update and –append-verify
--archive is all you really need. I actually wish --archive was the default because it is all most people need and with the exception of writing to a FAT filesystem it is almost always needed. --append is for very special cases and should only be used if you really know you need it and why. --append-verify only exists because people seem to think they need --append and get annoyed that it corrupts their files. --update is sometimes helpful however it interacts badly with --partial and --inplace. Since rsync doesn't set the timestamp of a file until it is finished then any file left incomplete by an aborted rsync will have the timestamp of the abort not the source file. Therefore it will be newer than the source file and unless the source is updated rsync --update will never complete the file. On 9/11/21 6:50 PM, hancooper via rsync wrote: > I am struggling to understand exactly what the rsync options --update and > --append-verify do. > > Doing info rsync gives > > -u, --update > This forces rsync to skip any files which exist on the destina‐ > tion and have a modified time that is newer than the source > file. (If an existing destination file has a modification time > equal to the source file’s, it will be updated if the sizes are > different.) > > --append-verify > This works just like the --append option, but the existing data > on the receiving side is included in the full-file checksum > verification step, which will cause a file to be resent if the > final verification step fails (rsync uses a normal, non-append‐ > ing --inplace transfer for the resend). > > I am using rsync to transfer directories recursively. There are times where I > have to stop the rsync transfer, and resume the transfer a few hours or days > later, without affecting the already transferred files at destination. > > I also got some files that return errors by rsync, such as > > rsync: read errors mapping > "/media/hc1/a1-chaos/amvib/IACL-2017-07-19T00:00:00-2017-07-19T23:59:59.mseed": > Input/output error (5) > > I would like to retry the transfer of these files, at a later time too, > without affecting the files that had been transferred successfully. > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question about rsync -uav dir1/. dir2/.: possib to link?
Yes, cpio -l can be useful since cpio can easily operate on the output from the very capable find command. On 9/4/21 8:34 PM, Dan Stromberg wrote: > > I was thinking --link-dest too. > > Sometimes this can be done with cpio too; check out the -pdlv options. > > On Sat, Sep 4, 2021 at 4:57 PM Kevin Korb via rsync > mailto:rsync@lists.samba.org>> wrote: > > Rsync does almost everything cp does but since it is designed to network > it never got that feature. I was thinking maybe --link-dest could be > tortured into doing it but if it can I can't figure out how. BTW, you > have some pointless dots in there. > > On 9/4/21 6:41 PM, L A Walsh via rsync wrote: > > I noticed in looking at download dirs for a project, that > > another mirror had "crept-in" for usage (where different mirrors > > are stored under mirror-URL names). To copy over the diffs, > > normally I'd do: > > rsync -uav dir1/. dir2/. > > (where dir1="the new mirror that I'd switched > > to by accident, and dir2=the original dir). > > > > The files were "smallish" so I just copied them, BUT I wass > > wondering if there was an option similar to using 'cp' for > > a dircopy, but instead of > > cp -a dr1 dr2 > > using: > > cp -al dr1 dr2 > > > > to just hard-link over files from "dir1" to "dir2" (both > > are on the same file system). > > > > I looked at (and tried) --link-dest=DIR > > (hardlink to files in DIR when unchanged), but either I had the syntax > > wrong, or didn't understand it as it didn't seem to do what I > > wanted: cp'ing the new files in dir1 into the orig dir). > > > > Does rsync have an option to just "copy" over the new > > files via a hardlink? > > > > Tnx! > > > > > > > > > > -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net (work) > Orlando, Florida k...@sanitarium.net > <mailto:k...@sanitarium.net> (personal) > Web page: https://sanitarium.net/ > <https://sanitarium.net/> > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > <https://lists.samba.org/mailman/listinfo/rsync> > Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > <http://www.catb.org/~esr/faqs/smart-questions.html> > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question about rsync -uav dir1/. dir2/.: possib to link?
Rsync does almost everything cp does but since it is designed to network it never got that feature. I was thinking maybe --link-dest could be tortured into doing it but if it can I can't figure out how. BTW, you have some pointless dots in there. On 9/4/21 6:41 PM, L A Walsh via rsync wrote: > I noticed in looking at download dirs for a project, that > another mirror had "crept-in" for usage (where different mirrors > are stored under mirror-URL names). To copy over the diffs, > normally I'd do: > rsync -uav dir1/. dir2/. > (where dir1="the new mirror that I'd switched > to by accident, and dir2=the original dir). > > The files were "smallish" so I just copied them, BUT I wass > wondering if there was an option similar to using 'cp' for > a dircopy, but instead of > cp -a dr1 dr2 > using: > cp -al dr1 dr2 > > to just hard-link over files from "dir1" to "dir2" (both > are on the same file system). > > I looked at (and tried) --link-dest=DIR > (hardlink to files in DIR when unchanged), but either I had the syntax > wrong, or didn't understand it as it didn't seem to do what I > wanted: cp'ing the new files in dir1 into the orig dir). > > Does rsync have an option to just "copy" over the new > files via a hardlink? > > Tnx! > > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Speeding up rsync
Most of the speedup is due to ssh being bound to a single CPU. You will get a little more speed out of a single CPU system but the speedup is unlikely to go past 2 in parallel. Even with that you may end up adding more latency via disk seeks to accomplish much. On 8/24/21 5:12 PM, hancooper wrote: > ‐‐‐ Original Message ‐‐‐ > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync > wrote: > >> In my experience backing up multiple hosts to a single host can speed >> things up. Especially if using rsync over ssh with multiple CPUs on the >> single host. You would need to do some experimentation to determine the >> best number for your hardware and network. Also, if you exceed that >> number you can put more stress on the backup machine lessening the >> impact to the hosts being backed up. >> >> On 8/24/21 3:58 PM, hancooper via rsync wrote: >> >>> It makes sense to we to run multiple rsync commands to speed up a backup. >>> At work, some have argued that if I sync the data all to the same host, >>> there is no advantage >>> in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes >>> no noticeable difference. >>> Speed in general does not increase by increasing the number of sync >>> processes. >>> However, if one want to sync to multiple hosts, the effort may be >>> worthwhile. >>> What is the current advice about speeding up data transfer ? >> >> -- >> Kevin Korb Phone: (407) 252-6853 > > > Hi Kevin. What about a single host to another with multiple CPU's ? > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Speeding up rsync
In my experience backing up multiple hosts to a single host can speed things up. Especially if using rsync over ssh with multiple CPUs on the single host. You would need to do some experimentation to determine the best number for your hardware and network. Also, if you exceed that number you can put more stress on the backup machine lessening the impact to the hosts being backed up. On 8/24/21 3:58 PM, hancooper via rsync wrote: > It makes sense to we to run multiple rsync commands to speed up a backup. > > At work, some have argued that if I sync the data all to the same host, > there is no advantage > in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes > no noticeable difference. > > Speed in general does not increase by increasing the number of sync > processes. > > However, if one want to sync to multiple hosts, the effort may be > worthwhile. > > What is the current advice about speeding up data transfer ? > > > > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: config
Rsync by default displays nothing. There are more than one options that tell it to display the files it is touching. There are other (and duplicate) options that tell it to show everything. You didn't say what options you are using so we have no idea. Except that you aren't using --itemize-changes which is the only option that will tell you why it is saying what it is saying. On 6/25/21 9:48 AM, Gerhard Obermayr via rsync wrote: > Hello everyone, > > Briefly to my problem - or wish: By lucky coincidence I came into > possession of an HP Proliant. > My junior installed OpenMediaVault for me. > I've been using it as a daily backup for my TeraStation ever since. > Rsync is set up for this. > Every day I get two emails from this backup, in which the complete > directory is listed that is due to be backed up. > But that's pretty confusing, because that's currently more than 8000 > lines in total. > And why? Because, among other things, I back up my directories with the > photos from one NAS to another. > And that's quite a lot of directories from the last 20 years ... > In between, individual files appear in individual directories that have > changed and have been saved as a result. > Isn't it possible that only those files appear in this mail that were > actually backed up because of the change? > That would be easier and clearer ... > Maybe there is already a solution - an additional option button - *to > only display the copied files in the report* ... > > What can i do? What parameter work right? Is there an option for that? > > -- > Liebe Grüße aus Haag - und Xund bleim! > > Gerhard Obermayr > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Permission denied errors with --append-verify run as non-root user
Short answer: Don't use --append[-verify] If you aren't syncing files that aren't tagged with 'chattr +a' you don't want --append. --append-verify is essentially "I think --append is a good idea to use on general files but rsync keeps corrupting stuff!" but it still allows files to be out of sync since it ignores any change to a file that doesn't make it a bigger file. Essentially it is for things like append only filesytem images and non-rotated log files. All of that being said, rsync by default only transfers what is different between the source and the target. If a file has gotten bigger only by adding stuff at the end rsync will figure that out and transfer only the stuff that matters. Any performance improvement you are getting from --append-verify is either because it is ignoring file changes that didn't result in a bigger file and because it causes file updates to work like --inplace (which is what is causing your problem). On 6/4/21 11:50 AM, Knight, Dave via rsync wrote: > I noticed that my latest "syncHome" script log scontained a number of > unexpected "Permission denied" errors for some modified files. The > cause seems to be that those files were read-only at the target. > > The syncHome script is used to maintain a partial mirror of my home > directory tree on another host. > > Further investigation and some experiments led me to suspect the > --append-verify functionality. Specifically, those files were not > rsync'd when the --append-verify rsync flag was specified but were > properly rsync'd when --append-verify was NOT specified. > > Other rsync scripts that use --append-verify and are run as root seem > not to exhibit this behavior. > > FYI, I downloaded and built the "latest" version (rsync version 3.2.3 > protocol version 31) and with this version of rsync, the modified files > are silently NOT rsync'd for both root and non-root users when > --append-verify is specified. When --append-verify is not specified, > the updated files are rsync'd correctly. > > That seems to me to be a serious bug, IMHO, because I "LIKE' to use > --append-verify to ensure that large files are copied entirely without > having to resync the entire file if there is an error. If the > --append-verify option */silently fails/* to rsync some updated files > that */should/* be rsync'd, I cannot use --append-verify! > > Please advise whether I am misunderstanding the purpose of, or misusing > the --append-verify flag or if this is a bug. It is really easy to > reproduce, BTW... > > Dave Knight > Radford, VA > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync: [generator] failed to set permissions : Operation not supported (95)
Those aren't really rsync errors they are just rsync telling you what
the kernel told it. The only thing I see wrong in your paste is that
you didn't use a trailing / on the source+target. Those do mean
something to rsync. The only thing I see wrong in the instructions is
the use of dd to make a file of a specific size. The truncate command
does that much better.
Anyway, check the mount options on the file system (use the mount
command it will show them) and the options on the filesystem itself with
tune2fs -l. My guess is that either ACLs or XATTRs are not enabled and
they probably are in use since the instructions told you to tell rsync
to copy those. You can also try running rsync without the -A and -X to
confirm.
On 4/6/21 9:50 AM, Mario Marietto via rsync wrote:
> Hello.
>
> I want to virtualize correctly Android 10 on top of my Jetson nano
> (arm64) using qemu and kvm on ubuntu 18.04. This is the tutorial that
> I'm following :
>
> https://github.com/antmicro/kvm-aosp-jetson-nano
> <https://github.com/antmicro/kvm-aosp-jetson-nano>
>
> everything went good until this command :
>
> |sudo rsync -avxHAX system-r{o,w}/
>
> |
>
> |something is not good because I get a lot of errors when I transfer the
> files and permissions from the source to the destination path (both are
> on the same disk and on the same ext4 partition. You can see the full
> log with the errors here :
>
> |
>
> https://pastebin.ubuntu.com/p/W9GjPCt8G4/
> <https://pastebin.ubuntu.com/p/W9GjPCt8G4/>
>
> the consequence of these errors is that when I try to emulate android
> with qemu like this :
>
> |qemu-system-aarch64 \ -enable-kvm \ -smp 4 \ -m 2048 \ -cpu host \ -M
> virt \ -device virtio-gpu-pci \ -device usb-ehci \ -device usb-kbd \
> -device virtio-tablet-pci \ -usb \ -serial stdio \ -display sdl,gl=on \
> -kernel aosp/Image \ -initrd aosp/ramdisk.img \ -drive
> index=0,if=none,id=system,file=aosp/system.img \ -device
> virtio-blk-pci,drive=system \ -drive
> index=1,if=none,id=vendor,file=aosp/vendor.img \ -device
> virtio-blk-pci,drive=vendor \ -drive
> index=2,if=none,id=userdata,file=aosp/userdata.img \ -device
> virtio-blk-pci,drive=userdata \ -full-screen \ -append
> "console=ttyAMA0,38400 earlycon=pl011,0x0900 drm.debug=0x0 rootwait
> rootdelay=5 androidboot.hardware=ranchu androidboot.selinux=permissive
> security=selinux selinux=1 androidboot.qemu.hw.mainkeys=0
> androidboot.lcd.density=160" |
>
>
> this is the error that I get :
>
> [ 2.532754] init: init first stage started!
> [ 2.535936] init: [libfs_mgr]ReadFstabFromDt(): failed to read fstab
> from dt
> [ 2.540632] init: [libfs_mgr]ReadDefaultFstab(): failed to find device
> default fstab
> [ 2.546246] init: Failed to fstab for first stage mount
> [ 2.549616] init: Using Android DT directory
> /proc/device-tree/firmware/android/
> [ 2.555116] init: [libfs_mgr]ReadDefaultFstab(): failed to find device
> default fstab
> [ 2.560762] init: First stage mount skipped (missing/incompatible/empty
> fstab in device tree)
> [ 2.566906] init: Skipped setting INIT_AVB_VERSION (not in recovery mode)
> [ 2.571227] init: execv("/system/bin/init") failed: No such file or
> directory
> [ 2.593768] init: #00 pc 000e90a0 /init
> [ 2.599958] reboot: Restarting system with command 'bootloader'
>
> --
> Mario.
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync support in authprogs - feedback requested
You should both look into rrsync. It comes with rsync and is designed to do exactly this. Unfortunately some Linux distros are maintained by insane people who install rrsync as if it was documentation (compressed and not executable) instead of a helper script which is what it is. On 2/18/21 10:28 AM, Karl O. Pinc via rsync wrote: > On Wed, 17 Feb 2021 21:52:06 -0800 > Bri Hatch via rsync wrote: > >> I recently added initial rsync support to authprogs. > >> I'd be very interested in feedback > > For some 15 years+ (?) I've had a /root/.ssh/authorized keys line > that starts with: > > "no-pty,no-agent-forwarding,no-port-forwarding,no-user-rc,no-X11-forwarding,command="rsync > --server --daemon ." > > Occasionally I frob the ssh restrictions as new ones are > introduced. > > The remote end uses rsync to backup (with --link-dest) the > entire file system. The idea (iirc) was to restrict > the given key so that it would only run rsync. > And I think this also forces the local end to use > /etc/rsyncd.conf, where there's an additional layer > of security via a secrets file and read-only can > be set to provide some control. > > The remote end always runs rsync -- the direction of > transfer is static, per-host-pair, but can be either > in or out. (Push or pull backups.) The above authorized_keys > line does not enforce direction, which might be useful. > > I only rarely think about tweaking the authorized_keys line, > and the rsync options used haven't changed since I got them to work. > Without really thinking about it it seems that your > authprogs development might be useful. > > My purpose with this email is to let you do all the > thinking and tell me of all the wonderful utility > your authprogs work can provides, either now or > in the future. ;-) Or at least give you some > background in case you want to develop in a direction > that you think would helpful to me. If something comes > of this I might even turn my brain on again and > modify my systems. :) > > Regards, > > Karl > Free Software: "You don't pay back, you pay forward." > -- Robert A. Heinlein > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync failed to set times (on temporary files)
I am not sure if I have missed information here but... Rsync only sets the mtime on files that it completely transferred. The back-dating of a file is rsync's guarantee that that file was completely transferred and verified. On 2/2/21 9:03 PM, Leon Vanderploeg via rsync wrote: > Windows Cygwin rsync is version 3.1.1 > > Ubuntu version is 3.1.2 > > Ubuntu file system is ext4 > > Rsync command main options -rtlg0D ( see attachment for full view) > > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Preserve Windows Symlinks
Unison doesn't use rsync it only uses librsync (rsync's file diffing algorithm). Also, unison has a native Windows version so no *nix translation is needed. On 12/29/20 12:48 PM, Edwin Bradford wrote: > Sorry, I should have realized, that makes complete sense in hindsight. > I deleted the original source volume symlink and restored from the > external volume symlink and the same problem exists... Windows doesn't > recognise it as a symlink or appear to know what kind of file it is. > > Previously I was using Unison File Sync which itself uses rsync and > doesn't support preserving symlinks between Windows and Unix so I'm > wondering if it's just not possible in which case maybe I'll just have > to live with following symlinks --copy-links. > > > > + + + + + + + + + + + + + + + + + + > > Email: edwinbradf...@gmail.com > > Tel.: +44 (0)7981 873 771 > > + + + + + + + + + + + + + + + + + + > > > On Tue, 29 Dec 2020 at 16:39, Kevin Korb wrote: >> >> Not quite. I meant rsync a symlink then delete the source symlink and >> use rsync to restore it from what rsync made. If the target of the >> symlink doesn't exist on the other system (in the same place) then the >> symlink shouldn't work on the remote but rsync can still backup it up >> and restore it. At least that is how it works in *nix. >> >> On 12/29/20 11:34 AM, Edwin Bradford wrote: >>> Yes I found --checksum is much slower so following your feedback I'll do >>> some more reading to see if I really need it thanks. Assuming I >>> understand correctly I did the following test and also replaced absolute >>> symlink paths with relative symlink paths which worked this time: >>> >>> 1. I deleted and recreated the Windows symlinks on the source volume >>> using relative paths. >>> 2. I deleted and recreated the same symlinks directly on the destination >>> volume. >>> 3. I ran the same rsync command with the --archive flag (preserve symlinks). >>> >>> The result was the symlinks on the destination volume were preserved or >>> at least not destroyed by the subsequent sync. If this test is what you >>> intended then it shows rsync can preserve the symlink between NTFS >>> volumes but I would have to manually recreate the symlinks on the >>> destination NTFS volume once. >>> >>> Have I understood correctly? >>> >>> >>> On Tue, 29 Dec 2020 at 13:53, Kevin Korb via rsync >>> mailto:rsync@lists.samba.org>> wrote: >>> >>> If you delete the link then restore it does it start working again when >>> in the correct place? >>> >>> Also, don't use --checksum unless you are really certain you understand >>> how terribly slow it is and how it doesn't actually accomplish much of >>> anything (certainly not any kind of data verification). >>> >>> On 12/29/20 7:29 AM, Edwin Bradford via rsync wrote: >>> > Is it possible to preserve Windows symlinks when backing up from >>> NTFS to >>> > NTFS volumes or any other for that matter? >>> > >>> > I am running rsync in Ubuntu in Windows Subsytem for Linux on >>> Windows 10 >>> > backing up a local NTFS volume to an external NTFS volume (and >>> also to a >>> > remote Unix server). >>> > >>> > The source NTFS volume has symlinks created in Windows using: >>> > >>> > mklink the\link the\file >>> > mklink /d the\link the\directory >>> > >>> > I am running rsync within a long shell script with the command: >>> > >>> > rsync >>> > --verbose --archive --checksum --itemize-changes --human-readable >>> > --delete --delete-excluded --partial --progress --stats >>> > --exclude-from='ignore.txt' >>> > source/ destination >>> > >>> > The symlinks are copied to the external NTFS partition and have >>> the same >>> > name but no functionality. Windows doesn't recognise them as symlinks, >>> > doesn't know what to do with them and there is no apparent difference >>> > between symlinked directories and files. >>> > >>> > I tried creating the symlinks with relative paths on the source volume >>> > but rsync generated an error and refu
Re: Preserve Windows Symlinks
Not quite. I meant rsync a symlink then delete the source symlink and use rsync to restore it from what rsync made. If the target of the symlink doesn't exist on the other system (in the same place) then the symlink shouldn't work on the remote but rsync can still backup it up and restore it. At least that is how it works in *nix. On 12/29/20 11:34 AM, Edwin Bradford wrote: > Yes I found --checksum is much slower so following your feedback I'll do > some more reading to see if I really need it thanks. Assuming I > understand correctly I did the following test and also replaced absolute > symlink paths with relative symlink paths which worked this time: > > 1. I deleted and recreated the Windows symlinks on the source volume > using relative paths. > 2. I deleted and recreated the same symlinks directly on the destination > volume. > 3. I ran the same rsync command with the --archive flag (preserve symlinks). > > The result was the symlinks on the destination volume were preserved or > at least not destroyed by the subsequent sync. If this test is what you > intended then it shows rsync can preserve the symlink between NTFS > volumes but I would have to manually recreate the symlinks on the > destination NTFS volume once. > > Have I understood correctly? > > > On Tue, 29 Dec 2020 at 13:53, Kevin Korb via rsync > mailto:rsync@lists.samba.org>> wrote: > > If you delete the link then restore it does it start working again when > in the correct place? > > Also, don't use --checksum unless you are really certain you understand > how terribly slow it is and how it doesn't actually accomplish much of > anything (certainly not any kind of data verification). > > On 12/29/20 7:29 AM, Edwin Bradford via rsync wrote: > > Is it possible to preserve Windows symlinks when backing up from > NTFS to > > NTFS volumes or any other for that matter? > > > > I am running rsync in Ubuntu in Windows Subsytem for Linux on > Windows 10 > > backing up a local NTFS volume to an external NTFS volume (and > also to a > > remote Unix server). > > > > The source NTFS volume has symlinks created in Windows using: > > > > mklink the\link the\file > > mklink /d the\link the\directory > > > > I am running rsync within a long shell script with the command: > > > > rsync > > --verbose --archive --checksum --itemize-changes --human-readable > > --delete --delete-excluded --partial --progress --stats > > --exclude-from='ignore.txt' > > source/ destination > > > > The symlinks are copied to the external NTFS partition and have > the same > > name but no functionality. Windows doesn't recognise them as symlinks, > > doesn't know what to do with them and there is no apparent difference > > between symlinked directories and files. > > > > I tried creating the symlinks with relative paths on the source volume > > but rsync generated an error and refused to complete. Using > --copy-links > > instead of --links (implicit in --archive) gives the expected > behaviour > > but I would prefer to preserve symlinks rather than replace them with > > their linked contents. > > > > At this point I'm not sure if it's a limitation of rsync and NTFS > > compatibility or whether I'm missing something. > > > > + + + + + + + + + + + + + + + + + + > > > > Email: edwinbradf...@gmail.com <mailto:edwinbradf...@gmail.com> > <mailto:edwinbradf...@gmail.com <mailto:edwinbradf...@gmail.com>> > > > > Tel.: +44 (0)7981 873 771 > > > > + + + + + + + + + + + + + + + + + + > > > > -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net (work) > Orlando, Florida k...@sanitarium.net > <mailto:k...@sanitarium.net> (personal) > Web page: https://sanitarium.net/ > <https://sanitarium.net/> > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > > -- > Please use reply-all fo
Re: Preserve Windows Symlinks
If you delete the link then restore it does it start working again when in the correct place? Also, don't use --checksum unless you are really certain you understand how terribly slow it is and how it doesn't actually accomplish much of anything (certainly not any kind of data verification). On 12/29/20 7:29 AM, Edwin Bradford via rsync wrote: > Is it possible to preserve Windows symlinks when backing up from NTFS to > NTFS volumes or any other for that matter? > > I am running rsync in Ubuntu in Windows Subsytem for Linux on Windows 10 > backing up a local NTFS volume to an external NTFS volume (and also to a > remote Unix server). > > The source NTFS volume has symlinks created in Windows using: > > mklink the\link the\file > mklink /d the\link the\directory > > I am running rsync within a long shell script with the command: > > rsync > --verbose --archive --checksum --itemize-changes --human-readable > --delete --delete-excluded --partial --progress --stats > --exclude-from='ignore.txt' > source/ destination > > The symlinks are copied to the external NTFS partition and have the same > name but no functionality. Windows doesn't recognise them as symlinks, > doesn't know what to do with them and there is no apparent difference > between symlinked directories and files. > > I tried creating the symlinks with relative paths on the source volume > but rsync generated an error and refused to complete. Using --copy-links > instead of --links (implicit in --archive) gives the expected behaviour > but I would prefer to preserve symlinks rather than replace them with > their linked contents. > > At this point I'm not sure if it's a limitation of rsync and NTFS > compatibility or whether I'm missing something. > > + + + + + + + + + + + + + + + + + + > > Email: edwinbradf...@gmail.com <mailto:edwinbradf...@gmail.com> > > Tel.: +44 (0)7981 873 771 > > + + + + + + + + + + + + + + + + + + > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync checksum issue
Either that or you have a hardware problem causing the checksum routine to return the wrong result. On 12/15/20 11:47 PM, Dan Stromberg via rsync wrote: > > On Tue, Dec 15, 2020 at 3:25 AM Laurent B via rsync > mailto:rsync@lists.samba.org>> wrote: > > Dear all, > > I'm encountering a problem with one of my backup. For some files, the > checksum calculation is failing leading to the following error : > > It sounds a little like a bug, but perhaps if you share the command > you're using for the backup? > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: A strange problem with my daily backups performed via rsync
A few things here... What filesystem is on /mnt/sony? Add a --itemize-changes to see what it says is going on. There really isn't a good reason to exclude lost+found. It should always be empty and take up no space. If there is something in it that is a sign that there is a problem with your filesystem and you may as well have rsync bring it to your attention. On 11/2/20 3:30 AM, Manish Jain via rsync wrote: > > Hi, > > I am facing a strange situation with my daily backups performed via > rsync. I primarily use Manjaro KDE Linux (LTS kernel), but also have > FreeBSD and Windows 10 bare-metal installations. > > I have an all-OS-writable ext2 partition /dev/sda2 mounted at /mnt/wall > > My USB backup device is a Sony SSD mounted at /mnt/sony > > This is the relevant line from my script which backs up /mnt/wall : > > rsync -aAHXv --delete \ > --exclude "lost+found" --exclude "System Volume Information" \ > /mnt/wall /mnt/sony/ > > Whenever the line above goes into action, it recursively deletes > /mnt/sony/wall and then copies /mnt/wall all over again. > > Any ideas why this might be happening ? Since /mnt/wall is 13GB, it > costs me time and SSD-wear to have /mnt/sony/wall recreated all over > again every time I do a backup. > > > Thanks, > Manish Jain > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: pattern geeks?
You need to include the dirs leading to there...
--include=httpd/, --include=httpd/conf.d/
Otherwise it will never look inside to see the cluster.d
(I am assuming you are right about lsyncd using the same patterns as rsync).
On 10/21/20 9:01 AM, lejeczek via rsync wrote:
> hi guys,
>
> I'm trying lsyncd with some patterns which should be just
> rsync's own patterns and I have these:
>
> rsync = { _extra = {
> "--include=samba**",
> "--include=postfix**",
> "--include=openvpn**",
> "--include=strongswan**",
> "--include=httpd/conf.d/cluster.d**",
> "--exclude=**",
> }
> }
>
> in my lsyncd.conf
> The rest of the config says the source dir, as well as the
> target, is '/etc'
> Now, stuff gets excluded and things as listed get included
> and go into target except for:
> "httpd/conf.d/cluster.d**"
>
> Obviously it isn't as simple as I presumed.
> How to make such sub-folder path included would you know?
> many thanks, L.
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Syncing multiple sub-directories to one directory
Interesting idea. It isn't something I have ever wanted to do. BTW, if
your find is recent use + instead of \;. + replaces {} with however
many entries fit in the command line length limit instead of running
individual rsync processes for each entry.
On 9/29/20 7:46 PM, Rob Campbell via rsync wrote:
> Thanks for your help. What you provided didn't work for me because that
> still placed things in subdirectories.
>
> I figured it out. This puts it all in the images directory.
>
> find /my/phone/root/dir/ \( -path '*Duo*' -o -path '*DCIM*' -o -path
> '*Pictures*' -o -path '*Camera*' -o -path "*Download*" -o -path
> "*textgram*" -o -path "*WhatsApp*" ! -path '*.textgram*' ! -path
> '*.thumbnails*' \) -type f \( -iname '*.jp*g' -o -iname '*.png' -o
> -iname '*.dng' -o -iname '*.raw' -o -iname '*.raw' \) -exec rsync -uz
> --progress "{}" /my/backup/directory/for/images/ \;
>
> ~
> When you are principled, set standards and stick to them some people will
> fall out of your life; let them!
>
>
> On Tue, Sep 29, 2020 at 1:33 PM Wayne Davison <mailto:wa...@opencoder.net>> wrote:
>
> On Tue, Sep 29, 2020 at 7:38 AM Rob Campbell wrote:
>
> I would like to sync many subdirectories into one directory with
> no subdirectories. I've tried
>
> rsync -rv --progress --include '*.jp*g' --include '*.png'
> --include '*.dng' --include '*.raw' --include '*.nef' --include
> 'Duo' --include 'DCIM' --include 'WhatsApp' --exclude '*'
> /my/phone/root/dir/ /my/backup/directory/for/images/
>
>
> You didn't specify what isn't working as you expect. That command
> works fine for the 3 listed dir names as long as you either don't
> have matching files inside /my/phone/root/dir (or as long as it's ok
> to also copy those files) and as long as there aren't subdirs or
> files named the same as the 3 top dirs you included. The latter can
> be fixed by anchoring your dir names and making them only match a
> dir (e.g. --include '/Duo/'). If you want to have the copy avoid
> files in the top dir you can either change all the file-based
> includes to have a "*/" prefix (such as "--include '*/*.png' ...")
> or you can change the dir includes into args and exclude '*/*'
> instead of "*". The last option would look like this (I also tossed
> in -i):
>
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --exclude '*/*'
> /my/phone/root/dir/{Duo,DCIM,WhatsApp} /my/backup/directory/for/images/
>
> That assumes you've got bash to do the brace expansion, but you
> could change that into 3 arg paths if you need to. You may also want
> to add --del if you want rsync to delete inside the 3 listed dirs.
>
> I'm not sure why you listed an extra command with "NewDir" when it's
> not mentioned in the first command. If that is an indication that
> you really want to copy all dirs under root/dir (not just the 3
> named dirs in the first command) then you could use an include of
> "/*/" to match any dir in the root of the transfer, like this:
>
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --include '/*/'
> --exclude '*' /my/phone/root/dir/ /my/backup/directory/for/images/
>
> One last suggestion, I like to make the args shorter by using -f
> (filter) commands, so an include example is -f '+ *.png' and an
> exclude example is -f '- /*/' (those are identical to the equivalent
> include/exclude args, so that's just personal preference).
>
> ..wayne..
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
signature.asc
Description: OpenPGP digital signature
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Back-up differences. Raspberry Pi and Fedora
On 9/29/20 7:07 AM, joe--- via rsync wrote: > unknown module Linux1 This would imply that "Linux1" is not defined in the rsyncd.conf file on the backup server. Of course since it is a NAS appliance I don't know if you have any access to the config file. 1 check you could run is 'rsync root@server::'. This would list the modules the server is configured with (assuming listing is allowed). -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Meaning of "failed verification -- update retained (will try again)."
Mostly it means that the file rsync ended up with on the target end didn't match the file on the source it started with. This is mostly caused by the file being modified on the source while rsync is copying it but it can also be a memory corruption problem. On 8/11/20 1:13 PM, Peng Yu via rsync wrote: > Hi, > > I see some warnings like the following. Could anybody explains what > they mean? Thanks. > > 19/31274477.pdf > 257,169,119 0%1.26MB/s0:03:15 (xfr#121, > to-chk=848309/1043298)WARNING: 18/32281577.pdf failed verification -- > update retained (will try again). > WARNING: 19/28879866.pdf failed verification -- update retained (will > try again). > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Daemon Remote Pull
user@x.x.x.x:port/ModuleName is not correct syntax. You may have something in your shell config translating that for you. Correct syntax is --port and user@host::module On 6/25/20 6:01 PM, Chandrasekar Natarajan via rsync wrote: > Hi, > > I am trying to pull folders from a windows remote machine using daemon > without SSH. > Below is the command that used: > rsync -vrtz --delete user@x.x.x.x:port/ModuleName '/cygdrive/d/backup/' > > No issue running the same command in Command Prompt.While executing this > command via c#.net Process startup, facing some issue. > > The exception is ': Unexpected remote arg: user@x.x.x.x:port/module > rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2] > > > Kindly help me to resolve this. > > -- > Thanks, > > *Chandrasekar N* > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: High memory usage - any way around it other than splitting jobs?
Unfortunately the hard links are the problem. In order to keep them straight rsync has to remember the details of every file it finds with a link count >1 making it grow and grow. Of course without -H rsync will end up duplicating them. On 6/25/20 10:30 AM, Andy Smith via rsync wrote: > Hi, > > I have a virtual machine with 2G of memory. On this VM there is a > directory tree with 33.3 million files in it. When attempting to > rsync (rsync -PSHav --delete /source /dest) this tree from one > directory to another on the same host, rsync uses all the memory and > is killed by oom-killer. > > This host is Debian oldstable so has > > $ rsync --version > rsync version 3.1.2 protocol version 31 > > The normal operation of this VM does not require more than 2G of > memory, but I doubled it to 4G anyway. Unfortunately rsync still > uses all the memory and is killed. > > Most advice I can find on decreasing rsync memory usage advises to > split the job up into batches. By issuing one rsync for each > directory within /source I was able to make this work. > > The interesting thing is though, the split of file numbers between > sub-directories is very uneven with the majority of them (31.5 > million of the 33.3 million) being in just one of the sub-directory > trees. I am kind of surprised that rsync has such a problem going > just that little bit further with the last 2 million. Is there any > scope for improvement with the incremental recursion code? > > If I upgraded the version of rsync could I expect this to work any > better? > > I could also give the host a massive swap file. It currently has > just 1G of swap, which all gets used in the failure case. I could > add more but I fear that the job will go so slow it will not > complete in a reasonable time. > > I don't know if the -H option is causing extra memory usage here; > unfortunately it is necessary as there are hardlinks in there. > > Some years old advice says to disable incremental recursion with > --no-i-r. As incremental recursion was added to reduce memory usage > this seems counter-intuitive to me, but this advice is all over the > Internet… > > These are all things I will investigate before settling for the > "split into multiple jobs" approach; just wondered if anyone has any > shortcuts for me. > > Thanks, > Andy > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync sometimes decides to delete all data
The only time I have seen something like that was when I was using rsync
to backup a snapshot and something went wrong in the snapshot creation
resulting in me rsyncing an empty dir instead of a mounted filesystem.
Does the --stats output imply that rsync saw an empty dir?
On 6/9/20 2:46 AM, freebsd--- via rsync wrote:
> Hello,
>
> I run into some really strange case I haven't seen so far over the years.
> I syncing a lot of different macines, raspberry pis to a backup server
> nightly.
> Lately one of them misbehaves.
>
> Server rsync: rsync version 3.1.2 protocol version 31
> Client rsync: rsync version 3.1.3 protocol version 31
>
> Command used:
> rsync --numeric-ids --delete --delete-excluded --delete-after -f "merge
> ${BUFFER}" --stats -e "${SSH_CMD}" root@${BACKUP_HOST}:/ $BACKUPROOT/
>
> Sometimes when I run this script from the server it just decides to
> delete ALL files what it already have and sync is done.
>
> I run it again couple of minutes after it syncs everything.
>
> What can cause such a behavior? I can of course remove the delete
> directives but then the local copy will become messy over time and this
> perfectly works with all other machines.
>
> It is not even that it cannot connect with rsync timeout and then it
> decides to delete the data. It can connect then decides to delete all
> data and done.
>
> Rsync is run by root in any case so it's not a privilege problem.
>
> Any ideas?
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
signature.asc
Description: OpenPGP digital signature
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to diff two directories?
On 5/6/20 11:49 AM, Peng Yu wrote: > On 5/5/20, Kevin Korb via rsync wrote: >> Rsync != diff. > > What do you mean? I only need to know what files are not the same, but > I don't need to know what the differences are with the files. See diff -q > >> However, if that command lists a file something is different about it. >> add --itemize-changes to find out what. > > I see this output. What does ">f.s 2.txt" mean? Thanks. There is a whole section in the man page that explains what all of the itemize output means. This means the file size is different. > > $ rsync -avun --delete --itemize-changes /tmp/tmp.GH6z0oN8rn/a/ > /tmp/tmp.GH6z0oN8rn/b > sending incremental file list >> f.s 2.txt > > sent 73 bytes received 19 bytes 184.00 bytes/sec > total size is 4 speedup is 0.04 (DRY RUN) > > > ==> /tmp/tmp.GH6z0oN8rn/a/1.txt <== > 1 > > ==> /tmp/tmp.GH6z0oN8rn/a/2.txt <== > 2 > > ==> /tmp/tmp.GH6z0oN8rn/b/1.txt <== > 1 > > ==> /tmp/tmp.GH6z0oN8rn/b/2.txt <== > 22 > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to diff two directories?
Rsync != diff. However, if that command lists a file something is different about it. add --itemize-changes to find out what. On 5/5/20 8:47 PM, Peng Yu via rsync wrote: > Hi, > > I use this command. But it also shows the files that are the same. Is > there an option that can be specified so that only the differences are > printed? Thanks. > > rsync -avun --delete src dir > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Would you expect --perms -M--fake-super to set the file mode to the original one?
I don't believe it is possible. I think the misunderstanding stems from the fact that the permissions are even stored in the xattr. They don't need to be there but they may as well be. They don't take much space. The real question would be when rsync reads the file to restore it and the file perms are different than the ones in the xattr which set does it use? On 3/16/20 10:01 AM, Dimitrios Apostolou via rsync wrote: > Thanks. This is a bit counter-intuitive to me. So how would you tell > rsync to store the original permissions in the xattr, but do not touch > the real file mode? > > On Thursday, March 12, 2020 6:26:18 PM CET, Kevin Korb via rsync wrote: >> I would expect that the sending rsync would only send the perms provided >> modified by the --chmod. I wouldn't expect the receiver to even know >> the other permissions. >> >> On 3/12/20 1:23 PM, Dimitrios Apostolou via rsync wrote: >>> Thank you for the feedback, I'm glad to see that different people see >>> the issue >>> differently. As a followup question, what would you expect this to do: >>> >>> rsync --perms --chmod g+rX -M--fake-super src dst >>> >>> I would expect it to store the original permissions in the xattr, >>> while ... >> > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Would you expect --perms -M--fake-super to set the file mode to the original one?
I would expect that the sending rsync would only send the perms provided modified by the --chmod. I wouldn't expect the receiver to even know the other permissions. On 3/12/20 1:23 PM, Dimitrios Apostolou via rsync wrote: > Thank you for the feedback, I'm glad to see that different people see > the issue > differently. As a followup question, what would you expect this to do: > > rsync --perms --chmod g+rX -M--fake-super src dst > > I would expect it to store the original permissions in the xattr, while > modifying the real file mode according to the chmod. > > On Thursday, March 12, 2020 6:06:34 PM CET, Kevin Korb via rsync wrote: >> Permissions don't require super. Any place where permissions can't be >> stored certainly can't handle xattrs either. So, I wouldn't expect >> --fake-super to affect --perms at all. >> >> On 3/12/20 12:46 PM, Dimitrios Apostolou via rsync wrote: >>> rsync --perms -M--fake-super src dst >>> >>> For me, this command means that rsync should save the original perms >>> in the >>> xattr, and leave the real file mode to the umask default. Currently >>> it also >>> modifies the real file mode, and there is no way to store something >>> different ... >> > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Would you expect --perms -M--fake-super to set the file mode to the original one?
Permissions don't require super. Any place where permissions can't be stored certainly can't handle xattrs either. So, I wouldn't expect --fake-super to affect --perms at all. On 3/12/20 12:46 PM, Dimitrios Apostolou via rsync wrote: > rsync --perms -M--fake-super src dst > > For me, this command means that rsync should save the original perms in the > xattr, and leave the real file mode to the umask default. Currently it also > modifies the real file mode, and there is no way to store something > different > in the xattr. > > According to an old bug report that I found, more people would like > --fake-super to be a complete attribute emulation layer. > > https://bugzilla.samba.org/show_bug.cgi?id=7112 > > Do you agree? I'm in the process of implementing this as a bug fix to > rsync, > and would like to know if everybody agrees with this behaviour. The patch > would also modify the man page to document it under --perms. > > > Regards, > Dimitris > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question/comment about -n (dry run) flag of rsync
If you used -v then the very last line rsync outputs is: total size is ### speedup is ### (DRY RUN) -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., On Tue, 10 Mar 2020, T. Shandelman via rsync wrote: Date: Tue, 10 Mar 2020 17:26:41 -0500 From: T. Shandelman via rsync To: rsync@lists.samba.org Subject: Question/comment about -n (dry run) flag of rsync Rsync is a remarkably handy tool that I use virtually every day. But there is one thing about rsync that drives me totally crazy. Under the -n (dry run) flag, rsync seems to produce exactly the same output as without that flag. I cannot tell you how many times I sit and scratch my head long and hard,after I discover that my intended rsync operations did not actually happen. Until I finally remember that I ran rsync in dry-run mode. That's why! This is especially a problem for very long-running rsync jobs run in dry-run mode. It seems to me that when run in dry-run mode, rsync should display a warning at the very, very end, something like: *WARNING: None of the above operations have been actually performed, * *because you ran rsync in dry-run mode.* Or does rsync already have such a feature, and I am not aware of it? But if not, that is my vote for the next feature to be added. It should be a very, very easy fix. Todd S. Austin, Texas, USA -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Exclude-from file format?
Either format is correct. However, there is no quoting or escaping in the file. So: - filename1 - file name 1 or without the - On 1/3/20 1:45 PM, @lbutlr via rsync wrote: > I have seen two main styles in examples for using rsync with exclude-from. > > The first is simply a list of filename, one per line. > > The second is a list of filenames, one per line, prefixed with either a - or > a + to indicate exclude/include. Which is correct? > > Also, none show the correct style with a filname containing spaces, though I > assumed quoting is OK > > Exclude1.txt: > - filename1 > - “file name 1” > > Exclude.txt: > filename1 > “file name 1” > > Which of these will exclude both filename1 and “file name 1” anywhere they > appear in the source? > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluding a . directory in a subdirectory
Order matters. --include overrides --exclude listed later. So, your exclude is nullified by both of your includes. So move the exclude before the includes. Of course then your includes will be irrelevant so just remove them unless you are leaving out details. On 12/28/19 9:15 PM, H via rsync wrote: > I am having problems getting a . directory in a subdirectory to be excluded > when rsyncing. I am using rsync 3.1.2 on CentOS 7 and the command line is: > > # all files from ~/bin/ and I use want to exclude the directory > ~/bin/docker/.git > rsync -vHrltDium -c --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats > --include='*' --include='*/' --exclude=".git/" -e "ssh -y -p 22" > /home/user/bin/ user@1.2.3.4:~/bin > > What am I doing wrong? I thought the exclusion pattern above would exclude > any .git subdirectory regardless of level found? > > Thank you. > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skip creating files in --backup-dir if content has not changed
The reason for that is pretty simple... Rsync isn't reading the existing file to find out that it is the same. Doing so would normally be a waste of rsync's time. A better question is why are your files changing timestamps when the data is the same. On 12/18/19 6:47 PM, Wayne Piekarski via rsync wrote: > I am using rsync with --backup --backup-dir to keep copies of files > which have changed as part of an incremental backup system. However, if > only the timestamp has changed, it creates a copy of the file in > --backup-dir, and if thousands of large files have their timestamps > changed, this can waste a lot of disk space on something which hasn't > really changed. > > Interestingly, if you use --checksum, rsync will not create a file in > --backup-dir unless the contents are truly different, but it will fix up > the timestamp on the remote end to match. This is what I want, but I > just don't want to pay the performance penalty of running --checksum all > the time. > > Here is an example that shows the problem: > > mkdir ./SRC > echo hello > ./SRC/a > echo hello > ./SRC/b > rsync -av ./SRC/ ./DEST/ > touch ./SRC/* > ls -al --full-time ./SRC/ ./DEST/ > # Creates copies in BACKUP, even though contents are the same > rsync -av --backup-dir=`pwd`/BACKUP/ ./SRC/ ./DEST/ > > After this run, the BACKUP directory will contain copies of both a and b > even though neither actually changed. If you add --checksum, then it > avoids creating a copy, but still syncs the timestamps correctly. > > touch ./SRC/* > # Does not create any copies in BACKUP since nothing changed > rsync -av --checksum --backup-dir=`pwd`/BACKUP/ ./SRC/ ./DEST/ > > The problem with --checksum is that for hundreds of gigabytes of data, > it can be very slow to run over every file, especially if the timestamps > are mostly actually the same. But without it, the delta algorithm in > rsync has already decided to make a backup copy before it realizes later > that nothing has changed. > > Is there a flag I can add to rsync that will tell it to only create a > backup file if something actually changed, saving lots of wasted backup > space? > > thanks, > Wayne > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How Can I save the original permissions while setting g+rX for all files
When I try it the chmod works on both the real permissions and the permissions in the xatttr. Maybe the behavior has changed since whatever version you have (3.1.3 here) but this probably wouldn't help you either. On 12/8/19 7:56 PM, Dimitrios Apostolou via rsync wrote: > Hello list! > > Combining -M--fake-super with --chmod ends up changing the permissions > stored in the fake-super xattrs. I.e. the permissions stored in the > xattr, are > affected by --chmod. > > The desirable behaviour for me would be for --chmod to modify the real > permissions of the destination files. The use case is writing a backup as a > non-root user, having it readable by the group (g+rX), while also saving > the > original permissions in the xattr. > > Is it a bug? Any suggestions on how I can achieve what I want? > > > Thanks in advance, > Dimitris > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Seemingly impossible bug: -v not always listing every copied file
It does seem impossible. I would suggest adding --itemize-changes (-v isn't really all that useful without it anyway). If entries are still missing then I would suspect that either log files are missing (maybe duplicate file names replacing the occasional log file?) or something other than rsync is doing things in the same dir. On 10/29/19 9:00 PM, raf via rsync wrote: > Hi, > > debian-9, rsync-3.1.2 (both ends) > > I have a task that rsyncs files from a list of > candidate files (--files-from=). It's verbose (-v) and > its stdout is captured to a file which is then sent to > the receiving host. The captured verbose output is > examined on the receiving host to know which files were > actually copied so that notification emails can be sent > to various people. > > The problem is that, sometimes, not all copied files > are listed in the verbose output, and so some > notification emails don't get sent out. At first, I > thought there was something wrong with the notification > emails not arriving, but the files in question, that > had definitely been copied, did not appear in any of > the captured verbose output files. > > This seems like an impossible bug but it really seems > to be happening. Has anyone else encountered behaviour > like this? I didn't have much luck searching the > internet for it. It would probably be hard to notice > if the verbose output wasn't being used for something > like triggering notifications whose absence might be > noticed. > > cheers, > raf > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: difference in behaviour of --filter --delete: cmdline vs. file
It is a little confusing but in order to have an exclude dir merge filter do both things (prevent transfer and prevent deletion) then it needs to be on both ends of the transfer. The file on each end only has partial effect on what rsync does. Think of it like a file you (as a regular user) can make in your home dir telling the backup system to not backup a file. If that file has already been backed up rsync will also remove it from the backup on the next run. On 8/7/19 1:04 PM, Madhu via rsync wrote: > Hello, > > I want to avoid transferring a file which is present both on the source > tree and on the target tree when calling rsync with -a --delete. > > mkdir -pv src/ dst/ > echo file1 > src/file > echo file2 > dst/file > > rsync -navi --delete --exclude file src/ dst/ > > works as expected. The file is not transferred nor is it deleted at > the receiver. > > rsync -navi --delete --filter '- file' src/ dst/ > > also works as expected. The file is not transferred nor is it deleted at > the receiver. > > echo '- file' > src/.rsync-filter > rsync -navi --delete -F -F src/ dst/ > > file is now deleted at the receiver, which is unexpected. I expect this > to be the same as the earlier two invocations. > > Am I missing something - or is this a bug? > > I think the manual could be clearer on the s and r modifiers with regard > to --delete. For now neither of the following works to protect file > from being deleted at the receiver > > echo '-s file' > src/.rsync-filter > rsync -navi --delete -c -F -F src/ dst/ > > echo '-r file' > src/.rsync-filter > rsync -navi --delete -c -F src/ dst/ > > ---Madhu > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Advanced rsync includes and excludes
I believe you can shorten that to: + /some/very/ + /some/very/deep/ + /some/very/deep/path/ + /some/very/deep/path/to/ + /some/very/deep/path/to/save/*** - /some/* You could also exclude /some and then use /some/very/deep/path/to/save as an additional source. I don't know if rsnapshot can handle multiple sources in 1 rsync but rsync itself can. On 8/7/19 9:54 AM, Hans-Peter Jansen via rsync wrote: > Hi, > > I'm a happy camper @ rsync (and rsnapshot) since years. Thanks for this major > piece of software. > > In an attempt to reorganize my rsnapshot backups, I stumbled across an issue, > that I'm trying to seeking a more sophisticated solution here. > > Given, I have a deeply branched tree, where I would like to include a > specific > directory deep under, while excluding anything else on that path, I find > myself doing: > > + /some/ > + /some/very/ > + /some/very/deep/ > + /some/very/deep/path/ > + /some/very/deep/path/to/ > + /some/very/deep/path/to/save/ > - /some/very/deep/path/to/* > - /some/very/deep/path/* > - /some/very/deep/* > - /some/very/* > - /some/* > > While it works, it feels rather awkward and gets very complicated, if you > have > more of such items to deal with. Sure, I could run this separately, but this > isn't the real McCoy either with a complicated tree structure. > > What are your favorite ways of doing such tasks with rsync? > > Thanks in advance, > Pete > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
In addition to the cron problems that others have pointed out there are rsync problems... #1 rsync's -c is almost always a bad idea and is extremely slow #2 if rsync isn't networking then -z is a waste of time On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote: > Hi all! :-) > > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in top) > and the backup takes ~25 min. But, when I add the script in cron to > start it at 1am at night it takes 7 - 9 hours and I see up to > 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - > 60 waits. Why? Can someone explain why it takes so long when it starts > with cron? > > This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" > This is the entry in cron (crontab -e): * 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > > When I look in the log files I see errors like this: > > rsync: > rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) > > rsync error: some files/attrs were not transferred (see previousrsync error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > > > Can you help me to solve this problem? > > > > regards, > > Hannes Hutmacher > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: "WARNING: failed verification -- update retained (will try again)" message
Usually it means that the file changed while rsync was running. But it can also mean that a hardware problem (usually RAM) is causing the hashing to return bogus results. On 6/3/19 6:09 AM, sambalist--- via rsync wrote: > Hi there! > > On a daily backup script, I am getting a lot of these messages: > > WARNING: -- update retained (will try again). > > I've tried googling, but i can't figure out what these messages exactly > mean and how i can solve them. > > Any hint? > > Thank you > > Gabriele > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-missing-args doesn't delete
--files-from will delete files from the destination that are listed but
not existing in the source. It isn't delete what isn't listed.
On 4/23/19 5:29 AM, MI via rsync wrote:
> I'm generating a list of files to sync, and would like all the files not
> in my list to be deleted from the destination. I thought that
> --delete-missing-args would do just that, but it doesn't delete anything.
>
> Would someone have an idea of what I'm doing wrong?
>
> This is what I tried :
>
> ~$ mkdir -p /tmp/source /tmp/dest
>
> ~$ for i in {1..3}; do echo "keep $i"Â Â Â Â Â > /tmp/source/keep$i; done
>
> ~$ for i in {4..5}; do echo "to remove $i" > /tmp/source/to-delete-$i; done
>
> ~$ rsync -va /tmp/source/ /tmp/dest/
> sending incremental file list
> ./
> keep1
> keep2
> keep3
> to-delete-4
> to-delete-5
>
> sent 410 bytes received 114 bytes 1,048.00 bytes/sec
> total size is 45Â speedup is 0.09
>
> ~$ for i in {1..3}; do touch /tmp/source/keep$i; done
>
> ~$ cd /tmp/source/
>
> /tmp/source$ find . -name "keep*" | tee /tmp/source/file-list
> ./keep2
> ./keep1
> ./keep3
>
> Now, using my file-list which does not include the "to-delete*" files, I
> hoped that these would be deleted. The new files were correctly copied,
> but nothing was deleted:
>
> /tmp/source$ rsync -vva --delete --delete-missing-args --force
> --files-from=/tmp/source/file-list . /tmp/dest/
> building file list ... done
> delta-transmission disabled for local transfer or --whole-file
> ./
> keep1
> keep2
> keep3
> total: matches=0Â hash_hits=0Â false_alarms=0 data=21
>
> sent 259 bytes received 143 bytes 804.00 bytes/sec
> total size is 21Â speedup is 0.05
>
> /tmp/source$ ls -Al /tmp/dest/
> total 20
> -rw-rw-r-- 1 mi mi 7 Apr 22 23:43 keep1
> -rw-rw-r-- 1 mi mi 7 Apr 22 23:43 keep2
> -rw-rw-r-- 1 mi mi 7 Apr 22 23:43 keep3
> -rw-rw-r-- 1 mi mi 12 Apr 22 23:42 to-delete-4
> -rw-rw-r-- 1 mi mi 12 Apr 22 23:42 to-delete-5
>
>
> The files not in my --files-from list are still at the destination.
>
> Thanks for any help,
>
> MI
>
>
> PS: my version of rsync :
>
> ~$ rsync --version
> rsync version 3.1.1 protocol version 31
> Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
> Web site: http://rsync.samba.org/
> Capabilities:
> Â Â Â 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
> Â Â Â socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
> Â Â Â append, ACLs, xattrs, iconv, symtimes, prealloc
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone:(407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
signature.asc
Description: OpenPGP digital signature
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Improved output (dry run)
Having dry-run not act like the same command without --dry-run would make less sense than people occasionally running it and discovering that they need more options. -v doesn't show things it didn't touch. So, it touched them for some reason. Without --itemize-changes you are left in the dark about what is happening or why. But what you probably want is --omit-dir-times That being said, -v is virtually useless without --itemize-changes anyway. On 4/14/19 6:30 AM, darkdragon via rsync wrote: > Please improve (default) output for /--dry-run/. > > Specifically, it would be nice to imply /-v/ since dry runs without > displaying anything make no sense. > > Further (independently from dry run), it would be nice to have an option > to hide directories which did *not* change. The need to /grep/ results > is really frustrating and error prone. > > Thanks a lot! > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Possible not expected behaviour with "delete" option
It doesn't seem like the trailing / on the target should matter. It only means something special on the source. However, if there is any error then rsync will abort the deletions for safety. On 4/3/19 2:10 PM, lugarci1 via rsync wrote: > Hi, I was trying to make a mirror backup of a folder in my machine with > the followin|g command:| > |rsync -avh --delete /source/ /destination| > | > | > |But the extra files that where on "destination" didn't got deleted. > Where as doing:| > ||rsync -avh --delete /source/ /destination/|| > ||worked as expected.|| > || > || > ||I wonder if this is correct, or it shouldn't be like that.|| > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync rewrites all blocks of large files although it uses delta transfer
It can't do what you want. The closest thing would be --compare-dest. On 2/13/19 5:26 PM, Delian Krustev wrote: > On Wednesday, February 13, 2019 11:29:44 AM EET Kevin Korb via rsync > wrote: >> With --backup in order to end up with 2 files it has to write out a >> whole new file. >> Sure, it only sent the differences (normally that means >> over the network but there is no network here) but the writing end was >> told to duplicate the file being updated before updating it. > > The copy is needed for the comparison of the blocks as "--inplace" overwrites > the destination file. I've tried without "--backup" but then the delta > transfers too much data - close to the size of the backed-up files. > > The copy is in a temp file system which is discarded after the backup (by "rm > -rf"). This temp filesystem is not log structured or copy-on-write so having > a > copy there is not a big problem. Although I don't want a backup of all files > which are modified but rather a TMPDIR. > > The ideal workflow would be to compare SRC and DST and write changed blocks > to > the TMPDIR, then read them from TMPDIR and apply it to DST. > > > > > Cheers > -- > Delian > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync rewrites all blocks of large files although it uses delta transfer
With --backup in order to end up with 2 files it has to write out a whole new file. Sure, it only sent the differences (normally that means over the network but there is no network here) but the writing end was told to duplicate the file being updated before updating it. On 2/13/19 10:47 AM, Delian Krustev via rsync wrote: > Hi All, > > For a backup purpose I'm trying to transfer only the changed blocks of > large files. Thus I've run "rsync" with the appropriate options: > > RSYNC_BKPDIR=`mktemp -d` > rsync \ > --archive \ > --no-whole-file \ > --inplace \ > --backup \ > --backup-dir="$RSYNC_BKPDIR" \ > --verbose \ > --stats \ > /var/backups/mysql-dbs/. \ > /mnt/bkp/var/backups/mysql-dbs/. > > The problem is that although "rsync" shows that delta transfer is used(when > run with -vv) and only small amount if data is transferred, the target files > look to be overwritten in full. > > Here is the output of "rsync" and some more debugging info: > > > > sending incremental file list > ./ > horde.data.sql > horde.schema.sql > LARGEDB.data.sql > LARGEDB.schema.sql > mysql.data.sql > mysql.schema.sql > phpmyadmin.data.sql > phpmyadmin.schema.sql > > Number of files: 9 (reg: 8, dir: 1) > Number of created files: 0 > Number of deleted files: 0 > Number of regular files transferred: 8 > Total file size: 1,944,522,704 bytes > Total transferred file size: 1,944,522,704 bytes > Literal data: 21,421,681 bytes > Matched data: 1,923,101,023 bytes > File list size: 0 > File list generation time: 0.001 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 21,612,218 > Total bytes received: 323,302 > > sent 21,612,218 bytes received 323,302 bytes 259,591.95 bytes/sec > total size is 1,944,522,704 speedup is 88.65 > > # du -m 1.9G /tmp/tmp.8gBzjNQOQZ > 1.9G /tmp/tmp.8gBzjNQOQZ > > # tree -a /tmp/tmp.8gBzjNQOQZ > /tmp/tmp.8gBzjNQOQZ > ├── horde.data.sql > ├── horde.schema.sql > ├── LARGEDB.data.sql > ├── LARGEDB.schema.sql > ├── mysql.data.sql > ├── mysql.schema.sql > ├── phpmyadmin.data.sql > └── phpmyadmin.schema.sql > > 0 directories, 8 files > > Free space at the beginning and end of the backup: > Filesystem 1M-blocks Used Available Use% Mounted on > /dev/mapper/bkp 102392 76872 20400 80% /mnt/bkp > /dev/mapper/bkp 102392 78768 18504 81% /mnt/bkp > > > > As can be seen "rsync" has sent about 20M and received 300K of data. However > the filesystem has allocated almost 2G, which is the total size of the files > being backed up. > > The filesystem mounted on "/mnt/bkp" is of type "nilfs2", which is a log > structured filesystem. I'm using its snapshotting feature to keep backups for > past dates. > > > Is there anything that can be done in order "rsync" to overwrite only the > changed blocks ? > > > > > P.S. I guess that it will be the same for copy-on-write filesystems, e.g. > BTRFS or ZFS. > > > > Cheers > -- > Delian > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --server options
I don't remember your original question but if I didn't suggest looking at rrsync I should have. It comes with rsync in the contrib dir and it knows which options to allow/deny and it can restrict the transfer to a specific dir and read-only or write-only. Since it comes with rsync it is kept current with the options rsync uses including the undocumented ones. Even if it doesn't do exactly what you are trying to do, it is a perl script so it should be easy enough to modify for your needs. Just keep a diff so you can easily analyze changes in new versions. On 2/7/19 10:57 PM, Richard Hector via rsync wrote: > On 21/01/19 8:07 PM, Richard Hector via rsync wrote: >> Hi all, >> >> I see a couple of earlier threads, particularly this one: >> >> https://www.mail-archive.com/rsync@lists.samba.org/msg32328.html >> >> partly answer my question, but not fully. >> >> I also am writing a wrapper, so that I can run multiple dirvish runs >> against the same server, using forced commands, but without using >> multiple keys to get multiple commands, which is my current workaround. >> >> But in the interests of locking things down, I'd like to know what the >> options are in the -eSTRING section, so I can choose which to >> allow/force. Is that documented anywhere? Or do I have to resort to the >> source? >> >> And/or can I safely assume that anything in the -eSTRING won't let a >> client get any more than I intended? Does the STRING just run from after >> the -e to the next whitespace? > > Anybody? Are these options/flags deliberately kept obscure? > > Cheers, > Richard > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest. Time to 'building file list' incrementing
It does normally take some time to analyze large trees of files. It has to call stat() on each file to get the size and timestamp. However, 15 hours seems a bit excessive even though I have never tried to do this on Windows or a NAS system. Just to be clear, is your --link-dest parameter a single directory or are you trying to tell it to use all of the previous backups? Also, have you deleted a backup yet? In my experience that takes a lot longer than running one so if you need 15 hours to run a backup I would expect deleting one to take about a week. On 1/3/19 4:23 AM, John Simpson via rsync wrote: > > > I've been running rsync as a cygwin task on Windows Server 2008 for about two > months now. I'm using the --link-dest option to do a daily 'snapshot' of the > contents of a server containing about 10TB of data, about 13 million files, > to a Linux based NAS server. Things started out great but I soon noticed that > the time take to complete was slowly incrementing. It started at around three > hours, but is now around fifteen. > > The command is as follows... > > rsync -rlptDhPR \ > --password-file=password \ > --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ > --Stats \ > --delete \ > --log-file=logfilename \ > --link-dest=linkdestpath \ > sourceDirectory \ > rsync@192.168.1.2::destinationDirectory > > I'm not using the full -a option as the differences between the Windows and > Linux ownership stuff messed things up. > > The first log file looked like this... > > 2018/10/01 23:00:14 [2164] building file list > ...transfer file list here > 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: > 1,499,821, link: 1) > 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) > 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 > 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes > 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes > 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes > 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes > 2018/10/02 02:11:30 [2164] File list size: 10.42M > 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds > 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds > 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M > 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M > 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes 21.17K > bytes/sec > 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 > > the most recent looks like this... > > 2018/11/24 23:00:15 [2924] building file list > 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ > 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: > 1,501,780, link: 1) > 2018/11/25 13:21:16 [2924] Number of created files: 0 > 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 > 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes > 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes > 2018/11/25 13:21:16 [2924] Literal data: 0 bytes > 2018/11/25 13:21:16 [2924] Matched data: 0 bytes > 2018/11/25 13:21:16 [2924] File list size: 10.35M > 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds > 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds > 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M > 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M > 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K > bytes/sec > 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96 > > As you can see the start time is 11:00PM (23:00) in both cases. The first log > shows that identifying the files to transfer took about three hours (I've > omitted the file list - it's quite long), the second log takes fourteen hours > to do the same job (in this case this was done at the weekend and I've > include the whole log file which correctly identifies that no files have > changed) > > The number of files is as might be expected as is everything else. It's just > the time taken "building file list" is significantly larger. > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
