Re: prevent filter excluded files from being deleted

[Robin Lee Powell via rsync]

On Tue, Mar 07, 2023 at 01:16:07PM +0100, Heiko Schlittermann via rsync wrote:
> Robin Lee Powell via rsync  (Di 07 Mär 2023 07:07:01 
> CET):
> > Read the "PER-DIRECTORY RULES AND DELETE" of the man page.  (And
> > don't feel bad, it took me a while to figure it out myself).
> 
> I did, but left with some uncertainty.
> 
> "H" hides the files from the transfer? What does it mean?
> "P" protects files from being deleted? 
> 
> I even tried '+r *', no luck.
> 
> > Short version: per-directory rules only apply on the side they're
> > *specified on*, but you need the exclusion to apply to *both* sides.
> 
> Hm. Maybe more reasoning behind.
> 
> I'm syncing a quite large tree, but some directories contain huuge VM images,
> which I need to transfer using another mechanism. Because rsync doesn't
> seem to handle filesizes of about 100…500GiB well.
> 
> These directories should contain a `.rsync-filter` to prevent them from
> the sync, but rsync should not remove these same-named files on the
> remote side.

First of all, I disagree that rsync handles very large files badly,
but that's not super relevant. :)  Consdier --partial for large
files.

Anyway.

Note that:

(1) my .rsync-filter file has "+ .rsync-filter" in it
and
(2) --delete-after means that the .rsync-filter is present on *both
sides* when the delete runs

Exclusion on the *sender* side means "don't send this"; exclusion on
the *receiver* side means "don't delete this".

*Normally*, they're the same, like if you do --exclude=foo in the
command, that affects *both* the sender and receeiver side, but with
-F, the sender side uses the .rsync-filter in its directory, and the
receiver side uses the .rsync-filter in *its* directory.

So if you want the files to be excluded from sending *and* deleting,
the .rsync-filter that excludes the files has to be present on
*both* sides.

You can do that by just manually making them in both places, or by
copying what I had in my email.


> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: prevent filter excluded files from being deleted

[Robin Lee Powell via rsync]

Read the "PER-DIRECTORY RULES AND DELETE" of the man page.  (And
don't feel bad, it took me a while to figure it out myself).

Short version: per-directory rules only apply on the side they're
*specified on*, but you need the exclusion to apply to *both* sides.

The following works, for the reasons specified there:

rlpowell@stodi> find a b -ls
703572129  0 drwxr-xr-x   2 rlpowell rlpowell   39 Mar  6 22:06 a
704372871  0 -rw-r--r--   1 rlpowell rlpowell0 Mar  6 21:56 a/file
704372873  4 -rw-r--r--   1 rlpowell rlpowell   20 Mar  6 22:06 
a/.rsync-filter
703572158  0 drwxr-xr-x   2 rlpowell rlpowell   39 Mar  6 22:06 b
704372870  0 -rw-r--r--   1 rlpowell rlpowell0 Mar  6 22:06 b/file
704372881  4 -rw-r--r--   1 rlpowell rlpowell   20 Mar  6 22:06 
b/.rsync-filter
rlpowell@stodi> cat a/.rsync-filter
+ .rsync-filter
- *
rlpowell@stodi> rsync -F --delete-after -iav a/ b/
building file list ... done

sent 102 bytes  received 11 bytes  226.00 bytes/sec
total size is 20  speedup is 0.18
rlpowell@stodi>

On Mon, Mar 06, 2023 at 10:04:52PM +0100, Heiko Schlittermann via rsync wrote:
> Hello,
> 
> given are 2 directories:
> 
> a
> ├── a-file
> └── .rsync-filter
> 
> b
> └── a-file
> 
> I'd like to sync a/ -> b/, but I'd like to *exclude* all files. But I do
> not want to delete the excluded files. (The real scenario is a way more
> complex, the above is my reproducer.) 
> 
> and the following rsync command: `rsync -F --del a/ b/`
> Note, there is *no* `--delete-excluded`:
> 
> $ rsync -F --del -inav a/ b/
> sending incremental file list
> *deleting   a-file
> .d..t.. ./
> 
> sent 62 bytes  received 29 bytes  182.00 bytes/sec
> total size is 0  speedup is 0.00 (DRY RUN)
> 
> Why? Because it is excluded? Using the `--exclude` option does what I
> expect:
> 
> $ rsync --exclude 'a-file' --del -inav a/ b/
> sending incremental file list
> .d..t.. ./
> >f+ .rsync-filter
> 
> sent 92 bytes  received 22 bytes  228.00 bytes/sec
> total size is 4  speedup is 0.04 (DRY RUN)
> 
> even does what I expect:
> 
> $ rsync --filter '- *' --del -inav a/ b/
> sending incremental file list
> .d..t.. ./
> 
> sent 58 bytes  received 19 bytes  154.00 bytes/sec
> total size is 0  speedup is 0.00 (DRY RUN)
> 
> Version:
> rsync  version 3.2.3  protocol version 31
> 
> Best regards from Dresden/Germany
> Viele Grüße aus Dresden
> Heiko Schlittermann
> --
>  SCHLITTERMANN.de  internet & unix support -
>  Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
>  gnupg encrypted messages are welcome --- key ID: F69376CE -



> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Trying to diagnose incomplete file transfer

[Robin Lee Powell via rsync]

Oh, yeah, I missed that part.  Yeah, don't do that; it's easy to add
a lock file to a shell script.

On Sun, Mar 05, 2023 at 12:30:16PM +0100, Hardy via rsync wrote:
> I second Francis here. You don't need to diagnose incomplete file transfers 
> as long as you have racing conditions as you described. This leads to strange 
> result inevitably.
> NEVER start several rsync jobs manipulating the same data - especially if 
> there are modifications to BOTH sides source and destination.
> 
> You do not necessarily define a service like Francis suggests. A simple 
> semaphore approach suffices. Perhaps even something like
> 
> # ps fax | grep -v grep | grep $0 && exit
> 
> to prevent this exact command "$0" to start concurrently.
> 
> Hardy
> 
> Am 04.03.23 um 08:38 schrieb Francis.Montagnac--- via rsync:
> > 
> > Hi.
> > 
> > On Sat, 04 Mar 2023 00:39:52 -0600 Albert Croft via rsync wrote:
> > 
> > > The rsync commands may be launched from command-line or cron, but use
> > > the same format and options in either case. As a result, there may be
> > > multiple rsync processes pulling files from the same remote path to the
> > > same local path.
> > 
> > I think you should first prevent this to happen.
> > 
> > If your receiving machine is using systemd:
> > 
> >- define a X.service for doing the rsync
> >- define a X.timer unit to replace using cron
> >- launch from the command line with: systemctl start X
> >  - this will not start a new rsync if one runs already (if X.service is
> >running)
> > 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Trying to diagnose incomplete file transfer

[Robin Lee Powell via rsync]

I think it's very hard to be sure what's going on with
--remove-source-files ; I think you should drop that option, look
for whether the problem continues, and if you need the files to be
cleaned up, do so in a separate step.

In particular as someone else suggested, are you *sure* the original
copies of the partial files are actually the size you think they
are?

I don't ever recall seeing a case where rsync just failed to
transfer a file and thought it had succeeded; there pretty much has
to be something else going on.

On Sat, Mar 04, 2023 at 12:39:52AM -0600, Albert Croft via rsync wrote:
> At $work I have an odd situation involving incomplete file transfers, but I
> am unsure where the issue may be occurring. Here is the scenario.
> 
> Problem:
> Sometimes the file transfer seems to have completed, but the file size does
> not match that on the remote system.
> 
> 
> Details:
> I transfer a number of large (>1GB) Tar-Gzipped (.tgz) files via SSH tunnels
> from $customer. Because of some previous issues, sometimes the SSH tunnels
> may be terminated externally. As a result, I am currently using the 'split'
> command to break the files into 1-GB "chunks" (ex.: foo.tgz.aa, foo.tgz.ab,
> ...).
> 
> For the rsync transfer, I am using the following options:
> rsync -az \
> -e "ssh ..." \
> --link-dest=/local/path1 \
> --link-dest=/local/path2 \
> --remove-source-files \
> user@remote:/path/to/files \
> /local/path1/
> 
> where
>   '-e "ssh ..."' is the set of SSH options (for tunneling, etc.).
>   '--link-dest=/local/path1' refers to a local directory that might 
> contain a
> copy of the file.
>   '--link-dest=/local/path2' refers to a local directory that might 
> contain a
> copy of the file.
> 
> I am frequently encountering times where the file appears to have been
> transferred but is incomplete. (Example: foo.tgz.ab now exists on the local
> system, has been removed from the remote, but is incomplete.)
> 
> 
> Additional notes:
> To my knowledge I do not know if the 'gzip' '--rsyncable' option is being
> used (but I do not think so--I suspect the file is created using a command
> similar to 'tar czf foo.tgz ...').
> 
> The rsync commands may be launched from command-line or cron, but use the
> same format and options in either case. As a result, there may be multiple
> rsync processes pulling files from the same remote path to the same local
> path.
> 
> I know that when rsync transfers a file (ex.: foo.tgz.ab) that during the
> transfer process it is named '.foo.tgz.ab.??' (where '.??' is a
> 6-character unique extension), and that upon completion the file is renamed
> to 'foo.tgz.ab'. (So I may see .foo.tgz.ab.4e67d0 and .foo.tgz.ab.fa7325 in
> the directory while the transfers are going.)
> 
> 
> I am unsure if this is a result of the combination of options I am using, or
> where to begin troubleshooting. Any guidance or direction would be
> appreciated.
> 
> -Albert C.
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync verify its writes?

[Robin Lee Powell via rsync]

That's not the same as a read-back write verification.

I believe that in general, rsync assumes that the disk actually
wrote whatever it was told to write.

However, a second pass with --checksum will, in fact, read the
entirety of both files; if a --checksum run doesn't actually
transfer anything, you can safely conclude the files are
byte-for-byte[1] identical (or you have a serious low-level problem
where the disk is lying to you, which rsync is definitely not going
to be able to fix).

[1]: Technically they might not be byte-for-byte identical if you
somehow get identical hashing blocks, but Schneier once compared
that sort of thing to the survival time of a mouse on the surface of
the sun, so  :D

On Fri, Feb 17, 2023 at 11:18:59AM +0100, anubis23 via rsync wrote:
> Hi,
> 
> you can use the switch --fsync to verify the transfer. From the manpage:
> 
> --fsync
>   Cause the receiving side to fsync each finished file.
> This may slow down the transfer, but can help to
>   provide peace of mind when updating critical files.
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Can rsync write to a FIFO?

[Robin Lee Powell via rsync]

Yeah, I was thinking tar, but it's effectively the same thing.

On Sat, Feb 11, 2023 at 07:14:57PM +0100, Hardy via rsync wrote:
> If this helps, in old days I used to use cpio for a similar thing.
> 
> I do not want to spam you with my whole script, but willing to share if you 
> want. I think you will get the hang of it by the following snippet. (Get 
> yourself man-knowledge about the -i -o -p mechanism of cpio and the use of 
> dd.) This was in the good (?) old days when rsh worked as simple (and 
> insecure) as this. In modern *n*x like systems rsh is a link to ssh, which is 
> (besides being entirely wrong!) a pitfall to finding correct cli arguments. 
> But it is manageable if you are aware of it.
> 
> CPIOP = parameter arguments to cpio
> /tmp/$$.f = list of files
> 
> Snippet:
> 
> case $CPIOP in
>   -i*) rsh -l $RUSER $RHOST dd if=$RDEV | cpio $CPIOP
> ;;
>   -o*)
> cpio $CPIOP  ;;
>   -p)
> cpio -ocv  ;;
>   *)  echo argument mismatch $CPIOP >&2
>   exit
>   ;;
> esac
> 
> Hope this gives an idea
> 
> Hardy
> 
> Am 10.02.23 um 10:31 schrieb Chris Green via rsync:
> > I have searched a little and read the man page but I can't really find
> > a good definite answer to this.
> > 
> > Can rsync write to a FIFO?  Obviously one needs the --inplace to do
> > this, does one also need --write-devices?
> > 
> > It would be very handy if one can do this, to use as a simple message
> > passing mechanism.  Write something to a file on system A and rsync it
> > to a FIFO on system B where there is a simple script reading the FIFO.
> > The script gets the contents of the file every time it's written.
> > 
> > (this is all within a LAN behind a reasonably secure firewall)
> > 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: quick question: --rsh="ssh -l root"

[Robin Lee Powell via rsync]

No.

On Sun, Aug 07, 2022 at 06:49:45AM +0200, Fourhundred Thecat via rsync wrote:
> Hello,
> 
> is there any difference/advantage between these two commands?
> 
>   rsync --rsh="ssh -l root"  my-host.com
>   rsync r...@my-host.com
> 
> thank you,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync slowness

[Robin Lee Powell via rsync]

I've definitely not seen that.  If you can produce a working example
and tar it up for us to look at, that might be interesting/useful.

Just to check, though: you do not have --checksum/-c on, right?

On Sat, Aug 06, 2022 at 05:54:12PM +, Gregory Heytings via rsync wrote:
> 
> I finally take the time to report an rsync slowness pattern that I've been
> seeing for years.
> 
> Assuming:
> 
> a directory with many (> 20K) files, for example a maildir, on the sending
> side, and
> 
> a partial copy of that directory on the receiving side, with "enough"
> missing missing (say 5K),
> 
> then the receiving side will do the following: it will take about one minute
> to start transferring the missing files, and will apparently hang about one
> minute every 200 files or so.  After a few (about 10 or 20) iterations, the
> receiving side apparently hangs.
> 
> On the receiving side rsync is using 100% CPU, on the sending side not more
> than a few percents.  In case it matters, both sides are using ext4
> filesystems, and Debian GNU/Linux.
> 
> I tried --msgs2stderr -M--msgs2stderr, but that does not print any error
> message, I also tried to disable compression, to use --whole-file, to use
> --no-inc-recursive, ..., to no avail.
> 
> Any hints?
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there a better way to transfer data that doesn't use so much cache?

[Robin Lee Powell via rsync]

On Wed, Aug 03, 2022 at 02:04:22PM -0400, Rob Campbell via rsync wrote:
> I've created a script that syncs (and removes) data from as many as 4
> places and puts them all in one of 2 directories.  The commands are:
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
> -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *'
> "$D850/DCIM/100ND850/" $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
> -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$Z9/DCIM/100NCZ_9/"
> $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ /*' -f'- *' "$DASHCAM/CARDV/VIDEO/" $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'-
> Screenshots/' -f'+ *.nef' -f'+ *.jpg' -f'+ *.jp*g' -f'+ *.png' -f'+ *.dng'
> -f'+ *.gif' -f'- *.thumbnails' -f'- *.android' -f'+ */' -f'+ DCIM/*' -f'+
> Snapbridge/*' -f'+ Pictures/*' -f'+ Download/*' -f'+ Textgram/*' -f'- *'
> $PHONE/ $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'- *'
> --files-from=<(find $PHONE -type f ! -path "*Download*" ! -path
> "*.trashed*" ! -iname .mp4 ! -iname
> '*.mp4\.*')/ $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'+ Movies/*' -f'+
> *Recordings/*' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'- */' -f'- *' $PHONE/
> $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'- *' --files-from=<(find $PHONE -iname
> .mp4) / $STAGINGV/TIKTOK/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *'
> $PHONE/Downloads/ $COMPUTER/Downloads/
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'-
> screenshot*' -'f- Screenshot*' -f'- Boondocks/' -f'- Dilbert/' -f'+ *.png'
> -f'+ *.jp*g' -f'+ *.dng' -f'+ *.gif' -f'- *20*/' -f'- *' -f'+ */' -f'-
> $STAGINGP/' $MYPICS/ $STAGINGP/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+
> Screenshot*.png' -f'- Staging/' -f'- *' $MYPICS/ $STAGINGP/Screenshots/ |
> tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.3gpp'
> -f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'- *'
> $HOME/Downloads $STAGINGV/ | tee -a $LOG
> 
> rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
> -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'+ *.3gpp' -f'- *'
> $MYVIDEOS/ $STAGINGV/ | tee -a $LOG
> 
> 
> The problem isn't that there are many syncs because the problem happens on
> the first one that runs.

You didn't actually say what the problem *is*.

I can infer from the subject that you think it's bad that rsync is
using a bunch of disk/buffer cache, but that's not rsync, that's
Linux, and it's by design; Linux uses as much RAM as it possibly can
for disk cache, always.  This improves performance.  In a
well-performing Linux system, the "free" column of "free -h" is very
low, and the "available" column is very high.

> Before any of them run I run:
> 
> sudo free -w -h;sync && echo 1 > /proc/sys/vm/drop_caches;free -w -h
> 
> I do not run this before each one because it sometimes takes a while to
> /proc/sys/vm/drop_caches

That's a great way to substantially reduce performance; why are you
doing that?

> Is there something in the logic that can be done to make this perform
> better or should I use something other than rsync or is what I am getting
> as good as it will get regardless of what I use?
> 
> Some of these directories can be over a gig.  Most of these are media files
> and should have exif data that has the timestamp so maybe I can get rid of
> -t but it is easier to keep the timestamp of the file rather than running
> exiftool to also use the create date to "touch" the file but maybe using
> exiftool is a faster way?
> 
> ~
> In all things, Be Intentional.

> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: do I always have to use either --checksum or --times ?

[Robin Lee Powell via rsync]

...

That's ...

Behaviour confirmed.  That is *not* cool.

On Fri, Jul 15, 2022 at 04:18:18AM -0400, Kevin Korb wrote:
> Yeah, unfortunately rsync's --checksum doesn't even employ obvious
> optimizations that the man page says it does.  There is no reason for it to
> checksum files that only exist on one end or are different in sizes yet it
> does.  Just try doing an rsync of a huge tree to an empty dir. You won't run
> out of space.  You will get tired of waiting for it to do something and
> abort it.
> 
> On 7/15/22 02:57, Robin Lee Powell wrote:
> > --checksum is only slower than re-copying if your network connection
> > between the hosts is similar in speed (or faster than) each host's
> > local disk access.
> > 
> > If local disk access is 10x your network link, it is definitely not
> > slower than re-copying.
> > 
> > Having said that, it really is *very* slow, and before you use it
> > you should have a clear situation in mind that makes it plausible that
> > two files could have the same size and last mod time and still not
> > have the same data.
> > 
> > Or, alternately, be in a situation where a bit flip would be
> > catastrophic.  Such situations are not common.
> > 
> > On Thu, Jul 14, 2022 at 04:26:48AM -0400, Kevin Korb via rsync wrote:
> > > You should almost never use --checksum.  It is slower than just re-copying
> > > everything.  You should almost always use --times (or --archive which
> > > includes --times).  Without this rsync is almost as dumb as cp.  Also, ssh
> > > has been the default --ssh for a long time.
> > > 
> > > On 7/14/22 04:22, Fourhundred Thecat via rsync wrote:
> > > > Hello,
> > > > 
> > > > I want to sync local folder to remote server. When I run follwing
> > > > command repeatedly, it always transfers everything each time again and
> > > > again:
> > > > 
> > > >     rsync --rsh='ssh' foo/ server:/foo/
> > > > 
> > > > does it mean I have to always use either --checksum or --times, to
> > > > prevent repeated transfer of files that have not changed ?
> > > > 
> > > > thank you,
> > > > 
> > > 
> > > -- 
> > > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> > >   Kevin Korb  Phone:(407) 252-6853
> > >   Systems Administrator   Internet:
> > >   FutureQuest, Inc.   ke...@futurequest.net  (work)
> > >   Orlando, Floridak...@sanitarium.net (personal)
> > >   Web page:   https://sanitarium.net/
> > >   PGP public key available on web site.
> > > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> > > 
> > > -- 
> > > Please use reply-all for most replies to avoid omitting the mailing list.
> > > To unsubscribe or change options: 
> > > https://lists.samba.org/mailman/listinfo/rsync
> > > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   https://sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: do I always have to use either --checksum or --times ?

[Robin Lee Powell via rsync]

--checksum is only slower than re-copying if your network connection
between the hosts is similar in speed (or faster than) each host's
local disk access.

If local disk access is 10x your network link, it is definitely not
slower than re-copying.

Having said that, it really is *very* slow, and before you use it
you should have a clear situation in mind that makes it plausible that
two files could have the same size and last mod time and still not
have the same data.

Or, alternately, be in a situation where a bit flip would be
catastrophic.  Such situations are not common.

On Thu, Jul 14, 2022 at 04:26:48AM -0400, Kevin Korb via rsync wrote:
> You should almost never use --checksum.  It is slower than just re-copying
> everything.  You should almost always use --times (or --archive which
> includes --times).  Without this rsync is almost as dumb as cp.  Also, ssh
> has been the default --ssh for a long time.
> 
> On 7/14/22 04:22, Fourhundred Thecat via rsync wrote:
> > Hello,
> > 
> > I want to sync local folder to remote server. When I run follwing
> > command repeatedly, it always transfers everything each time again and
> > again:
> > 
> >    rsync --rsh='ssh' foo/ server:/foo/
> > 
> > does it mean I have to always use either --checksum or --times, to
> > prevent repeated transfer of files that have not changed ?
> > 
> > thank you,
> > 
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   https://sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Merging three slightly different directories

[Robin Lee Powell via rsync]

So I'm not quite following your Windows rsync problem.
Specifically, do you mean:

"I copied files A, B, and C, and there's still 3 files but each file
is bigger"

or

"I copied files A, B, and C to a directory containing files that I
thought were identical using the -b flag but now there's twice as
many files because apparently rsync thinks none of them are
identical"

?

In the former case: Windows and Linux store files on disk
differently; I would expect files to be slightly larger on Windows,
but like 10%, not double.

In the latter case: By default, rsync checks for last-mod-time and size and if 
those
are identical, it treats the file as identical.  This will almost
never be the case across Linux and windows because they store files
differently on disk, so rsync is going to think all the files are
different, so if you're using -b it'll take backups of everything.

Another things to consider: if a directory is named differently on
Windows vs. Linux (i.e. different case on the directory name), rsync
will not merge those directories.

I suggest walking the directory tree to see where the size
differences are, exactly; in Windows I use https://windirstat.net/ ,
there are various equivalents on Linux (including just using du).

On Thu, Jun 09, 2022 at 09:42:27AM -0400, hputn3 wrote:
>  On Thu, 09 Jun 2022 02:20:02 -0400 Robin Lee Powell 
>  wrote 
> 
> 
> 
> > It would help if you gave us an example of what you'd *want* to have 
> 
> > happen in different situations, but what about the -b option?  This 
> 
> > will do nothing with identical files but keep both versions of 
> 
> > non-identical ones. 
> 
> 
> 
> 
> Sorry It seemed obvious to me.  Myopia is a sometime hazard for me.
> 
> 
> 
> 
> 
> I hoped by merging the slightly different directories that have grown up 
> partially independent but largely the same.
> 
> 
> 
> I wanted to pick up what ever changes has occurred in the three different 
> directories over time, into one directory that has everything.
> 
> 
> 
> 
> 
> 
> I have actually done what I was proposing and in the end the size of the 
> Merged Images directory has doubled.
> 
> 
> 
> My original scheme:
> 
> Seemed to work as expected between two Linux HOSTS:  There was a size 
> increase of about 2-300MB
> 
> 
> 
> But something screwy happens when rsyncing from windows to linux or unix.  I 
> doubled the size and I'm certain any actual changes would have been fairly 
> small. However, I suppose if I had decided to move a major internal directory 
> inside Images or rename one or something... that might cause a serious 
> problem for rsync to surmount.
> 
> 
> 
> On Wed, Jun 08, 2022 at 12:24:16AM +, hput via rsync wrote: 
> 
> 
> 
> 
> 
> 
> 
> [...]
> 
> > How can I make rsync do the work for me?  So I don't end up loosing files. 
> 
>  > 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Robin wrote
> 
>  > but what about the -b option?  This 
> 
> > will do nothing with identical files but keep both versions of 
> 
> > non-identical ones.
> 
> 
> 
> After pulling out the man pages and looking at that,  It might be just the 
> thing.
> 
> I still have all original unmolested directories so I can try that and see 
> where it goes.
> 
> 
> 
> But what about the thing with windows...causing a doubling of the size of 
> Images?
> 
> It made me remember having rynced stuff from windows before, long ago, and 
> seeing some kind of nasty behavior
> 
> I might have to hand merge that directory... and egad at 285G that sounds a 
> bit daunting.
> 
> 
> 
> Thank you Robin for your thoughtful and helpful reply

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Merging three slightly different directories

[Robin Lee Powell via rsync]

It would help if you gave us an example of what you'd *want* to have
happen in different situations, but what about the -b option?  This
will do nothing with identical files but keep both versions of
non-identical ones.

On Wed, Jun 08, 2022 at 12:24:16AM +, hput via rsync wrote:
> I want to merge 3 slightly different directories of mostly images.
> 
> Not just mostly but the vast majority are images files.
> 
> Each directory has about 285 GB of files.
> 
> At first I thought I would just run a straightish rsync from each directory
> inturn starting with the biggest which is not much bigger ... maybe
> a few MB.
> 
> Like:
> 
> rsync -vvrptgoD --stats /biggest/ /emptydir/
> 
> rsync -vvrptgoD --stats /next-biggest/ /same-dir/
> 
> rsync -vvrptgoD --stats /smallest/ /same-dir
> 
> But after some thought I'm guessing that might be wrong headed way to go.
> 
> All three dir have mostly the same stuff in them and in the same
> places but a close inspection, given the 285 GB would be pretty much a
> non-starter.
> 
> There will be thousands that have matching names maybe newer or older
> bigger etc.  And maybe some of the same stuff but in slightly  different 
> places.
> 
> How can I make rsync do the work for me?  So I don't end up loosing files.
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: suppress message: is uptodate

[Robin Lee Powell via rsync]

On Sun, Aug 15, 2021 at 07:03:18AM +0200, Fourhundred Thecat via rsync wrote:
> Hello,
> 
> is there a way to suppress this message when syncing files?
> 
>is uptodate
> 
> I would like to see only files that have been synced.
> 
> The internet forums are full of people asking how to get rid of this,
> even as far back as 10 years, and only hack I could find is to pipe it
> to grep.
> 
>   rsync ... | grep -v ' is uptodate$'
> 
> This is a terribly dirty hack, and creates problems of its own.
> 
> Is there a better solution? Can better solution be implemented, perhaps
> by some new option?
> 
> What is the opinion of this community? I assume someone has to have
> mentioned this problem here throughout the years.
> 
> In case it is relevant to my question, I am using rsync with these options:

The options you're using are *definitely* relevant, as I have never
seen such a message in over 20 years of using rsync.

>   rsync --info=name2,del2 --checksum -rlpgo -AHX --rsh=ssh foo/ server:/

I have never used --info; I didn't even know it was there.

And in "rsync --info=help", we have:

NAME   Mention 1) updated file/dir names, 2) unchanged names

So, yeah, rsync is giving you that message because you added an
extra flag to specifically tell it to do so.  :)  Try "name1" and
see if that fixes it.

Your comment about forums being "full" of people asking how to get
rsync to not do that makes me wonder how many people have been
cargo-culting these options over the years ;(

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html