Re: Why try to update (some) permissions which are the same?

[Perry Hutchison via rsync]

Kevin Korb  wrote:
> On Sun, 3 Sep 2023, Perry Hutchison via rsync wrote:
> > On the source system:
> > ...
> > $ ll -d fcst-200[89] fcst-201[01]
> > dr-xr-xr-x  2 perryh  perryh  7168 Nov 27  2009 fcst-2008
> > dr-xr-xr-x  2 perryh  perryh  9216 Jul 21  2010 fcst-2009
> > drwxr-xr-x  2 perryh  perryh  9216 Jul  7  2011 fcst-2010
> > drwxr-xr-x  2 perryh  perryh  4608 Jul  7  2011 fcst-2011
> >
> > $ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] 
> > fbsd81:/home/perryh
> > building file list ... done
> > rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation 
> > not permitted (1)
> > rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation 
> > not permitted (1)
> > ...
> >
> > On the destination system:
> >
> > $ ll -d fcst-200[89] fcst-201[01]
> > dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 7168 Nov 27  2009 fcst-2008
> > dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul 21  2010 fcst-2009
> > drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul  7  2011 fcst-2010
> > drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 4608 Jul  7  2011 fcst-2011
> > ...
> >
> > The question is:  why does rsync attempt (and fail) to change the
> > permissions of two destination directories, and not the other two,
> > when the permissions of all four destination directories already
> > match the corresponding source directories?  (The change attempt
> > fails because of the destinations' uchg,uunlnk flags, but as far
> > as I can see the change should never have been attempted in the
> > first place.)
>
> You have --itemize-changes but either it didn't or you filtered it out.

I had noticed that adding --itemize-changes to the command line did
not result in any additional output.  Dunno why, nor whether the two
issues might be related.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Why try to update (some) permissions which are the same?

[Perry Hutchison via rsync]

On the source system:

$ rsync --version
rsync  version 2.6.8  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

Capabilities: 64-bit files, socketpairs, hard links, ACLs, symlinks, batchfiles,
  inplace, IPv6, file flags, 32-bit system inums, 64-bit internal 
inums
...

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  4608 Jul  7  2011 fcst-2011

$ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] 
fbsd81:/home/perryh
building file list ... done
rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation not 
permitted (1)
rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation not 
permitted (1)

sent 15794 bytes  received 20 bytes  31628.00 bytes/sec
total size is 278224  speedup is 17.59
rsync error: some files could not be transferred (code 23) at main.c(892) 
[sender=2.6.8]


On the destination system:

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 4608 Jul  7  2011 fcst-2011

$ rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, no iconv, symtimes
...


The question is:  why does rsync attempt (and fail) to change the
permissions of two destination directories, and not the other two,
when the permissions of all four destination directories already
match the corresponding source directories?  (The change attempt
fails because of the destinations' uchg,uunlnk flags, but as far
as I can see the change should never have been attempted in the
first place.)

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Perry Hutchison via rsync]

Perry Hutchison via rsync  wrote:

> Fourhundred Thecat via rsync <400the...@lists.samba.org> wrote:
>
> > I am copying /mnt/foo to /mnt/bar/
> >
> >rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/
> >
> > /mnt/foo contains deep directory structure, ie:
> >
> >/mnt/foo/aaa/
> >/mnt/foo/aaa/somestuff/
> >/mnt/foo/aaa/somestuff/file1
> >
> >/mnt/foo/bbb/
> >/mnt/foo/bbb/someotherstuff/
> >/mnt/foo/bbb/someotherstuff/file2
> >
> > I am not interested in details which individual files were copied, just
> > the main directory. Is it somehow possible for rsync to only report the
> > first level directory?
> >
> > ie, to have output like this when copying:
> >
> >/mnt/foo/aaa/
> >/mnt/foo/bbb/
>
> Do you need this reported in real time (e.g. to monitor progress),
> or just as a logfile?  If the latter, filtering the output with grep
> might work -- something like this (untested):
>
>   rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep '^/.*/.*/.*/$'

On second thought, that grep will match any directory name having 3
*or more* levels.  This:

  rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep '^/[^/]*/[^/]*/[^/]*/$'

should match only those with exactly 3 levels.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Perry Hutchison via rsync]

Fourhundred Thecat via rsync <400the...@lists.samba.org> wrote:

> I am copying /mnt/foo to /mnt/bar/
>
>rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/
>
> /mnt/foo contains deep directory structure, ie:
>
>/mnt/foo/aaa/
>/mnt/foo/aaa/somestuff/
>/mnt/foo/aaa/somestuff/file1
>
>/mnt/foo/bbb/
>/mnt/foo/bbb/someotherstuff/
>/mnt/foo/bbb/someotherstuff/file2
>
> I am not interested in details which individual files were copied, just
> the main directory. Is it somehow possible for rsync to only report the
> first level directory?
>
> ie, to have output like this when copying:
>
>/mnt/foo/aaa/
>/mnt/foo/bbb/

Do you need this reported in real time (e.g. to monitor progress),
or just as a logfile?  If the latter, filtering the output with grep
might work -- something like this (untested):

  rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep '^/.*/.*/.*/$'

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to tune rsync to speed up?

[Perry Hutchison via rsync]

Sebastian G??decke via rsync  wrote:

> We're facing some flapping traffic when rsyncing atm 70T from
> one server to an DELL Isilon.
> Both systems are connected with 10G Fiber (not Channel).
> So we started with one simple "rsync -a /src /dest" to the DELL
> by using NFS3.
> ...
> I always thought (and had observed it so far with rsync) that it
> makes full use of the network card ...

If you are using NFS, it is NFS -- not rsync -- which is managing the
network traffic.  In such a setup rsync will perform about as well as
cp(1).

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync sends again already existing files

[Perry Hutchison via rsync]

Stephane Ascoet via rsync  wrote:
> Kevin Korb  le 29/06/2023 22:43:
> > Are you so sure rsync actually copies the file? It should
> > correct the timestamp and tell you it did.
>
> Of that what it should do! But I'm sure not: the target is a very 
> low-quality-and-performance USB key ... less than 3 kb/s. And
> looking the FS during the transfer, I saw the temporary files
> being written. And since I was using --backup, the files has been
> backuped...  only to be replaced by exactly the same thing :-(
>
> So this disable a lot of interest in Rsync :-( Isn't there a way
> to disable "--whole-file"?

Short answer:  Last I heard, no, for (what rsync sees as) "local"
transfers.  And I agree that there should be, at least when the
target is a flash device -- but it might not help much with your
situation.

Longer answer, with details:  rsync believes that, for any transfer
which it sees as local (and this includes NFS, CIFS, etc. which are
mapped into the local namespace although the data may in fact be
accessed over a network), overwriting the destination -- even with
what may be the same data -- is at least as fast as reading it first
to determine whether it needs to be overwritten.  This belief was
usually accurate before the advent of read-mostly technologies (like
flash).

However, when the target is on a flash device, --whole-file creates
a couple of problems:

* Most flash devices, including but not limited to USB "thumb drives,"
  can be read substantially faster than they can be written.

* Even if there were no speed difference, overwriting -- even with
  the same data -- typically imposes wear on the flash device.
  (There may be a few which internally do a read-before-write, and
  avoid overwriting the same data.)

As to your situation, I suspect you may be dealing more with transfer
speed (via a USB 1.1 port?) than with read/write speed differences.
The rsync assumption that reading is no faster than writing may well
be accurate with this "low-quality-and-performance USB key."

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What could cause rsync to kill ssh?

[Perry Hutchison via rsync]

Maurice R Volaski  wrote:

> Rsync 3.2.7 is running on the Gentoo computer, which doesn't have
> a version, other than it's "current". I'm running the script from
> this computer.
>
> Rsync 3.1.2 is on the source computer, where the files come from,
> which is Ubuntu 18.0.4.6.
>
> I'm copying to a CIFS share mounted on the Gentoo computer.
>
> The rsync scripts are all similar to this one:
>
> /usr/bin/rsync -v -a --progress --exclude-from=${exclude} --safe-links 
> --itemize-changes --no-perms --no-owner --progress --stats \
> al...@labadmin-precision-tower-3620.montefiore.org:/home/alexa/ 
> /mnt/data.einstein/luke/all_but_dat/alexa/desktop_bkup/profile \
> >> /home/maurice/logs/rsync-client-alexa.log
>
> I re-ran the scripts skipping this one. The next one was running
> and during that period, ssh stopped responded to new connections,
> so it may be the case that the failure is taking place across
> time, and it doesn't fail wholesale immediately.
>
> However, I have other scripts like these copying from other
> sources (not Ubuntu) and they are not causing these failures.

You have several moving parts, which complicates figuring out which
of the various interactions is contributing to the problem.

BTW anyone else on the list is more than welcome to weigh in.  I am
hardly an expert on rsync, and not at all familiar with the ins and
outs of either Gentoo or CIFS.

One thing which I think is most likely _not_ involved in the problem is
sshd on the Gentoo system, and this is consistent with the observation
that restarting sshd did not help.  (If I'm reading the rsync command
correctly, rsync on the Gentoo system is establishing the ssh
connection and transferring the files over it.  Gentoo's sshd would
be involved only if the client were initiating the connection.)

Is there anything interesting in the rsync logfile, especially near the
end, or in the any of the involved machines' system logs (including the
CIFS host) around the time of the hang?

Does the Gentoo system have enough space for rsync to copy the files
to a local drive, so that rsync and Samba are not both working on the
same transfer at the same time?  (The files can then be copied to the
CIFS share in a separate step, using "cp -r" or some such.)  If rsync
still fails when arranged that way it would tend to eliminate CIFS as
a factor (and it will simplify the environment); OTOH if that "solves"
the problem you'll at least have a workaround.

Totally separate from that, is this Ubuntu system the only client using
Ubuntu 18.0.4.6 and/or rsync 3.1.2?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What could cause rsync to kill ssh?

[Perry Hutchison via rsync]

Maurice R Volaski via rsync  wrote:

> I have an rsync script that it is copying one computer (over ssh)
> to a shared CIFS mount on Gentoo Linux, kernel 6.3.4. The script
> runs for a while and then at some point quits knocking my ssh
> session offline on all terminals and it blocks ssh from being able
> to connect again. Even restarting sshd doesn't help. Rsync has
> apparently killed it. I have to reboot.

For starters:

  What OS and version is the rsync script running on?

  Which end do you have to reboot?  The machine running the script,
  or the Gentoo Linux?

  What versions of rsync are running on each end?

  Can you show the command line that fails?

  Based on the mention of multiple terminals, it sounds as if you
  have a fairly complex ssh environment.  Can you get it to fail in
  a simpler environment, ideally with only one terminal?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Trying to diagnose incomplete file transfer

[Perry Hutchison via rsync]

Albert Croft via rsync  wrote:

> ... I am currently using the 'split' command to break the files
> into 1-GB "chunks" (ex.: foo.tgz.aa, foo.tgz.ab, ...).
> ...
> I am frequently encountering times where the file appears to
> have been transferred but is incomplete. (Example: foo.tgz.ab
> now exists on the local system, has been removed from the remote,
> but is incomplete.)

One thing to check, not in rsync itself but in the preparation of
the data:  not all versions of "split" support files that aren't
text.  In particular, some will silently drop null bytes.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Multiple paths in a module?

[Perry Hutchison via rsync]

Glen Huang via rsync  wrote:

> ... the mounting paths are dynamically configured in another
> service, so I need to remount whenever the paths change
> if I mount them beforehand.

Could the remount be done in that other service?

> With the "early exec" hook, I can avoid such remounting,
> but it's only doable with root...

General approach when only part of a job requires root:

1.  Set the remount program permissions to
-r-sr-xr--  root  foo
(i.e. setuid root, executable only by group foo).
The program must be an executable binary, not a script.

2.  Ensure that the service which will run the remounter
-- and only that service -- runs as group "foo".

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Workaround for rsyncing when fileflag schg is set?

[Perry Hutchison via rsync]

That sort of snafu is why find(1) has the -depth directive.
Does rsync have anything similar?

Kevin Korb via rsync  wrote:

> There maybe a proper solution but an obvious workaround would be to run
> rsync twice.  The first time without the --fileflags option.
>
> --no-perms wouldn't help.  That is only the standard unix permissions.
>
> On 10/30/21 8:04 PM, Fred Fugate via rsync wrote:
> > Hi,
> > 
> > I have some subdirectories within a home directory that are chflagged
> > schg, aka system immutable.
> > 
> > When I rsync the home directory to another machine, I get lots of errors
> > for these subdirectories and they fail to copy to the target. NOTE: The
> > target is empty; rsync is copying to an empty remote machine, not trying
> > to overwrite anything.
> > 
> > I understand why this is happening; it's the schg flag. But is there a
> > workaround using some combination of rsync options or multiple passes?
> > 
> > Here is an example.
> > 
> > The source directory is /Users/redacted.
> > 
> > Within /Users/redacted is a subdirectory foo that has flag schg set:
> > /Users/redacted/Documents/artwork/foo
> > 
> > # ls -laO /Users/redacted/Documents/artwork
> > total 80
> > drwxr-xr-x ?? 20 redacted ??staff ??- ?? ?? ?? ??680 Sep 21 21:06 .
> > drwx--@ 609 redacted ??staff ??- ?? ?? ??20706 Oct 29 16:07 ..
> > -rw-r--r--@ ?? 1 redacted ??staff ??- ?? ?? ??18436 Sep 21 20:55 somefile
> > drwxrwxrwx@ ??18 redacted ??staff ??schg ?? ?? 612 Apr 12 ??2006 foo
> > 
> > 
> > My rsync args are:
> > 
> > --verbose --archive --one-file-system --acls --hard-links --xattrs
> > --protect-args --delete-after --numeric-ids --itemize-changes --crtimes
> > --fileflags --force-change
> > --rsync-path=/opt/rsync323/bin/rsync
> > 
> > My rsync version is 3.2.3
> > 
> > My execution and output looks like this, run as root:
> > 
> > # /opt/rsync323/bin/rsync [ARGS ABOVE] /Users/redacted
> > remotemachine.domain.com:/Users
> > .
> > .
> > rsync: [receiver] mkstemp
> > "/Users/redacted/Documents/artwork/foo/bar/.background.tiff.vYOAS2"
> > failed: Operation not permitted (1)
> > rsync: [receiver] mkstemp
> > "/Users/redacted/Documents/artwork/foo/bar/.dc4.orange.CMYK.tiff.rYCmAP"
> > failed: Operation not permitted (1)
> > rsync: [receiver] mkstemp
> > "/Users/redacted/Documents/artwork/foo/bar/.barcode.tiff.2E4mec" failed:
> > Operation not permitted (1)
> > .
> > .
> > 
> > In a nutshell:
> > 
> > Subdirectory foo gets created on the receiver, but it gets created with
> > the schg flag set.
> > 
> > No further copying into foo can happen after that, because the schg flag
> > prevents that.
> > 
> > Subdirectory bar cannot be created under foo, and
> > .background.tiff.vYOAS2 and other files cannot be created under bar, etc.
> > 
> > How can I force the schg flags to be set on the receiver AFTER
> > everything has been copied from the source?
> > 
> > I really don't want to remove all of my schg settings on the source
> > before rsyncing to the target.
> > 
> > Would --no-perms allow this? But what would it break?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync 3.2.3 released

[Perry Hutchison via rsync]

Rupert Gallagher  wrote:

> ... I see this ball bounched betwen rsync, openbsd and supermicro
> ... I see a large cache delivered by the OS on server hardware and
> a program unable to use it.

It sounds as if the problem may be OpenBSD "delivering" the cache
rather than "utilizing" it.  What does the documentation -- or the
OpenBSD support community -- say about configuring OpenBSD to use
that sort of cache for filesystem optimization?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync 3.2.3 released

[Perry Hutchison via rsync]

Rupert Gallagher via rsync  wrote:

> On 7 Aug 2020, 23:44, Wayne Davison < wa...@opencoder.net> wrote:
>
> >> Also, I have 12GB of cache in ecc ram that rsync is not using.
>
> >It uses whatever memory it needs plus whatever filesystem caching
> >your OS provides.
>
> Hmmm... bad day today...
>
> No, it is not using all available resources. It is doing frantic
> I/O instead of buffering. The result is a appalling low transfer
> rate.

This is an OS configuration/tuning problem, not an rsync problem.

Application-level code (such as rsync) does not directly deal with
NUMA features such as external RAM banks.  Support for that sort of
thing would normally be provided by the OS memory management and
filesystem code -- but (depending on the hardware details) it might
need configuration or even a specialized driver.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: sync of ordinary files from lvm snapshot fails with --sparse, 3.2.2pre2

[Perry Hutchison via rsync]

Scott Mcdermott via rsync  wrote:

> Getting some puzzling errors doing the following backup procedure
> on a root filesystem that's on LVM, when using sparse flag:
>
> lvcreate --snapshot --name baksnap --size 3G /dev/vg0/root
> mount -o ro /dev/vg0/baksnap /var/tmp/snapmnt
> rsync -izaHx \
> --delete --delete-excluded --delete-after \
> --partial --inplace --sparse \
> --numeric-ids --bwlimit=640k --rsh=ssh \
> /var/tmp/snapmnt \
> bakhost:/path/to/backups/root/
>
> This is causing the following errors:
>
> .d..t.. etc/lvm/archive/
>   .d..t.. etc/lvm/backup/
>ERROR: etc/lvm/backup/vg0 failed verification -- update retained.
>  ERROR: root/.bash_history failed verification -- update retained.
> rsync error: some files/attrs were not transferred (see previous
> errors) (code 23) at main.c(1287) [sender=3.2.2pre2]
>
> It repeats the above as many times as I run it.  Simply removing the
> "--sparse" flag makes the error go away.
>
> This makes no sense to me in the first place, because I'm mounting the
> snapshot readonly, so how can it change during the copy (presumably
> this is why it fails verification)? And in the second place, I don't
> see how sparse flag would affect anything at all.  I'm using --sparse
> --inplace because I have large VM images on this filesystem, and
> transfers are sometimes interrupted.  However these are only very
> small text files showing the error (and again, they are in a readonly
> mount...)

I suspect rsync is implementing --sparse in the straightforward way:
whenever the sender reads a large sequence of all-zero bytes -- for
some definition of "large" -- the receiver does a seek forward of
that amount instead of writing the zeros.  This is exactly what is
wanted when the output file is new, but it is naive if the output
file already exists and is being updated in place.  In the latter
case, we either need to read the part of the output file that
we're about to seek over -- and overwrite anything that reads back
non-zero with zeros -- or we need to temporarily turn off --sparse
when the output file already exists.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: High memory usage - any way around it other than splitting jobs?

[Perry Hutchison via rsync]

Andy Smith via rsync  wrote:

> I have a virtual machine with 2G of memory. On this VM there is a
> directory tree with 33.3 million files in it. When attempting to
> rsync (rsync -PSHav --delete /source /dest) this tree from one
> directory to another on the same host, rsync uses all the memory and
> is killed by oom-killer.
>
> This host is Debian oldstable so has
>
> $ rsync --version
> rsync  version 3.1.2  protocol version 31

Since this is all taking place on a single VM (thus there is no
network involved), it's possible that rsync is not the best tool
for the job.  Had you considered something like:

$ ( cd /source && find . -depth -print0 | cpio -p -0l /dest ) && rm -rf /source

One advantage of "cpio -p -l" is that it avoids copying any of the
files -- it just makes a new directory tree containing hardlinks to
the original files.

(I am guessing that, since this is a VM, it is likely to have been
set up with a single, large filesystem on a single (virtual) drive
rather than the older approach of creating multiple partitions --
so that /source and /dest are in the same filesystem.)

For that matter, what about:

# rm -rf /dest
# mv /source /dest

i.e. just rename the source tree?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: High memory usage - any way around it other than splitting jobs?

[Perry Hutchison via rsync]

Kevin Korb via rsync  wrote:

> Unfortunately the hard links are the problem.  In order to keep
> them straight rsync has to remember the details of every file it
> finds with a link count >1 making it grow and grow.

I _hope_ it is only remembering the source and destination inode
numbers, and pruning that list when possible[*], as opposed to
storing all of "the details" for the duration of the operation.

[*] If the source file has (say) 3 hardlinks, it can be deleted
from the list as soon as the 3rd hardlink has been copied.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: osx permission issue

[Perry Hutchison via rsync]

Roland via rsync  wrote:
> does somebody know how to circumvent that "extra file access restriction
> feature" introduced in osx some time ago ?

It may not be possible.

Based on experience with FreeBSD, from which much of OSX is derived,
I suspect you may be running into issues with "file flags"; check the
OSX documentation for the chflags system call.  In FreeBSD, I've seen
that same error message when rsync attempts an operation that would
violate chflags restrictions.

> i already tried adding rsync binary to programms with "full disk access"
> privilege ( system-prefs -> security & privacy -> privacy -> full disk
> access)?? , since running as root is not sufficient - but it does not work.

"full disk access" likely refers to the ability to read the disk
directly, bypassing the filesystem.  It's used by maintenance
programs like fsck and fsdb, and by some full-disk backup programs,
but it won't help rsync.

> i want to make sure that every file on osx is getting backup
>
> i run rsync on linux to remotely backup osx system (via ssh).
>
> regards
> roland
>
> rsync:
> readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/0/com.apple.routined")
> failed: Operation not permitted (1)
> ...

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 13901] New: Empty quotes adds cwd to SRC directories

[Perry Hutchison via rsync]

Joe via rsync  wrote:

> I brought this up on the list years ago and was told it's a feature,
> not a bug.

That is a really lame excuse.
http://qph.fs.quoracdn.net/main-qimg-6349815aff502d70e502e228f07d8cd4

> Even if some other GNU or Linux commands have this "feature", it still
> violates the principle of least surprise.

It is certainly not expected behavior for "normal" Unix commands.
A couple of examples, on FreeBSD:

$ ls ""
ls: fts_open: No such file or directory

$ cat ""
cat: : No such file or directory

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync client for Chrome OS?

[Perry Hutchison via rsync]

Uxio Prego  wrote:
> Hi, hey android allows execution of _legacy_ static
> binary programs, via unpacking of binary as asset,
> then chmod ing it properly and finally running it
> through a system call [from e.g. the Java userland
> of a vanilla android application package].
>
> I doubt that Chrome OS doesn't include some
> equivalent facility!

Chrome OS does allow for running Android assets, but:

* They run in an Android container, and do not have access to Chrome
  OS configuration and user files.

* This feature was added relatively recently, and enabling it requires
  specific support code to be written (and released) for each particular
  platform.  It hasn't been done for the platform I'm using.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync client for Chrome OS?

[Perry Hutchison via rsync]

Parke  wrote:
> On Sat, Jan 5, 2019 at 9:54 PM Perry Hutchison via rsync
>  wrote:
> > ... I'm wondering if anyone here
> > knows of an implementation of rsync client for Chrome OS in normal
> > (not developer) mode, i.e. as an app or extension.
>
> Have you tried compiling rsync statically (on a non-Chrome OS system)
> and seeing if it will run on Chrome OS?  Linux is Linux, after all.
> ...
> (I also don't know if you can even run any third-party binary
> executables on Chrome OS if you are not in developer mode.)

I very much doubt that Chrome OS would make this possible, given its
focus on security.  (If it _were_ possible to exec a user-supplied
file in normal mode, one could do all sorts of interesting things
with executable shell scripts -- like setting DISPLAY to point to
some other machine.)

While it would certainly be possible to import a blob that happened
to be a binary for some CPU type (into, say, the Downloads directory),
without developer mode there is no chmod command and thus no way to
mark that binary as executable.  Also, I suspect that user-accessible
filesystems are mounted no-exec.

I suspect what I'm looking for is a reimplementation of rsync (or a
useful subset, e.g. I doubt I need ACL support) in Java.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync client for Chrome OS?

[Perry Hutchison via rsync]

Having found no mention of Google's Chrome OS in the rsync archives,
nor any useful* pointers via Google, I'm wondering if anyone here
knows of an implementation of rsync client for Chrome OS in normal
(not developer) mode, i.e. as an app or extension.

* It seems that rsync is available in "test" builds of Chrome OS, but
  not in normal release builds.  As far as I can tell, installing a
  "test" build requires putting the device in Developer Mode, which
  requires doing a "powerwash," which wipes local storage -- totally
  defeating the purpose of installing rsync in the first place.

* rsync is available in Termux, but presumably would only have access
  to the Termux container, not to the user's Chrome OS configuration
  or files.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync between 2 datacenters not working

[Perry Hutchison via rsync]

Marc Roos via rsync  wrote:
> Hmmm, looks indeed like changing the mtu helps, strange thing is that 
> with the bigger mtu sometimes I had no problems and often TB are copied 
> without problems. 
> Problem is I can't test the connection that much because the connection 
> has been established by a third party and it is just some vlan without 
> any hops. 

VLANs are notorious for reducing the MTU by adding metadata to the
packets.

2 possible reasons why it sometimes works OK:

* Overall net performance is sometimes good enough that the
  fragmentation introduced by the VLAN has little impact.

* The 3rd party who sets it up may not always do it in
  exactly the same way.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

[Perry Hutchison via rsync]

leonv12 via rsync  wrote:

> I don't get why it runs from the command line but not from a scheduled
> task.  Any suggestions for a fix or a work-around?

Check the environment settings, which are often the cause of differences
in behavior between running from the command line vs. as scheduled.

You may need to have the scheduler run a script that sets up the
environment and runs rsync, rather than having the scheduler run
rsync directly.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html