[Bug 12732] New: hard links can cause rsync to block or to silently skip files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12732

Bug ID: 12732
   Summary: hard links can cause rsync to block or to silently
skip files
   Product: rsync
   Version: 3.1.2
  Hardware: x64
OS: Linux
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: sa...@hlipp.de
QA Contact: rsync...@samba.org

Overview


Hard link handling seems to be broken when using "rsync -aH --compare-dest". I
found two possible scenarios:

1) rsync completes without error message and exit code 0, although some files
are missing from the backup
2) rsync blocks and must be interrupted/killed

I found this bug when tracking down random hangs of rsync. It turned out that
most, but not all of these hangs occur when hard links are present. Therefore,
I hope the latter case might give some hints to a larger problem that might be
triggered by this hard link bug.


How to reproduce (1)


[ Using Linux on e.g. ext4 ]

mkdir srclt
cd srclt
echo x > a
ln a b
echo x > c
ln c d
cd ..

cp -aix srclt dstlt
rm dstlt/{b,c}

mkdir baklt

rsync -aHvv --compare-dest=$PWD/dstlt/. $PWD/srclt/. $PWD/baklt/. >> testlt.log
2>&1



Actual Results (1)
==

cat testlt.log
#
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
a is uptodate
d is uptodate
b
a => b
total: matches=0  hash_hits=0  false_alarms=0 data=2

sent 173 bytes  received 160 bytes  666.00 bytes/sec
total size is 8  speedup is 0.02
#
ls -il srclt dstlt baklt
#
baklt:
total 8
3249818642 -rw-r--r-- 2 X X 2 2017-04-04 X a
3249818642 -rw-r--r-- 2 X X 2 2017-04-04 X b

dstlt:
total 8
2205741698 -rw-r--r-- 1 X X 2 2017-04-04 X a
2205741699 -rw-r--r-- 1 X X 2 2017-04-04 X d

srclt:
total 16
1138988347 -rw-r--r-- 2 X X 2 2017-04-04 X a
1138988347 -rw-r--r-- 2 X X 2 2017-04-04 X b
1138988348 -rw-r--r-- 2 X X 2 2017-04-04 X c
1138988348 -rw-r--r-- 2 X X 2 2017-04-04 X d
#


Expected Results (1)


The directory baklt should contain the entries b and c.
Entry c is completely ignored and does not show up in the log.
Entry a does not need to appear in the backup, as it is present in both
directories srclt and dstlt.


How to reproduce (2)


[ Using Linux on e.g. ext4 ]

mkdir srclt2
cd srclt2
echo x > a
ln a b
cd ..

cp -aix srclt2 dstlt2
rm dstlt2/b

mkdir baklt2

rsync -aHvv --compare-dest=$PWD/dstlt2/. $PWD/srclt2/. $PWD/baklt2/. >>
testlt2.log 2>&1



Actual Results (2)
==

=> rsync hangs and must be interrupted/killed.

cat testlt2.log
#
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
a is uptodate
b
a => b
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(636)
[sender=3.1.2]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504)
[generator=3.1.2]
#
ls -il srclt2 dstlt2 baklt2
#
baklt2:
total 8
2191211 -rw-r--r-- 2 X X 2 2017-04-04 X a
2191211 -rw-r--r-- 2 X X 2 2017-04-04 X b

dstlt2:
total 4
2191208 -rw-r--r-- 1 X X 2 2017-04-04 X a

srclt2:
total 8
2191206 -rw-r--r-- 2 X X 2 2017-04-04 X a
2191206 -rw-r--r-- 2 X X 2 2017-04-04 X b
#


Expected Results (2)


rsync should not block.
Entry a does not need to appear in the backup, as it is present in both
directories srclt and dstlt.


Further information
===

This problem exists at least for rsync versions 3.1.0, 3.1.1, and 3.1.2 for
different Linux varieties using various file systems:
https://lists.samba.org/archive/rsync/2015-April/030092.html

Latest test on openSUSE 42.2 (x86_64) on ext4 + on nfs with
rsync --version
#
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
#

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12732] hard links can cause rsync to block or to silently skip files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12732

--- Comment #1 from Hansjoerg Lipp  ---
Am 05.04.2017 um 22:05 schrieb L A Walsh via rsync:
>I ran rsync 3.1.1 for over a year to help generate
> snapshots.  I can't say if it copied all the files or not, as
> it was backing up a large "/home" partition, BUT, it never hung.
> It did take 45min to a few hours to do the compare, but it
> was comparing a large amount of data (>750G) w/a snapshot
> (another 750G) to dump diffs to a third, and my /home partion
> has a *very* large number of hard links.

I've been using rsync for many years and it works fine most of the time.
I'm not sure if all of the occasional hangs have the same reason, these
are really hard to track down as they usually occur during large
transfers (e.g. when synchronizing large backup disks). That's why I was
happy that I could find a small test case which triggers this problem.

Does your rsync hang after the sequence of commands described in section
"How to reproduce (2)"?

>> Latest test on openSUSE 42.2 (x86_64) on ext4 + on nfs with
>>   
> 
>Ah...  I'd suspect nfs...
>  Why are you using nfs?

In order to find out if there is a difference when using another file
system type. The most recent tests were on ext4 and on nfs
(independently), older tests were on at least ext3 and xfs. IIRC I only
tested on different OpenSUSE and Debian versions on x86_64 systems, though.

>Just checked my /home partition.
>find shows 9295431 names (of any type), but du shows
> (using du --inodes) shows 4407458 inodes.  That means over
> half of the filenames are hard linked.  While my home
> partition takes up 60% more space now, even cutting
> those counts in half would still a large number of
> hard links -- and rsync didn't crash doing an
> rsync of the partition to an empty one, but first comparing
> to a previous snapshot (the empty partition ended up
> with differences between the main partition & the snapshot.

Probably using different options? Can this be some sort of Heisenbug,
nobody can reproduce? Do the two sequences of shell commands work for
you as expected? Please note that both rsync commands in the mail
generated by bugzilla are split into two lines (each): Both rsync
commands should read
   rsync PARAMS DIRS >> XXX.log 2>&1

Kind regards,
Hansjoerg

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12741] New: stop rsync on "No space left on device"

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12741

Bug ID: 12741
   Summary: stop rsync on "No space left on device"
   Product: rsync
   Version: 3.1.2
  Hardware: x86
OS: Linux
Status: NEW
  Severity: enhancement
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ry...@caramail.com
QA Contact: rsync...@samba.org

rsync show me :
rsync: delete_file: make_backup(...) failed: No space left on device (28)
rsync: delete_file: make_backup(...) failed: No space left on device (28)
...
in output.
Rsync is right, but I would like it to stop on first error, because I think
that it isn't necessary that it populate my log moreover and use my CPU and my
network vainly.
Despite I understand that another backup of a more little file could succeed, I
think it could be useful.

Thanks for your work.

Regards.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12519] [PATCH] Failed to open lock file, add reason

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12519

Ben RUBSON  changed:

   What|Removed |Added

 Resolution|--- |LATER
 Status|NEW |RESOLVED

--- Comment #1 from Ben RUBSON  ---
I close this as I don't need it anymore, it will help cleaning the queue :)
Feel free to re-open it if needed.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12742] New: a proposal: fix bogus nanosecond mtimes on transfer (patch included)

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12742

Bug ID: 12742
   Summary: a proposal: fix bogus nanosecond mtimes on transfer
(patch included)
   Product: rsync
   Version: 3.1.1
  Hardware: All
OS: All
Status: NEW
  Severity: minor
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: gfedo...@unimelb.edu.au
QA Contact: rsync...@samba.org

Created attachment 13152
  --> https://bugzilla.samba.org/attachment.cgi?id=13152=edit
a proposed patch to ignore bogus .st_mtim.tv_nsec values by resetting them to 0

This suggestion is actually in a grey area between a proposed enhancement and a
minor fix.

Basically utimensat() sometimes fails to transfer mtimes for some bogus mtime
values, so then rsync tries to transfer the same files over and over again.

However, a simple utime() call works just fine, what leads to the whole
suggestion.

I'm attaching a proposed patch for rsync 3.1.1, but I think v. 3.1.2 can be
adjusted in a similar vein ; not sure about 3.1.3, since I thought it is in
development.

Now I will go into details, and then briefly discuss the alternatives.


Details
===

First of all, this is not strictly speaking an rsync problem: we are talking
about a marginally small number of files ( a few hundred of at least hundreds
of thousands, sometimes millions ), with their mtimes messed up possibly
because of some subtle fault in the filesystem implementation. 

However, these very few files make rsync to set return code RERR_PARTIAL (23)
for the whole transfer, what leaves us with no good options.

In my experience, on a Linux system with utimensat() / futimens() the latter
fails to transfer a negative .st_mtim.tv_nsec, or a value greater than
10 with errno 22 (EINVAL).

However, since rsync 3.1.1 seems to compare only seconds, and rsync 3.1.2 seems
to still use only seconds comparison when nominating the files for a transfer,
it is very tempting to ignore bogus nanosecond values by setting them to zero,
because in that case futimens() will just do the job, and we'll have no further
transfer attempts.

I must admit that I do not know how and why these false mtime values appear; my
another guess is that there may be a minor bug in some of utimes()
implementations, since all the bogus nanosecond times that I see are a multiple
of a 1000.

However, struct timespec in my opinion is just a plain invitation for this kind
of error, since it defines tv_nsec as a long, which is plain insane: you do not
want to define a filesystem-related field to have anything but a fixed length
size, when at least for GCC a long is basically a machine word, that is, its
size depends on the CPU architecture. ( Windows long type definition is better
in that particular case, making it AFAIR an int32. )

Rsync, by the way, takes a much more reasonable approach by defining mod_nsec
in set_modtime() as an uint32 ; however, this sanity is incompatible with
insanity in the standard ( see e.g.
http://pubs.opengroup.org/onlinepubs/7908799/xsh/time.h.html ), what will of
course break the bogus nanosecond mtimes anyway.

For a discussion, please refer to a following rsync lists message:
https://lists.samba.org/archive/rsync/2017-April/031177.html , but in short --
nanosecond mtime values:

(a) are not supposed to exist, since we need to define at least UTIME_OMIT and
UTIME_NOW as long values, and have no other choice but to put these constants
somewhere outside of 0 .. 1 000 000 000 interval ;

(b) can still appear on a filesystem nevertheless due to some possible
imperfections in the OS code or filesystem implemenation ;

(c) are not really taken into account by rsync 3.1.1 / 3.1.2 when it compares
files ;

(d) however, can still break an rsync transfer -- that is, lead to an error
code 23.

Therefore, this request suggests that we ignore them on a transfer by setting
set them to 0.


Alternatives


One could of course recompile rsync to make it ignore ns precision at all, but
we can probably do much better -- that is, transfer all the proper mtime values
with the proper precision, and ignore nanoseconds for the messed up ones.

So an obvious alternative to changing mtimes would be to allow the user to
ignore them -- but since we still probably want to transfer mtimes for 99.99%
of files, this shall rather be an option to ignore a few mtime transfer errors,
and I will fill another enhancement request to address that.

Finally, running two subsequent rsync transfers -- one to transfer the file
contents ignoring mtimes, and another to transfer mtimes, where we will ignore
the return code, is possible, but has two drawbacks:

(1) this may double the time needed for the transfer: e.g. one one rsync
session runs overnight, there would be no place for another rsync session on
the same day ;

(2) rsync will still try to transfer the same files over and over 

[Bug 12755] [patch] Improve execution speed on Windows; with Win32 API calls

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12755

--- Comment #1 from Joseph Benden  ---
If anyone wishes to test the compiled binary, it's available for download here:

https://github.com/jbenden/rsync/releases

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12755] New: [patch] Improve execution speed on Windows; with Win32 API calls

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12755

Bug ID: 12755
   Summary: [patch] Improve execution speed on Windows; with Win32
API calls
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: Windows 10
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: j...@benden.us
QA Contact: rsync...@samba.org

Created attachment 13169
  --> https://bugzilla.samba.org/attachment.cgi?id=13169=edit
Improve wall-time execution speed on Windows

The attached patch improves execution speed under Windows, by sprinkled usage
of the Win32 API. The patch is against the Git repository master branch.

When compiled with MSYS or Cygwin, this patch massively improves on total wall
time of the synchronization. By utilizing the Win32 API, I was able to bypass
lots of POSIX-to-Win32 translations occurring coupled with reusing given
stat()-equivalent data returned during the file-system directory walks.

Comments, suggestions, ideas are welcomed!

-Joe

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12940] New: rsync: -C/--cvs-exclude does not ignore SCM ignore files (patch)

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12940

Bug ID: 12940
   Summary: rsync: -C/--cvs-exclude does not ignore SCM ignore
files (patch)
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: minor
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: austinengl...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13436
  --> https://bugzilla.samba.org/attachment.cgi?id=13436=edit
add gitignore to cvs_excludes

Howdy,

First, thanks for the excellent rsync tool. It's been invaluable over the
years!

I was recently working on a project where I wanted to rsync a copy of a git
repo, and exclude SCM files. I used rsync -C, but noticed that while .git was
removed, .gitignore remained.

I've written a small patch to also exclude .bzrignore, .hgignore, and
.gitignore, so that bazaar, git and mercurial SCMs also have their ignore
files, ignored (as they are practically useless without the other SCM files,
e.g., for a tarball release).

Before:

austin@austin2:~/src/rsync$ rsync -CavPz /home/austin/src/foobar $(mktemp -d)
sending incremental file list
foobar/
foobar/.gitignore
  6 100%0.00kB/s0:00:00 (xfr#1, to-chk=3/5)
foobar/README.md
 63 100%   61.52kB/s0:00:00 (xfr#2, to-chk=2/5)
foobar/setup-flash-drive.sh
  8,119 100%7.74MB/s0:00:00 (xfr#3, to-chk=1/5)
foobar/setup-kit.sh
613 100%  598.63kB/s0:00:00 (xfr#4, to-chk=0/5)

sent 3,849 bytes  received 96 bytes  7,890.00 bytes/sec
total size is 8,801  speedup is 2.23

After:

austin@austin2:~/src/rsync$ ./rsync -CavPz /home/austin/src/foobar $(mktemp -d)
sending incremental file list
foobar/
foobar/README.md
 63 100%0.00kB/s0:00:00 (xfr#1, to-chk=2/4)
foobar/setup-flash-drive.sh
  8,119 100%7.74MB/s0:00:00 (xfr#2, to-chk=1/4)
foobar/setup-kit.sh
613 100%  598.63kB/s0:00:00 (xfr#3, to-chk=0/4)

sent 3,778 bytes  received 77 bytes  7,710.00 bytes/sec
total size is 8,795  speedup is 2.28

Note: I searched bugzilla and found a few other bugs that reference .gitignore,
but I believe none are the same:
bug 5418/bug 10995: .gitignore is casually mentioned in a list of files in the
rsync command, unrelated
bug 6120: is about a .rsyncignore file (default --exclude-from location),
roughly equivalent to .gitignore
bug 9744: is about parsing git/svn/hg/bzr ignore lists and having rsync ignore
THOSE files

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12942] New: Traffic shaping: Make --bwlimit dynamic

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12942

Bug ID: 12942
   Summary: Traffic shaping: Make --bwlimit dynamic
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: kapo...@gmail.com
QA Contact: rsync...@samba.org

It would be great to be able to modify the value of --bwlimit while rsync is
already running. Perhaps with a signal sent to rsync?

Other idea:
--bwlimit-from-file=/tmp/bwlimitvalue.txt
$ echo "200K" > /tmp/bwlimitvalue.txt

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12942] Traffic shaping: Make --bwlimit dynamic

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12942

--- Comment #2 from kapo...@gmail.com ---
Hi Roland, thanks for your help. I looked at your solution:


export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh"

Of course using $$ was the lazy way out, should kill pv instead but how to
obtain the pid in a oneliner?


Two remarks:

(1) You can easily get the pids of pv with: pv --pidfile /tmp/foo; and then you
will do: kill $(cat /tmp/foo).

(2) The solution does not work well. If I start a transfer at low speed and I
raise the speed (with pv -R [pid] -L [speed]), it works. But if a network
transfer runs with high speeds and if I decrease the speed afterwards, I don't
see any immediate effect. Sometimes I see it 20 minutes later... So I cannot
regard this as a good solution; a "dynamic" --bwlimit option is needed.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12955] [patch] Fix rsync -A on AIX

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12955

Cun Gong  changed:

   What|Removed |Added

  Attachment #13458|0   |1
is obsolete||
 CC||gongcunj...@gmail.com

--- Comment #1 from Cun Gong  ---
Created attachment 13478
  --> https://bugzilla.samba.org/attachment.cgi?id=13478=edit
Fix rsync ACL files on AIX

Because the chmod(2) will disable ACL permission on AIX (see
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.security/access_control_list.htm),
so this is a full patch for fixing the problem that can't synchronize ACL files
on AIX.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12964] New: Maybe we can add the '--bind-cpu' option

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12964

Bug ID: 12964
   Summary: Maybe we can add the '--bind-cpu' option
   Product: rsync
   Version: 3.1.2
  Hardware: All
OS: All
Status: NEW
  Severity: trivial
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: gongcunj...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13472
  --> https://bugzilla.samba.org/attachment.cgi?id=13472=edit
add '--bind-cpu' option to rsync

We use rsync to take daily backup or log synchronization, but rsync
often trigger high CPU load. I tried to find a solution through
Google, but didn't find a satisfactory answer. Many people suggested
using the '--whole-file' option to reduce the CPU load, or use 'nice'
to lower the rsync execution priority. So I have another idea:

  Maybe we can add the '--bind-cpu' option to tell rsync to run
  on specified processor, like the 'worker_cpu_affinity' in nginx.

Although I'm not sure that is a good idea, because the core issue is
the improvement of the rsync protocol, but I still have made a few
attempts. I made some changes to rsync 3.1.2 to enable it to support
binding CPUs on GNU/Linux, AIX, FreeBSD & Solaris, please see the
attachment. For example, the following option tells rsync to run on
CPUs 0, 2-5 and 7:

  --bind-cpu=0,2-5,7

I also used maketree.py script to test the synchronization of a large
number of files (1 files), and drew some statistics curves (see
the cpu_time.pdf in attachment). I was just testing on a machine (AIX
7.2 with 8 processors) without testing the remote synchronization, but
I think the conclusion is worth reference. As the size of file
increases, the CPU load will continue to increase by default, but the
CPU load will slowly increase until it is around 55% when binding a
single processor. The disadvantage of binding CPUs is that when file
sizes are growing, they are slower to process than non-binding (it's
about two times slower in my test result). Of course, if I/O or
network bandwidth is the bottleneck, then binding CPUs is not
significant because most of time is waiting.

Is it necessary to add the '--bind-cpu' option ? Your comments are
welcome, happy to answer any question I can !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12920] New: Invalid path from sender

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12920

Bug ID: 12920
   Summary: Invalid path from sender
   Product: rsync
   Version: 3.1.2
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: crazyjim...@hotmail.com
QA Contact: rsync...@samba.org

Working on 3.1.1 correctly


rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, no xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.



How to reproduce:


cd /tmp

mkdir -p a/a a/b a/c

rsync -axR /tmp/a /tmp/b# This
works fine
rsync --no-inc-recursive -avxR a /tmp/b /tmp/c# This works fine at this
moment

mkdir a/a/a
mkdir a/a/b

rsync --no-inc-recursive -avxR a /tmp/b /tmp/c #  Same command fails
after creating new directories
building file list ... done
ABORTING due to invalid path from sender: a/b
rsync error: protocol incompatibility (code 2) at generator.c(1270)
[generator=3.1.2]

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] New: --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

Bug ID: 12915
   Summary: --modify-window should default to 1 for fat
filesystems
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: i...@fsf.org
QA Contact: rsync...@samba.org

The manual on --modify-window says "Specifying 1 is useful for copies
to/from MS Windows FAT filesystems, because FAT represents times with a
2-second resolution (allowing times to differ from the original by up to
1 second)."

So make 1 the default if we can detect that it's a FAT filesystem.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12916] New: when rsync fails to chown, it gives up and does not set timestamp

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12916

Bug ID: 12916
   Summary: when rsync fails to chown, it gives up and does not
set timestamp
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: minor
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: i...@fsf.org
QA Contact: rsync...@samba.org

If I do

rsync -ot VFAT_FS_MOUNT

It outputs

rsync: chown "SOME_PATH" failed: Operation not permitted (1)

And it's timestamp is the current time, not the source time as specified
by -t. Rsync should not give up on setting the time just because it
couldn't set the owner. rsync -t works fine.

I tested on git master from today, ce854cf021cb7207ee504c6badf97b4ffbb2021a

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #1 from Kevin Korb  ---
This would not actually be very helpful.  Going between a *nix style timestamp
and a FAT one has other issues.  Especially if your TZ changes or you are in a
TZ where the clock changes twice per year.  Your modify windows will need to be
2 + (number of hours difference * 3600) which is kind of an unsafe thing to do.
 FAT just isn't capable of keeping proper timestamps.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12916] when rsync fails to chown, it gives up and does not set timestamp

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12916

--- Comment #1 from Kevin Korb  ---
If you mount your FAT filesystems with the quiet option the FAT driver will
just ignore the operations it can't do instead of throwing fatal errors at you.
 This will resolve problems in many tools including rsync.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #2 from Ian Kelling  ---
> This would not actually be very helpful.

Why? You haven't said. The manual says it's helpful
and it's helpful for me.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #5 from Kevin Korb  ---
Actually, I looked it up.  It wasn't actually my request I had already dealt
with the problem.

This is the ticket from the person I was helping: 
https://bugzilla.samba.org/show_bug.cgi?id=6927

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #3 from Brian K. White  ---
(In reply to Ian Kelling from comment #2)

The manuals says useful not helpful. The manual does not say it's good or bad
or recommended or any other value judgement. It only describes it's function
and the reason for it's existence. It does not say you automatically always
want it, even in that special case it was created for.

I'm not sure why it would automatically be a bad default either, just pointing
out that the manual makes no such claim that it is a good default.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #4 from Kevin Korb  ---
The manual says that with FAT you need --modify-window=2 not 1.  But here in
the US for half of the year you need --modify-window=3602 instead.  The same
goes for many other locations.  The simple fact of the matter is that FAT is an
obsolete filesystem designed for 5.25 inch floppy disks that is not in any way
compatible to doing 1990's vintage things.  I know there are devices out there
that are stuck with FAT but it is the user's job to adapt to their
incapabilities not rsync's.

Years ago, when I still had a Win98 computer to backup I suggested that rsync
should have a --fat option.  The --fat option would would have omitted --owner,
--group, --permissions from --archive and would have ignored an exactly 1 hour
difference in time stamp (back then rsync only supported whole second time
stamps anyway).  I was told no.  It isn't rsync's job to work around garbage
filesystems that have been obsolete for decades.

Rsync should not be responsible for detecting what a filesystem can handle. 
Many things can obscure those facts from rsync on a per-directory basis  (or
even a per-file basis).  Rsync should do what it is told to do or it should
report failure.  That is what it is doing.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #6 from Ben RUBSON  ---
Just to clarify, is FAT32 also impacted and requires --modify-window option, or
only the old one FAT is ? Thx !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12769] error allocating core memory buffers (code 22) depending on source file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #3 from Roland Haberkorn  ---
Ok, I digged somewhat deeper. I've found a second difference between my two
sources. The one is the original data, the other one is a diffential rsync
backup with hard links.
I then built a testcase with about 50 million dummy files with something like
this:

#!/bin/bash
for i in {1..50}
do
mkdir $i
#cd $i
for j in {1..1000}
do
mkdir $i/$j
#cd $j
for k in {1..1000}
do
touch $i/$j/$k
done
done
done

Rsyncing this testfolder work fine from and to any of the tested file systems
(ext4 64Bit, XFS, btrfs). This is true for with and without the Option -H and
as long as in the source file system there is no hard linked copy of the source
folder. 
In the moment when there is at least one hard linked copy the option -H breaks
the run:

roland@msspc25:~$ stat /mnt/rsynctestsource/1/1/1
  File: /mnt/rsynctestsource/1/1/1
  Size: 0   Blocks: 0  IO Block: 4096   regular empty file
Device: 811h/2065d  Inode: 153991349   Links: 2
Access: (0644/-rw-r--r--)  Uid: ( 2001/  roland)   Gid: (  100/   users)
Access: 2017-05-09 10:54:10.341300841 +0200
Modify: 2017-05-08 09:33:51.535967423 +0200
Change: 2017-05-26 16:21:57.610628573 +0200
 Birth: -
roland@msspc25:~$ rsync -rlptgoDxAn --info=name,progress2  --delete
--link-dest=/mnt2/link3/ /mnt/rsynctestsource/ /mnt2/link1/.
  0 100%0.00kB/s0:00:00 (xfr#0, to-chk=0/49049050)   
roland@msspc25:~$ rsync -rlptgoDxHAn --info=name,progress2  --delete
--link-dest=/mnt2/link3/ /mnt/rsynctestsource/ /mnt2/link1/.
  0 100%0.00kB/s0:00:00 (xfr#0, ir-chk=1000/25191050)
ERROR: out of memory in hashtable_node [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106)
[sender=3.1.2]

You can see, the first run without -H works, the last one with -H doesn't. 

So I would have to somewhat rename the bug report into "-H breaks the
incremental recursion on hard linked sources". This is as well true for all the
three file systems tested.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12916] when rsync fails to chown, it gives up and does not set timestamp

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12916

--- Comment #2 from Ian Kelling  ---
Great tip. If we can solve it that would be
good too.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #8 from Kevin Korb  ---
The 3602 problem happens when the clocks change not the entire time the clocks
are different.  If you do a daily rsync then rsync will see 1 hour off
timestamps the day after both clock changes.

Yes, this is all versions of FAT including FAT32.  Nobody really cares about
FAT16 anymore.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12915] --modify-window should default to 1 for fat filesystems

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12915

--- Comment #7 from Ian Kelling  ---
> But here in the US for half of the year you need --modify-window=3602 instead.

I think people use vfat because there are a lot of devices out there
like android which require it, and they only care about timestamps in so
much as they can tell rsync which files need to be copied.  We are in
daylight savings right now, and syncing to filesystem made with
mkfs.vfat seems to work fine with less than 3602 modify window.

> Rsync should not be responsible for detecting what a filesystem can handle.  
> Many things can obscure those facts from rsync on a per-directory basis  (or 
> even a per-file basis).

That's the only good argument I see. The bug asks if we can
detect it. If it's not feasable in normal efficient operation,
case closed.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11866

--- Comment #1 from Ben RUBSON  ---
I can easily reproduce this dangerous bug.
Both sides running rsync 3.1.2.

# cd /tmp/ && mkdir -p test && cd test && rm -f *
# ssh $usr@$srv "rm -rf /tmp/dst*"

# for i in `seq 0 9`
  do
  echo content_$i > $i
  done

# rsync -a . $usr@$srv:/tmp/dst1/

# rsync -a --link-dest=../dst1 . $usr@$srv:/tmp/dst2/

# ssh $usr@$srv "ls -lin /tmp/dst1 /tmp/dst2"
/tmp/dst1:
total 45
660319 -rw-r--r--  2 501  0  10  4 Jul 12:55 0
660320 -rw-r--r--  2 501  0  10  4 Jul 12:55 1
660321 -rw-r--r--  2 501  0  10  4 Jul 12:55 2
660322 -rw-r--r--  2 501  0  10  4 Jul 12:55 3
660323 -rw-r--r--  2 501  0  10  4 Jul 12:55 4
660324 -rw-r--r--  2 501  0  10  4 Jul 12:55 5
660325 -rw-r--r--  2 501  0  10  4 Jul 12:55 6
660326 -rw-r--r--  2 501  0  10  4 Jul 12:55 7
660327 -rw-r--r--  2 501  0  10  4 Jul 12:55 8
660328 -rw-r--r--  2 501  0  10  4 Jul 12:55 9
/tmp/dst2:
total 45
660319 -rw-r--r--  2 501  0  10  4 Jul 12:55 0
660320 -rw-r--r--  2 501  0  10  4 Jul 12:55 1
660321 -rw-r--r--  2 501  0  10  4 Jul 12:55 2
660322 -rw-r--r--  2 501  0  10  4 Jul 12:55 3
660323 -rw-r--r--  2 501  0  10  4 Jul 12:55 4
660324 -rw-r--r--  2 501  0  10  4 Jul 12:55 5
660325 -rw-r--r--  2 501  0  10  4 Jul 12:55 6
660326 -rw-r--r--  2 501  0  10  4 Jul 12:55 7
660327 -rw-r--r--  2 501  0  10  4 Jul 12:55 8
660328 -rw-r--r--  2 501  0  10  4 Jul 12:55 9

// Perfect, --link-dest correctly works.

# for i in `seq 0 9`
  do
  cp -a $i $i.orig
  echo content_$i >> $i
  done

# rsync -a -y --progress --stats --link-dest=../dst1 . $usr@$srv:/tmp/dst2/
./
0
 20 100%0.00kB/s0:00:00 (xfr#1, to-chk=19/21)
0.orig
 10 100%9.77kB/s0:00:00 (xfr#2, to-chk=18/21)
1
 20 100%   19.53kB/s0:00:00 (xfr#3, to-chk=17/21)
1.orig
 10 100%9.77kB/s0:00:00 (xfr#4, to-chk=16/21)
2
 20 100%   19.53kB/s0:00:00 (xfr#5, to-chk=15/21)
2.orig
 10 100%9.77kB/s0:00:00 (xfr#6, to-chk=14/21)
3
 20 100%   19.53kB/s0:00:00 (xfr#7, to-chk=13/21)
3.orig
 10 100%9.77kB/s0:00:00 (xfr#8, to-chk=12/21)
4
 20 100%   19.53kB/s0:00:00 (xfr#9, to-chk=11/21)
4.orig
 10 100%9.77kB/s0:00:00 (xfr#10, to-chk=10/21)
5
 20 100%9.77kB/s0:00:00 (xfr#11, to-chk=9/21)
5.orig
 10 100%4.88kB/s0:00:00 (xfr#12, to-chk=8/21)
6
 20 100%9.77kB/s0:00:00 (xfr#13, to-chk=7/21)
6.orig
 10 100%4.88kB/s0:00:00 (xfr#14, to-chk=6/21)
7
 20 100%9.77kB/s0:00:00 (xfr#15, to-chk=5/21)
7.orig
 10 100%4.88kB/s0:00:00 (xfr#16, to-chk=4/21)
8
 20 100%9.77kB/s0:00:00 (xfr#17, to-chk=3/21)
8.orig
 10 100%4.88kB/s0:00:00 (xfr#18, to-chk=2/21)
9
 20 100%9.77kB/s0:00:00 (xfr#19, to-chk=1/21)
9.orig
 10 100%3.26kB/s0:00:00 (xfr#20, to-chk=0/21)
Number of files: 21 (reg: 20, dir: 1)
Number of created files: 10 (reg: 10)
Number of deleted files: 0
Number of regular files transferred: 20
Total file size: 300 bytes
Total transferred file size: 300 bytes
Literal data: 190 bytes
Matched data: 110 bytes
File list size: 0
File list generation time: 0.002 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,422
Total bytes received: 549
sent 1,422 bytes  received 549 bytes  1,314.00 bytes/sec
total size is 300  speedup is 0.15

# ssh $usr@$srv "ls -lin /tmp/dst2"
total 90
660330 -rw-r--r--  1 501  0  20  4 Jul 12:55 0
660331 -rw-r--r--  1 501  0  10  4 Jul 12:55 0.orig
660332 -rw-r--r--  1 501  0  20  4 Jul 12:55 1
660333 -rw-r--r--  1 501  0  10  4 Jul 12:55 1.orig
660334 -rw-r--r--  1 501  0  20  4 Jul 12:55 2
660335 -rw-r--r--  1 501  0  10  4 Jul 12:55 2.orig
660336 -rw-r--r--  1 501  0  20  4 Jul 12:55 3
660343 -rw-r--r--  1 501  0  10  4 Jul 12:55 3.orig
660344 -rw-r--r--  1 501  0  20  4 Jul 12:55 4
660345 -rw-r--r--  1 501  0  10  4 Jul 12:55 4.orig
660346 -rw-r--r--  1 501  0  20  4 Jul 12:55 5
660347 -rw-r--r--  1 501  0  10  4 Jul 12:55 5.orig
660348 -rw-r--r--  1 501  0  20  4 Jul 12:55 6
660349 -rw-r--r--  1 501  0  10  4 Jul 12:55 6.orig
660350 -rw-r--r--  1 501  0  20  4 Jul 12:55 7
660351 -rw-r--r--  1 501  0  10  4 Jul 12:55 7.orig
660352 -rw-r--r--  1 501  0  20  4 Jul 12:55 8
660353 -rw-r--r--  1 501  0  10  4 Jul 12:55 8.orig
660354 -rw-r--r--  1 501  0  20  4 Jul 12:55 9
660355 -rw-r--r--  1 501  0  10  4 Jul 12:55 9.orig

// Perfect, update correctly done.

# rsync -a -yy --progress --stats --link-dest=../dst1 . $usr@$srv:/tmp/dst3/
sending incremental file list
./
0
 20 100%0.00kB/s0:00:00 (xfr#1, to-chk=19/21)
1
 20 100%   19.53kB/s0:00:00 (xfr#2, to-chk=17/21)
2
 20 100%   19.53kB/s0:00:00 (xfr#3, to-chk=15/21)
3
 20 100%   19.53kB/s0:00:00 (xfr#4, 

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11866

--- Comment #2 from Ben RUBSON  ---
Created attachment 13342
  --> https://bugzilla.samba.org/attachment.cgi?id=13342=edit
rsync_double_fuzzy_11866

Bug found, patch attached.
Wayne could you please review and commit please ?
Thank you very much !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12942] Traffic shaping: Make --bwlimit dynamic

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12942

--- Comment #1 from roland  ---
see https://bugzilla.samba.org/show_bug.cgi?id=7120

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12955] New: [patch] Fix rsync -A on AIX

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12955

Bug ID: 12955
   Summary: [patch] Fix rsync -A on AIX
   Product: rsync
   Version: 3.1.2
  Hardware: PPC
OS: AIX
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: gongcunj...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13458
  --> https://bugzilla.samba.org/attachment.cgi?id=13458=edit
Fix rsync -A on AIX

Hello,

When use rsync (version 3.1.2) to synchronize ACLs files (rsync -A) on
AIX 7.2(also on AIX 6.1 and AIX 5.3), the errors would occur as
follows:

  rsync: change_sacl_perms: sys_acl_get_tag_type(): No such file or directory
(2)

or

  rsync: change_sacl_perms: sys_acl_get_tag_type(): Error 0 (0)

This patch fixes the problem and has been tested on AIX 5.3, 6.1 and 7.2.

It also has been tested to synchronize ACLs files from Linux to AIX.

Thanks.

Cun

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12781] New: rsync library

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12781

Bug ID: 12781
   Summary: rsync library
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: enhancement
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: il...@altlinux.ru
QA Contact: rsync...@samba.org

I suggest the feature was requested for tens of times before me. But I couldn't
find it in the Bugzilla.
RSync as the protocol may be useful to many other applications. There even is a
librsync (https://github.com/librsync/librsync) which is used by a few
applications, yet that librsync is incompatible with this rsync
Do you think it's possible to create an API and extract the library out of the
rsync code?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #4 from Heinz-Willi Wyes  ---
I got a personal message from Lars Ellenberg that went also to
samba-b...@samba.org and rsync...@samba.org. He wrote:

--
Calling chmod only on (optimization: non-empty) directories would fix this.
I don't need to chmod a *file* before unlinking it, I just need write
permission on the directory it is located in.

(Now you have to convince the "appliance" to use a patched rsync ...)

Cheers,

Lars Ellenberg


diff --git a/delete.c b/delete.c
index 88e4230..223b6d2 100644
--- a/delete.c
+++ b/delete.c
@@ -97,10 +97,10 @@ static enum delret delete_dir_contents(char *fname, uint16
flags)
 }

 strlcpy(p, fp->basename, remainder);
-if (!(fp->mode & S_IWUSR) && !am_root && fp->flags & FLAG_OWNED_BY_US)
-do_chmod(fname, fp->mode | S_IWUSR);
 /* Save stack by recursing to ourself directly. */
 if (S_ISDIR(fp->mode)) {
+if (!(fp->mode & S_IWUSR) && !am_root && fp->flags &
FLAG_OWNED_BY_US)
+do_chmod(fname, fp->mode | S_IWUSR);
 if (delete_dir_contents(fname, flags | DEL_RECURSE) != DR_SUCCESS)
 ret = DR_NOT_EMPTY;
 }
@@ -138,14 +138,13 @@ enum delret delete_item(char *fbuf, uint16 mode, uint16
flags)
 fbuf, (int)mode, (int)flags);
 }

-if (flags & DEL_NO_UID_WRITE)
-do_chmod(fbuf, mode | S_IWUSR);
-
 if (S_ISDIR(mode) && !(flags & DEL_DIR_IS_EMPTY)) {
 /* This only happens on the first call to delete_item() since
  * delete_dir_contents() always calls us w/DEL_DIR_IS_EMPTY. */
 ignore_perishable = 1;
 /* If DEL_RECURSE is not set, this just reports emptiness. */
+if (!(mode & S_IWUSR) && !am_root && flags & DEL_NO_UID_WRITE && flags
& DEL_RECURSE)
+do_chmod(fbuf, mode | S_IWUSR);
 ret = delete_dir_contents(fbuf, flags);
 ignore_perishable = 0;
 if (ret == DR_NOT_EMPTY || ret == DR_AT_LIMIT)
-

As far as I understand, there actually *is* a design flaw in the rsync
implementation that causes the behavior I described in my original post.

I suggested the patch to my provider. They replied that they rather wait for a
new release of rsync with that patch officially applied.

Now I wonder what the status is. Will there be a patched version of rsync?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12173] memory leak around poptGetOptArg()

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12173

T Yamada  changed:

   What|Removed |Added

Version|3.0.6   |3.1.1

--- Comment #1 from T Yamada  ---
3.1.1 is still affected. How is it going?

==22846== 4 bytes in 1 blocks are definitely lost in loss record 2 of 8
==22846==at 0x4C28BED: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22846==by 0x524049E: ??? (in /lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==22846==by 0x5241994: poptGetNextOpt (in
/lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==22846==by 0x13AE04: ??? (in /usr/bin/rsync)
==22846==by 0x114D7E: main (in /usr/bin/rsync)
==22846== 
==22846== LEAK SUMMARY:
==22846==definitely lost: 4 bytes in 1 blocks
==22846==indirectly lost: 0 bytes in 0 blocks
==22846==  possibly lost: 0 bytes in 0 blocks
==22846==still reachable: 890 bytes in 7 blocks
==22846== suppressed: 0 bytes in 0 blocks

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12173] memory leak around poptGetOptArg()

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12173

--- Comment #2 from Paul Slootman  ---
I think there are more urgent problems than a memory leak of less than 1kB,
which I expect isn't really a leak but memory which isn't freed at the end of
execution but may be used up to that time. So is it really a leak or a bit of
bad housekeeping which doesn't matter at all?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12173] memory leak around poptGetOptArg()

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12173

--- Comment #3 from T Yamada  ---
> definitely lost: 4 bytes in 1 blocks
For entire program.
But maybe this priority is low.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12769] error allocating core memory buffers (code 22) depending on source file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #2 from Roland Haberkorn  ---
I did some further investigation... 
First thing to add: The ext4 file systems are hard-linked differential rsync
backups of the real data on XFS.
I changed the testcase by deleting the --link-dest option.
When rsyncing from an XFS, the rsync process on the client uses about 3% RAM
(of total 8GB). When rsyncing from an ext4, it uses up to about 50% RAM.
This picture totally changes when I delete the option -H. In this case also
copying from an ext4 uses only less than 2% RAM. 
My guess would be that perhaps -H breaks the incremental recursion when copying
from an ext4.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12781] rsync library

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12781

--- Comment #1 from Axel Kittenberger  ---
I asked exactly this 10 years ago to the mailing list.

The answer was no.

This is not a simple to do.

You could do it, I decided that kind of project would move me too far away from
what I actually wanted to achieve. I went instead for calling rsync from a
daemon and handing data via pipes. ( https://github.com/axkibe/lsyncd ). You
just need some process management.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10738] report --stats output when termination signal arrives

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10738

Ben RUBSON  changed:

   What|Removed |Added

 CC||ben.rub...@gmail.com

--- Comment #4 from Ben RUBSON  ---
Hello,

I'm facing this "issue" too, I really would like to have stats displayed when
rsync receives a termination signal.
Sometimes rsync runs all night long and is killed in the morning to save
bandwidth, would then really be interesting to have stats to help with
reporting.

Any progress in this then ?

Many thx !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] New: Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

Bug ID: 12806
   Summary: Deleting in a row of hardlinked snapshots resets file
permissions.
   Product: rsync
   Version: 3.1.0
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: h...@delember-wyes.de
QA Contact: rsync...@samba.org

Created attachment 13239
  --> https://bugzilla.samba.org/attachment.cgi?id=13239=edit
Log file of bug reproduction

Overview:
In an differential backup system with hard-linked copies of unchanged files,
deleting one snapshot resets the permission bit on all files.

Steps to reproduce:
1) Set up a source directory, fill in a couple of files. Some text is just
fine.
Make one of these files have 444 permissions.

2) Set up a different directory to hold snapshots of the source directory.

3) Make a unique name for a snapshot, using e.g. the current date/time.

4) Make a snapshot using rsync:
rsync -ar --itemize-changes src/ snapshots/`echo $snap`/
$snap contains the name of the snapshot. Save this name in a file
"lastsnapshot".

5) Make a new name for a new snapshot and make it using:
rsync -ar --itemize-changes --link-dest=`pwd`/snapshots/`cat lastsnapshot`/
src/ snapshots/`echo $snap`/. Again, save the name of the last snapshot.

6) Repeat step 5) a couple of times. In the snapshot directory, you will get a
row of snapshots. All files in each snapshot is hard-linked to the counterparts
in the other snapshots. Watch the permission bits of the one file with 444
permissions: Everything is ok.

7) Now, deliberately delete one of the snapshots using e.g.
rsync -r --delete `mktemp -d`/ snapshots/2017-05-26-15-00-53/

8) Now all of the files in all snapshots have 644 permissions. This will cause
the re-transmission of that file with the following snapshot.

Actual results: On deletion of a snapshot, hardlinked files get the wrong
permissions.

Expected results: Permission bits should be left intact.

Attached is a log file where I documented this behaviour.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #1 from Heinz-Willi Wyes  ---
Addendum.
Thought that the --inplace option could play the trick. I just employed the
following:

rsync -r --delete --inplace `mktemp -d`/ snapshots/2017-05-28-08-10-11/

But this does not help either. Permission bits get reset to 644 anyway.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #2 from Kevin Korb  ---
Just use rm -rf to delete a backup.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10738] report --stats output when termination signal arrives

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10738

--- Comment #6 from Ben RUBSON  ---
Mmmh however some of the statistics seem to be wrongly calculated.

### Example with tsync 3.1.2 :

Number of files: 8,732 (reg: 7,568, dir: 1,163, link: 1)
Number of created files: 1,232 (reg: 1,187, dir: 45)
Number of deleted files: 0
Number of regular files transferred: 1,187
Total file size: 17.92G bytes
Total transferred file size: 1.48G bytes
Literal data: 914.70M bytes
Matched data: 0 bytes
File list size: 27.07K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 0
Total bytes received: 0

sent 0 bytes  received 0 bytes  -0.00 bytes/sec
total size is 17.92G  speedup is -9,223,372,036,854,775,808nf

### Example with an old rsync 2.6.9 :

Number of files: 24626 
Number of files transferred: 827 
Total file size: 69­.44G bytes 
Total transferred file size: 1­.20G bytes 
Literal data: 633­.50M bytes 
Matched data: 441­.70K bytes 
File list size: 526135 
File list generation time: 0­.588 seconds 
File list transfer time: 0­.000 seconds 
Total bytes sent: 0 
Total bytes received: 0 

sent 0 bytes  received 0 bytes  -0­.0 bytes/sec 
total size is 69­.44G  speedup is inf

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10738] report --stats output when termination signal arrives

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10738

--- Comment #5 from Ben RUBSON  ---
So I just went through the code and found that this feature request is already
implemented.
We just have to use SIGUSR2 and rsync will stop, displaying stats.
kill -USR2 
Perfect !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #3 from Heinz-Willi Wyes  ---
(In reply to Kevin Korb from comment #2)
I'd like to, but I can't. I set up a local scenario in order to make the
problem most easily reproducible. But the real use case involves rsync in a
cloud backup application where I get no acces to the cloud provider's shell.
Using any other possible protocol so as e.g. ftp or cifs is practically not
possible. Each file operation with these protocols requires a roundtrip to my
local server. Such an attempt lasts for hours considering that there are some
100.000 files per snapshot. rsync does this in less than two minutes - but
resets the file permissions.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12817] New: [PATCH] Allow daemon itself to chroot

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12817

Bug ID: 12817
   Summary: [PATCH] Allow daemon itself to chroot
   Product: rsync
   Version: 3.1.2
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ben.rub...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13248
  --> https://bugzilla.samba.org/attachment.cgi?id=13248=edit
rsync_daemon_chroot

Hello,

Here is a patch which adds 3 new parameters to rsyncd.conf :
daemon chroot
daemon gid
daemon uid

The first one is a path to a directory the daemon itself will chroot to before
beginning communication with clients.
The 2 others are the uid/gid the daemon itself will switch to before beginning
communication with clients.

These parameters can improve security.
For example, using daemon via a restricted remote-shell connection, for
security reasons, if we want whole rsync to be chrooted, we can now use :

daemon chroot = /home/%SUDO_USER%/rsync/
daemon uid = %SUDO_UID%
daemon gid = %SUDO_GID%

With of course rsync being sudo-called by the restricted shell (to configure
properly).

We could already do this without this patch, using the "use chroot" parameter,
but then the daemon itself is not chrooted and remains run by root.

Thank you !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12817] [PATCH] Allow daemon itself to chroot

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12817

Ben RUBSON  changed:

   What|Removed |Added

  Attachment #13250|0   |1
is obsolete||

--- Comment #3 from Ben RUBSON  ---
Created attachment 13251
  --> https://bugzilla.samba.org/attachment.cgi?id=13251=edit
rsync_daemon_chroot

Minor issue corrected : do not forget to not sanitize_paths if daemon is
chrooted.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12817] [PATCH] Allow daemon itself to chroot

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12817

Ben RUBSON  changed:

   What|Removed |Added

  Attachment #13249|0   |1
is obsolete||

--- Comment #2 from Ben RUBSON  ---
Created attachment 13250
  --> https://bugzilla.samba.org/attachment.cgi?id=13250=edit
rsync_daemon_chroot

Minor issue corrected : do not forget to init log before chrooting + typo.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12817] [PATCH] Allow daemon itself to chroot

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12817

Ben RUBSON  changed:

   What|Removed |Added

  Attachment #13248|0   |1
is obsolete||

--- Comment #1 from Ben RUBSON  ---
Created attachment 13249
  --> https://bugzilla.samba.org/attachment.cgi?id=13249=edit
rsync_daemon_chroot

Minor issue corrected : do not forget to init log before chrooting.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] New: [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

Bug ID: 12819
   Summary: [PATCH] sync() on receiving side for data consistency
   Product: rsync
   Version: 3.1.2
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ben.rub...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13253
  --> https://bugzilla.samba.org/attachment.cgi?id=13253=edit
rsync_sync

Hello,

Here is a patch which sync() once files received, for data consistency.

Thank you !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12838] New: [PATCH] Log sent/received bytes even in case of error

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12838

Bug ID: 12838
   Summary: [PATCH] Log sent/received bytes even in case of error
   Product: rsync
   Version: 3.1.2
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ben.rub...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13271
  --> https://bugzilla.samba.org/attachment.cgi?id=13271=edit
rsync_log_daemon

Hello,

This patch allows the daemon to log data sent/received even in case of error,
which can be interesting to have when the transfer has been interrupted by the
client.

Thank you !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #7 from Ben RUBSON  ---
And what about a power failure between 2 ZFS transaction groups ?

Note that my patch simply adds a sync() just after recv_files(), so one sync()
per connection, not per write operation.
Quite low workload actually :)

But we could make this a rsync option, so that one can enable / disable it on
its own.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #8 from Brian K. White  ---
You tell me, what ABOUT a power failure between 2 zfs, or any other fs
operations?

This does not improve or solve any problem that the fs and all the other layers
aren't already handling. This is simply a misguided idea, however sensible and
attractive it seems.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #5 from Heinz-Willi Wyes  ---
Lars Ellenberg provided a workaround for the behaviour. Using

rsync -d --delete --super `mktemp -d`/ snapshots/2017-05-26-15-00-53/

plays the trick.

Not sure whether the --super option has implications not suitable for other
scenarios. For me it works just fine. Nevertheless, there should be a patch for
rsync in order to overcome the questionable behaviour.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #5 from Brian K. White  ---
Any program could make this same "just to be safe" argument practically every
time they ever close-on-write for any reason. If they wrote anything, it was
always for some reason, and they want to know for sure that it really got
safely written. There is nothing special about rsync in that regard. cp might
as well have it. The ">" operator in bash might as well have it.

The kernel and vfs and hardware drivers all already do whatever is necessary in
that regard, and it's generally wrong for any application to try to do it
itself. Otherwise the disk would be in a constant state of sync()'ing and never
actually manage to get any other work done. Consider a multiuser host with 500
rsync receivers. Each individual sync() is incredibly disruptive to all other
processes. "Everyone hold up while we flush the disk buffer...". The entire
system waits while that happens.

That way just leads to things like the example you just used, lower layers that
just start lying about sync() to upper layers because too many apps use it when
they shouldn't. "Fine, if apps are going to sync all the time, that ends up
being 86 times a second between all procs running at any given moment, which is
unsupportable, so we'll just make sync() a no-op stub and we'll do it when
it's' actually required, and apps can sync()-away to their hearts content".

I think the only reason rsync might have to sync is if you built rsync as a
self-contained bootable executable like memtest86, or possibly as an MS-DOS
executable.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #6 from Brian K. White  ---
Think of it this way, write() already makes a certain promise that it will not
return until it's done it's job, and it will not assert success when it can't.
Essentially the man page for any syscall is a contract. In fact all API's are
contracts.

write() in turn is relies on various other calls to even lower layers to keep
their promises too, to manage the in-kernel buffer or the cache on a raid card
etc.

All of these things MUST be relied on rather than second-guessed. It would be
insane for example, for write() to say "I can't really be sure this disk driver
has really done it's thing. I better force it to sync before I return to the
application." or "I can't really be sure malloc() really allocated the memory,
I better malloc 3 or 4 copies and compare them and use whichever copies agree
with each other... It's insane.

You write(), you check the return value, and you're done. The low level
hardware is someone else's job, and you won't be doing a better job than they
already did.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #2 from Ben RUBSON  ---
Thank you for your feedback Brian.
I don't have any problem.
I just want to be sure that when client (sender) has finished its transfer, its
data is on server's (receiver) disks, before it disconnects.
So that when it correctly / successfully disconnects, its data is for sure on
disks.

On disks means on platters, so that if there is a failure (hardware, power...),
data is safe, not lost.

Of course disks which do not lie about sync() command must be used (data must
be on platters, not only in disks' cache). As well as a robust filesystem, some
redundancy... (but here that's off-topic).

Perhaps we could make it an option, so that those who have OS failing to manage
write buffers would not be degraded even more... But certainly they should have
a look to their performance issue first.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12835] New: Allow --link-dest to link to an optionally unexisting directory

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12835

Bug ID: 12835
   Summary: Allow --link-dest to link to an optionally unexisting
directory
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: enhancement
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: julien+bugzilla-sa...@palard.fr
QA Contact: rsync...@samba.org

Found myself trying to hardlink between versions, but as I may rsync to a new
server I may not having an old version to point to, so I had to use:

  rsync -rl --link-dest=$(ssh prod "cd /var/www; readlink -e current || echo
.") static index.html prod:/var/www/$CI_COMMIT_SHA/
  ssh prod "ln -nfs $CI_COMMIT_SHA" current 

Which is ugly.

I'd prefer something like:

  rsync -rl --link-dest=current static index.html prod:/var/www/$CI_COMMIT_SHA/
  ssh prod "ln -nfs $CI_COMMIT_SHA" current

But warning / erroring on missing --link-dest sound legitimate, and we may not
want to change this. If not, why not introduce a --link-dest-if-exists? Or a
--ignore-missing-link-dest (in combination with the normal --link-dest)?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #1 from Brian K. White  ---
This seems wrong to me. If the OS is failing to manage write buffers and file
access between processes, you would have a lot bigger problems in every process
all through the system, and this wouldn't fix it.

Similarly, if rsync were corrupting data, a lot of people would already know
about it. It gets used way too much and too heavily for anything like this to
go unnoticed for more than a day, let alone 15 or more years.

It's almost axiomatic: No matter what problem you think you have, no matter
what language or OS or platform, if you think it's fixed by either sleep() or
sync(), it's not.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #3 from Paul Slootman  ---
How about just using a post-xfer command on the server side that does 'sync'?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12819] [PATCH] sync() on receiving side for data consistency

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12819

--- Comment #4 from Ben RUBSON  ---
Yes Paul I thought about it but sync command may not be available if the server
(receiver) is chrooted (for example using patch proposed in #12817).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12838] [PATCH] Log sent/received bytes even in case of error

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12838

--- Comment #1 from Ben RUBSON  ---
Mmmmh unfortunately numbers are not correct.

# Sender side (which interrupts the transfer receiving SIGUSR2) :
Total file size: 16­.85G bytes 
Total transferred file size: 183­.38M bytes 
Literal data: 182­.32M bytes 

# Receiver side : 
rsyncd: rsync: connection unexpectedly closed (202529 bytes received so far)
[generator]
rsyncd: sent 24749 bytes  received 202529 bytes  total size 16845198594

Perhaps related to bug #10738 where stats are also not correctly calculated.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12583] [PATCH] Add new daemon option "syslog tag"

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12583

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #1 from Wayne Davison  ---
Thanks for the patch!  I tweaked it a bit and committed the result to git.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12755] [patch] Improve execution speed on Windows; with Win32 API calls

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12755

--- Comment #2 from Joseph Benden  ---
Please disregard the patch attachment as the actual definitive solution and
rather visit my GitHub repository for the current working source and binaries.

If this solution is accepted for inclusion in to Rsync or the Rsync-Patches
repository, I will re-create a viable patch. I will also do so if asked...

See GitHub: https://github.com/jbenden/rsync/

Thank you! Comments, suggestions, and ideas are always welcomed!
-Joe

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12755] [patch] Improve execution speed on Windows; with Win32 API calls

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12755

Wayne Davison  changed:

   What|Removed |Added

  Attachment #13169|application/mbox|text/plain
  mime type||
  Attachment #13169|0   |1
   is patch||

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12583] [PATCH] Add new daemon option "syslog tag"

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12583

--- Comment #2 from Ben RUBSON  ---
Wayne thank you very much !!
Really perfect :)
Many thx !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12769] error allocating core memory buffers (code 22) depending on source file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #1 from Roland Haberkorn  ---
If you want me to run further testings with other file systems I am totally
willing to produce fake data and run tests. I just haven't done yet because of
my lack of knowledge about the underlying mechanisms and because I am not
totally sure whether this is a rsync's problem or a kernel issue.
To add two more things: We saw this issue also when having mounted the data
with NFSv3 or v4. The target file system does not matter, we've had this issue
with btrfs, Ext4 and XFS.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12769] New: error allocating core memory buffers (code 22) depending on source file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12769

Bug ID: 12769
   Summary: error allocating core memory buffers (code 22)
depending on source file system
   Product: rsync
   Version: 3.1.0
  Hardware: All
OS: Linux
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: mss...@fau.de
QA Contact: rsync...@samba.org

We run an openSuSE Leap 42.2 and an Ubuntu 14.04.5 on two servers. Copying a
large number of files (in this case about 28 million) leads to different
results depending on the source file system. 
We copy with rsync -rlptgoDAxHnP --info=progress2 --delete
--link-dest=$LINK_DEST root@$SERVER:/$FOLDER /backup/rsynctest/ . Replacing
--delete by --delete-delay doesn't change the behaviour as expected. The error
occurs with and without the option -n, in this case it is just for testing
reasons included.
In case the source is located on an Ext4 file system we run into the following
error message after about 26 million files copied:
ERROR: out of memory in hashtable_node [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102)
[sender=3.1.0]
In case the source is located on an XFS file system the above command copies
all files without error.
Both of the file systems hold the same data as the one is the backup copy of
the other. The behaviour appears as well when we use rsync via an rsync server
and not via SSH and as well when we copy locally on one of the two machines.
And it appears regardless of the operating system (openSuSE 42.2 or Ubuntu
14.04.5). 
I did not try in the last time but replaying the backup from an Ext4 showed
this error at least one year ago as well. With the change to XFS on the source
file system the error suddenly disappeared.
As the error appears even if just doing a --dry-run it seems to be related to
the way rsync handles metadata. The data size seems to be unimportant.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12820] New: rsync always try change owner and group of symlink in --fake-super mode

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12820

Bug ID: 12820
   Summary: rsync always try change owner and group of symlink in
--fake-super mode
   Product: rsync
   Version: 3.0.9
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: pa...@hubbitus.info
QA Contact: rsync...@samba.org

Hello.

I long time discover strange to me behavior. If I add --fake-super
option on both sides rsync start always set owner and group of symlink.
What interesting there no error, but it done each time as it has been
changed.

Simple reproduce:
$ mkdir rsync.symlinks
$ echo 'file content' > rsync.symlinks/file.itself
$ ln -s file.itself rsync.symlinks/file.symlink

Then try to transfer it on remote host:
$ rsync -apr --links --itemize-changes --fake-super -M--fake-super
rsync.symlinks [hidden email]:~/temp/rsync.symlinks
cd+ rsync.symlinks/
 file.itself

And again:
.Log... rsync.symlinks/file.symlink -> file.itself


I had tried use --rsync-path="rsync --fake-super" instead of
-M--fake-super what unsurprisingly give me same result:
$ rsync -apr --links --itemize-changes --fake-super --rsync-path="rsync
--fake-super" rsync.symlinks
[hidden email]:~/temp/rsync.symlinks
.Log... rsync.symlinks/file.symlink -> file.itself


Please note all symlinks correct:
$ LANG=en_US.utf-8 ls -l rsync.symlinks
total 4
-rw-rw-r--. 1 pasha pasha 13 Oct 11 06:10 file.itself
lrwxrwxrwx. 1 pasha pasha 11 Oct 11 06:11 file.symlink -> file.itself

$ cat rsync.symlinks/file.symlink
file content

It also correct on receiver side.


P.S. Asked long time ago in ML -
http://samba.2283325.n4.nabble.com/rsync-always-try-change-owner-and-group-of-symlink-in-fake-super-mode-td4692664.html

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

--- Comment #4 from chd...@gmail.com ---
Communicating with Homebrew today, it's still not clear whether this because of
anything they're doing, or whether rsync will have to do something to
accommodate the combination of Xcode 9 (which only comes with a 10.13 SDK) with
macOS 10.12.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

--- Comment #2 from Andrey Gursky  ---
utimensat() specified in POSIX.1-2008 (about 9! years ago) has been (at last!)
added in macOS 10.13. So if you've compiled binary using latest Xcode i.e. for
macOS 10.13, then it depends on the existence of utimensat() in the libc, which
is missing on pre-10.13.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

--- Comment #3 from Ben RUBSON  ---
Nearly off topic, but what is the embedded rsync version in Mac OS 10.13 ?
10.11 still uses 2.6.9 :| ...
Thx !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] New: On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

Bug ID: 13044
   Summary: On macOS 10.12.6 with the new Xcode 9, `make check` is
full of failures
   Product: rsync
   Version: 3.1.2
  Hardware: x86
OS: Mac OS X
Status: NEW
  Severity: regression
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: chd...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13614
  --> https://bugzilla.samba.org/attachment.cgi?id=13614=edit
Log for `make check`

macOS 10.12.6, `make check` is full of failures using Xcode 9, but all those
same tests pass when using Xcode 8.3.3.  I'm attaching a log, but here's an
example:

created directory
/Users/chdiza/.tmp/tmpdir/rsync-20170920-66113-1rmtqxr/rsync-3.1.2/testtmp/exclude/chk
./
.filt
dyld: lazy symbol binding failed: Symbol not found: _utimensat
  Referenced from:
/Users/chdiza/.tmp/tmpdir/rsync-20170920-66113-1rmtqxr/rsync-3.1.2/rsync
  Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: _utimensat
  Referenced from:
/Users/chdiza/.tmp/tmpdir/rsync-20170920-66113-1rmtqxr/rsync-3.1.2/rsync
  Expected in: /usr/lib/libSystem.B.dylib

rsync: connection unexpectedly closed (132 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226)
[sender=3.1.2]
- exclude log ends

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

--- Comment #1 from chd...@gmail.com ---
Hmmm, I can only reproduce this when using Homebrew.  I've reported this to
them, so we'll see whether it's their "fault" or whether rsync needs to adjust
for Xcode 9.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

--- Comment #6 from chd...@gmail.com ---
Homebrew's builds for different macOS versions will use as few versions of
Xcode as possible.  Sometimes one version of Xcode is used on two different
macOS VMs with differing macOS versions.  Anyway, those are for Homebrew's
pre-compiled binaries (and in the case of rsync, they don't run `make check`
when building said binaries).  I want to build rsync from source.

I've found that if one manually runs `xcode-select` to point to the folder
where the Command Line Tools live (as opposed to pointing at the tools included
in Xcode.app), there is no problem with rsync and its tests.

As far as I can see now, this is a bug in Xcode 9.0.

Apple is known to be stubborn about this sort of thing though, so it might
require rsync to do some workarounds.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13044] On macOS 10.12.6 with the new Xcode 9, `make check` is full of failures

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13044

--- Comment #5 from Andrey Gursky  ---
> Xcode 9 (which only comes with a 10.13 SDK)

I guess you obviously will not be able to compile programs for other macOS
versions with such Xcode 9. Unless you install the appropriate SDK manually and
point Xcode to use it (if this is possible). From what I know: homebrew has 3
different builds for every package. It is because Apple supports, sure, the
very last release, the release many people still have installed and the
previous "LTS" release. I suppose each build is made with a suitable SDK,
otherwise why to have 3 different builds?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12417] Sametimes currupted file after disconnect

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12417

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |WORKSFORME
 Status|NEW |RESOLVED

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11656] Escaping broken with --files-from

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11656

Wayne Davison  changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution|--- |WORKSFORME

--- Comment #10 from Wayne Davison  ---
The consistent and documented escaping is that some characters get ouput with a
backslash+hash+3-digit octal number. This includes control chars, some
backslashes, and (without -B) high-bit chars.  From the man page:

"The escape idiom that started in 2.6.7 is to output a literal backslash (\)
and a hash (#), followed by exactly 3 octal digits.  For example, a newline
would output as "\#012".  A literal backslash that is in a filename is not
escaped unless it is followed by a hash and 3 digits (0-9)."

One easy way to unescape is thus to filter names through something like this:

perl -pe 's/\\#(\d\d\d)/chr(oct($1))/eg'

(...after any necessary parsing of the output to find the names or twiddle
newlines into nulls).  You'll note that this only matches exactly 3-digits, as
rsync will leave something like "\#5" alone in the output, since it cannot be
confused with an actual escaped char.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11866

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #4 from Wayne Davison  ---
Thanks for figuring out the issue and providing a patch. I've committed it to
git.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12568] Integer overflow still exists in xattrs.c, leading to buffer overflow

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12568

Wayne Davison  changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution|--- |FIXED

--- Comment #1 from Wayne Davison  ---
I've committed a fix for this into git. Many thanks for pointing this out.
Sorry for how slow I've been lately.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11866

--- Comment #5 from Ben RUBSON  ---
Thank you very much for the merge Wayne !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11571] rsync-3.1.1 --link-dest arbitrary limit

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11571

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #1 from Wayne Davison  ---
The limit is now documented.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11338] Rsync Crash - Segmentation fault

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #2 from James Stevenson  ---

I seem to see the same crash. Here is some additional details with symbols. It
seems to happen during network timeouts.

#0  0x7fcc2e167b05 in utf8_internal_loop (step=0x560710d73f90,
step_data=0x560710d740c0, irreversible=0x7ffccf327d28, outend=0x560710d7dad0
"", outptrp=, 
inend=0x7ffccf3287f8 "\n", inptrp=0x7ffccf327e80) at ../iconv/loop.c:325
#1  __gconv_transform_utf8_internal (step=0x560710d73f90,
data=data@entry=0x560710d740c0, inptrp=inptrp@entry=0x7ffccf327e80,
inend=inend@entry=0x7ffccf3287f8 "\n", 
outbufstart=outbufstart@entry=0x0,
irreversible=irreversible@entry=0x7ffccf327df8, do_flush=do_flush@entry=0,
consume_incomplete=consume_incomplete@entry=0)
at ../iconv/skeleton.c:613
#2  0x7fcc2e162722 in __gconv (cd=0x560710d740b0,
inbuf=inbuf@entry=0x7ffccf327e80, inbufend=0x7ffccf3287f8 "\n",
outbuf=outbuf@entry=0x7ffccf327e98, outbufend=, 
irreversible=irreversible@entry=0x7ffccf327df8) at gconv.c:79
#3  0x7fcc2e161fa1 in iconv (cd=cd@entry=0x560710d740b0,
inbuf=inbuf@entry=0x7ffccf327e80, inbytesleft=inbytesleft@entry=0x7ffccf327e88,
outbuf=outbuf@entry=0x7ffccf327e98, 
outbytesleft=outbytesleft@entry=0x7ffccf327e90) at iconv.c:52
#4  0x56070fa1954f in iconvbufs (ic=ic@entry=0x560710d740b0,
in=in@entry=0x7ffccf327f20, out=out@entry=0x7ffccf327f00, flags=0) at
rsync.c:208
#5  0x56070fa30191 in rwrite (code=, code@entry=FERROR,
buf=, 
buf@entry=0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes
received so far) [sender]\n", len=, is_utf8=,
is_utf8@entry=0) at log.c:374
#6  0x56070fa307e3 in rprintf (code=code@entry=FERROR,
format=format@entry=0x56070fa677b0 "rsync: connection unexpectedly closed (%s
bytes received so far) [%s]\n") at log.c:433
#7  0x56070fa39064 in whine_about_eof (allow_kluge=allow_kluge@entry=0) at
io.c:231
#8  0x56070fa3c992 in read_buf (f=f@entry=5, buf=buf@entry=0x7ffccf329cd0
"\037", len=len@entry=4) at io.c:1834
#9  0x56070fa3caa1 in read_int (f=f@entry=5) at io.c:1729
#10 0x56070fa3e48c in setup_protocol (f_out=4, f_in=5) at compat.c:158
#11 0x56070fa2d4b0 in client_run (f_in=5, f_out=4, pid=9022, argc=1,
argv=0x560710d74760) at main.c:1110
#12 0x56070fa109a1 in start_client (argv=0x560710d74760, argc=1) at
main.c:1420
#13 main (argc=2, argv=0x560710d74760) at main.c:1648
(gdb) f 4
#4  0x56070fa1954f in iconvbufs (ic=ic@entry=0x560710d740b0,
in=in@entry=0x7ffccf327f20, out=out@entry=0x7ffccf327f00, flags=0) at
rsync.c:208
208in rsync.c
(gdb) p flags
$15 = 0
(gdb) p xbuf
Attempt to use a type name as an expression
(gdb) p in
$16 = (xbuf *) 0x7ffccf327f20
(gdb) p *in
$17 = {buf = 0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes
received so far) [sender]\n", pos = 18889, len = 18446744073709532799, size =
18446744073709551615}
(gdb) p *out
$18 = {buf = 0x7ffccf327f40 "\"\023", pos = 0, len = 0, size = 1024}

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10776] SIGSEGV in utf8_internal_loop()

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10776

--- Comment #2 from James Stevenson  ---

This looks like a possible duplicate of
https://bugzilla.samba.org/show_bug.cgi?id=11338

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13082] New: [REQ] Hardware / SSE based MD5 operations

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13082

Bug ID: 13082
   Summary: [REQ] Hardware / SSE based MD5 operations
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ben.rub...@gmail.com
QA Contact: rsync...@samba.org

Hi,

I made some performance tests and ses that rsync can be a bottleneck while it
is calculating hashes of basis files (not a surprise).

For example, while I can read a big test file directly from the storage at 300
MB/s, rsync checksums it at about 200 MB/s.

Thus impact of calculating hashes for big files can be quite important (some
minutes for files of several hundreds of GB).

Then, what about MD5 implementation based on SSE ?
Do you think we would have a performance improvement ?

Thank you !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13061] New: File lost on case-insensitive file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13061

Bug ID: 13061
   Summary: File lost on case-insensitive file system
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: Windows 7
Status: NEW
  Severity: major
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: t...@towo.net
QA Contact: rsync...@samba.org

On a case-insensitive file system (Windows/cygwin):
I renamed a file to upper case, then ran my backup script using
rsync -r --delete-after src /backup

I got this error message:
rsync: send_files failed to open "/cygdrive/d/home/tools/mined/src/n/x":
Permission denied (13)
IO error encountered -- skipping file deletion

The old (lower case) file was finally deleted in the backup area, but the new
(upper case) file had not been copied to it.
(It was copied of course when I ran rsync again.)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13061] File lost on case-insensitive file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13061

--- Comment #1 from Kevin Korb  ---
Are you saying that it deleted a file AFTER saying "IO error encountered --
skipping file deletion"?

Maybe you need some --itemize-changes to make sure.  Unfortunately, this is a
limitation of the filesystem and rsync *should* have simply left the file
untouched (and never be able to update it) unless --ignore-errors.  The
behaviour you describe is what --ignore-errors should do.

The --fuzzy option might possibly help here.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13061] File lost on case-insensitive file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13061

--- Comment #2 from t...@towo.net ---
Actually, I was too eager minimizing the test case, sorry.
I can only reproduce it reliably with additional options -vltoD :
rsync -vrltoD --delete-after src /backup

Also sorry for the bogus error message, actually it looks like:
building file list ... done
deleting src/vivaldi

(where a file src/Vivaldi exists, a file /backup/src/vivaldi existed before and
is then deleted)

Attempts to remove options for reproducing the issue expose weird behaviour;
for example, if I strip "ltoD":
rsync -vr --delete-after src /backup

it also happens initially, but if I repeat the test, it does not happen again
until I revert to the previous options (of course with a fresh test case setup
each time).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13061] File lost on case-insensitive file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13061

--- Comment #3 from Kevin Korb  ---
That makes even less sense.  Rsync doesn't do anything to the source at all
unless --remove-source-files and it only does that after it successfully
transfers a file.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13061] File lost on case-insensitive file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13061

--- Comment #5 from Kevin Korb  ---
OK, now I understand what is going on.  It is a 2-part problem...
Rsync sees the source file as new because it does not exist in the target but
it fails to copy the file because the name conflicts with an existing file. 
That should abort the deletions (not sure why it isn't) but in the end rsync
sees the existing file as superfluous because there is no matching filename on
the source.

The --fuzzy option might fix this.  Using --delete-before instead of
--delete-after would delete then replace the file.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13061] File lost on case-insensitive file system

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13061

--- Comment #4 from t...@towo.net ---
The file /backup/src/vivaldi is deleted, not src/Vivaldi

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10715] IPv6 configure test fails on musl c-library

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10715

Pierre-Olivier Mercier  changed:

   What|Removed |Added

 CC||nemuna...@nemunai.re

--- Comment #1 from Pierre-Olivier Mercier  ---
Created attachment 13546
  --> https://bugzilla.samba.org/attachment.cgi?id=13546=edit
Proposed patch to fix the issue

Please find attached a proposal patch following Timo's suggestion: I replace
the long specific C library detection (which in our case forgot musl's users)
by a single feature test that all C libraries have to implement in order to
have a compiling rsync with IPv6 support.

After applying this patch, I was able to compile an IPv6 ready rsync with musl,
while keeping the same behaviour with the glibc under Linux. This should be
tested against others libc I don't have access to.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12817] [PATCH] Allow daemon itself to chroot

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12817

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #4 from Wayne Davison  ---
Thanks for the patch! I've tweaked it a little bit and committed it to git.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12817] [PATCH] Allow daemon itself to chroot

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12817

--- Comment #5 from Ben RUBSON  ---
Many thanks Wayne for having reworked & merged it !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10719] Error with cached effective process gid

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10719

--- Comment #1 from lonerr  ---
Three years later, any updates? :)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 10719] Error with cached effective process gid

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=10719

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #2 from Wayne Davison  ---
I've just committed a fix for this to git.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13071] New: [PATCH] Allow --partial-dir with --inplace

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13071

Bug ID: 13071
   Summary: [PATCH] Allow --partial-dir with --inplace
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ben.rub...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13649
  --> https://bugzilla.samba.org/attachment.cgi?id=13649=edit
Allow partial-dir with inplace

Hello,

Here is a patch to allow partial-dir with inplace.

Why ?

Quoting modified man-page :
If --inplace is used with --partial-dir, or with any option implying it
(--delay-update), the inplace behavior will take place in the partial
directory, each file being moved to its final destination once completed.

It really helps when one want to keep partial files but don't want rsync server
to duplicate them remotely when transfer is restarted. Partial files will then
be used "inplace", in the partial directory.
Really useful in conjunction with for example --link-dest.
More generally this helps saving space / dealing with short free-space targets.

Could we then think about reviewing and merging this please ?

Thank you very much !

Ben

(this patch has been written so that current client versions can easily benefit
from this using --temp-dir to indicate their partial directory, server code
being really easy to modify to take tmpdir as partial_dir)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7123] Use both old dest file and partial file as basis data

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7123

--- Comment #7 from Ben RUBSON  ---
I should have a look at this.
This would complete my work in https://bugzilla.samba.org/show_bug.cgi?id=13071
to help dealing with big files.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11866] rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=11866

--- Comment #3 from Ben RUBSON  ---
Hi,

Could it be possible to merge this please ?
It's really tiny (one character) and easily understandable :)

And it avoids silent data loss !

Thank you very much !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13083] New: [PATCH] Use both partial file + basis file

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13083

Bug ID: 13083
   Summary: [PATCH] Use both partial file + basis file
   Product: rsync
   Version: 3.1.3
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: ben.rub...@gmail.com
QA Contact: rsync...@samba.org

Created attachment 13681
  --> https://bugzilla.samba.org/attachment.cgi?id=13681=edit
Use both partial file + basis file

Hi,

In case of a transfer restart, if a partial file has been kept on receiving
side, this patch makes rsync to use the usual basis file (if found) in addition
to the partial file to speed-up the transfer.

Technically :
- if generator founds a partial file and a basis file, it will symlink the
basis file next the the partial file ;
- generator will send checksums to the sender from both files, as if there were
only one big basis file ;
- between the checksums of the 2 files, generator will send some dummy 0
checksums so that sender will see the second file's blocks as if they were
after the end of the final file, thus sender will be able to fully use them,
especially useful using --inplace ;
- receiver, if working on a partial file as basis file, will try to open the
second basis file thanks to the symlink ;
- receiver, if asked to handle a checksum with an offset greater than the
partial file's length, will take the corresponding block from the second basis
file.

Compatible with all sender's versions, as modification only takes place on
receiver side.

Fully compatible with the "allow --partial-dir with --inplace" patch :
https://bugzilla.samba.org/show_bug.cgi?id=13071

Hope to see both in 3.1.3 :)
Of course feel free to comment !

Thx !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7123] Use both old dest file and partial file as basis data

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7123

--- Comment #8 from Ben RUBSON  ---
See https://bugzilla.samba.org/show_bug.cgi?id=13083 for a patch !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html