[Bug 7120] Variable bandwidth limit .. bwlimit
https://bugzilla.samba.org/show_bug.cgi?id=7120 --- Comment #7 from roland --- please mind the note from this posting when using pv: http://superuser.com/questions/778066/using-pv1-to-limit-rsync-speed --snipp-- It seems that pv is waiting for data from rsync, and rsync is waiting for data too (stuck in select()) and not closing the input to pv. So it's a deadlock. Same happens when you substitute pv with something else (like dd). It seems that those commands just don't behave like rsync expects them to. Haven't found a workaround short of killing everything: export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh" kill is not a solution I'd be happy with. But I haven't found another. Of course using $$ was the lazy way out, should kill pv instead but how to obtain the pid in a oneliner? pidof just gives any pv pids. Querying their respective parent pids seemed a bit odd for a oneliner. Looking forward to a real solution... ;) --snipp-- -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 Wayne Davison changed: What|Removed |Added Resolution|--- |FIXED Status|ASSIGNED|RESOLVED --- Comment #11 from Wayne Davison --- I have added some code that makes sure that --daemon and --server can never be aliased to any other value. The code has always disabled popt aliases during the parsing of a daemon or server command-line, but if the main option itself was removed then rsync wouldn't know to disable aliases. This new code overrides any system or user alias to ensure that can never happen. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 Wayne Davison changed: What|Removed |Added Status|NEW |ASSIGNED --- Comment #10 from Wayne Davison --- One thing I was thinking is that the popt code could have a sanity check when reading in user aliases from a different user when running as root. So, the code would check that it was running as uid 0 and just ignore a $HOME-based popt aliases file if the file wasn't also uid 0. However, I do also like the idea of not using popt aliases for the --server side (including --daemon) as that could also cause rsync some problems in how things get setup between the 2 sides (in addition to being a potential security issue). -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #9 from Paul Donohue --- popt ticket requesting a solution in popt itself: http://rpm5.org/cvs/tktview?tn=98 -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #8 from Paul Donohue --- I agree with the general philosophy that it isn't rsync's problem to secure the command line. However, I don't see any good way that sudo can secure the rsync command line unless rsync provides some mechanism for disabling popt aliases, hence why I'm proposing these patches. Note that popt aliases are an optional feature of popt and are not automatically enabled when popt is used, so this problem will not necessarily affect every program that uses popt. However, perhaps an alternative solution would be to extend popt itself to allow popt aliases to be disabled in some standard way in any application that uses them. (Maybe a standard command line option that applies to all programs using popt, or an environment variable, or some code to automatically detect when popt is running under sudo and disable aliases?) -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12583] New: [PATCH] Add new daemon option "syslog tag"
https://bugzilla.samba.org/show_bug.cgi?id=12583 Bug ID: 12583 Summary: [PATCH] Add new daemon option "syslog tag" Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Created attachment 12933 --> https://bugzilla.samba.org/attachment.cgi?id=12933&action=edit Add new daemon option "syslog tag" Hello, This patch adds a new option "syslog tag". It allows to specify the syslog tag to use when the daemon uses syslog logging. Useful when daemon is used by several users : syslog tag = rsyncd.%USER% By default set to "rsyncd". Thank you ! Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12527] Sender waits for timeout when fuzzy basis file found
https://bugzilla.samba.org/show_bug.cgi?id=12527 --- Comment #3 from Ben RUBSON --- So sounds like that when the generator sends too much info (too many checksums) to the sender, the sender does not receive / see (important) messages anymore. Perhaps we could solve this by making the generator wait before calling generate_and_send_sums(), until the sender really needs this info, i.e. until the sender is ready to send this file. I think however I'll need your help Kevin / Wayne for such a modification. Perhaps there's also a better way to solve this issue ? Thank you again, Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #7 from Kevin Korb --- I have been thinking about this a bit and I believe it is a sudo problem and not an rsync problem. It is not rsync's job to secure the command line. Plus rsync is far from the only program that uses popt to parse the command line and therefore not the only program that would be affected by this problem. However, I do think that Wayne should add at least one of your patches since this would also affect rrsync and other forms of ssh ForcedCommands. Note that I don't know much about popt and might be missing something obvious/simple. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12527] Sender waits for timeout when fuzzy basis file found
https://bugzilla.samba.org/show_bug.cgi?id=12527
--- Comment #2 from Ben RUBSON ---
I'm troubleshooting and found that this happens when generate_and_send_sums()
writes too many bufs.
If I replace :
for (i = 0; i < sum.count; i++) {
by :
for (i = 0; i < 4500; i++) {
(sum.count > 15000 in this test)
It works as expected, sender exits correctly writing down all error messages
from the server.
If I replace :
for (i = 0; i < sum.count; i++) {
by :
for (i = 0; i < 4900; i++) {
It fails as explained above.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #6 from Paul Donohue --- This all makes sense, I appreciate the suggestions, and I may actually implement some of this. However, the existence of this rrsync solution doesn't change the fact that there exists another simple and obvious solution using sudo which has a giant undocumented security hole related to an unusual, undocumented, and not widely used feature of rsync (popt aliases). My goal for this bug report is to either get a disclaimer added to the rsync man page (which documents popt aliases feature, explains the security implications, and suggests mitigations and/or alternative solutions to avoid security issues, including this rrsync solution), or to get the attached trivial patches merged to help mitigate this security issue without requiring users to wrap complicated scripts around rsync or avoid the use of sudo. Security is hard enough to get right when everything works in a consistent and intuitive manner. Having an unusual, unintuitive, and undocumented feature with significant undocumented security implications is just asking for trouble. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #5 from Karl O. Pinc --- On Sun, 12 Feb 2017 02:18:35 + samba-b...@samba.org wrote: > https://bugzilla.samba.org/show_bug.cgi?id=12576 > > --- Comment #4 from Paul Donohue --- > That's an interesting solution, but it doesn't really work well for > my use case. I would like my users to be able to maintain their own > SSH keys (this solution would require me to manage users' SSH keys in > /root/.ssh/authorized_keys), and I don't particularly want to set > "PermitRootLogin yes" in /etc/ssh/sshd_config. I also already have > scripts to manage sudo permissions, and I would have to make some > significant changes to support centrally managing authorized_keys. You don't have to set "PermitRootLogin yes". You can set "PermitRootLogin forced-commands-only". Far more secure. Yes, you probably would need a script, or something, that takes a ssh key and somehow modifies /root/.ssh/authorized_keys. One secure way to do this is to have a script (or any other interface) that drops a file containing the key (and other requisite information) into a directory that is watched with, e.g., incron. incron can then run a script that parses the file, pulls out the requisite information, and has the permissions to modify /root/.ssh/authorized_keys. (And then cleans up by deleting the file.) This way the process with permissions to modify /root/.ssh/authorized_keys can verify the data it's putting in there and validate the credentials of the user putting the data in. You could instead leverage your sudo management scripts to give users permissions to run scripts as root that modify /root/.ssh/authorized_keys. Here it's sudo, instead of the script run by incron, controlling access. Anybody without authorization gets a permission error. As above, this approach avoids the insecurity of suid root scripts. Either way, the script the user runs must pick up the user's username (or home directory, or whatever) itself. This avoids the problem of allowing one user to set permissions for another. The trick for the script run via sudo would be to use logname(1). So, writing a script (or scripts) to add/remove/replace lines in /root/.ssh/authorized_keys does not sound all that much work, although of course there are always devilish details. flock(1) is probably your friend if you write in shell. You probably want to log with logger(1), etc. Regards, Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #4 from Paul Donohue --- That's an interesting solution, but it doesn't really work well for my use case. I would like my users to be able to maintain their own SSH keys (this solution would require me to manage users' SSH keys in /root/.ssh/authorized_keys), and I don't particularly want to set "PermitRootLogin yes" in /etc/ssh/sshd_config. I also already have scripts to manage sudo permissions, and I would have to make some significant changes to support centrally managing authorized_keys. I think the rsyncd+sudo solution actually works pretty well except for the non-obvious fact that popt lets the user override the sudo restrictions. There are are lots of rsync users out there who are running rsync through sudo, so even if there happens to be a better way to handle my specific use case, it seems to me that there either needs to be a giant disclaimer somewhere that says running rsync in sudo is dangerous and suggests alternative solutions, or rsync needs to provide some reasonably intuitive mitigations. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #3 from Kevin Korb --- There is no reason to involve rsyncd (or even sudo). See the rrsync script in the support directory. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #2 from Paul Donohue --- Created attachment 12916 --> https://bugzilla.samba.org/attachment.cgi?id=12916&action=edit Add a new --no-popt-aliases option to explicitly disable popt aliases -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] New: popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 Bug ID: 12576 Summary: popt aliases allow users to bypass sudo argument restrictions Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: samba-b...@paulsd.com QA Contact: rsync...@samba.org My goal is to allow a specific user to read but not write a specific file as root using rsync via SSH. The obvious solution was to configure /etc/rsyncd.conf to allow read-only access to the file, then add "user ALL=(root) NOPASSWD:/usr/bin/rsync --server --daemon ." to /etc/sudoers, then have the user run `rsync --rsh=ssh --rsync-path='sudo rsync' host::module/file .` However, this is not as secure as it appears. Unfortunately, the popt alias feature allows the user to replace the rsync arguments with almost any other arbitrary rsync arguments, which effectively gives the user full root access to the system. For example, the user can add the following to /home/user/.popt and run `sudo rsync --server --daemon .` to read the contents of /etc/shadow: rsync alias --server -v rsync alias --daemon /etc/shadow This can be prevented by setting the 'always_set_home' option in sudoers (so that only root's popt config is read), although this setting is global and may not be desirable in all cases. This can also be prevented by creating a shell script that overrides $HOME then runs rsync, and using that shell script instead of rsync in both sudoers and --rsync-path. However, that is an unintuitive solution that few users are likely to implement unless a giant disclaimer is added to the documentation. This really seems like a problem that should be solved in rsync itself. In the rsync code, popt aliases are explicitly disabled when '--server' or '--daemon' is used, but only after those arguments have been parsed with popt aliases enabled, which is why the above example is able to use popt aliases to override those arguments. The first attached patch checks for '--server' or '--daemon' before enabling popt aliases, which fixes this issue when '--server' or '--daemon' are used. The second attached patch adds a new '--no-popt-aliases' argument. This explicitly disables popt aliases and may be used to allow rsync to be safely run using sudo with an argument list that does not include '--server' or '--daemon'. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12576] popt aliases allow users to bypass sudo argument restrictions
https://bugzilla.samba.org/show_bug.cgi?id=12576 --- Comment #1 from Paul Donohue --- Created attachment 12915 --> https://bugzilla.samba.org/attachment.cgi?id=12915&action=edit Do not enable popt aliases if --server or --daemon is specified -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12570] Problems with --checksum --existing
https://bugzilla.samba.org/show_bug.cgi?id=12570 --- Comment #1 from Kevin Korb --- Unfortunately rync's --checksum is just that dumb. It checksums *EVERYTHING* on the source and the target before it does anything else. Since --checksum is almost always the wrong thing to do nobody seems to be willing to add basic intelligence to it. Unfortunately, what you are trying to do is one of those few instances when --checksum is the right thing to use. So, that is just the way it works. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12570] New: Problems with --checksum --existing
https://bugzilla.samba.org/show_bug.cgi?id=12570
Bug ID: 12570
Summary: Problems with --checksum --existing
Product: rsync
Version: 3.1.1
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: way...@samba.org
Reporter: a...@smasher.org
QA Contact: rsync...@samba.org
Problem:
I've got an sd-card with some movies, a few of which are corrupted files.
I want to copy only the files that don't match the good files.
command:
rsync --checksum --existing -vhriP /movies/ /media/128-SD/Movies/
The problem here is that *all* files in "/movies/" are hashed before anything
else happens. This can be verified with lsof: "lsof +D /movies".
I've got <100GB in "/media/128-SD/Movies/".
I've got >1.5TB in "/movies/", and hashing all of those files is just a huge
waste of time and system resources.
When "--existing" and "--checksum" are both used, the algorithm should first
make a list of candidate files, then start hashing. It should *not* start
hashing everything on the send-side and then figure out which files might be
needed.
Workaround for me:
diff -r /movies/ /media/128-SD/Movies/ | grep differ | awk '{print "pv " $3" >
"$5}' | sh
nb, that workaround requires "pv" and only works with file-names that do not
contain spaces, but for me it's a quick and easy way to see progress while
files are being copied. "cp" would work fine in place of "pv".
On my system, that workaround saved my about 1-2 days of hashing, and completed
in less than an hour.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12569] New: Missing directory errors not ignored
https://bugzilla.samba.org/show_bug.cgi?id=12569
Bug ID: 12569
Summary: Missing directory errors not ignored
Product: rsync
Version: 3.1.2
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: way...@samba.org
Reporter: axk...@gmail.com
QA Contact: rsync...@samba.org
When using --delete-missing-args and --ignore-errors with a a file list it
terminates anyway if one directory, that is not a sub of root, is not existing.
See:
~$ mkdir -p src/a trg/a
~$ echo "/a/b/c" > list
~$ /usr/local/bin/rsync -slt --ignore-errors --force --ignore-missing-args
--delete-missing-args --files-from=list --rsync-path=/usr/local/bin/rsync src
localhost:`pwd`/trg
file has vanished: "/home/axel/src/a/b"
ABORTING due to invalid path from sender: a/b/c
rsync error: protocol incompatibility (code 2) at generator.c(1270)
[generator=3.1.2]
~$ /usr/local/bin/rsync --version
rsync version 3.1.2 protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, no ACLs, xattrs, iconv, symtimes, prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
Following patch fixes it for me, albeit it ignores this kind of error now
always:
--- rsync-3.1.2-org/generator.c2015-12-05 20:10:24.0 +0100
+++ rsync-3.1.2/generator.c2017-02-07 10:58:33.768453242 +0100
@@ -1264,10 +1264,11 @@
&& (*dn != '.' || dn[1]) /* Avoid an issue with --relative and
the "." dir. */
&& (prior_dir_file && strcmp(dn, f_name(prior_dir_file, NULL)) !=
0)
&& flist_find_name(cur_flist, dn, 1) < 0) {
+io_error |= IOERR_VANISHED;
rprintf(FERROR,
-"ABORTING due to invalid path from sender: %s/%s\n",
+"WARNING: invalid path from sender: %s/%s\n",
dn, file->basename);
-exit_cleanup(RERR_PROTOCOL);
+return;
}
if (relative_paths && !implied_dirs
&& do_stat(dn, &sx.st) < 0) {
Now the exit code is 24, partial transfer, as would be expected.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12568] New: Integer overflow still affects xattrs.c
https://bugzilla.samba.org/show_bug.cgi?id=12568
Bug ID: 12568
Summary: Integer overflow still affects xattrs.c
Product: rsync
Version: 3.1.2
Hardware: All
OS: All
Status: NEW
Severity: critical
Priority: P5
Component: core
Assignee: way...@samba.org
Reporter: shqk...@gmail.com
QA Contact: rsync...@samba.org
A suspicious integer overflow is found in xattrs.c:692.
The code snippet is as follows.
684 for (num = 1; num <= count; num++) {
685char *ptr, *name;
686rsync_xa *rxa;
687size_t name_len = read_varint(f);
688size_t datum_len = read_varint(f);
689size_t dget_len = datum_len > MAX_FULL_DATUM ? 1 + MAX_DIGEST_LEN :
datum_len;
690size_t extra_len = MIGHT_NEED_RPRE ? RPRE_LEN : 0;
691if ((dget_len + extra_len < dget_len)
692 || (dget_len + extra_len + name_len < dget_len))
693overflow_exit("receive_xattr");
694ptr = new_array(char, dget_len + extra_len + name_len);
695if (!ptr)
696out_of_memory("receive_xattr");
697name = ptr + dget_len + extra_len;
698read_buf(f, name, name_len);
>From the code we can see that the security checks at line 691 and line 692 aim
to filter integer overflows. Specifically, a handler, i.e. function
"overflow_exit" will be invoked if the first addition "dget_len + extra_len"
overflows (protected by the check at line 691) or the second addition "dget_len
+ extra_len + name_len" overflows (protected by the check at line 692).
Here, we want to say that the later check at line 692 is insufficient to catch
integer overflow. That means, there exist some integer overflows, which can
bypass the later check.
Assume that it's on a 32-bit machine, and "dget_len" is 100, "extra_len" is
also 100, whereas "name_len" takes a very big integer value, e.g., 0x .
Hence, "dget_len + extra_len + name_len" overflows to 199, which is bigger than
"dget_len", i.e. 100. As a result, an integer overflow indeed happens here,
however, the overflow check at line 692 doesn't catch it. Furthermore, buffer
overflow would occur at line 698.
One possible workaround is to use a much stricter overflow check at line 692,
as below:
"dget_len + extra_len + name_len < dget_len + extra_len".
Thanks.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12544] New: Confusing return codes on unauthorized connections
https://bugzilla.samba.org/show_bug.cgi?id=12544 Bug ID: 12544 Summary: Confusing return codes on unauthorized connections Product: rsync Version: 3.1.3 Hardware: All OS: Linux Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: mrupr...@redhat.com QA Contact: rsync...@samba.org I have seen a couple of discussions here about whether certain bugs are related to rsync or rather systemd. Right now I am dealing with a problem regarding running rsync as a systemd socket activated daemon. Basically systemd runs a daemon only after it is contacted through socket, daemon performs a task and then finishes. systemd only collects return value. If I use "allow hosts" option in rsyncd.conf file and then I receive a connection through the socket from host which is not on the list, the connection is rejected. This is completely fine but rsync ends with -1 in such a case. From my point of view it should end with other value because rsync ended correctly as instructed through the conf file. If I receive lots of unauthorized connections, rsync leaves behind a failed unit. After some time these failed units seem to influence performance of the system until at some point the system literally crawls to a halt. My suggestion would be to perhaps try to return something else than -1. Maybe not 0 because in that case every unauthorized connection would be simply ignored but at least some return value which would distinguish this use-case from all the other general -1 errors. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2294] Detect renamed files and handle by renaming instead of delete/re-send
https://bugzilla.samba.org/show_bug.cgi?id=2294 --- Comment #27 from Wolfgang Hamann --- Hi, I recently ran into the problem that a large file set got renamed and then re-sent. I tried to fix after the fact, so I went the obvious way of comparing sizes and modtimes on the destination and calculate checksums for potential matches. I would have preferred to use a list of inode numbers and files for the old and new file sets instead... So I wonder whether a different approach to the problem could make sense: a) the filelist contains inode numbers, and after a successful rsync, a file is generated in the target dir listing inodes and names of all files transferred b) when receiving to the same dir, if a target file does not exist, the inode in the filelist is used to look up the previous filename. If it exists and matches in size and modtime, it could be hardlinked. Deleting files from the target that are no longer in the source would take care of the old file. When the sync is completed without error, the list of inodes and file names would be updated c) when receiving in link-dest mode, the file in the old dir would be consulted for a potential match, and the new list would be created in the target dir Of course this only makes sense if inode numbers are reliable, as on all standard local file systems or nfs. I do not know whether the new storage arenas preserve inodes. It is obvious that the same inode may appear more than once in a source file set Regards Wolfgang -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12530] New: [REQ] Improve fuzzy using files being uploaded
https://bugzilla.samba.org/show_bug.cgi?id=12530 Bug ID: 12530 Summary: [REQ] Improve fuzzy using files being uploaded Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Hello, Let's imagine the sender is uploading a bunch of files which are quite similar. For example, the following dir : /directory |-backup1.iso |-backup2.iso |-backup3.iso |-backup4.iso |-backup5.iso For the moment, if no remote fuzzy basis is found at the very beginning of the transfer, every file will be fully uploaded. Goal would then be to improve rsync so that once the first file has been uploaded, fuzzy algorithm could look at this new file as a fuzzy basis file for the other new files arriving. Same thing once the second file has been uploaded etc... Perhaps it could be done once for all at the very beginning of the transfer, also taking the list of files which will be uploaded (sent by the sender), and their properties, to feed the fuzzy algorithm. This would speed-up transfer in a number of situations. Thank you very much ! Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12527] Sender waits for timeout when fuzzy basis file found
https://bugzilla.samba.org/show_bug.cgi?id=12527 --- Comment #1 from Ben RUBSON --- Same issue with rsync-HEAD-20161010-1854GMT. My quick previous test uses these local files : -rw--- 1 root root 1177136858 Jan 10 2017 bf.iso -rw--- 1 root root68558591 Jan 10 2017 sf2.dmg And this remote file : -rw--- 1 root root68558591 Jan 10 2017 sf1.dmg bf.iso (local) is big enough to make the remote FS full. sf2.dmg (local) and sf1.dmg (remote) are equal. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12527] New: Sender waits for timeout when fuzzy basis file found
https://bugzilla.samba.org/show_bug.cgi?id=12527 Bug ID: 12527 Summary: Sender waits for timeout when fuzzy basis file found Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: critical Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Hello, Trying to make the sender get the last error messages from the generator/receiver (https://bugzilla.samba.org/show_bug.cgi?id=12522), I found one case where sender exits with timeout instead of correctly exiting. It is when a fuzzy basis file has been selected. Here is an example where we send 2 files, but as the remote FS gets full, transfer properly exits. Perfect. sending incremental file list [sender] change_dir(/home/prd/bkp) send_files starting server_recv(2) starting pid=45749 get_local_name count=3 /home/prd/bkp/ [Receiver] change_dir(/home/prd/bkp) generator starting pid=45749 delta-transmission enabled recv_generator(.,0) set modtime of . to (1484751636) Wed Jan 18 16:00:36 2017 recv_generator(.,1) recv_generator(bf.iso,2) recv_generator(sf2.dmg,3) recv_files(3) starting generate_files phase=1 send_files(0, /home/prd/bkp/.) ./ send_files(2, /home/prd/bkp/bf.iso) bf.iso 549,158,912 74% 12.12MB/s0:00:14 rsync: [sender] write error: Broken pipe (32) [sender] got msg=2, len=14 recv_files(.) [sender] got msg=2, len=19 recv_files(bf.iso) [sender] got msg=2, len=27 [receiver] send_msg(1, 94) [sender] got msg=2, len=28 [generator] send_msg(1, 94) [sender] got msg=1, len=94 rsync: write failed on "/home/prd/bkp/bf.iso": Disc quota exceeded (69) [sender] got msg=2, len=33 [generator] converted msg len=94 [sender] got msg=2, len=27 [receiver] send_msg(3, 76) [sender] got msg=2, len=28 [generator] send_msg(3, 76) [sender] got msg=3, len=76 rsync error: error in file IO (code 11) at receiver.c(400) [receiver=3.1.2] [sender] got msg=2, len=33 [generator] converted msg len=76 [sender] got msg=2, len=85 [receiver] _exit_cleanup(code=11, file=receiver.c, line=400): about to call exit(11 [sender] got msg=2, len=32 [receiver] send_msg_int(86, 11) [sender] got msg=2, len=33 [generator] send_msg_int(86, 11) [sender] got msg=86, len=4 [sender] send_msg(86, 0) [sender] _exit_cleanup(code=11, file=io.c, line=1633): about to call exit(11) Exactly same command launched, but adding -y so that fuzzy basis may be found. sending incremental file list [sender] change_dir(/home/prd/bkp) send_files starting server_recv(2) starting pid=48003 get_local_name count=3 /home/prd/bkp/ [Receiver] change_dir(/home/prd/bkp) generator starting pid=48003 delta-transmission enabled recv_generator(.,0) set modtime of . to (1484751636) Wed Jan 18 16:00:36 2017 recv_generator(.,1) recv_generator(bf.iso,2) recv_generator(sf2.dmg,3) fuzzy basis selected for sf2.dmg: sf1.dmg send_files(0, /home/prd/bkp/.) ./ send_files(2, /home/prd/bkp/bf.iso) bf.iso 519,077,888 70%8.14MB/s0:00:25 [sender] io timeout after 30 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(195) [sender=3.1.2] [sender] _exit_cleanup(code=30, file=io.c, line=195): about to call exit(30) As you can see, a fuzzy basis has been selected for the second file. However, as the remote FS gets full, the transfer of the first file stops, but instead of correctly exiting, the process hangs and waits for the configured timeout (30 seconds here), or forever if no timeout configured. Same behaviour with protocol 30 and 31. I went through the code but was not able to find out why. Wayne, could you (or someone else of course) help me solving this issue ? Would be perfect if this could be solved without having to modify the sender code. Thank you very much for your support ! Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12525] [PATCH] Avoid uploading whole file when transfer fails
https://bugzilla.samba.org/show_bug.cgi?id=12525 Ben RUBSON changed: What|Removed |Added Attachment #12839|0 |1 is obsolete|| --- Comment #1 from Ben RUBSON --- Created attachment 12842 --> https://bugzilla.samba.org/attachment.cgi?id=12842&action=edit Avoid uploading whole file when transfer fails -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12522] [PATCH] Send last error messages to sender
https://bugzilla.samba.org/show_bug.cgi?id=12522 Ben RUBSON changed: What|Removed |Added Attachment #12838|0 |1 is obsolete|| --- Comment #2 from Ben RUBSON --- Created attachment 12841 --> https://bugzilla.samba.org/attachment.cgi?id=12841&action=edit Send last error messages to sender Error correction on sender side. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12525] New: [PATCH] Avoid uploading whole file when transfer fails
https://bugzilla.samba.org/show_bug.cgi?id=12525 Bug ID: 12525 Summary: [PATCH] Avoid uploading whole file when transfer fails Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Created attachment 12839 --> https://bugzilla.samba.org/attachment.cgi?id=12839&action=edit Avoid uploading whole file when transfer fails Hello, Here is a patch which stops file upload when it fails. This can be useful for example when the remote FS is full, and client is uploading big files. First apply https://bugzilla.samba.org/show_bug.cgi?id=12522 Then the attached patch. Thank you ! Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12522] [PATCH] Send last error messages to sender
https://bugzilla.samba.org/show_bug.cgi?id=12522 Ben RUBSON changed: What|Removed |Added Attachment #12832|0 |1 is obsolete|| --- Comment #1 from Ben RUBSON --- Created attachment 12838 --> https://bugzilla.samba.org/attachment.cgi?id=12838&action=edit Send last error messages to sender Nicer / smarter version. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12522] New: [PATCH] Send last error messages to sender
https://bugzilla.samba.org/show_bug.cgi?id=12522 Bug ID: 12522 Summary: [PATCH] Send last error messages to sender Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Created attachment 12832 --> https://bugzilla.samba.org/attachment.cgi?id=12832&action=edit Send last error messages to sender Hello, Here is a patch which sends last error messages to sender for a file-transfer which failed. It helps especially with protocol < 31. This can be useful for example to know that a FS is full. Thank you ! Ben ### Without : /repository/mybigfile.iso 539.26M 6% 11.21MB/s0:11:01 rsync: [sender] write error: Broken pipe (32) rsync error: error in file IO (code 11) at io.c(831) [sender=3.1.2] ### With (of course the interesting message here is the first one) : /repository/mybigfile.iso 539.26M 6% 11.21MB/s0:11:01 rsync: write failed on "/repository/mybigfile.iso" (in bkp): Disc quota exceeded (69) rsync error: error in file IO (code 11) at receiver.c(400) [receiver=3.1.2] rsync error: error in file IO (code 11) at io.c(1633) [generator=3.1.2] rsync: [sender] write error: Broken pipe (32) rsync error: error in file IO (code 11) at io.c(831) [sender=3.1.2] -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12519] New: Failed to open lock file, add reason
https://bugzilla.samba.org/show_bug.cgi?id=12519 Bug ID: 12519 Summary: Failed to open lock file, add reason Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Created attachment 12828 --> https://bugzilla.samba.org/attachment.cgi?id=12828&action=edit Failed to open lock file, add reason Hello, Here is a patch which gives more details to the client about why rsync failed to open lock file. This can be useful for example to know that a FS a full. Thank you ! Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12508] New: fileflags & forcechange don't work for hardlinks
https://bugzilla.samba.org/show_bug.cgi?id=12508
Bug ID: 12508
Summary: fileflags & forcechange don't work for hardlinks
Product: rsync
Version: 3.1.2
Hardware: All
OS: FreeBSD
Status: NEW
Severity: normal
Priority: P5
Component: core
Assignee: way...@samba.org
Reporter: eha...@freebsd.org
QA Contact: rsync...@samba.org
I've received the following bug report in FreeBSD's bugzilla:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215688
Use case:
# mkdir -pv /root/debug/{src,dst}
# touch /root/debug/src/0x
# ls -lio /root/debug/src/0x
330810 -rw-r--r-- 1 root wheel uarch 0 Jan 10 12:40 /root/debug/src/0x
# ln /root/debug/src/0x /root/debug/src/1x
# ln /root/debug/src/0x /root/debug/src/2x
# ln /root/debug/src/0x /root/debug/src/3x
# ls -lio /root/debug/src/
total 2
330810 -rw-r--r-- 4 root wheel uarch 0 Jan 10 12:40 0x
330810 -rw-r--r-- 4 root wheel uarch 0 Jan 10 12:40 1x
330810 -rw-r--r-- 4 root wheel uarch 0 Jan 10 12:40 2x
330810 -rw-r--r-- 4 root wheel uarch 0 Jan 10 12:40 3x
# chflags schg /root/debug/src/0x
# ls -lio /root/debug/src/
total 2
330810 -rw-r--r-- 4 root wheel schg,uarch 0 Jan 10 12:40 0x
330810 -rw-r--r-- 4 root wheel schg,uarch 0 Jan 10 12:40 1x
330810 -rw-r--r-- 4 root wheel schg,uarch 0 Jan 10 12:40 2x
330810 -rw-r--r-- 4 root wheel schg,uarch 0 Jan 10 12:40 3x
# rsync -aHXSE --partial --timeout=60 --fileflags --force-change --numeric-ids
--force --super -8 /root/debug/src/. /root/debug/dst/
rsync: link "/root/debug/dst/2x" => 3x failed: Operation not permitted (1)
rsync: link "/root/debug/dst/1x" => 3x failed: Operation not permitted (1)
rsync: link "/root/debug/dst/0x" => 3x failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1200) [sender=3.1.2]
# ls -lio /root/debug/dst/
total 1
330826 -rw-r--r-- 1 root wheel schg,uarch 0 Jan 10 12:40 3x
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12489] --fuzzy --fuzzy does not work with daemon
https://bugzilla.samba.org/show_bug.cgi?id=12489 Ben RUBSON changed: What|Removed |Added Severity|critical|normal --- Comment #1 from Ben RUBSON --- As a workaround I found a way to enable "use chroot = yes" in my production modules. I then put the Importance of this bug report back to "normal", but keep it opened as the issue is real :) -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12498] New: --fuzzy --fuzzy hugely impacts performance even if its' not needed
https://bugzilla.samba.org/show_bug.cgi?id=12498 Bug ID: 12498 Summary: --fuzzy --fuzzy hugely impacts performance even if its' not needed Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Hello, I make backups to a Rsync daemon. Each backup is made to an empty directory, using 20 --link-dest parameters, so that already uploaded files are hard-linked. Perfect. I made some tests adding --fuzzy --fuzzy so that non-existing files can use a base-file, when possible. It works perfectly, really good thing to find a base-file within link-dest directories. However, it really slows down the backup process, even when it's not needed. For example, it terribly impacts performance even for a backup with no new files at all, for which all files will be hard-linked. Here are some backup durations for a backup with no new files : - without -yy : 9 seconds - with -y : 9 seconds - with --y and 1 --link-dest : 11 seconds - with --y and 8 --link-dest : 22 seconds - with --y and 20 --link-dest : 37 seconds My question is then, could it be possible for --fuzzy --fuzzy (-yy) to only work for files which really need to be uploaded ? So to avoid working for files which will simply be hard-linked. Would such a modification only impact daemon side ? (would be great) Thank you very much ! Best regards, Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10312] Rsync times out during deletion on big folders
https://bugzilla.samba.org/show_bug.cgi?id=10312 --- Comment #2 from Ben RUBSON --- Regarding --link-dest, I made further testing with rsync daemon 3.1.2. I added a usleep(5000) at the beginning of try_dests_reg() in generator.c, so that hard-linking all the files takes more than 2 minutes instead of some seconds only. Client, with a very low --timeout=10, does not timeout on this. So it's OK, no more issue with --link-dest ! Same experience with --delete (--delete-after in my tests), adding some usleep() so that deletion takes around 4 minutes did not make client with --timeout=10 to abort. Perfect ! -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2294] Detect renamed files and handle by renaming instead of delete/re-send
https://bugzilla.samba.org/show_bug.cgi?id=2294 --- Comment #26 from Ben RUBSON --- ### What's the diff between --fuzzy and --detect-renamed ? If I understand correctly, --fuzzy looks only in destination folder, for either a file that has an identical size and modified-time, or a similarly-named file, and uses it as a basis file. Whereas --detect-renamed looks everywhere for files that either match in size & modify-time, or match in size & checksum (when --checksum is used), and uses each match as a basis file. So the main difference is destination_folder_only vs everywhere, am I right ? ### Some questions : # --fuzzy can be used twice to look in --link-dest folders, useful when backing-up to an empty directory. What about --detect-renamed ? # Don't these 2 options kill memory when backing-up many many files (furthermore when also looking in --link-dest folders) ? Don't they maintain in-memory list of files ? # Will these options only do their job when needed (need to find a basis file), or every time ? # Do these options impact destination performance, or do they benefit from already-done scans ? For example, will -yy scan all --link-dest dirs (disk IO intensive) even if perhaps it's not needed ? # About --detect-renamed, let's imagine foo/A has been foud in bar/. Will it be smart enough to directly search for foo/B in bar/, instead of restarting a whole lookup ? # These 2 options use found file as a basis file. Let's imagine the found file totally matches, and we are using --link-dest. Could we think about linking the file instead of copying it ? # Last, do you have plans for --detect-renamed onto the trunk ? Thank you very much for this deep-analysis ! Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12489] New: --fuzzy --fuzzy does not work with daemon
https://bugzilla.samba.org/show_bug.cgi?id=12489 Bug ID: 12489 Summary: --fuzzy --fuzzy does not work with daemon Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: critical Priority: P5 Component: core Assignee: way...@samba.org Reporter: ben.rub...@gmail.com QA Contact: rsync...@samba.org Hello, I try to use --fuzzy --fuzzy with --link-dest : rsync -aR -yy --link-dest="../2016-12-28/" --link-dest="../2016-12-27/" /my/folder srv::daemon/computer/2016-12-29/ But : --fuzzy --fuzzy eligible files are not detected (--debug=FUZZY2 gives nothing at all). Files are correctly linked, but for every file, rsync daemon logs the following : file has vanished: "/computer/2016-12-29/2016-12-28/my/folder/file" file has vanished: "/computer/2016-12-29/2016-12-27/my/folder/file" I tried with the following command : rsync -aR -yy --link-dest="/computer/2016-12-28/" --link-dest="/computer/2016-12-27/" /my/folder srv::daemon/computer/2016-12-29/ Which gives : file has vanished: "/computer/2016-12-29/srvpath/computer/2016-12-28/my/folder/file" file has vanished: "/computer/2016-12-29/srvpath/computer/2016-12-27/my/folder/file" I don't have these error messages with only one --fuzzy. I don't have the problem using a SSH server as the target instead of a Rsync daemon. And I don't have the problem setting chroot to yes in the Rsync daemon. So sounds like these are related to --fuzzy --fuzzy talking to a Rsync daemon with chroot=no. Tested in several environments, with Rsync 3.1.1 and 3.1.2. I'm also surprised that, according to these log lines, --fuzzy --fuzzy tries to reach every file in link-dest directories, even if they are absolutely not needed (simple test with destination already in-sync, or with all destination files to be simply linked (same name/date/perms etc...)). Thank you for your support ! Best regards, Ben -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10338] Start deletion from the top of the hierarchy
https://bugzilla.samba.org/show_bug.cgi?id=10338 Ben RUBSON changed: What|Removed |Added Resolution|WORKSFORME |WONTFIX -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10338] Start deletion from the top of the hierarchy
https://bugzilla.samba.org/show_bug.cgi?id=10338 Ben RUBSON changed: What|Removed |Added Resolution|--- |WORKSFORME Status|NEW |RESOLVED --- Comment #4 from Ben RUBSON --- Using a workaround, so closing this for the moment. Thank you ! -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 5324] Reduce the performance penalty of --xattrs on Mac OS X
https://bugzilla.samba.org/show_bug.cgi?id=5324 --- Comment #11 from Björn Jacke --- no, because the problem was fixed with metzes patch. if you see a need for the ctime compare, then please open a new bug report for it. this bug here is closed and fixed. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 5324] Reduce the performance penalty of --xattrs on Mac OS X
https://bugzilla.samba.org/show_bug.cgi?id=5324 --- Comment #10 from Oren Kishon --- > how about adding an option like "--use-ctime-before-xattr-compares", which > only > reads and compares EAs for files where the ctime on the source side is > newer > than the ctime on the target side. EA modifications update the ctime > ususally. > This would be a way to speed up syncing with EAs quite a lot I > think. Has anyone started developing this? Thanks you Oren Kishon, Ctera -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12472] invalid rsync-command syntax or options with —stats
https://bugzilla.samba.org/show_bug.cgi?id=12472 --- Comment #3 from rutleco...@gmail.com --- (In reply to Paul Slootman from comment #1) Also, no error when the server side is 3.1.2, only when the server side is 3.0.9 -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12472] invalid rsync-command syntax or options with —stats
https://bugzilla.samba.org/show_bug.cgi?id=12472 --- Comment #2 from rutleco...@gmail.com --- (In reply to Paul Slootman from comment #1) For the bug reporter, not in my rsync command. (My rsync command is the same script that I used w/o error with 3.0.9) I think the error has something to do with what's mentioned in the 3.1.2 man page re: "âinfo=FLAGS" -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12472] invalid rsync-command syntax or options with —stats
https://bugzilla.samba.org/show_bug.cgi?id=12472 --- Comment #1 from Paul Slootman --- Did you cut&paste the --stats from some webpage? As you can see from the emailed version of your report: Summary: invalid rsync-command syntax or options with â\200\224stats There's your problem! You're not using the ASCII '-' sign, you're using some UTF8 abomination that represents a dash only to humans. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12472] New: invalid rsync-command syntax or options with —stats
https://bugzilla.samba.org/show_bug.cgi?id=12472 Bug ID: 12472 Summary: invalid rsync-command syntax or options with âstats Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 Component: core Assignee: way...@samba.org Reporter: rutleco...@gmail.com QA Contact: rsync...@samba.org Client version 3.1.2 Server version 3.0.9 When using rsync 3.1.2 gives the error: /usr/local/s3rsync/prod-ssh-rsync3: invalid rsync-command syntax or options By removing options, I was able to track the problem to the âstats flag. It would be useful if the error message could report that âstats was the cause of the error. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 3444] Deal with case-insensitive file-systems better (perhaps by adding an option)
https://bugzilla.samba.org/show_bug.cgi?id=3444 Dmitry changed: What|Removed |Added CC||dm...@mail.ru --- Comment #12 from Dmitry --- I believe that bug #7951 and bug #10448 are the duplicates of this one. Would be nice to have an ability to define more readable masks unlike "*.[Pn][Nn][Gg]". -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12454] New: rsync server is not going down.
https://bugzilla.samba.org/show_bug.cgi?id=12454 Bug ID: 12454 Summary: rsync server is not going down. Product: rsync Version: 3.1.3 Hardware: x64 OS: Linux Status: NEW Severity: major Priority: P5 Component: core Assignee: way...@samba.org Reporter: lior...@amdocs.com QA Contact: rsync...@samba.org Hello, we are using rsync in production to copy data from main servers to replication servers. the command we are using is: rsync -az -h --delete --include="*/" --include="*UUP_FILE*.dat" --exclude="*" "/opt/app/es/appahome/pnmesp1/var/m3g/projs/apr/interfaces/output/UUP/23112016/" -e "/usr/bin/ssh -q -o StrictHostKeyChecking=no -o PasswordAuthentication=no" pnge...@target.server.com:/tmp/UUP/ --log-file=rsync_nbies1_uup.log the result of this command is that I can see 2 processes running on the target server: pngesp1 9065 9064 0 Nov23 ?00:00:03 rsync --server -lOogDtprze.is --delete . /opt/app/es/appahome/pngesp1/var/m3g/projs/apr/interfaces/output/UUP/ pngesp1 9070 9065 29 Nov23 ?00:02:57 rsync --server -lOogDtprze.is --delete . /opt/app/es/appahome/pngesp1/var/m3g/projs/apr/interfaces/output/UUP/ the problem starts when the command on source ends, but I can still see the command on the server running. then my process is executing a new rsync command on source 5 minutes later and I can see 2 servers running on target (total of 4 processes). example: pngesp1 9065 9064 0 Nov23 ?00:00:03 rsync --server -lOogDtprze.is --delete . /opt/app/es/appahome/pngesp1/var/m3g/projs/apr/interfaces/output/UUP/ pngesp1 9070 9065 29 Nov23 ?00:02:59 rsync --server -lOogDtprze.is --delete . /opt/app/es/appahome/pngesp1/var/m3g/projs/apr/interfaces/output/UUP/ pngesp1 20097 20096 92 00:00 ?00:00:01 rsync --server -lOogDtprze.is --delete . /opt/app/es/appahome/pngesp1/var/m3g/projs/apr/interfaces/output/UUP/ pngesp1 20102 20097 0 00:00 ?00:00:00 rsync --server -lOogDtprze.is --delete . /opt/app/es/appahome/pngesp1/var/m3g/projs/apr/interfaces/output/UUP/ when that happens I get many errors on the new process stating that the temp file, rsync is creating, doesn't exist. probably the process that should have gone down, already deleted it. more info: rsync version 3.0.6 protocol version 30 uname -a -> Linux olpvces1 2.6.18-416.el5 #1 SMP Wed Oct 26 12:04:18 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12440] make "rsync -N" == "rsync --numeric-ids" ?
https://bugzilla.samba.org/show_bug.cgi?id=12440 Wayne Davison changed: What|Removed |Added Resolution|--- |WONTFIX Status|REOPENED|RESOLVED -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12440] make "rsync -N" == "rsync --numeric-ids" ?
https://bugzilla.samba.org/show_bug.cgi?id=12440 Harri changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|WONTFIX |--- --- Comment #4 from Harri --- PS: The popt feature is interesting, but I made this suggestion for the benefit of *all* rsync users. Not to mention that there would be no error message if the /etc/popt and ~/.popt files are missing and rsync would silently use --crtimes instead, mapping it to NOOP. Please reconsider. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12440] make "rsync -N" == "rsync --numeric-ids" ?
https://bugzilla.samba.org/show_bug.cgi?id=12440 --- Comment #3 from Harri --- I have several objections: * --numeric-ids is in wide use, while there is no --crtimes in rsync yet * --crtimes is useful only on MacOS, --numeric-ids on all Unix-like platforms. The probality for someone using --numeric-ids is much higher. * There is no "n" in "crtimes". The letter was chosen just because it was free. Another char could be used just as easy. But "N" == "numeric"? Thats easy to remember. Of course the --crtimes is a useful option for the MacOS command line users, but maybe Apple is dropping this feature in their new file system next week. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12440] make "rsync -N" == "rsync --numeric-ids" ?
https://bugzilla.samba.org/show_bug.cgi?id=12440 Wayne Davison changed: What|Removed |Added Resolution|--- |WONTFIX Status|NEW |RESOLVED --- Comment #2 from Wayne Davison --- The -N option is the letter that the crtimes patch uses for --crtimes, so I consider it already allocated. One thing you can use is the popt (option parsing) library's support for aliases in both /etc/popt and ~/.popt -- e.g.: rsync alias -N --numeric-ids rsync alias -9 -ai --numeric-ids --chmod=o-rwx --omit-link-times As long as you allocate your hosts with a consistent /etc/popt file, you can treat the option the same as if it were compiled into the executable. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12440] make "rsync -N" == "rsync --numeric-ids" ?
https://bugzilla.samba.org/show_bug.cgi?id=12440 --- Comment #1 from Harri --- Created attachment 12689 --> https://bugzilla.samba.org/attachment.cgi?id=12689&action=edit proposed patch -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12440] New: make "rsync -N" == "rsync --numeric-ids" ?
https://bugzilla.samba.org/show_bug.cgi?id=12440 Bug ID: 12440 Summary: make "rsync -N" == "rsync --numeric-ids" ? Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 Component: core Assignee: way...@samba.org Reporter: hdun...@aixigo.de QA Contact: rsync...@samba.org Hi folks, "--numeric-ids" is a lot of text and easy to be forgotten or misspelled. Since it is a highly important option for making backups of remote systems via rsync I wonder if "-N" could be introduced as an abbreviation for "--numeric-ids"? "-N" is not in use yet, afaics. Just a suggestion, of course. Keep on your good work. Regards Harri -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12430] New: rsync daemon chooses wrong place for pathes containing space and the module name
https://bugzilla.samba.org/show_bug.cgi?id=12430 Bug ID: 12430 Summary: rsync daemon chooses wrong place for pathes containing space and the module name Product: rsync Version: 3.1.1 Hardware: All OS: Linux Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: thimo.neuba...@cst.com QA Contact: rsync...@samba.org When uploading our internal test data (which features a rich set of filenames and paths) I've discovered that some files ended up in the wrong place. Here's a reduced setup to reproduce the problem: - create directory /tmp/foo - create a minimal rsyncd.conf, e.g. """ use chroot = false uid = thimo [foo] path = /tmp/foo read only = false """ - start rsyncd with "sudo rsync --daemon --no-detach --config=rsyncd.conf" Now the testcase is to copy some file into arbitrary directories. As a working case perform: - mkdir -p '/tmp/foo/c boo' - rsync '/etc/issue' 'rsync://localhost/foo/c boo/issue' - as expected a file '/tmp/foo/c boo/issue' will appear Now if the module name (in this case "foo") appears after the space you can see the bug: - mkdir -p '/tmp/foo/c foo' - rsync '/etc/issue' 'rsync://localhost/foo/c foo/issue' - contrary to the case above you'll notice that the file ended up in '/tmp/foo/c' instead of '/tmp/foo/c foo/issue' Quoting wasn't a problem in the first case, hence it shouldn't be one in the second case -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 7249] Add an option to use O_NOATIME
https://bugzilla.samba.org/show_bug.cgi?id=7249 --- Comment #7 from cyril...@gmail.com --- +1 I'm currently using a locally patched version, because I'm using rsync to mirror my whole data daily, plus using a deduplicating backup tool weekly that stores atime (so if rsync sets atime, the metadata are never deduplicated). It is logical for a backup tool to avoid setting atime, and logical that it is an option for rsync that may or may not be used as a backup tool. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12417] Sametimes currupted file after disconnect
https://bugzilla.samba.org/show_bug.cgi?id=12417 --- Comment #1 from Wayne Davison --- Rsync 3.1.2 doesn't leave its tmp files around unless it is killed by an uncaught signal, such as a sigbus. You might want to enable core files and/or monitor the rsync process to see how it is dying. As for the corruption, rsync uses a full-file checksum on the resulting file to see if it was constructed correctly, so I'd suspect that your system is having some kind of a hardware issue, possibly connected to the above abnormal terminations. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12417] New: Sametimes currupted file after disconnect
https://bugzilla.samba.org/show_bug.cgi?id=12417 Bug ID: 12417 Summary: Sametimes currupted file after disconnect Product: rsync Version: 3.1.2 Hardware: x64 OS: All Status: NEW Severity: major Priority: P5 Component: core Assignee: way...@samba.org Reporter: anatoly.pen...@kaspersky.com QA Contact: rsync...@samba.org Hello. Could you help with our bug. On client host: rsync version 3.1.2 protocol version 31 Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, no xattrs, iconv, symtimes, prealloc SSH: OpenSSH_7.2p1, OpenSSL 1.0.2h 3 May 2016 OS Windows 2012 R2 Standard 64-bit, rsync & ssh from Cygwin. Server side: rsync version 3.1.2 protocol version 31 Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 32-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, no iconv, symtimes, no prealloc, file-flags SSH: OpenSSH_7.2p2, OpenSSL 1.0.1s-freebsd 1 Mar 2016 From client start upload to server several files (big - more than 100MB and small – less than 1 mb). Command for start: rsync.exe -rlDi -z -t --no-h --progress --stats --chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r --delay-updates --delete-after --force --ignore-errors -e "'ssh.exe' -vv -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 -F '.ssh/ssh_config' -i '/cygdrive/C/Windows/TEMP/DRSA4A3.tmp' " -f '- *.lock' "/cygdrive/E/LocalStorage/FILE_ PR_DL - 991486649-0/shares/release/pr_dl/" u...@server.kaspersky-labs.com:"/data/pr_dl/" Sometimes, connection is disconnected between server & client, in rsync log we see: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12367] temporary lines in --progress output are not cleared
https://bugzilla.samba.org/show_bug.cgi?id=12367 --- Comment #3 from Wayne Davison --- Actually, I diagnosed this further, and noticed that the file counts shouldn't be output at all -- the check for progress output was being fooled by a temporary setting of the xfer_dirs var when -R is used without -d or -r. I've changed the show_filelist_p function into a var that is set during the init call so that it is always evaluated the same way. This git commit also contains a fix for the newline that could get output with --quiet: ff66fd4bb -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12367] temporary lines in --progress output are not cleared
https://bugzilla.samba.org/show_bug.cgi?id=12367 Wayne Davison changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #2 from Wayne Davison --- This is apparently only happening with no -r combined with --progress. In this case there can be a series of file-count outputs (with CR) prior to the start of the transfer, but no LF before it moves on to other output. Git commit e02b89d0d fixes this. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12386] Copy Loop
https://bugzilla.samba.org/show_bug.cgi?id=12386 Wayne Davison changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |INVALID --- Comment #2 from Wayne Davison --- You'd probably be better served by asking about something like this on the mailing list, as there is no issue with .* concerning rsync -- this is entirely a shell issue. In this instance you're probably using bash, which is so stupid that it expands ".*" to include "..", which is obviously not what you want. I personally use zsh, which doesn't have this issue. When using bash, you could instead try a "path/.??*" (if all your dot files are at least 2 chars after the dot) or something like ".[a-z]*" (though you might also need A-Z and 0-9 etc added). -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12386] Copy Loop
https://bugzilla.samba.org/show_bug.cgi?id=12386 --- Comment #1 from Kevin Korb --- That isn't a bug you are using it wrong. drop the .* and it will do what you expect. There should almost never be a * in the source parameter. Also, don't rsync without --times unless you have a really good reason. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12386] New: Copy Loop
https://bugzilla.samba.org/show_bug.cgi?id=12386 Bug ID: 12386 Summary: Copy Loop Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: crysant...@gmail.com QA Contact: rsync...@samba.org If you use .* as a rsync source it will be looped until end of disk ie. rsync -rvhIW crp/.* rohanamobile/public/ -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #6 from Kevin Korb --- If you want to sync files newer than say 3 days ago that is what --files-from is for... cd /source find . -mtime -3 -print | rsync -vai --files-from=- . /target The primary purpose of --files-from is to give rsync the power of find. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #5 from Alessio --- (In reply to Wayne Davison from comment #4) Well if it's by design I will not go further. But for me it's a missing "feature" because one has to prepare the list beforehand while rsync could do the job "while it's there". It's like the impossibility to sync files newer than specific date, which I really miss, but it's another story. Thanks for your help! (In reply to Kevin Korb from comment #3) Thanks for the detailed example and explanation it's exactly what happens to me. And if you remove the "--files-from" you see that "cache" folder is really excluded from the copy. I was wondering why the behavior is different applying the "--files-from" or not and unfortunately WayneD said it's by design. Thanks for your help -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 Wayne Davison changed: What|Removed |Added Resolution|--- |WORKSFORME Status|NEW |RESOLVED --- Comment #4 from Wayne Davison --- Command args (which includes names inside a files-from file) are never excluded by an exclude directive. You told rsync to copy it, so it copies it. Excludes only affect matching of files that rsync finds inside directories that you told it to copy. It is expected that you already trimmed any unwanted files/dirs from your args before you called rsync, since those names are under your control. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #3 from Kevin Korb --- I didn't say anything specific about a remote host. You excluded .cache then you told it to copy specific files some of which are in .cache. If there was a file named .cache inside of a directory that you told it to copy then it would be excluded. If you excluded userName/.cache/winetricks/msls31/InstMsiW.exe then it would be excluded. Exclude is not patern matched against parts of things you explicitly tell rsync to copy... kmk@dementia[1%]> cd /tmp kmk@dementia[2%]> mkdir test kmk@dementia[3%]> cd !$ cd test kmk@dementia[4%]> mkdir src dst kmk@dementia[5%]> cd src kmk@dementia[6%]> mkdir -p a b c d e cache/c kmk@dementia[7%]> cd .. kmk@dementia[8%]> /bin/echo -ne "a\nb\nc\nd\ne\ncache/c\n" > list kmk@dementia[9%]> cat list a b c d e cache/c kmk@dementia[10%]> rsync -vain --files-from=list --exclude=cache src/ dst/ building file list ... done cd+ a/ cd+ b/ cd+ c/ cd+ cache/ cd+ cache/c/ cd+ d/ cd+ e/ sent 157 bytes received 37 bytes 388.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) kmk@dementia[11%]> rsync -vain --files-from=list --exclude=c src/ dst/ building file list ... done cd+ a/ cd+ b/ cd+ d/ cd+ e/ sent 106 bytes received 28 bytes 268.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) kmk@dementia[12%]> rsync -vain --files-from=list --exclude=cache/c src/ dst/ building file list ... done cd+ a/ cd+ b/ cd+ c/ cd+ d/ cd+ e/ sent 121 bytes received 31 bytes 304.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) Cache isn't really being copied it is just being created to make a place for cache/c to be stored when cache/c or c is not excluded. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #2 from Alessio --- (In reply to Kevin Korb from comment #1) Excuse me, but i don't get it, what's the difference between excluding (matching) the pattern from a --files-from and a remote host? debugging the rsync transfer from the remote host I get: [sender] hiding directory home/userName/.cache because of pattern .cache in the other case nothing is being hidden instead but .cache is inside my path: userName/.cache/winetricks/msls31/InstMsiW.exe -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 --- Comment #1 from Kevin Korb --- It did not copy the directory you excluded it copied the files within that directory that you explicitly told it to copy and created the appropriate directories to allow that to happen. IOW, .cache is not relative to userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15 -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12378] New: why i cannot exclude dir/files if using option "--files-from"
https://bugzilla.samba.org/show_bug.cgi?id=12378 Bug ID: 12378 Summary: why i cannot exclude dir/files if using option "--files-from" Product: rsync Version: 3.1.2 Hardware: x64 OS: Linux Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: is...@sissa.it QA Contact: rsync...@samba.org I don't understand if this is intended or not, but when I use "--files-from" and "--exclude" options together all files are synced anyhow. eg.: rsync -av -c --delete --relative --delete-excluded --exclude=".cache" --exclude="mydirToExclude" /home . it correctly doesn't copy the .cache folder. but: rsync -av -c --delete --relative --delete-excluded --files-from=myFileList.txt --exclude=".cache" --exclude="mydirToExclude" /home . does copy everything. instead using a pattern such as --exclude="**myDirToExclude**" doesn't copy all lines with that pattern myFileList.txt is something like: userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/D3BD7BB89AFC5E5C3921C55D38B7DB4F6A6A8C55 userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15 userName/.cache/mozilla/firefox/e9dwu0jm.default/directoryLinks.json userName/.cache/winetricks/dotnet40/dotNetFx40_Full_x86_x64.exe userName/.cache/winetricks/dotnet30/dotnetfx3.exe userName/.cache/winetricks/dotnet20sp2/NetFx20SP2_x86.exe userName/.cache/winetricks/msls31/InstMsiW.exe userName/Downloads/Sentinel_LDK_Run-time_setup/HASPUserSetup.exe userName/Downloads/Sentinel_LDK_Run-time_setup/readme.html Thanks, alessio -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #25 from Коренберг Марк --- What about fallcate()d area beyond file size ? Will they be synchronized ? Just curious. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #24 from Theodore Ts'o --- So a simple workaround would be to use fallocate with KEEP_SIZE at first, then use punch whole, write the blocks, etc., and then use either truncate to set i_size, or seek to the desired size minus one and write a single byte. Seeking to the desired size minus one is more portable, but if you want to avoid allocating an extra 4k block, you could try using truncate, and if that doesn't set i_size (it's not guaranteed by POSIX, but I believe all Linux file systems will set i_size), seeking to size-1 and writing a single zero byte is guaranteed to work. That being said, I agree that ext4 should allow punch hole to work beyond i_size, if there are blocks allocated using fallocate(2). We'll fix that for the future, but for now, the workaround suggested above is probably the simplest way to work around the issue in a way that's compatible with both the current and future behavior. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #23 from Wayne Davison --- > Continuing to think aloud. It's not really a hole, it's already reserved > space. Exactly, and it's impossible to punch holes in that allocation. I'm changing my patch to give the file a size to deal with this anomaly. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #22 from Andrey Gursky --- (In reply to Andrey Gursky from comment #21) Continuing to think aloud. It's not really a hole, it's already reserved space. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #21 from Andrey Gursky --- (In reply to Andrey Gursky from comment #20) Sorry, it is indeed preallocated. Other still holds: hole-punch doesn't fail because the file already consists of only hole. Such file I would call a preallocated data-sparse. In opposite to the usual not preallocated space-sparse with truncate: $ truncate -s 1048576 test-sparse $ ls -ls test-sparse 0 -rw-r--r-- 1 andrey andrey 1048576 Oct 9 15:34 test-sparse $ /usr/sbin/filefrag -v test-sparse Filesystem type is: ef53 File size of test-sparse is 1048576 (256 blocks of 4096 bytes) test-sparse: 0 extents found -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #20 from Andrey Gursky --- (In reply to Wayne Davison from comment #17) From what I see, it doesn't fail, since the file is not preallocated at all with FALLOC_FL_KEEP_SIZE, but just a fully sparse file is created (consisting of only one big hole): $ ls -ls test-file 1024 -rwx-- 1 andrey andrey 0 Oct 9 14:47 test-file $ /usr/sbin/filefrag -v test-file Filesystem type is: ef53 File size of test-file is 0 (0 blocks of 4096 bytes) ext: logical_offset:physical_offset: length: expected: flags: 0:0.. 255: 14828800.. 14829055:256: last,unwritten,eof test-file: 1 extent found -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #19 from Wayne Davison --- Also, to be more like rsync would do you can follow the hole-punch with a seek and a write so that the file ends up with a non-zero size. Apparently if I change the order to do the seek & the write first and THEN punch hole it works. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #18 from Wayne Davison --- FYI, I tested on Linux 4.2.0 and 3.10.0 (I don't have a newer kernel running here to try). -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #17 from Wayne Davison --- Take the test program and change the SYS_fallocate to use the FALLOC_FL_KEEP_SIZE flag (don't forget to "rm test-file") and it will fail. Rsync always pre-allocates with FALLOC_FL_KEEP_SIZE when the flag is available. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #16 from Andrey Gursky --- (In reply to Theodore Ts'o from comment #15) Theo, thanks for taking time to test it! This works for me too (Debian Testing 4.7.4-2, ext4): $ /usr/sbin/filefrag -v test-file Filesystem type is: ef53 File size of test-file is 1048576 (256 blocks of 4096 bytes) ext: logical_offset:physical_offset: length: expected: flags: 0:0.. 7: 14482176.. 14482183: 8: unwritten 1: 16.. 255: 14482192.. 14482431:240: last,unwritten,eof test-file: 1 extent found $ -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #15 from Theodore Ts'o --- Created attachment 12557 --> https://bugzilla.samba.org/attachment.cgi?id=12557&action=edit Test program to show that fallocate followed by punch hole works just fine -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #14 from Theodore Ts'o ---
>I believe "on the same file handle" is the unusual prerequisite to trigger the
>>behavior described by Wayne.
I was fairly sure that was a red herring, so I was trying to save myself some
time, but no, it doesn't replicate even if you use the same file descriptor
{/usr/projects/linux/ext4} (origin)
1009% strace /tmp/test-fallocate
execve("/tmp/test-fallocate", ["/tmp/test-fallocate"], [/* 64 vars */]) = 0
brk(NULL) = 0x187e000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f20e7d2a000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=101609, ...}) = 0
mmap(NULL, 101609, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f20e7d11000
close(3)= 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\3\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1685264, ...}) = 0
mmap(NULL, 3791264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f20e776d000
mprotect(0x7f20e7902000, 2093056, PROT_NONE) = 0
mmap(0x7f20e7b01000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x194000) = 0x7f20e7b01000
mmap(0x7f20e7b07000, 14752, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f20e7b07000
close(3)= 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f20e7d0f000
arch_prctl(ARCH_SET_FS, 0x7f20e7d0f700) = 0
mprotect(0x7f20e7b01000, 16384, PROT_READ) = 0
mprotect(0x7f20e7d2d000, 4096, PROT_READ) = 0
munmap(0x7f20e7d11000, 101609) = 0
open("test-file", O_WRONLY|O_CREAT|O_TRUNC, 0700) = 3
fallocate(3, 0, 0, 1048576) = 0
fallocate(3, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 32768, 32768) = 0
close(3)= 0
exit_group(0) = ?
+++ exited with 0 +++
{/usr/projects/linux/ext4} (origin)
1010% filefrag -v test-file
Filesystem type is: ef53
File size of test-file is 1048576 (256 blocks of 4096 bytes)
ext: logical_offset:physical_offset: length: expected: flags:
0:0.. 7: 62130432.. 62130439: 8: unwritten
1: 16.. 255: 62130448.. 62130687:240:
last,unwritten,eof
test-file: 1 extent found
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #13 from Andrey Gursky --- (In reply to Theodore Ts'o from comment #11) Theo, I believe "on the same file handle" is the unusual prerequisite to trigger the behavior described by Wayne. Or such a test is already contained in e2fsprogs (in a C test program, not a shell script)? Regards, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #12 from Andrey Gursky --- (In reply to Wayne Davison from comment #9) >> Hole-punch works only for full filesystem blocks > That has nothing to do with it. Wayne, OK, might be. I haven't tested it the exactly way you're doing it now, since I did it other way. But this was important to take care in order to fix the bugs related to sparse with inplace [1], [2]. But due to missing response I've hold back the fix (not yet throughly tested) for me. [1] About data/token send/receive protocol part and more https://lists.samba.org/archive/rsync/2015-December/030471.html [2] Status of --inplace and --sparse in rsync or alternative? https://lists.samba.org/archive/rsync/2015-December/030472.html -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588
--- Comment #11 from Theodore Ts'o ---
Re: #9. I'm not able to reproduce the described behavior. If you want to
follow up on what you think is a kernel bug, please send a simple repro program
or script and what version of the kernel you are using to the linux-ext4
mailing list. Thanks!!
Cheers,
{/usr/projects/docker/dropbox} (master)
1009% fallocate -o 0 -l 128M test.file
{/usr/projects/docker/dropbox} (master)
1010% filefrag -v test.file
Filesystem type is: ef53
File size of test.file is 134217728 (32768 blocks of 4096 bytes)
ext: logical_offset:physical_offset: length: expected: flags:
0:0..4095: 55990272.. 55994367: 4096: unwritten
1: 4096..8191: 56025088.. 56029183: 4096: 55994368: unwritten
2: 8192.. 10239: 56170496.. 56172543: 2048: 56029184: unwritten
3:10240.. 14335: 56180736.. 56184831: 4096: 56172544: unwritten
4:14336.. 16383: 56252416.. 56254463: 2048: 56184832: unwritten
5:16384.. 20479: 56229888.. 56233983: 4096: 56254464: unwritten
6:20480.. 28671: 56305664.. 56313855: 8192: 56233984: unwritten
7:28672.. 32767: 56352768.. 56356863: 4096: 56313856:
last,unwritten,eof
test.file: 8 extents found
{/usr/projects/docker/dropbox} (master)
1011% xfs_io -c "fpunch 65536 65536" test.file
{/usr/projects/docker/dropbox} (master)
1012% filefrag -v test.file
Filesystem type is: ef53
File size of test.file is 134217728 (32768 blocks of 4096 bytes)
ext: logical_offset:physical_offset: length: expected: flags:
0:0.. 15: 55990272.. 55990287: 16: unwritten
1: 32..4095: 55990304.. 55994367: 4064: unwritten
2: 4096..8191: 56025088.. 56029183: 4096: 55994368: unwritten
3: 8192.. 10239: 56170496.. 56172543: 2048: 56029184: unwritten
4:10240.. 14335: 56180736.. 56184831: 4096: 56172544: unwritten
5:14336.. 16383: 56252416.. 56254463: 2048: 56184832: unwritten
6:16384.. 20479: 56229888.. 56233983: 4096: 56254464: unwritten
7:20480.. 28671: 56305664.. 56313855: 8192: 56233984: unwritten
8:28672.. 32767: 56352768.. 56356863: 4096: 56313856:
last,unwritten,eof
test.file: 8 extents found
{/usr/projects/docker/dropbox} (master)
1013% uname -a
Linux callcc 4.8.0-00041-gecd2f69 #3 SMP Mon Oct 3 02:56:05 EDT 2016 x86_64
GNU/Linux
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #10 from Wayne Davison --- > ... I can share my work. Sounds interesting! Looking forward to seeing what you've come up with. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #9 from Wayne Davison --- > Hole-punch works only for full filesystem blocks That has nothing to do with it. If you fallocate() the full file length and then (on the same file handle) try to punch out parts of the allocated file, no blocks change away from becoming allocated. Looks like a bug in Linux to me. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] better handling for --preallocate with --sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #8 from Andrey Gursky --- (In reply to Wayne Davison from comment #7) Wayne, since this bug made rsync unusable for me, I fixed that and implemented additional checks needed for ext4 a month or two after I reported this bug and saw no reaction at all. Now a couple of people got interested and you also, so I can share my work. It's not tiny. > In my testing, using both a pre-allocate call on a file followed by a > hole-punch call has no effect on the allocation of the blocks (though it does > zero them). Yes, this is tricky. Hole-punch works only for full filesystem blocks (e.g., default 4K). Issuing few partial hole-punch requests wouldn't work, even if they cover the whole block. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12305] --fallocate and --sparse work wrong
https://bugzilla.samba.org/show_bug.cgi?id=12305 --- Comment #5 from Коренберг Марк --- Fallocated: areas of the file that has been fallocate()d, but stillnot written. Technically, on sender, even written parts that was written, but contain zeroes may be considered as fallocated areas. I mean that receiver should call fallocate() on that region instead writing zeroes. How to determine: How to determine if area is alocated: lseek() + SEEK_DATA/SEEK_HOLE. And after that, check if non-sparse area contains zeroes. Another way - is to examine fiemap ( https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt ) -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #7 from Wayne Davison --- Created attachment 12556 --> https://bugzilla.samba.org/attachment.cgi?id=12556&action=edit Preliminary patch to support punching holes In my testing, using both a pre-allocate call on a file followed by a hole-punch call has no effect on the allocation of the blocks (though it does zero them). I tested --sparse and --inplace with this, and it worked fine on one system (with a new enough linux kernel). There are cases where the sparseness will be lost, though, depending on OS & filesystem. I'm thinking we just update the docs to mention that if you combine --sparse with --inplace (and/or --preallocate) that you might not get the sparseness preserved. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #6 from Carson Gaspar --- For punching holes, Solaris and UnixWare support F_FREESP(64) in fcntl(). Windows supports both reporting and punching holes, but I don't know if cygwin (or any other rsync on windows platform) implements it. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #5 from Carson Gaspar --- (In reply to Carson Gaspar from comment #4) Actually, you never want the sender to scan for zero regions if SEEK_HOLE isn't supported, as performance would then be terrible. And a given filesystem may not support SEEK_HOLE, even if lseek() does. So we're really back to picking (1), (2), or (3), as I don't see unreliable sender sparse mapping as sensible (although using SEEK_HOLE to save source file read time and provide a hint to the receiver may be nice). -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #4 from Carson Gaspar --- rsync currently just has the receiver turn "long" sequences of zeroes into sparse regions when --sparse is specified. If --preallocate is also specified, what would you like rsync to do? No wire protocol change required (please pick which behaviour you prefer even if you'd rather a protocol bump, as we may negotiate an older wire protocol): 1) Emit an error (as with --in-place and --sparse) 2) Disable one of the options (which one?) 3) Pre-allocate the file, but when zero regions are detected then ftruncate() it and create the sparse region. Reallocate the rest of the file space after creating the sparse region or not? (IFF receiver is on Linux and on a supported filesystem, fallocate(,FALLOC_FL_PUNCH_HOLE,...) could be used to create sparse regions without truncating the file) Wire protocol change required, have the sender determine the sparse regions, using SEEK_HOLE if available, otherwise scanning for all-zero regions: A) Preallocate 1st data region, create 1st sparse region, preallocate 2nd data region, ... -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12305] --fallocate and --sparse work wrong
https://bugzilla.samba.org/show_bug.cgi?id=12305 --- Comment #4 from Carson Gaspar --- Please define what you mean by 'fallocated' in (1) and (2). Please also specify how you're determining that something has been 'fallocated'. I agree that (3) is a bug, and as the only real one that I can see, and is correctly marked a s a dup. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12367] temporary lines in --progress output are not cleared
https://bugzilla.samba.org/show_bug.cgi?id=12367 --- Comment #1 from Andrey Gursky --- Hi Paul, you reported something similar on the list (https://lists.samba.org/archive/rsync/2015-December/030478.html). I've instantly recalled that, because this issue also annoys me. If I redirect rsync output via pipe and tee the resulting file is polluted with percentage progress info. Regards, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12367] New: temporary lines in --progress output are not cleared
https://bugzilla.samba.org/show_bug.cgi?id=12367
Bug ID: 12367
Summary: temporary lines in --progress output are not cleared
Product: rsync
Version: 3.1.2
Hardware: All
OS: All
Status: NEW
Severity: minor
Priority: P5
Component: core
Assignee: way...@samba.org
Reporter: p...@debian.org
QA Contact: rsync...@samba.org
A bug report received by Debian
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749165). I can verify this.
emit_filelist_progress() in flist.c needs something like a
"output_needs_newline_maybe" variable... I can't quite see how to implement
that cleanly.
>>---
Some temporary lines output by --progress are not cleared, so that
the output looks strange. Here's a testcase.
#!/bin/sh
set -e
export LC_ALL=C
t=$HOME/rsync-test
mkdir "$t"
cd "$t"
if [ ! -d a1 ]; then
mkdir a1
touch a1/foo
fi
if [ ! -d a2 ]; then
mkdir a2
for i in `seq 1 250`
do
mkdir -p a2/dir$i
for j in `seq 1 20`
do
mktemp --tmpdir=a2/dir$i > /dev/null
done
done
fi
mkdir b
for i in 1 2
do
mkdir -p c/a$i
ln -ns ../c/a$i b
done
rs()
{
rsync -KRt "$@" a2/*/* a1/foo "localhost:$t/b/"
}
rs
rs --progress
rm -r "$t"
I get:
a1/00 files...
a2/
(sometimes without the "a2/" line).
If I redirect the output to some file, I can see with "less":
0 files...^M 400 files...^M 2500 files...^M 4600 files...^Ma1/
a2/
The problem is that when "a1/" is output, the previous " 4600 files..."
line was not cleared and one can still see the end of it.
<<---
I can verify this.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12336] New: SKYPE password reset((+1 (866)*769 (8I27) @SKYPE Tech Support number
https://bugzilla.samba.org/show_bug.cgi?id=12336 Bug ID: 12336 Summary: SKYPE password reset((+1 (866)*769 (8I27) @SKYPE Tech Support number Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: erajasthan...@gmail.com QA Contact: rsync...@samba.org ❶❽667698127 SKYPE phone number 18667698127 SKYPE customer service phone number 18667698127 SKYPE desktop support, SKYPE laptop customer service, contact SKYPE support, SKYPE pc support, SKYPE laptop customer care number 18667698127, SKYPE support for , SKYPE customer care, SKYPE customer care phone number 18667698127, hewlett packard help, phone number 18667698127 for SKYPE, SKYPE online help, SKYPE laptop customer care, SKYPE helpline phone number 18667698127, SKYPE customer support, SKYPE technical support chat, SKYPE computer help, SKYPE support number 18667698127s, SKYPE technical support contact number 18667698127, SKYPE telephone number 18667698127, SKYPE technical support phone number 18667698127, SKYPE helpline, SKYPE support , SKYPE support online, SKYPE contact number 18667698127, SKYPE help phone number 18667698127, SKYPE customer care 18667698127SKYPE helpline, SKYPE telephone support, SKYPE online support, SKYPE support contact,SKYPE chat support, hewlett packard phone number, SKYPE 18667698127 customer service, SKYPE tech support number,18667698127 SKYPE product support, hewlett packard customer service phone number18667698127,SKYPE computer support number, SKYPE support contact number,18667698127 SKYPE support , SKYPE computer support, SKYPE tech support chat,18667698127 1888 468 0321SKYPE helpline number, 18667698127 SKYPE laptop support, hewlett packard tech support, SKYPE online chat, hewlett packard technical support, SKYPE help line, phone number for SKYPE support, hewlett packard support phone number, SKYPE technical support, hewlett packard customer service number,1888 468 0321 SKYPE service number, hewlett packard 18667698127 helpline, SKYPE customer care no, SKYPE customer service number, SKYPE help number, 18667698127 SKYPE customer service phone number, SKYPE number, SKYPE support phone, SKYPE support line, hewlett packard contact number, SKYPE tech support phone number, SKYPE customer support phone number, SKYPEs help, call SKYPE support, ##SKYPE support## chat, hewlett packard support number, hewlett packard tech support number, SKYPE support telephone number, hewlett packard tech support phone number, call SKYPE, SKYPE contact support, hewlett packard technical support phone number, SKYPE support centre, hewlett packard customer support, SKYPE desktop support, SKYPE laptop customer service, contact SKYPE support, SKYPE pc support, SKYPE laptop customer care number, SKYPE support for s, SKYPE customer care, SKYPE customer care phone number, hewlett packard help, phone number for SKYPE, SKYPE online help, SKYPE laptop customer care, SKYPE helpline phone number, SKYPE customer support, SKYPE technical support chat, SKYPE computer help, SKYPE support numbers, SKYPE technical support contact number, SKYPE telephone number, SKYPE technical support phone number, SKYPE helpline, SKYPE support s, SKYPE support online, SKYPE contact number, SKYPE help phone number, SKYPE customer care number, contact hewlett packard by phone, SKYPE phone support, hewlett packard s support, SKYPE tech support phone, SKYPE technical help, SKYPE laptop tech support number, contact SKYPE by phone, SKYPE support call, SKYPE computers support, hewlett packard customer service telephone number, phone number for hewlett packard, SKYPE online support chat, SKYPE laptop customer service number, SKYPE online chat support, SKYPEs customer service, hewlett packard customer service phone, SKYPE laptop tech support, SKYPE service phone number, hewlett packard help, phone number for SKYPE s, SKYPE troubleshooting phone number, SKYPE number,http://bit.ly/2bwGOcj hewlett packard technical support number, contact SKYPE support phone, phone number for SKYPE support, SKYPE customer support chat, SKYPE help and support number, contact hewlett packard, SKYPE laptop support phone number, SKYPEs customer service phone number, SKYPE laptop customer service phone number, SKYPE computer support phone number, SKYPE pavilion support, SKYPE computer customer service, SKYPE customer services, hewlett packard telephone number, SKYPE helpline no, SKYPE help desk number, contact SKYPE support phone number, hewlett packard contact, SKYPE phone numbers, SKYPEs customer care number, SKYPE help and support, contact SKYPE technical support, SKYPE contact numbers, contact SKYPE support chat, call SKYPE tech support, SKYPE customer service phone, SKYPE help support, SKYPE computer tech support, SKYPE assistance phone number
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #3 from Andrey Gursky --- (In reply to Коренберг Марк from comment #2) Марк, nevertheless it is still the same issue. You're welcome to add the details here. Please, be sure, to fix the typo: instead of --fallocate it should be --preallocate. Thanks, Andrey -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 --- Comment #2 from Коренберг Марк --- Bug 12305 has much more detailed description -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 12305] --fallocate and --sparse work wrong
https://bugzilla.samba.org/show_bug.cgi?id=12305 Wayne Davison changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |DUPLICATE --- Comment #3 from Wayne Davison --- *** This bug has been marked as a duplicate of bug 11588 *** -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11588] missing option: preallocate for all files except for sparse
https://bugzilla.samba.org/show_bug.cgi?id=11588 Wayne Davison changed: What|Removed |Added CC||socketp...@gmail.com --- Comment #1 from Wayne Davison --- *** Bug 12305 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
