Re: [feature request?]: Show progress only for big files

[Lars Ellenberg via rsync]

On Fri, Feb 17, 2023 at 01:21:40PM +0100, anubis23 via rsync wrote:
> Hi,
> 
> I've read through the rsync manpage, this mailing list, asked Google and
> studied lots of posts on stackexchange.com (stackoverflow,
> superuser...), askubuntu.com and some others, concerning rsync's
> capabilities of showing progress information. But all I've found was
> what I already knew: --progress (or -P) shows a progress information for
> *every* file transmitted, --info=progress2 shows an overall progress of
> the whole transfer (more useful, but slower, if combined with
> --no-inc-recursive/--no-i-r...).
> 
> What I am looking for is a way to show the progress on file-level (like
> --progress or -P) but only for files bigger than a certain threshold,
> say for example >25 MB for a transfer to a remote host or >250 MB for a
> local copy. Printing the progress of files which are transfered within
> just one or two seconds is not really necessary. Is this possible? If
> not, the best I can imagine, would be a switch, maybe called
> "--progress-threshold=x", where 'x' is the threshold level (e.g. 25m for
> 25 MB...).

I'd base the threshold on time, not size.

Could look like below,
though PROGRESS_REPORT_THRESHOLD_MS would need to become an option,
and default to 0 (and, if paranoid about time jumps, possibly be
explicitly tested against 0) to not change current behaviour.

Cheers,
Lars


diff --git a/progress.c b/progress.c
index 87207fbf..22de7723 100644
--- a/progress.c
+++ b/progress.c
@@ -35,6 +35,7 @@ extern struct file_list *cur_flist;
 BOOL want_progress_now = False;
 
 #define PROGRESS_HISTORY_SECS 5
+#define PROGRESS_REPORT_THRESHOLD_MS 2500
 
 #ifdef GETPGRP_VOID
 #define GETPGRP_ARG
@@ -173,7 +174,9 @@ void end_progress(OFF_T size)
rprint_progress(stats.total_transferred_size,
stats.total_size, , True);
} else {
-   rprint_progress(size, size, , True);
+   /* Do not report progress for "quick" transfers, based 
on starting time. */
+   if (msdiff(_start.time, ) >= 
PROGRESS_REPORT_THRESHOLD_MS)
+   rprint_progress(size, size, , True);
memset(_start, 0, sizeof ph_start);
}
}
@@ -237,5 +240,9 @@ void show_progress(OFF_T ofs, OFF_T size)
return;
 #endif
 
+   /* Do not report progress for "quick" transfers, based on starting 
time. */
+   if (msdiff(_start.time, ) < PROGRESS_REPORT_THRESHOLD_MS)
+   return;
+
rprint_progress(ofs, size, , False);
 }

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion?

[Lars Ellenberg via rsync]

On Mon, Sep 04, 2017 at 10:45:26PM +, Don Kuenz via rsync wrote:
> Greetings,
>  
> Is it possible to use rsync to transmit a large, dynamic 2TB file in a 
> piecemeal fashion during daylight hours over the course of a dozen days?
> On a good day, about 200GB of data can be transferred before rsync times
> out to enable a nightly local backup to complete. The local backup
> changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
> is dynamic.
> 
> As an experiment, rsync was started on the client to sync the 2TB file.
> At that point, rsyncd on the backup server created a temporarily file
> with the 2TB file name appended with a dot followed by six characters.
> 
> After about 100MB was transferred the rsync client was interrupted and
> stopped in order to simulate the nightly shutdown. When rsync was 
> restarted it erased the original 100MB temporary file and started over
> with a new, empty temporarily file. 
> 
> In summary:
> 
> * rsync apparently erases temporarily files upon restart
> 
> * it takes about a dozen days to transfer the 2TB file
> 
> Unless there's an option to have rsync pick up where it left off it
> seems impossible to transmit a large file in a piecemeal fashion. 
> Because the (nearly) same 200GB at the beginning of the 2TB file gets
> retransmitted over and over with each restart.

Would --partial work for you?
Possibly combine with --partial-dir, --temp-dir,
--delay-updates, and/or --inplace.

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[Lars Ellenberg via rsync]

On Thu, Jun 08, 2017 at 07:09:24AM +, just subscribed for rsync-qa from 
bugzilla via rsync wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=12806
> 
> --- Comment #4 from Heinz-Willi Wyes  ---
> I got a personal message from Lars Ellenberg that went also to
> samba-b...@samba.org and rsync...@samba.org. He wrote:
> 
> --
> Calling chmod only on (optimization: non-empty) directories would fix this.
> I don't need to chmod a *file* before unlinking it, I just need write
> permission on the directory it is located in.
> 
> (Now you have to convince the "appliance" to use a patched rsync ...)

Note that as long as you have only files (not directories) without write
permissions, you can potentially work around this using "--super" (or 
"--fake-super"),
because in that case rsync sets "am_root" to 2 (or -1),
"!am_root" becomes false in the various conditions,
and the do_chmod() will not even be called.

Not sure though what other implications --[fake-]super will have for your 
usecase.

Lars Ellenberg


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[Lars Ellenberg via rsync]

Calling chmod only on (optimization: non-empty) directories would fix this.
I don't need to chmod a *file* before unlinking it,
I just need write permission on the directory it is located in.

(Now you have to convince the "appliance" to use a patched rsync ...)

Cheers,

Lars Ellenberg


diff --git a/delete.c b/delete.c
index 88e4230..223b6d2 100644
--- a/delete.c
+++ b/delete.c
@@ -97,10 +97,10 @@ static enum delret delete_dir_contents(char *fname, uint16 
flags)
}
 
strlcpy(p, fp->basename, remainder);
-   if (!(fp->mode & S_IWUSR) && !am_root && fp->flags & 
FLAG_OWNED_BY_US)
-   do_chmod(fname, fp->mode | S_IWUSR);
/* Save stack by recursing to ourself directly. */
if (S_ISDIR(fp->mode)) {
+   if (!(fp->mode & S_IWUSR) && !am_root && fp->flags & 
FLAG_OWNED_BY_US)
+   do_chmod(fname, fp->mode | S_IWUSR);
if (delete_dir_contents(fname, flags | DEL_RECURSE) != 
DR_SUCCESS)
ret = DR_NOT_EMPTY;
}
@@ -138,14 +138,13 @@ enum delret delete_item(char *fbuf, uint16 mode, uint16 
flags)
fbuf, (int)mode, (int)flags);
}
 
-   if (flags & DEL_NO_UID_WRITE)
-   do_chmod(fbuf, mode | S_IWUSR);
-
if (S_ISDIR(mode) && !(flags & DEL_DIR_IS_EMPTY)) {
/* This only happens on the first call to delete_item() since
 * delete_dir_contents() always calls us w/DEL_DIR_IS_EMPTY. */
ignore_perishable = 1;
/* If DEL_RECURSE is not set, this just reports emptiness. */
+   if (!(mode & S_IWUSR) && !am_root && flags & DEL_NO_UID_WRITE 
&& flags & DEL_RECURSE)
+   do_chmod(fbuf, mode | S_IWUSR);
ret = delete_dir_contents(fbuf, flags);
ignore_perishable = 0;
if (ret == DR_NOT_EMPTY || ret == DR_AT_LIMIT)

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync buffer overflow detected

[Lars Ellenberg via rsync]

On Fri, Apr 14, 2017 at 08:22:29PM +0300, Boris Savelev via rsync wrote:
> I use rsync from python on my Debian Jessie amd64 and get this error:
> *** buffer overflow detected ***: /rsync terminated

> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from 
> gdb:
> (gdb) bt

> #5  0x7791ca17 in __fdelt_chk (d=d@entry=1606) at fdelt_chk.c:25
> #6  0x55584c78 in safe_read (fd=fd@entry=1606, 
> buf=buf@entry=0x7fffa810 "\037", len=len@entry=4) at io.c:245

That is FD_SET(fd, _fds); with fd >= FD_SETSIZE, which is 1024.
You cannot use select with file descriptor numbers >= FD_SETSIZE (or < 0),
and glibc is catching that.

The "buffer" that would overflow is the fd_set.

Maybe rsync could simply close all inherited file descriptors,
first things first, before it does anything else,
possibly after making sure fds 0,1,2 are open to somewhere,
to avoid any output to "supposedly" stdout/stderr to clobber
fds opened only later.  Similar to what lvm tools do in their
_check_standard_fds() and _close_stray_fds()?

But of course rsync could also say: not my problem, *you* (whatever
entity was spawning rsync) leaked file descriptors, learn to use
O_CLOEXEC resp. set FD_CLOEXEC, so only 0,1,2 will be inherited.

quick and dirty workaround:
use a wrapper script, close all fds >= 3 "just in case",
then exec rsync.

> It looks like a bug, but I'm not sure)

Thanks,

Lars Ellenberg


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html