[PATCH 6/7] lei up: avoid excessively parallel --all

We shouldn't dispatch all outputs right away since they can be expensive CPU-wise. Instead, rely on DESTROY to trigger further redispatches. This also fixes a circular reference bug for the single-output case that could lead to a leftover script/lei after MUA exit. I'm not sure how --jobs/-j

[PATCH 7/7] lei: drop redundant WQ EOF callbacks

Redundant code is noise and therefore confusing :< --- lib/PublicInbox/LeiImportKw.pm | 8 +--- lib/PublicInbox/LeiMirror.pm| 10 ++ lib/PublicInbox/LeiNoteEvent.pm | 8 +--- lib/PublicInbox/LeiPmdir.pm | 8 +--- script/public-inbox-clone | 2 +- 5 files

[PATCH 4/7] lei: dclose: do not close unnecessarily

The bit about reap_compress is no longer true since LeiXSearch->query_done triggers it, instead. I only noticed this while working on "lei up". --- lib/PublicInbox/LEI.pm | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm index 29293e6c..a1cab55a

[PATCH 5/7] inbox: do not waste hash slot on httpbackend_limiter

A few dozen bytes saved here can add up when we have thousands of inboxes. It also makes Data::Dumper debug output a bit cleaner. --- lib/PublicInbox/Inbox.pm | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm index

[PATCH 0/7] lei bugfixes and other fixes

How did I forget 2/7 for all these months? :< 6/7 took a while for me to figure out, I only noticed it because of background processes running after MUA exit (from "lei up --mua=... MFOLDER"), but I got 4/7 and 7/7 fixes out of it. Eric Wong (7): ipc: do not add "0" to $0 of solo workers

[PATCH 3/7] script/lei: describe purpose of sleep loop

It looks dumb, but I'm not about to take a runtime penalty to use signalfd|EVFILT_SIGNAL, here, either. --- script/lei | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/script/lei b/script/lei index 399296ba..bc437798 100755 --- a/script/lei +++ b/script/lei @@ -137,6 +137,6 @@

[PATCH 2/7] treewide: fix %SIG localization, harder

This fixes the occasional t/lei-sigpipe.t infinite loop under "make check-run". Link: http://nntp.perl.org/group/perl.perl5.porters/258784 Followup-to: b552bb9150775fe4 ("daemon+watch: fix localization of %SIG for non-signalfd users") --- lib/PublicInbox/Admin.pm| 2 +-

[PATCH 1/7] ipc: do not add "0" to $0 of solo workers

It's needless noise and misleads users reading "ps" into thinking there's more workers when there's only one. --- lib/PublicInbox/IPC.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm index add5f3df..1c699d76 100644 ---

Re: Holding on to deleted packfiles

Konstantin Ryabitsev wrote: > Well, it may also not be something that's the responsibility of public-inbox > either, e.g. other long-running daemons don't perform such checks. We can just > issue a reload after we've done repacking. The less operational knowledge required, the better; especially

Re: Holding on to deleted packfiles

On Tue, Sep 21, 2021 at 07:06:53PM +, Eric Wong wrote: > Was this from /all/ (ALL.git using batch-file) or Gcf2? I believe this was from Gcf2, though I can't go back and check, unfortunately. > The old stuff has timers to do periodic cleanup, but the new > stuff is trickier as the cost of a

Re: Holding on to deleted packfiles

Konstantin Ryabitsev wrote: > Hello: > > A large git repack job that ran over the weekend revealed a minor problem -- > public-inbox daemon processes will hold on to deleted pack files until they > are restarted. Is there any way to gracefully recognize and handle this > condition? It's not

Holding on to deleted packfiles

Hello: A large git repack job that ran over the weekend revealed a minor problem -- public-inbox daemon processes will hold on to deleted pack files until they are restarted. Is there any way to gracefully recognize and handle this condition? It's not quite benign, as this ended up keeping 40GB+

[PATCH 3/3] lei: umask(077) before opening errors.log

There's a chance some sensitive information (e.g. folder names) can end up in errors.log, though $XDG_RUNTIME_DIR or /tmp/lei-$UID/ will have 0700 permissions, anyways. --- lib/PublicInbox/LEI.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/PublicInbox/LEI.pm

[PATCH 2/3] script/lei: handle SIGTSTP and SIGCONT

Sometimes it's useful to pause an expensive query or refresh-mail-sync to do something else. While lei-daemon and lei/store can't be paused since they're shared across clients, per-invocation WQ workers can be paused safely using the unblockable SIGSTOP. While we're at it, drop the ETOOMANYREFS

[PATCH 0/3] lei: a few more annoyances fixed

The more I use it, the more I find wrong... Eric Wong (3): t/lei-up: use '-q' to silence non-redirected test script/lei: handle SIGTSTP and SIGCONT lei: umask(077) before opening errors.log lib/PublicInbox/LEI.pm | 20 +--- script/lei | 9 - t/lei-up.t

[PATCH 1/3] t/lei-up: use '-q' to silence non-redirected test

We could redirect, too, but just use -q since we don't care for the output with run_mode => 0. --- t/lei-up.t | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/lei-up.t b/t/lei-up.t index 8937cfb1..fc369156 100644 --- a/t/lei-up.t +++ b/t/lei-up.t @@ -35,7 +35,7 @@

[PATCH 11/12] lei q: update messages to reflect --save default

I wanted to try --dedupe=none for something, but it failed since I forgot --no-save :x So hint users towards --no-save if necessary. --- lib/PublicInbox/LeiSavedSearch.pm | 4 ++-- t/lei-q-save.t| 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git

[PATCH 10/12] search: drop reopen retry message

It's needless noise in syslogs for daemons and unnecessarily alarming to users on the command-line. --- lib/PublicInbox/Search.pm | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm index af0a35d9..1d1cd2f5 100644 ---

[PATCH 12/12] lei q: improve --limit behavior and progress

Avoid slurping gigantic (e.g. 10) result sets into a single response if a giant limit is specified, and instead use 1 as a window for the mset with a given offset. We'll also warn and hint towards about the --limit= switch when the estimated result set is larger than the default limit.

[PATCH 09/12] lei q: show progress on >1s preparation phase

Overwriting existing destinations safe (but slow) by default, so show a progress message noting what we're doing while a user waits. --- lib/PublicInbox/LeiToMail.pm | 23 ++- lib/PublicInbox/LeiXSearch.pm | 2 +- t/lei-watch.t | 2 +- 3 files changed, 24

[PATCH 08/12] lei: various completion improvements

"lei export-kw" no longer completes for anonymous sources. More commands use "lei refresh-mail-sync" as a basis for their completion work, as well. ";AUTH=ANONYMOUS@" is stripped from completions since it was preventing bash completion from working on AUTH=ANONYMOUS IMAP URLs. I'm not sure if

[PATCH 06/12] doc: lei-security: section for WIP auth methods

Lots of stuff out there that becomes a pain to setup configuration for and test... --- Documentation/lei-security.pod | 8 1 file changed, 8 insertions(+) diff --git a/Documentation/lei-security.pod b/Documentation/lei-security.pod index 4b712c2d..02305b90 100644 ---

[PATCH 05/12] lei lcat: use single queue for ordering

If lcat-ing multiple argument types (blobs vs folders), maintain the original order of the arguments instead of dumping all blobs before folder contents. --- lib/PublicInbox/LeiLcat.pm| 13 ++--- lib/PublicInbox/LeiXSearch.pm | 16 2 files changed, 14 insertions(+),

[PATCH 04/12] lei: simplify internal arg2folder usage

We can set opt->{quiet} for (internal) 'note-event' command to quiet ->qerr, since we use ->qerr everywhere else. And we'll just die() instead of setting a ->{fail} message, since eval + die are more inline with the rest of our Perl code. --- lib/PublicInbox/LEI.pm| 2 +-

[PATCH 03/12] lei_mail_sync: account for non-unique cases

NNTP servers, IMAP servers, and various MUAs may recycle "unique" identifiers due to software bugs or careless BOFHs. Warn about them, but always be prepared to account for them. --- lib/PublicInbox/LeiImport.pm | 8 ++-- lib/PublicInbox/LeiImportKw.pm| 12

[PATCH 02/12] lei inspect: support NNTP URLs

No reason not to support them, since there's more public-inbox-nntpd instances than -imapd instances, currently. --- lib/PublicInbox/LeiInspect.pm | 48 +++--- lib/PublicInbox/LeiMailSync.pm | 40 +++- lib/PublicInbox/TestCommon.pm | 11

[PATCH 00/12] lei: fix various annoyances

Eric Wong (12): lei inspect: convert to WQ worker lei inspect: support NNTP URLs lei_mail_sync: account for non-unique cases lei: simplify internal arg2folder usage lei lcat: use single queue for ordering doc: lei-security: section for WIP auth methods lei lcat: support NNTP URLs

[PATCH 01/12] lei inspect: convert to WQ worker

Xapian and SQLite access can be slow when a DB is large and/or on high-latency storage. --- lib/PublicInbox/LeiInspect.pm | 31 +-- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm index