[PATCH 27/29] t/indexlevels-mirror*: switch to run_script

This more than doubles the speed of these tests --- t/indexlevels-mirror.t | 24 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/t/indexlevels-mirror.t b/t/indexlevels-mirror.t index 40afe4e9..d129237e 100644 --- a/t/indexlevels-mirror.t +++ b/t/indexlevels

[PATCH 24/29] t/nntpd: use run_script for -init

This only gives a 5% speedup or so, but anything helps. --- t/nntpd.t | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/t/nntpd.t b/t/nntpd.t index 462e2da9..b516ffd1 100644 --- a/t/nntpd.t +++ b/t/nntpd.t @@ -30,7 +30,6 @@ my $inboxdir = "$tmpdir/main.git"; my $group = 'te

[PATCH 19/29] doc: remove IPC::Run as a dev and test dependency

It's no longer needed and we're able to speed up some of our tests as a result. --- INSTALL | 4 ci/deps.perl | 1 - 2 files changed, 5 deletions(-) diff --git a/INSTALL b/INSTALL index aad52c7b..4d54e6a0 100644 --- a/INSTALL +++ b/INSTALL @@ -162,10 +162,6 @@ Uncommonly needed modules:

[PATCH 17/29] t/mda: switch to run_script for testing

Another noticeable speedup, this test is roughly ~3x faster now. --- t/mda.t | 53 ++--- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/t/mda.t b/t/mda.t index 3e03a25a..89dedd4a 100644 --- a/t/mda.t +++ b/t/mda.t @@ -8,11 +8,7 @@ us

[PATCH 18/29] t/mda_filter_rubylang: drop IPC::Run dependency

This test runs more than twice as fast, now. --- t/mda_filter_rubylang.t | 17 - 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/t/mda_filter_rubylang.t b/t/mda_filter_rubylang.t index f7d872c9..e971b440 100644 --- a/t/mda_filter_rubylang.t +++ b/t/mda_filter_rubylang

[PATCH 21/29] t/convert-compact: convert to run_script

While this didn't use IPC::Run, having to reload several Perl modules and scripts is slow and inefficient, so roughly double the speed of this test. --- t/convert-compact.t | 18 -- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/t/convert-compact.t b/t/convert-compa

[PATCH 25/29] t/watch_filter_rubylang: run_script for -init and -index

This nets us a 20% speedup or so. --- t/watch_filter_rubylang.t | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/t/watch_filter_rubylang.t b/t/watch_filter_rubylang.t index 4b88d670..57ab3b91 100644 --- a/t/watch_filter_rubylang.t +++ b/t/watch_filter_rubylang.t @@ -36,11

[PATCH 29/29] t/common: start_script replaces spawn_listener

We can shave several hundred milliseconds off tests which spawn daemons by preloading and avoiding startup time for common modules which are already loaded in the parent process. This also gives ENV{TAIL} support to all tests which support daemons which log to stdout/stderr. --- t/common.perl

[PATCH 23/29] t/watch_maildir_v2: use run_script for -init

This only gives a small 10% speedup or so, but anything helps. --- t/watch_maildir_v2.t | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/watch_maildir_v2.t b/t/watch_maildir_v2.t index ccc85c17..7fe23521 100644 --- a/t/watch_maildir_v2.t +++ b/t/watch_maildir_v2.t @@ -21,1

[PATCH 26/29] t/v2mda: switch to run_script in many places

This more than doubles the speed of the test. --- t/v2mda.t | 38 ++ 1 file changed, 10 insertions(+), 28 deletions(-) diff --git a/t/v2mda.t b/t/v2mda.t index ebcbd1f4..0cd852b1 100644 --- a/t/v2mda.t +++ b/t/v2mda.t @@ -34,8 +34,6 @@ my $mime = PublicInbox::M

[PATCH 28/29] t/xcpdb-reshard: use run_script for -xcpdb

This more than doubles the speed of the test, since we make many invocations of -xcpdb. --- t/xcpdb-reshard.t | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/t/xcpdb-reshard.t b/t/xcpdb-reshard.t index 43e08639..88e6c3dc 100644 --- a/t/xcpdb-reshard.t +++ b/t/xcpdb-reshard

[PATCH 22/29] t/httpd: use run_script for -init

This only gives a small ~10% speedup, since -httpd still needs execve, but any speedup is welcome. --- t/httpd.t | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/t/httpd.t b/t/httpd.t index 15984a78..e7527ed6 100644 --- a/t/httpd.t +++ b/t/httpd.t @@ -22,7 +22,6 @@ my $gr

Re: message threading in the web UI

Ralph Siemsen wrote: > The way that threads are displayed seems to differ between the "frontpage" > (eg. https://public-inbox.org/meta/) versus how they are shown in the > "Thread Overview" (at the bottom of the page when viewing a specific > thread). > > On the frontpage, it seems only a subset

[PATCH 0/3] start tidying up gzip-related code

Starting with the mbox.gz stuff, first. Gettig rid of Plack::Middleware::Deflater is a long-term goal since we can take advantage of doing gzip during HTML/XML rendering to reduce memory usage. Eric Wong (3): mbox: unused mid_clean import mbox: split mboxgz out into a separate file mboxgz

[PATCH 1/3] mbox: unused mid_clean import

We're gradually phasing mid_clean out (in favor of mids()). --- lib/PublicInbox/Mbox.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/PublicInbox/Mbox.pm b/lib/PublicInbox/Mbox.pm index 67b671f5..9e808c09 100644 --- a/lib/PublicInbox/Mbox.pm +++ b/lib/PublicInbox/Mbox.pm

[PATCH 3/3] mboxgz: use Compress::Raw::Zlib instead of IO::Compress::Gzip

IO::Compress::Gzip is a wrapper around Compress::Raw::Zlib, anyways, and being able to easily detach buffers to return them via ->getline is nice. This results in a 1-2% performance improvement when fetching giant mboxes. --- lib/PublicInbox/Mbox.pm | 2 +- lib/PublicInbox/MboxGz.pm | 41 +

[PATCH 2/3] mbox: split mboxgz out into a separate file

It'll make using Compress::Raw::Zlib easier, since we can use that and import constants more easily. --- MANIFEST | 1 + lib/PublicInbox/Mbox.pm | 64 ++- lib/PublicInbox/MboxGz.pm | 64 +++ 3 files changed

Re: libgmime experiences? (was: [PATCH] doc: update HACKING and TODO ...)

Eric Wong wrote: > I've never used gmime as a user or hacker... I still haven't used it as a user. > However, looking at the Debian changelogs, there's only been one > CVE (CVE-2010-0409) from nearly a decade ago. Maybe all the > bugs are fixed, or nobody really a

Re: [PATCH 29/29] t/common: start_script replaces spawn_listener

Eric Wong wrote: > diff --git a/t/nntpd.t b/t/nntpd.t > index b516ffd1..eb9be9b7 100644 > --- a/t/nntpd.t > +++ b/t/nntpd.t > @@ -339,6 +336,8 @@ Date: Fri, 02 Oct 1993 00:00:00 + > unlike($eout, qr/wide/i, 'no Wide character warnings'); > } > >

Re: [PATCH 29/29] t/common: start_script replaces spawn_listener

Eric Wong wrote: > "$n" (Net::NNTP) and "$td" (TestDaemon) going out of scope at > the same time seems to trigger some strange ->DESTROY > interaction since both classes have a ->DESTROY method. Nope, I was wrong about that :x > This causes t/v2writable

Re: Filter example for ML footers

Konstantin Ryabitsev wrote: > Hello: > > Groups.io adds a super-obnoxious footer to all outgoing messages, and I > would like to be able to filter that out. Example: > > https://lore.kernel.org/keys/2019161300.hc7vb7rcb45gsqmg@chatter.i7.local/ > > The obnoxious footer can be either part o

Re: [PATCH 3/3] mboxgz: use Compress::Raw::Zlib instead of IO::Compress::Gzip

SZEDER Gábor wrote: > I've just stumbled upon an issue that I suspect to be related to this > patch series (or maybe just a strange coincidence...). > > When trying to download a mbox.gz with 'wget' I get a "501 Not > Implemented", e.g.: Thanks, fixed now. It's a bug in the build/install since

[PATCH 01/17] tests: disable daemon workers in a few more places

There were still a few places where we used worker processes unnecessarily in tests, causing a small amount of unnecessary overhead. Followup-to: ad221e9b2852f6c5 ("t/*.t: disable nntpd/httpd worker processes in most tests") --- t/httpd-unix.t | 1 + t/nntpd.t | 2 +- 2 files changed, 2 ins

[PATCH 00/17] test fixes and cleanups

There's some fixes for race conditions around daemon startup and shutdown and resurrects start_script for slightly improved test performance. And slowly eliminating all END{} block usages Eric Wong (17): tests: disable daemon workers in a few more places tests: use strict everywhere

[PATCH 02/17] tests: use strict everywhere

The "strict" pragma makes code easier to debug, and we had undeclared variables as a result in t/watch_maildir_v2.t. So use it everywhere to be consistent with the rest of our code. --- t/mid.t | 1 + t/qspawn.t | 1 + t/watch_maildir.t| 1 + t/watch_maildir_v2.t | 7 +++

[PATCH 13/17] xapcmd: replace Xtmpdirs with File::Temp->newdir

Since we're using Perl 5.10.1 and File::Temp 0.19+, we don't need Xtmpdirs at all for cleaning up tempdirs on failure and can just rely on the DESTROY handler provided by File::Temp. --- lib/PublicInbox/Xapcmd.pm | 73 +++ 1 file changed, 27 insertions(+), 46 de

[PATCH 05/17] t/httpd-corner: wait for worker process death

We need to ensure the worker process is terminated before starting a new connection, so leave a persistent HTTP/1.1 connection open and wait for the SIGKILL to take effect and drop the client. --- t/httpd-corner.psgi | 2 +- t/httpd-corner.t| 19 --- 2 files changed, 13 insert

[PATCH 04/17] t/xcpdb-reshard: test xcpdb --compact

We did not have a test for this, and need to guard against regressions when changing Xapcmd to use File::Temp->newdir in future commits. --- t/xcpdb-reshard.t | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/t/xcpdb-reshard.t b/t/xcpdb-reshard.t index 88e6c3dc..9335843d 10064

[PATCH 14/17] tests: use File::Temp->newdir instead of tempdir()

We'll also introduce a tmpdir() API to give tempdirs consistent names. --- t/admin.t | 4 ++-- t/altid.t | 4 ++-- t/altid_v2.t | 3 +-- t/cgi.t | 3 +-- t/common.perl | 10 ++ t/config.t| 4 ++-

[PATCH 07/17] .gitignore: ignore local prove(1) files

As described in prove(1), .prove is storage for --state=save and .proverc allows per-worktree customizations. --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 167d08bf..9eb97751 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ +/.prove +/.p

[PATCH 12/17] t/nntpd-validate: get rid of threads dependency

Threads are officially discouraged by perl5-porters and proves problematic with my Perl installation when using run_mode=1 to speed up tests. So just use fork() and pipes to share results from Net::NNTP. --- t/nntpd-validate.t | 39 +++ 1 file changed, 23 inser

[PATCH 08/17] daemon: use sigprocmask to block signals at startup

`$SIG{FOO} = "IGNORE"' will cause the daemon to miss signals entirely. Instead, we can use sigprocmask to block signal delivery until we have our signal handlers setup. This closes a race where a PID file can be written for an init script and a signal to be dropped via "IGNORE". --- lib/PublicIn

[PATCH 03/17] t/v1-add-remove-add: quiet down "git init"

Use the "-q" flag like everywhere else. --- t/v1-add-remove-add.t | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/v1-add-remove-add.t b/t/v1-add-remove-add.t index 3facd87e..035fba5c 100644 --- a/t/v1-add-remove-add.t +++ b/t/v1-add-remove-add.t @@ -13,7 +13,7 @@ foreach my $

[PATCH 09/17] daemon: use sigprocmask when respawning workers

We need to block signals in workers during respawns until they're ready to receive signals. --- lib/PublicInbox/Daemon.pm | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/Daemon.pm b/lib/PublicInbox/Daemon.pm index e830a98f..90f11137 100644 --- a/lib/Public

[PATCH 10/17] daemon: avoid race when quitting workers

While the master process has a self-pipe to avoid missing signals, worker processes lack that aside from a pipe to detect master death. That pipe doesn't exist when there's no master process, so it's possible DS::close never finishes because it never woke up from epoll_wait. So create a pipe on t

[PATCH 06/17] t/nntpd-tls: sometimes SSL_connect succeeds quickly

It seems caching can happen within OpenSSL or negotiation can be delayed in some cases. In any case, don't barf on PublicInbox::TLS::epollbit() when connect_SSL succeeds unexpectedly. --- t/nntpd-tls.t | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/t/nntpd-tls.t b

[PATCH 15/17] tests: quiet down commit graph

Newer versions of git enable the commit graph by default. Since we blow away our temporary directories every test, generating graphis is a waste and clutters stderr with "Computing commit graph generation numbers" messages. --- MANIFEST | 1 + t/.gitconfig | 4 t/purge.t | 1 + t

[PATCH 16/17] t/perf-*.t: use $ENV{GIANT_INBOX_DIR} consistently

It's more consistent with our current terminology and "PI_DIR" is already used to override ~/.public-inbox/ (which holds "config" and possibly other files which affect all inboxes for a particular user, but is not an inbox itself); so stop advertising GIANT_PI_DIR in skip messages. --- t/perf-msgv

[PATCH 17/17] tests: move giant inbox/git dependent tests to xt/

xt/ is typically reserved for "eXtended tests" intended for the maintainers and not ordinary users. Since these require special configuration and do nothing by waste cycles during startup, they qualify. --- MANIFEST | 10 +- {t => xt}/git-http-backend.t | 0 {t => xt}

[PATCH 11/17] t/common: start_script replaces spawn_listener

We can shave several hundred milliseconds off tests which spawn daemons by preloading and avoiding startup time for common modules which are already loaded in the parent process. This also gives ENV{TAIL} support to all tests which support daemons which log to stdout/stderr. --- t/common.perl

[PATCH] check for File::Temp 0.19 for ->newdir method

This is distributed with Perl 5.10.1 and onwards, so it should not be an installation burden for any users. I'm planning to move away from tempdir() entirely and use File::Temp->newdir to remove dependencies on END{} blocks. --- Makefile.PL | 2 +- lib/PublicInbox/SolverGit.pm

[PATCH] mboxgz: fix compiler parse error under under Perl 5.16.3

Perl 5.16.3 (and possibly older versions) fails with the following errors (from CentOS7): Use of ?PATTERN? without explicit operator is deprecated Search pattern not terminated --- lib/PublicInbox/MboxGz.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/PublicInbox/Mb

[PATCH 1/3] t/msgtime: show date in test descriptions

Otherwise it's hard to figure what fails. --- t/msgtime.t | 21 + 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/t/msgtime.t b/t/msgtime.t index 6b396602..7bf73185 100644 --- a/t/msgtime.t +++ b/t/msgtime.t @@ -58,7 +58,7 @@ for (my $min = -1440; $min <= 1440; $

[PATCH 0/3] msgtime: tests and wonky offsets

I've been working on making Date::Parse optional, but found some existing bugs and improvements to test cases along the way. Eric Wong (3): t/msgtime: show date in test descriptions t/msgtime: add more checks for known cases msgtime: deal with strange minutes in TZ offsets lib/Public

[PATCH 3/3] msgtime: deal with strange minutes in TZ offsets

I'm not sure if TZ minute offsets aside from '00' or '30' exist, but lets just deal with them properly when negative. Examples taken from various inboxes on lore.kernel.org. These are mostly message from spammers, but some are legitimate messages. --- lib/PublicInbox/MsgTime.pm | 4 t/msgti

[PATCH 2/3] t/msgtime: add more checks for known cases

Broken email clients sent the darndest things, make sure we can still support them when we make Date::Parse optional. --- t/msgtime.t | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/t/msgtime.t b/t/msgtime.t index 7bf73185..f969fa25 100644 --- a/t/msgtime.t +++ b/t/msgtime.

Re: [PATCH 10/17] daemon: avoid race when quitting workers

Eric Wong wrote: > While the master process has a self-pipe to avoid missing > signals, worker processes lack that aside from a pipe to > detect master death. > > That pipe doesn't exist when there's no master process, > so it's possible DS::close never finishe

[PATCH 0/2] fix kqueue support and missed signal wakeups

don't think it matters in practice; since we already account for coalescing in handling SIGCHLD. Eric Wong (2): dskqxs: fix missing EV_DISPATCH define httpd|nntpd: avoid missed signal wakeups MANIFEST | 3 + lib/PublicInbox/DS.pm | 6 +- lib/PublicInbox/DSKQXS

[PATCH 1/2] dskqxs: fix missing EV_DISPATCH define

Oops, IO::KQueue support was broken due to this missing constant. Add a new ds-kqxs.t test case to ensure we test the IO::KQueue path if IO::KQueue is available. --- MANIFEST | 1 + lib/PublicInbox/DSKQXS.pm | 2 ++ t/ds-kqxs.t | 14 ++ t/ds-poll.t

[PATCH 2/2] httpd|nntpd: avoid missed signal wakeups

Our attempt at using a self-pipe in signal handlers was ineffective, since pure Perl code execution is deferred and Perl doesn't use an internal self-pipe/eventfd. In retrospect, I actually prefer the simplicity of Perl in this regard... We can use sigprocmask() from Perl, so we can introduce sig

[PATCH] Makefile.PL: MANIFEST dependency fix

We need to force an update to Makefile (not Makefile.PL) when MANIFEST changes. Since "Makefile" (aka. "$(FIRST_MAKEFILE)") is already a single-colon make target; we can't create a double-colon rule to augment it. So we'll continue using a "Makefile.PL" rule, but have it recreate the resulting Ma

[PATCH] t/httpd-unix: FreeBSD expects to fail with EADDRINUSE

Tested FreeBSD 11.2. I'm starting to think I'm too conservative with this check and it could be safely expanded to cover any OS with UNIX sockets. --- t/httpd-unix.t | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/httpd-unix.t b/t/httpd-unix.t index f7881cfa..ceec127c 100644

[PATCH 1/2] t/replace: quiet "git fsck" invocation

Test output can be a terminal if running as "perl -I lib t/$FOO.t", and showing fsck progress is pointless for tests. --- t/replace.t | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/t/replace.t b/t/replace.t index e9361856..57290f96 100644 --- a/t/replace.t +++ b/t/replace

[PATCH 2/2] replace: quiet "git gc" invocation

Since we give users no indication or control of how "git gc" runs, showing its progress is confusing. --- lib/PublicInbox/Import.pm | 2 +- lib/PublicInbox/V2Writable.pm | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm

[PATCH 0/2] replace: quiet down git commands

git fsck and gc are noisier when not run under "prove", as git will decide to emit progress depending on whether or not stderr is a TTY (prove makes stderr a pipe). Eric Wong (2): t/replace: quiet "git fsck" invocation replace: quiet "git gc" invocation lib

[PATCH 5/7] tests: don't repeatly validate NEWS.atom

We can create a stamp to avoid rerunning the check unless NEWS.atom changes (and it will, soon, I hope :>). --- .gitignore | 1 + Documentation/include.mk | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 9eb97751..bdb8cf15 100644

[PATCH 7/7] import: (cleanup) drop redundant env arg to run_die

run_die() doesn't require an $env arg, so there's no point passing "undef" to it. --- lib/PublicInbox/Import.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm index 8a369ee4..46de09c4 100644 --- a/lib/PublicInbox/Import

[PATCH 2/7] t/common: set $0 when running script w/o fork

We can localize changes to $0 so $0 is restored when the "script" sub is done. This will be helpful when we encounter a stuck/slow processes during our tests (hopefully never!) --- t/common.perl | 1 + 1 file changed, 1 insertion(+) diff --git a/t/common.perl b/t/common.perl index 0ff5de4a..288a

[PATCH 4/7] TODO: update and add a few more items

SpamAssassin has used re2c (via sa-compile) for many years, now, and it seems to work fine, there. GMime also looks promising when combined with Inline::C since GMime can operate on mmap-ed regions. Given the inevitable demise of many .orgs when price rise; supporting a URL rewriter similar to .m

[PATCH 1/7] t: localize the PI_CONFIG env

We don't want the user's ~/.public-inbox/config to be read from during tests. I only noticed this because I had a non-existent pathname for one of my inboxes :x I've also verified this change by running "inotifywait ~/.public-inbox/config -m" in another terminal while running "make check"; (perha

[PATCH 0/7] some low impact changes and cleanups

Some other changes coming in the next few days, I hope; but the straightforward stuff, first... Eric Wong (7): t: localize the PI_CONFIG env t/common: set $0 when running script w/o fork ds: ->Reset initializes $nextq TODO: update and add a few more items tests: don't repeatly

[PATCH 6/7] spawn: remove support for clearing the env

It's unnecessary code which I'm not sure we ever used. In retrospect, completely clearing the environment doesn't make sense for the processes we spawn. We don't need to clobber individual environment variables in our code, either (and if we did for tests, we can use 'local'). --- lib/PublicInbo

[PATCH 3/7] ds: ->Reset initializes $nextq

I haven't noticed this being a problem in practice, but be consistent with the rest of the singleton stuff. Since we always call Reset() at load time, only do initialization in that sub and not at declaration. --- lib/PublicInbox/DS.pm | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) di

[PATCH 1/2] Makefile.PL: avoid non-portable $

OpenBSD make(1) complains, but FreeBSD make(1) is fine. --- Makefile.PL | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.PL b/Makefile.PL index 4aa0caa7..4d83a198 100644 --- a/Makefile.PL +++ b/Makefile.PL @@ -94,7 +94,7 @@ lib/PublicInbox/UserContent.pm :: contrib/css/

[PATCH 2/2] doc: don't attempt to install non-existent manpages

More effort needs to be made to get the full doc build portable across different "make" implementations. I assume BSD users don't want GNU make installed, at all... --- Documentation/include.mk | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Documentation/include.mk b/

[PATCH 0/2] minor fixes for OpenBSD make(1)

GNU make is one of my favorite tools (or even languages), but "make" is not always the GNU implementation... So, throw in some minor fixes before I write more Perl5 to write portable make... Eric Wong (2): Makefile.PL: avoid non-portable $< doc: don't attempt to install non

[PATCH 0/4] drop Date::Parse dependency

caes when checks are done running... Eric Wong (4): git: async batch interface add msgtime_cmp maintainer test msgtime: drop Date::Parse for RFC2822 Date::Parse is now optional INSTALL| 9 ++- MANIFEST | 2 + Makefile.PL

[PATCH 4/4] Date::Parse is now optional

-mda should not be dealing with broken Date: headers nowadays, and deprioritize it in our documentation and internal checks. --- INSTALL | 9 + Makefile.PL | 1 - TODO | 4 ci/deps.perl | 2 +- lib/PublicInbox/Admin.pm | 2

[PATCH 2/4] add msgtime_cmp maintainer test

Changes will be coming for MsgTime to stop depending on Date::Parse due to lack of package availability on OpenBSD and suboptimal performance on RFC822 dates. --- MANIFEST | 1 + xt/msgtime_cmp.t | 161 +++ 2 files changed, 162 insertions(+) c

[PATCH 3/4] msgtime: drop Date::Parse for RFC2822

Date::Parse is not optimized for RFC2822 dates and isn't packaged on OpenBSD. It's still useful for historical email when email clients were less conformant, but is less relevant for new emails. --- lib/PublicInbox/MsgTime.pm | 118 - t/msgtime.t

[PATCH 1/4] git: async batch interface

This is a transitionary interface which does NOT require an event loop. It can be plugged into in current synchronous code without major surgery. It allows HTTP/1.1 pipelining-like functionality by taking advantage of predictable and well-specified POSIX pipe semantics by stuffing multiple git ca

[PATCH] build: support doc generation w/o GNU make

Eric Wong wrote: > GNU make is one of my favorite tools (or even languages), > but "make" is not always the GNU implementation... So, > throw in some minor fixes before I write more Perl5 to > write portable make... Actually, I think replacing the previous 2 patch seri

[PATCH 5/4] msgtime: avoid obviously out-of-range dates (for now)

Wacky dates show up in lore for valid messages. Lets ignore them and let future generations deal with Y10K and time-travel problems. --- lib/PublicInbox/MsgTime.pm | 6 +- t/msgtime.t| 14 -- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/Publ

Re: [PATCH 5/4] msgtime: avoid obviously out-of-range dates (for now)

Eric Wong wrote: > diff --git a/t/msgtime.t b/t/msgtime.t > index 1452dc97d5b0b..cecad775769e1 100644 > --- a/t/msgtime.t > +++ b/t/msgtime.t > @@ -17,7 +17,11 @@ sub datestamp ($) { > Subject => 'this is a subject', >

[PATCH] mbox: do not try to create filename from empty string

This was causing warnings to pop up in syslogs for messages with empty Subject headers. --- lib/PublicInbox/Mbox.pm | 2 +- lib/PublicInbox/MboxGz.pm | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/Mbox.pm b/lib/PublicInbox/Mbox.pm index 42cedd15..c50a118b

[PATCH 2/1] Makefile.PL: fix "dsyn" target

The "dsyn" target needs to remain working, despite still being dependent on GNU-isms at the moment. Fixes: 73fe3421f1ecbdc8 ("build: support doc generation w/o GNU make") --- Makefile.PL | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile.PL b/Makefile.PL index 5faced63..0daa6107 100644

[PATCH 0/3] there is no END{}

END{} blocks make code harder to test and reuse in long-lived processes. Get rid of them in preparation for allowing daemons to be restarted in tests without new processes. This may make our code more multiplicity-friendly in the future :> Eric Wong (3): ds: move EvCleanup code into DS

[PATCH 3/3] daemon: use DESTROY for unlinking --pid-file

This gets rid of the last "END{}" block in our code and cleans up a (temporary) circular reference. Furthermore, ensure the cleanup code still works in all configurations by adding tests and testing both the -W1 (default, 1 worker) and -W0 (no workers) code paths. --- lib/PublicInbox/Daemon.pm |

[PATCH 1/3] ds: move EvCleanup code into DS

EvCleanup only existed since Danga::Socket was a separate component, and cleanup code belongs with the event loop. --- MANIFEST | 1 - lib/PublicInbox/DS.pm| 27 ++- lib/PublicInbox/Daemon.pm| 2 -- lib/PublicInbox/EvCleanup.pm | 30 ---

[PATCH 2/3] ds: move NNTP-only expiration code into DS

We'll be supporting idle timeout for the HTTP code in the future to deal directly with Internet-exposed clients w/o Varnish or nginx. --- lib/PublicInbox/DS.pm | 41 +-- lib/PublicInbox/NNTP.pm | 48 + 2 files changed, 44 in

[PATCH 2/2] address: use Email::Address::XS if available

Email::Address::XS is a dependency of modern versions of Email::MIME, so it's likely loaded and installed on newer systems, already; and capable of handling more corner-cases than our pure-Perl fallback. We still fallback to the imperfect-but-good-enough-in-practice pure-Perl code while avoiding t

[PATCH 0/2] use Email::Address::XS if available

Email::Address::XS is loaded by default on modern Email::MIME (which we depend on), so we might as well use it. 1/2 also improves the pure-Perl fallback to match optimal ::XS behavior. Eric Wong (2): address: use comment as name if no phrase available address: use Email::Address::XS if

[PATCH 1/2] address: use comment as name if no phrase available

Some users will set their From: headers in the form of: " (A U Thor)", where their name is in the parenthesized comment. Use that instead of the email address, if available. --- lib/PublicInbox/Address.pm | 30 -- t/address.t| 7 --- 2 files change

[PATCH 3/2] address: explicitly reject local-only addresses

Apparently, neither our previous address parsing code nor Email::Address::XS recognizes local, username-only addresses in the form of (without "@host"). Without this change, Email::Address::XS->address would return "undef", so we need to filter it out via "grep { defined }" It seems the cases wh

[PATCH] Makefile.PL: allow overriding "prove" from make CLI

Development versions of Perl install "prove$VERSION" where $VERSION is something like "5.31.7". This makes it easier to test everything we have against development versions of Perl5. Note: I could not find a way to get quoting right to use the "--exec $(PERL)" option of prove(1), but that would b

[PATCH] searchidx: do not modify read-only $1 via git_unquote

git_unquote works in-place, and we sometimes see strange filenames, or badly munged diffs with terminal escape characters (for colorization) end up in emails. --- lib/PublicInbox/SearchIdx.pm | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/SearchIdx.pm b/l

[PATCH] inbox: fix periodic git process cleanup

We need to use $PublicInbox::DS::in_loop instead of ::running(). The latter is not valid for systems with signalfd or kqueue and is now gone, completely. Not needing periodic cleanups at all to deal with unlinked pack indices will be a tougher task... --- lib/PublicInbox/DS.pm| 2 -- lib/Publ

[PATCH 2/2] daemon: drop listeners early in master on graceful shutdown

For users not relying on socket activation via systemd (or similar), we want to drop listeners ASAP so another process can bind to their address. While we're at it, disable TTIN and HUP handlers since we have no chance of starting usable workers without listeners. --- lib/PublicInbox/Daemon.pm |

[PATCH 0/2] daemon: kill listener early at shutdown

Not everybody uses systemd (or similar) which allow socket activation. So we now get rid of listeners ASAP during graceful shutdown to ensure replacement processes can bind to the same listen socket w/o requiring SO_REUSEPORT. Eric Wong (2): daemon: shorten lifetime of listener_names mapping

[PATCH 1/2] daemon: shorten lifetime of listener_names mapping

Keeping a ref to the IO::Socket handle was preventing close(2) from being invoked on graceful shutdown of worker. --- lib/PublicInbox/Daemon.pm | 15 --- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/PublicInbox/Daemon.pm b/lib/PublicInbox/Daemon.pm index dd9e7780..

[PATCH] t/edit.t: drop redundant "delete local $ENV{...}"

"delete local" is only in Perl v5.11.0, and we only depend on Perl v5.10.1. We already localize and delete it as two separate statements immediately above. I wish this was hidden behind a "use feature" flag like other new-fangled things :<. Oh well, I think the oldest Perl actually in use for th

[PATCH 0/6] test updates and speedups

up the "solver" tests to cover the ViewVCS PSGI frontend, more work there on the horizon... Eric Wong (6): TODO: add UUCP address item viewvcs: flesh out some functionality and test Makefile.PL: sort target and var lists t/*.t: avoid sharing "my" variables in subs t

[PATCH 5/6] tests: move t/common.perl to PublicInbox::TestCommon

We want to be able to use run_script with *.t files, so t/common.perl putting subs into the top-level "main" namespace won't work. Instead, make it a module which uses Exporter like other libraries. --- MANIFEST | 2 +- t/common.perl => lib/PublicInbox/TestC

[PATCH 6/6] t/run.perl: to avoid repeated process spawning for *.t

Spawning a new Perl interpreter for every test case means Perl has to reparse and recompile every single file it needs, costing us performance and development time. Now that we've modified our code to avoid global state, we can preload everything we need. The new "check-run" test target is now 20

[PATCH 1/6] TODO: add UUCP address item

We should support historical archives from the old days, but I'm not sure how to best go about it, for now, given how tricky correct handling of modern email addresses is. We can deal with it if/when somebody decides to import some ancient archives... --- TODO | 2 ++ 1 file changed, 2 insertions(

[PATCH 2/6] viewvcs: flesh out some functionality and test

Expose MAX_SIZE via "our" will make it possible to use in tests, and configure, later. Additionally, returning HTTP 500 code for big files is not an Internal Server Error, just a memory limit... Some browsers won't show our HTML response with the link to the raw file in case of errors, either, so

[PATCH 3/6] Makefile.PL: sort target and var lists

Sorting makes it easier to review the generated result. --- Makefile.PL | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile.PL b/Makefile.PL index 3020f25a..33688095 100644 --- a/Makefile.PL +++ b/Makefile.PL @@ -83,13 +83,13 @@ $v->{rsync_xdocs} = [ @{$v->{gz_xdoc

[PATCH 4/6] t/*.t: avoid sharing "my" variables in subs

These usages of file-local global variables make the *.t files incompatible with run_script(). Instead, use anonymous subs, "our", or pass the parameter as appropriate. --- t/httpd-corner.t | 8 t/indexlevels-mirror.t | 10 - t/mda.t| 46 +++

Re: [PATCH 3/6] Makefile.PL: sort target and var lists

Eric Wong wrote: > Sorting makes it easier to review the generated result. > --- a/Makefile.PL > +++ b/Makefile.PL > @@ -83,13 +83,13 @@ $v->{rsync_xdocs} = [ @{$v->{gz_xdocs}}, > @{$v->{xdocs_html}}, @{$v->{xdocs}} ]; > my $TGTS = join("\n", map {;

[PATCH] msgiter: msg_part_text returns undef on text/html

We want HTML parts to be downloadable, but not displayed as unreadable (but injection-safe) HTML source in our own web and Atom interfaces. This affects indexing, too, as HTML tags/comments won't be indexed anymore, but existing indices are only cleaned after --reindex. HTML-only mail won't be in

<    1   2   3   4   5   6   7   8   9   10   >