Eric Wong <e...@80x24.org> wrote:
> Without concurrent connections; I can't see that happening
> unless there's a single message which is gigabytes in size. I'm
> already irked that Email::MIME requires slurping entire emails
> into memory; but it should not be using more than one
> Email::MIME object in memory-at-a-time for a single client.
Giant multipart messages do a lot of damage. Maybe concurrent
clients hitting the same endpoints will do more damage.
Largest I see in LKML is 7204747 bytes (which is frightening).
That bloats to 21626795 bytes when parsed by Email::MIME.
I thought it was bad enough that all Perl mail modules seem to
require slurping 7M into memory...
-------8<--------
use strict; use warnings;
require Email::MIME;
use bytes ();
use Devel::Size qw(total_size);
my $in = do { local $/; <STDIN> };
print 'string: ', total_size($in), ' actual: ', bytes::length($in), "\n";
print 'MIME: ', total_size(Email::MIME->new(\$in)), "\n";
-------8<--------
That shows (on amd64):
string: 7204819 actual: 7204747
MIME: 21626795
Maybe you have bigger messages outside of LKML.
This prints all objects >1MB in a git dir:
git cat-file --buffer --batch-check --batch-all-objects \
--unordered | awk '$3 > 1048576 { print }'
And I also remember you're supporting non-vger lists where HTML
mail is allowed, so that can't be good for memory use at all :<
Streaming MIME handling has been on the TODO for a while,
at least...
|* streaming Email::MIME replacement: currently we generate many
| allocations/strings for headers we never look at and slurp
| entire message bodies into memory.
| (this is pie-in-the-sky territory...)
--
unsubscribe: meta+unsubscr...@public-inbox.org
archive: https://public-inbox.org/meta/
