Aaron Stone wrote:
On Thu, Feb 10, 2005, ""mc"" <[EMAIL PROTECTED]> said:
Recently I am planning to move from vpopmail to dbmail.
After I did a quick scan on dbmail's code, I've found that whenever a MTA
sends a mail to dbmail, dbmail would try to read everything into
memory...and then it would allocate another piece of memory for the INSERT
query. Did I miss something or was it meant to be like that? It seems to me
that this approach could potentially lead to massive memory consumption,
especially on systems where large size attachments are common. I still
haven't got the time to read the code in detail though. I apologize if I am
talking nonsense. :)
This is how I orignally wrote the message receiving code for DBMAil 2.0,
and you can see it in the first couple of "rc" releases (which were all
horribly broken)... in order to get 2.0 out the door, Ilja rewrote it.
The problem was that the MIME parser didn't know what to do with messages
that contained \n (unix line endings) or \r\n ('network' line endings) or
both. So it was exploding on all of the mail received through LMTP through
my code.
And of course I rewrote it again for 2.1 to get rid of the internal mime-parser by using gmime. Still, reading
the messages into memory as a whole is at present the only workable solution because that's what mime parsing
requires. Assuming there are no leaks, memory consumption will be fairly short lived.
Your idea to insert in blocks will likely create some problems:
>> insert msg set id=xxx, body="first 512 bytes of msg goes here...";
>> update msg set body=body+"another 512 bytes..." where id=xxx;
>> update msg set body=body+"yet another 512bytes" where id=xxx;
this will create massive amounts of insert queries if you're indeed receiving a lot of large attachments. This
will either pound your disk IO during insertions, or if your using transactions, you're simply shifting the
memory consumption to the database server.
Also, your syntax wont do. Though 'set body=concat(body,"more bodytext")' would, I'm not sure that's
compatible sql syntax.
In sum: what's your use case? Have you tested and ran into trouble? If not, I'd say let's not get into any
premature optimizations at this point.
--
________________________________________________________________
Paul Stevens mailto:[EMAIL PROTECTED]
NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED]
The Netherlands________________________________http://www.nfg.nl
