On Wed, 2007-02-07 at 03:34 +0100, Martin Furter wrote: > > On Tue, 6 Feb 2007, Aaron Stone wrote: > > > > > Hey folks, I've updated the multifoo architecture page on the wiki to > > reflect some of my latest work on the subject. At this point I'm starting > > to build the framework for the threaded server core. I think I've > > eliminated the need for a thread pool library and for any super > > complicated back and forth message passing between threads. > > > > http://www.dbmail.org/dokuwiki/doku.php?id=multifoo_architecture > > I don't know how dbmail stores mail now but reading that wiki page... > > Well, the handler thread has been given a complete command. It no longer > needs to read anything from the socket. > > So that means you store the whole email in RAM until it's put into the > database? Wouldn't it be better to stream the data from the socket > directly to the DB ? (assuming it supports that, I know Oracle OCI does, > but I don't know about PostgreSQL and MySQL).
We already store the whole message in RAM in 2.2, to some extent GMime requires this, and if we change to messageblk-per-mime-part, then we need everything in memory to be able to split it up. > The main thread would just read the first line of a command, clear the > read bit in the poll struct and feed that line to a worker thread (or into > a queue if no worker is free). > The worker reads the rest and processes it. When it's done it tells the > main thread to set the read bit again. (that might need some tricks) > That way each socket can have only one worker thread so there shouldn't be > any race conditions. I want to protect against slow clients that might send most of a command, but not the whole command. If we pass off the whole client connection to the worker, it might sit there waiting on more data. Instead I want to pass full commands so that the worker can go right at it, no waiting on further communication. Did my well-ordered command paragraph on the wiki make sense? I might have to rewrite it. I'm trying to get at the issue that some commands can be executed in parallel while others cannot. There would have to be a flag indicating which state the client is in, and a flag indicating if a command is already being processed by a worker thread or not. Aaron _______________________________________________ Dbmail-dev mailing list [email protected] http://twister.fastxs.net/mailman/listinfo/dbmail-dev
