Re: [Dbmail-dev] Multifoo architecture update
On Tue, 6 Feb 2007, Aaron Stone wrote: Hey folks, I've updated the multifoo architecture page on the wiki to reflect some of my latest work on the subject. At this point I'm starting to build the framework for the threaded server core. I think I've eliminated the need for a thread pool library and for any super complicated back and forth message passing between threads. http://www.dbmail.org/dokuwiki/doku.php?id=multifoo_architecture I don't know how dbmail stores mail now but reading that wiki page... Well, the handler thread has been given a complete command. It no longer needs to read anything from the socket. So that means you store the whole email in RAM until it's put into the database? Wouldn't it be better to stream the data from the socket directly to the DB ? (assuming it supports that, I know Oracle OCI does, but I don't know about PostgreSQL and MySQL). The main thread would just read the first line of a command, clear the read bit in the poll struct and feed that line to a worker thread (or into a queue if no worker is free). The worker reads the rest and processes it. When it's done it tells the main thread to set the read bit again. (that might need some tricks) That way each socket can have only one worker thread so there shouldn't be any race conditions. Just a random idea... Martin ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Multifoo architecture update
On Tue, Feb 6, 2007, Leif Jackson <[EMAIL PROTECTED]> said: > On Tue, February 6, 2007 3:35 pm, Aaron Stone wrote: >> On Tue, Feb 6, 2007, Aaron Stone <[EMAIL PROTECTED]> said: >>> eliminated the need for a thread pool library and for any super >> >> Sorry, I *do* need a thread pool library -- I'm going to use Glib's. I >> won't need a database pool library; each thread will own its own database >> handle directly. > > If you do use a thread pool then the threads in the pool will have to know > how to handle both pop and imap, as GLib's thread pool is a little > limited. I have had some issue with it as well in how it spreads work over > the pool and some other things, my advise test a LOT before you commit to > glib's thread pool. Pthread portablity might be the only issue to writing > your own but they are not hard to do. We can do a multiprotocol server, or continue to have several binaries that each have code for a single service. I don't feel strongly either way. If performance of Glib's thread pool is below par, we can push them to make it faster. I'd prefer to continue to leverage other people's API's wherever possible. Any experience you can share will be greatly appreciated! Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Multifoo architecture update
On Tue, Feb 6, 2007, Leif Jackson <[EMAIL PROTECTED]> said: > Aaron, > > From your wiki page: > >>On the issue of a database connection pool, I think that issue is mooted > by >this architecture. There are relatively few commands that do not > require >any database interaction, and the ones that dont are so fast to > process as >to not warrant the additional abstraction so that they wont > tie up a >database handle for their 0.05ms of processing time. > > Sorry not familar with editing the wiki so I will respond here. That fine, I want to discuss things on the mailing list and post the general consensus ideas on the wiki. Wikis kinda suck for active discussion. > The need > for thread pools is limited but the database pool is a good thing to have > due to the fact that you will end up making a new connection when you get > a new pop3/imap connect, sql connections take some time vs requesting the No new threads get spun up on connections. That the whole point of this architecture :-) A single thread manages all incoming socket events and only when the events result in a complete command is that command placed into the work queue. > data in the response and so if you have a pool of open database > connections waiting there you remove that connect lag, which gets large > when MySQL gets busy. The thread of execution then just returns the > connection back to the pool when the request is completed and thus you > elemenate the churn of opening and closing DB connections. I'll eliminate the churn by having as long-lived a worker thread pool as possible. Some sizing up and down will take place, as it will in any system, but no churn. Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Multifoo architecture update
On Tue, February 6, 2007 3:35 pm, Aaron Stone wrote: > On Tue, Feb 6, 2007, Aaron Stone <[EMAIL PROTECTED]> said: > > > [snip] > >> eliminated the need for a thread pool library and for any super > > Sorry, I *do* need a thread pool library -- I'm going to use Glib's. I > won't need a database pool library; each thread will own its own database > handle directly. If you do use a thread pool then the threads in the pool will have to know how to handle both pop and imap, as GLib's thread pool is a little limited. I have had some issue with it as well in how it spreads work over the pool and some other things, my advise test a LOT before you commit to glib's thread pool. Pthread portablity might be the only issue to writing your own but they are not hard to do. -leif > > Aaron > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev > > ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Multifoo architecture update
Aaron, From your wiki page: >On the issue of a database connection pool, I think that issue is mooted by >this architecture. There are relatively few commands that do not require >any database interaction, and the ones that dont are so fast to process as >to not warrant the additional abstraction so that they wont tie up a >database handle for their 0.05ms of processing time. Sorry not familar with editing the wiki so I will respond here. The need for thread pools is limited but the database pool is a good thing to have due to the fact that you will end up making a new connection when you get a new pop3/imap connect, sql connections take some time vs requesting the data in the response and so if you have a pool of open database connections waiting there you remove that connect lag, which gets large when MySQL gets busy. The thread of execution then just returns the connection back to the pool when the request is completed and thus you elemenate the churn of opening and closing DB connections. just my .02 -Leif On Tue, February 6, 2007 3:28 pm, Aaron Stone wrote: > > Hey folks, I've updated the multifoo architecture page on the wiki to > reflect some of my latest work on the subject. At this point I'm starting > to build the framework for the threaded server core. I think I've > eliminated the need for a thread pool library and for any super > complicated back and forth message passing between threads. > > http://www.dbmail.org/dokuwiki/doku.php?id=multifoo_architecture > > > Aaron > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev > > ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Multifoo architecture update
On Tue, Feb 6, 2007, Aaron Stone <[EMAIL PROTECTED]> said: [snip] > eliminated the need for a thread pool library and for any super Sorry, I *do* need a thread pool library -- I'm going to use Glib's. I won't need a database pool library; each thread will own its own database handle directly. Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] Multifoo architecture update
Hey folks, I've updated the multifoo architecture page on the wiki to reflect some of my latest work on the subject. At this point I'm starting to build the framework for the threaded server core. I think I've eliminated the need for a thread pool library and for any super complicated back and forth message passing between threads. http://www.dbmail.org/dokuwiki/doku.php?id=multifoo_architecture Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] [DBMail 0000486]: Many duplicate key errors from MySQL running "dbmail-util -by"
A NOTE has been added to this issue. == http://www.dbmail.org/mantis/view.php?id=486 == Reported By:zak Assigned To: == Project:DBMail Issue ID: 486 Category: Database layer Reproducibility:always Severity: minor Priority: normal Status: new target: == Date Submitted: 07-Jan-07 21:07 CET Last Modified: 06-Feb-07 12:45 CET == Summary:Many duplicate key errors from MySQL running "dbmail-util -by" Description: I have just attempted an upgrade from dbmail 2.0.11 to 2.2.1 on a fairly low-volume system, and (after running the migrate_from_2.0_to_2.2.mysql.gz script) I now get the lots of the following errors every time I run "dbmail-util -by": ... Error insert datefield failed [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (1976,'1970-01-01 00:00:00')] .Error [Duplicate entry '1975-1970-01-01 00:00:00' for key 2] [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (1975,'1970-01-01 00:00:00')] Error insert datefield failed [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (1975,'1970-01-01 00:00:00')] .Error [Duplicate entry '1974-1970-01-01 00:00:00' for key 2] [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (1974,'1970-01-01 00:00:00')] Error insert datefield failed [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (1974,'1970-01-01 00:00:00')] .Error [Duplicate entry '1973-1970-01-01 00:00:00' for key 2] [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (1973,'1970-01-01 00:00:00')] ... All the dates are '1970-01-01 00:00:00'. However, the IMAP daemon has started and appears to be functioning fine. I'm using MySQL 4.1.21. I'm not sure what additional information might be useful in tracking this down, but I'm happy to provide any that is likely to help. == -- aaron - 19-Jan-07 02:50 -- Dump out a couple of messages and their respective messageblks and attach them to the bug. The first easiest thing will be to re-insert your messageblks into another database and see if they still act up. Also, what version of GMime are you using? -- Anne - 19-Jan-07 16:49 -- I have the same problem. My errors also include the dbmail_subjectfield table. I get these errors sometimes when doing a daily dbmail-util -ay Ok. Found [2] un-cached physmessages. Error [Duplicate entry '8127044-2007-01-16 09:13:32' for key 2] [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (8127044,'2007-01-16 09:13:32')] Error insert datefield failed [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (8127044,'2007-01-16 09:13:32')] Error [Duplicate entry '8127044-Lang zullen ze lezen!' for key 2] [INSERT INTO dbmail_subjectfield (physmessage_id, subjectfield) VALUES (8127044,'Lang zullen ze lezen!')] Error insert subjectfield failed [INSERT INTO dbmail_subjectfield (physmessage_id, subjectfield) VALUES (8127044,'Lang zullen ze lezen!')] .Error [Duplicate entry '8127043-2007-01-16 09:13:32' for key 2] [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (8127043,'2007-01-16 09:13:32')] Error insert datefield failed [INSERT INTO dbmail_datefield (physmessage_id, datefield) VALUES (8127043,'2007-01-16 09:13:32')] Error [Duplicate entry '8127043-Lang zullen ze lezen!' for key 2] [INSERT INTO dbmail_subjectfield (physmessage_id, subjectfield) VALUES (8127043,'Lang zullen ze lezen!')] Error insert subjectfield failed [INSERT INTO dbmail_subjectfield (physmessage_id, subjectfield) VALUES (8127043,'Lang zullen ze lezen!')] . -- aaron - 06-Feb-07 01:08 -- Has this been resolved by Paul's recent patch? -- pgilmore-ellis - 06-Feb-07 12:45 -- Sorry filed a dupe bug for this with id 503. Nube! We're using MySQL 5.0.18 and GMime 2.2.3 and dbmail-2.2.2. Tried to patch with PostgreSQL patch but to no awail. The error log is attached t
[Dbmail-dev] [DBMail 0000503]: Error running dbmail-util -by after upgrade from 2.0.7 to 2.2.2
The following issue has been SUBMITTED. == http://www.dbmail.org/mantis/view.php?id=503 == Reported By:pgilmore-ellis Assigned To: == Project:DBMail Issue ID: 503 Category: Command-Line programs (dbmail-users, dbmail-util) Reproducibility:always Severity: minor Priority: normal Status: new target: == Date Submitted: 06-Feb-07 12:38 CET Last Modified: 06-Feb-07 12:38 CET == Summary:Error running dbmail-util -by after upgrade from 2.0.7 to 2.2.2 Description: This bug was discussed and solved on the mailing list yesterday but for PostgreSQL, but it seems to effect MySQL as well. We use mysql 5.0.18 on a box standard Slackware machine. Attached you will find the error messages found in the log when log level i set to 5. == Issue History Date Modified Username FieldChange == 06-Feb-07 12:38 pgilmore-ellis New Issue 06-Feb-07 12:38 pgilmore-ellis File Added: dbmail_log.txt == ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
