Re: [Dbmail-dev] refactoring plans for 2.1
Paul J Stevens <[EMAIL PROTECTED]> said: > I could some threaded model. I've been thinking about header caching: We > implement a cache table with fields for every header you wish to cache. > If a sysadmin adds a column for some funky X-Field he wants to cache, the > insertion code immediatly starts using the extra header field, while the > retrieval code degrades gracefully if a message is not yet in the cache. > It would be nice to be able to launch a slow priority thread that > gradually starts to analyze all older messages without interfering with > the imap interaction. It would be interesting to have a "side process" running that had various threads for on-the-fly maintenance and update tasks... but I'd rather stick to using threading as a more efficient way of handling connections socket and database (especially for Oracle, where one process with ten threads handling ten users needs only one OCI connection and saves considerable database overhead). We could figure out that "side process" might look like, but I don't think it could be part of any of the server processes without hugely rethinking how, for example, IMAPD and POP3D let each other's "cleanup threads" know what they're doing. And, before anyone suggests it, "pop3 never needs header caching, so only imapd would have the add-a-header thread" I *really* don't like that idea -- as much as possible, we should keep the daemons similar, on a common codebase, and only differing in their protocol handling and communications routines. By refactoring the IMAP code to eliminate globals and use single-output routines that take a high level list structure and produce correct protocol formatted parenthesis and whatnot, we're almost done with threading the application. Just slap threading into the process manager, double check for stupidity and corner cases (80% of the effort anyways) and that's it. Right!? Aaron --
Re: [Dbmail-dev] refactoring plans for 2.1
Leif Jackson wrote: If we're using Glib, then let's also use Glib's threading interface. It gives us transparency over a variety of thread implementations and platforms, and provides for graceful failure in case there is no thread implementation at all. The issue is that glib's thead pool won't work for an application server like dbmail, it is a pool or worker functions I wrote another program using glib's thread interface and it is a nightmare. I have a thread pool implementaion that is mostly from the pthreads book by o'reilly with real world updates and improments to it. It should be portable and if we make the threads configurable that would take care of the systems that don't allow for threads. From my perspective this is a matter of reputation. I don't know anything much about writing multi-threaded implementations in c. My experience is limited to java. But any code we maintain, must be maintainable. Being open to new things, I'm perfectly willing to take your word for it if you say your code is better than't glib's implementation, but I rather take your code for it :-) I could some threaded model. I've been thinking about header caching: We implement a cache table with fields for every header you wish to cache. If a sysadmin adds a column for some funky X-Field he wants to cache, the insertion code immediatly starts using the extra header field, while the retrieval code degrades gracefully if a message is not yet in the cache. It would be nice to be able to launch a slow priority thread that gradually starts to analyze all older messages without interfering with the imap interaction. In the event of no thread implementation, or threads turned off, there would basically be one thread per process and we could act as we do now. In such a case, I think it would be ideal if the main parent process could keep track of how many threads are running, and spawn processes until the proper thread count is reached. On another note, dbmail shouldn't be run on systems without threads as off the top of my head I can't think of one that it would even compile on right now. Sun Solaris, linux and the bsd's have pthreads without any issues it is more of an issue to support glib on other system other than linux than it is to support pthreads. GLib's aim is to be portable and gracefully degrading. But pthreads does have a serious industrial grade image. And the lack of priorities in gthreads is kind of disappointing. -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl
RE: [Dbmail-dev] refactoring plans for 2.1
> ""Leif Jackson"" <[EMAIL PROTECTED]> said: > >>> If we're using Glib, then let's also use Glib's threading interface. It >>> gives us transparency over a variety of thread implementations and >>> platforms, and provides for graceful failure in case there is no thread >>> implementation at all. >> >> The issue is that glib's thead pool won't work for an application server >> like dbmail, it is a pool or worker functions I wrote another program >> using glib's thread interface and it is a nightmare. I have a thread >> pool >> implementaion that is mostly from the pthreads book by o'reilly with >> real >> world updates and improments to it. It should be portable and if we make >> the threads configurable that would take care of the systems that don't >> allow for threads. > > > There are two sets of thread API's in Glib: Threads and ThreadPools. I > haven't used either, but Threads looks like an appropriate wrapper around > PThreads and Solaris threads. Does that address your thread pool concerns? > No unfortunatly it doesn't exactly it makes the code a bit harder, I will work on tring to use the glib threads with my thread pool code and see what happens, it just adds an extra layer of code that isn't needed as threads are really not that hard to write well, primarly with the glib stuff it makes it 3 times as hard to debug and that is my main concern. -leif > > Aaron > > -- > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev >
RE: [Dbmail-dev] refactoring plans for 2.1
""Leif Jackson"" <[EMAIL PROTECTED]> said: >> If we're using Glib, then let's also use Glib's threading interface. It >> gives us transparency over a variety of thread implementations and >> platforms, and provides for graceful failure in case there is no thread >> implementation at all. > > The issue is that glib's thead pool won't work for an application server > like dbmail, it is a pool or worker functions I wrote another program > using glib's thread interface and it is a nightmare. I have a thread pool > implementaion that is mostly from the pthreads book by o'reilly with real > world updates and improments to it. It should be portable and if we make > the threads configurable that would take care of the systems that don't > allow for threads. There are two sets of thread API's in Glib: Threads and ThreadPools. I haven't used either, but Threads looks like an appropriate wrapper around PThreads and Solaris threads. Does that address your thread pool concerns? Aaron --
RE: [Dbmail-dev] refactoring plans for 2.1
> If we're using Glib, then let's also use Glib's threading interface. It > gives us transparency over a variety of thread implementations and > platforms, and provides for graceful failure in case there is no thread > implementation at all. The issue is that glib's thead pool won't work for an application server like dbmail, it is a pool or worker functions I wrote another program using glib's thread interface and it is a nightmare. I have a thread pool implementaion that is mostly from the pthreads book by o'reilly with real world updates and improments to it. It should be portable and if we make the threads configurable that would take care of the systems that don't allow for threads. > > > In the event of no thread implementation, or threads turned off, there > would basically be one thread per process and we could act as we do now. > In such a case, I think it would be ideal if the main parent process could > keep track of how many threads are running, and spawn processes until the > proper thread count is reached. On another note, dbmail shouldn't be run on systems without threads as off the top of my head I can't think of one that it would even compile on right now. Sun Solaris, linux and the bsd's have pthreads without any issues it is more of an issue to support glib on other system other than linux than it is to support pthreads. Just my thoughts as always -leif
RE: [Dbmail-dev] refactoring plans for 2.1
If we're using Glib, then let's also use Glib's threading interface. It gives us transparency over a variety of thread implementations and platforms, and provides for graceful failure in case there is no thread implementation at all. In the event of no thread implementation, or threads turned off, there would basically be one thread per process and we could act as we do now. In such a case, I think it would be ideal if the main parent process could keep track of how many threads are running, and spawn processes until the proper thread count is reached. Aaron ""Wolfram A. Kraushaar"" <[EMAIL PROTECTED]> said: > Would be nice if this gets configurable like with > the mpm modules in apache 2, so systems without pthreads > are also supported. > > just my 2 cent > Wolfram > > >> -Original Message- >> From: [EMAIL PROTECTED] >> [mailto:[EMAIL PROTECTED] On Behalf Of Leif Jackson >> Sent: Montag, 20. September 2004 19:46 >> To: DBMAIL Developers Mailinglist >> Subject: Re: [Dbmail-dev] refactoring plans for 2.1 >> >> Along with this I would like to start a discussion on the >> possiblity for >> doing away with the fork code and going to a pthread model >> with a thread >> pool, I have already starting some proof of concept code on >> my own, but >> with the performance on threads in linux 2.6.8 and the performance of >> threads on my Sun boxes this would give DBmail a leap ahead. >> Let me know >> the thoughts ...etc from everyone on if you guys think it >> would make any >> since to switch to a threading pooled model. >> >> Thanks, >> Leif >> > > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev > --
RE: [Dbmail-dev] refactoring plans for 2.1
Would be nice if this gets configurable like with the mpm modules in apache 2, so systems without pthreads are also supported. just my 2 cent Wolfram > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Leif Jackson > Sent: Montag, 20. September 2004 19:46 > To: DBMAIL Developers Mailinglist > Subject: Re: [Dbmail-dev] refactoring plans for 2.1 > > Along with this I would like to start a discussion on the > possiblity for > doing away with the fork code and going to a pthread model > with a thread > pool, I have already starting some proof of concept code on > my own, but > with the performance on threads in linux 2.6.8 and the performance of > threads on my Sun boxes this would give DBmail a leap ahead. > Let me know > the thoughts ...etc from everyone on if you guys think it > would make any > since to switch to a threading pooled model. > > Thanks, > Leif >
Re: [Dbmail-dev] refactoring plans for 2.1
Along with this I would like to start a discussion on the possiblity for doing away with the fork code and going to a pthread model with a thread pool, I have already starting some proof of concept code on my own, but with the performance on threads in linux 2.6.8 and the performance of threads on my Sun boxes this would give DBmail a leap ahead. Let me know the thoughts ...etc from everyone on if you guys think it would make any since to switch to a threading pooled model. Thanks, Leif > > Hi all, > > As promised I've been busy reworking imapcommands.c lately. Since what I'm > doing > will be pretty invasive wrt the imap code I want to share with you some of > the > progress made and what I plan to do... > > 1) change the signature for all _ic_XXX functions > 2) change the way results are gathered and flushed to to client > 3) use glib and gmime extensively > > > > ad 1) > > Currently all _ic_XXX functions look like: > > int _ic_XXX(char *tag, char **args, ClientInfo * ci) > > I plan to change this to > > int _ic_XXX(struct ImapSession *self) > > where a struct ImapSession contains much of the currently globalized > variables > in imapcommands.c and imaputil.c > > This will enable splitting up of _ic_fetch and friends, and generally make > life > easier for people who want to refactor the imap code. > > > > ad 2) > > Currently all results are written to the client by calling fprintf all > over the > place (or ci_write which is a wrapper). > > I'm almost finished with rewriting the imapcode and replace all those > fprintf's > with a GList setup. This helps building up clean lists of items to be > returned > to the client, and sanatize their formatting. No more trailing spaces in > parentisized lists, etc... > > > > ad3) > > While working at (2) I began using GLib extensively, but now I've also > started > working on the actual message-retrieval code. Using gmime there makes a > lot of > sense, and I've already rewritten some the rfcsize code. > > Using GMime will help clean up the message parsing *a lot*. > > All this is being worked on on the nfg-0-1 branch. Code is currently > working > well, in sync with head as of sept 16, and tested with both the testframe > and > thunderbird. > > Of course, doing this started out with a plan to phase-in the is_header > field, > and start working toward a header-caching setup. Those plans will largely > determine what I'll be working on in this branch. > > > -- > >Paul Stevens [EMAIL PROTECTED] >NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 >The Netherlands___www.nfg.nl > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev >
[Dbmail-dev] refactoring plans for 2.1
Hi all, As promised I've been busy reworking imapcommands.c lately. Since what I'm doing will be pretty invasive wrt the imap code I want to share with you some of the progress made and what I plan to do... 1) change the signature for all _ic_XXX functions 2) change the way results are gathered and flushed to to client 3) use glib and gmime extensively ad 1) Currently all _ic_XXX functions look like: int _ic_XXX(char *tag, char **args, ClientInfo * ci) I plan to change this to int _ic_XXX(struct ImapSession *self) where a struct ImapSession contains much of the currently globalized variables in imapcommands.c and imaputil.c This will enable splitting up of _ic_fetch and friends, and generally make life easier for people who want to refactor the imap code. ad 2) Currently all results are written to the client by calling fprintf all over the place (or ci_write which is a wrapper). I'm almost finished with rewriting the imapcode and replace all those fprintf's with a GList setup. This helps building up clean lists of items to be returned to the client, and sanatize their formatting. No more trailing spaces in parentisized lists, etc... ad3) While working at (2) I began using GLib extensively, but now I've also started working on the actual message-retrieval code. Using gmime there makes a lot of sense, and I've already rewritten some the rfcsize code. Using GMime will help clean up the message parsing *a lot*. All this is being worked on on the nfg-0-1 branch. Code is currently working well, in sync with head as of sept 16, and tested with both the testframe and thunderbird. Of course, doing this started out with a plan to phase-in the is_header field, and start working toward a header-caching setup. Those plans will largely determine what I'll be working on in this branch. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
[Dbmail-dev] [DBMail 0000087]: Possible memory leak in session and db code for pop3d
A BUGNOTE has been added to this bug. == http://dbmail.org/mantis/bug_view_advanced_page.php?bug_id=087 == Reported By:ljackson Assigned To:ilja == Project:DBMail Bug ID: 87 Category: Database layer Reproducibility:always Severity: major Priority: normal Status: feedback == Date Submitted: 13-Sep-04 23:35 CEST Last Modified: 20-Sep-04 15:59 CEST == Summary:Possible memory leak in session and db code for pop3d Description: I have been able to reproduce in both cvs from 080304 and 2.0rc8 an issue where normal connects & lists to the pop3d leaks memory, I have traced it to session code not being freed in the connection cleanup code. == -- ilja - 15-Sep-04 12:20 CEST -- I'm taking this patch one step at a time. first: I've added db_session_cleanup() to pop3.c. I've put it in a different place than in the patch. It should clean up the session after errors too. So, it's added after the switch statement in the patch. -- ilja - 15-Sep-04 14:20 CEST -- The auth/authsql.c fix is also in. -- ilja - 15-Sep-04 14:51 CEST -- db.c fixed -- ilja - 15-Sep-04 15:03 CEST -- the fix for mysql/dbmysql.c isn't necessary: mysql_store_result() will return NULL if the query isn't a SELECT query. On top of that mysql_num_fields doesn't work in current versions of MySQL. -- ilja - 15-Sep-04 15:04 CEST -- All relevant fixes are now in dbmail_2_0_branch and HEAD. -- ljackson - 15-Sep-04 19:53 CEST -- Thanks for appling this patch, while I aggree that in Mysql 3.x might not use the mysql_store_result but this works for Mysql 4.x and higher and is recomended from thier documentation. This solves the need for a lot of extra db_free_result's in each of the sql statments that are not selects. Otherwise we still have a memory leak for each on of the statments. Please consider a ifdef or conditional that adds it if your using a Mysql version >= 4.x thanks, leif -- ljackson - 15-Sep-04 19:54 CEST -- also has this patch been merged into 2.1 cvs head? I havn't checked but I belive it is an issue there as well. -- aaron - 15-Sep-04 20:01 CEST -- I thought that DBMail required MySQL 4.0 or higher... if not, it should, IMHO. -- ljackson - 15-Sep-04 20:15 CEST -- In the back of my mind I thought it did as well that is why I didn't feel bad adding that bit of code, *shrug* I thought it made sence. There are still a number of minor memory leaks that are reported from valgrind in thread watching mode if anyone is interested use the latest version of valgrind 2.2.0 and the following cmd line from the compiled directory. valgrind --tool=memcheck --logfile=/tmp/pop3d --leak-check=yes --show-reachable=yes --trace-children=yes -v ./dbmail-pop3d or if you want to really dig on it use valgrind --tool=memcheck --logfile=/tmp/pop3d --leak-check=yes --show-reachable=yes --trace-children=yes --track-fds=yes --time-stamp=yes --num-callers=6 --error-limit=no --show-below-main=yes -v ./dbmail-pop3d it is minor but there are still some cases where memory is being malloced for list node_add and not being freed even on Sig 15. just my 0.02 again. -- ljack
[Dbmail-dev] [DBMail 0000087]: Possible memory leak in session and db code for pop3d
A BUGNOTE has been added to this bug. == http://dbmail.org/mantis/bug_view_advanced_page.php?bug_id=087 == Reported By:ljackson Assigned To:ilja == Project:DBMail Bug ID: 87 Category: Database layer Reproducibility:always Severity: major Priority: normal Status: feedback == Date Submitted: 13-Sep-04 23:35 CEST Last Modified: 20-Sep-04 15:45 CEST == Summary:Possible memory leak in session and db code for pop3d Description: I have been able to reproduce in both cvs from 080304 and 2.0rc8 an issue where normal connects & lists to the pop3d leaks memory, I have traced it to session code not being freed in the connection cleanup code. == -- ilja - 15-Sep-04 12:20 CEST -- I'm taking this patch one step at a time. first: I've added db_session_cleanup() to pop3.c. I've put it in a different place than in the patch. It should clean up the session after errors too. So, it's added after the switch statement in the patch. -- ilja - 15-Sep-04 14:20 CEST -- The auth/authsql.c fix is also in. -- ilja - 15-Sep-04 14:51 CEST -- db.c fixed -- ilja - 15-Sep-04 15:03 CEST -- the fix for mysql/dbmysql.c isn't necessary: mysql_store_result() will return NULL if the query isn't a SELECT query. On top of that mysql_num_fields doesn't work in current versions of MySQL. -- ilja - 15-Sep-04 15:04 CEST -- All relevant fixes are now in dbmail_2_0_branch and HEAD. -- ljackson - 15-Sep-04 19:53 CEST -- Thanks for appling this patch, while I aggree that in Mysql 3.x might not use the mysql_store_result but this works for Mysql 4.x and higher and is recomended from thier documentation. This solves the need for a lot of extra db_free_result's in each of the sql statments that are not selects. Otherwise we still have a memory leak for each on of the statments. Please consider a ifdef or conditional that adds it if your using a Mysql version >= 4.x thanks, leif -- ljackson - 15-Sep-04 19:54 CEST -- also has this patch been merged into 2.1 cvs head? I havn't checked but I belive it is an issue there as well. -- aaron - 15-Sep-04 20:01 CEST -- I thought that DBMail required MySQL 4.0 or higher... if not, it should, IMHO. -- ljackson - 15-Sep-04 20:15 CEST -- In the back of my mind I thought it did as well that is why I didn't feel bad adding that bit of code, *shrug* I thought it made sence. There are still a number of minor memory leaks that are reported from valgrind in thread watching mode if anyone is interested use the latest version of valgrind 2.2.0 and the following cmd line from the compiled directory. valgrind --tool=memcheck --logfile=/tmp/pop3d --leak-check=yes --show-reachable=yes --trace-children=yes -v ./dbmail-pop3d or if you want to really dig on it use valgrind --tool=memcheck --logfile=/tmp/pop3d --leak-check=yes --show-reachable=yes --trace-children=yes --track-fds=yes --time-stamp=yes --num-callers=6 --error-limit=no --show-below-main=yes -v ./dbmail-pop3d it is minor but there are still some cases where memory is being malloced for list node_add and not being freed even on Sig 15. just my 0.02 again. -- ljack
[Dbmail-dev] [DBMail 0000070]: Fails to give reponse upon completion of rcpt
The following bug has been REOPENED. == http://dbmail.org/mantis/bug_view_advanced_page.php?bug_id=070 == Reported By:danweber Assigned To:aaron == Project:DBMail Bug ID: 70 Category: LMTP daemon Reproducibility:always Severity: major Priority: normal Status: feedback == Date Submitted: 23-Aug-04 01:43 CEST Last Modified: 20-Sep-04 14:33 CEST == Summary:Fails to give reponse upon completion of rcpt Description: A responce is needed directly after rcpt command. Otherwise its incompatible with the rfc, and it also makes python hang.I stumbled upon this when I was adding lmtp support for mailbox2dbmail. == -- ilja - 23-Aug-04 13:32 CEST -- AFAIK the client _should_ support pipelining, which means the respons comes after the DATA command, not after every RCPT command. -- aaron - 23-Aug-04 18:11 CEST -- Actually, pipelining means that neither side should block or wait for anything during the RCPT stage. So both behaviors are slightly wrong. -- danweber - 24-Aug-04 18:50 CEST -- Its good practice to send a response after rcpt. This way it allows a compatible smtp client to be used with it, like python. Dan -- aaron - 24-Aug-04 19:28 CEST -- Oh, that's a good point. LMTP is a dialect of SMTP, the only difference is the HELO/EHLO/LHLO greeting. Simple scripts that speak simple SMTP can be easily modified to work with LMTP, but adding in the pipelining is more hassle than needed, since the particular behavior we are using isn't quite correct anyhow. So I agree we should change the responses to happen immediately following each RCPT. I'll post a patch this evening. -- aaron - 26-Aug-04 06:08 CEST -- This turns out to be fairly hard because I abstracted the list code out to a separate function, and so I'll need to un-abstract it a little bit. I'm still working on it, and expect to be done tomorrow evening. I'll commit the changes to HEAD and Ilja can patch it back to the release branch once we're 110% sure it's a correct patch. -- aaron - 29-Aug-04 11:11 CEST -- Fixed in HEAD. Refactoring turned up two memory leaks, too. -- ilja - 20-Sep-04 14:33 CEST -- This should be updated in dbmail_2_0_branch. It works very well for 2.1. Bug History Date Modified Username FieldChange == 23-Aug-04 01:43danweber New Bug 23-Aug-04 13:32ilja Bugnote Added: 182 23-Aug-04 18:11aaron Bugnote Added: 191 24-Aug-04 18:50danweber Bugnote Added: 197 24-Aug-04 19:28aaron Bugnote Added: 199 25-Aug-04 11:04ilja Assigned To => aaron 25-Aug-04 11:04ilja Status new => assigned 26-Aug-04 06:08aaron Bugnote Added: 207 29-Aug-04 11:11aaron Bugnote Added: 208 29-Aug-04 11:11aaron Resolution open => fixed 29-Aug-04 11:11aaron Status assigned => resolved 20-Sep-04 14:33ilja Bugnote Added: 275 20-Sep-04 14:33ilja Resolution fixed => reopened 20-Sep-04 14:33ilja Status resolved => feedback
