Re: [Dbmail-dev] Any reason not to put the prefix patch into RC8 for the 2.0 release?
On Fri, 24 Sep 2004 22:53:25 +0200, Paul J Stevens <[EMAIL PROTECTED]> wrote: > 2.0 is frozen solid. Bugfixes only (so unless you want to file a bug against > the dbmail_ prefix...). And about > time we ship that product. I don't see anything major at the moment holding > up 2.0.0-final. Yep. No more new stuff in 2.0. > > As much as I like the prefix patch, it came too late for 2.0.0. I'd rather > focus on getting as much features > as possible into 2.1. Then branch 2.1 asap, and continue working on 3.0. I think we're well on our way of getting interesting stuff in 2.1 :) > > My vision for 3.0: multi-master replication safe, modular sql backend, > high-performance imap implementation > with starttls capability (rfc3501), full ldap support. I'd also like > distributed storage (delegate different > users to different db-storage servers/clusters), but that may have to wait a > little longer :-) > > Btw, Ilja -- I kind of miss the wiki. hint, hint. I'll add a Wiki. The current website is a bit of a hack. I just needed something quick to replace the olf one.. I'm thinking of using some kind of CMS to make it easier for other people to add content to the website. This might make www.dbmail.org a better way of attracting new users and developers. > > Leif, I see your point, but I feel that migration from 1.2 to 2.0 will be and > remain non-trivial for all > users. In fact, so much so, that I feel that dbmail-smtp/imapd/pop3d actually > should refuse to run at all > against a database in 1.2 state. The fact that they do run are cause for > serious concern from my point of view > as debian maintainer. Agreed. Perhaps we can do something simple like this: check if the physmessage table is present. If it's not -> refuse to run. Ilja > > -- > >Paul Stevens mailto:[EMAIL PROTECTED] >NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] >The Netherlandshttp://www.nfg.nl > > > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev >
Re: [Dbmail-dev] pthreads support?
Gerrit P. Haase wrote: Leif Jackson wrote: Hello, please consider using one single process with threads, for each client one thread, instead the slower 'fork a child for every client' or Apache like pre-fork technique. Our current 2.1 cvs head has pre-fork and I am currently working on a high performance version with pthreads which is coming along well but still is a while out. Hey, this is great news, I just started using lighttpd which uses pthreads too and it is a lot faster than Apache therefore. So I thought there are chances that it will speed up dbmail too. Threads may speed up dbmail a little, but I don't think the forking model currently is much of a bottleneck. Combined with a redesign of the imap codebase however, dbmail on pthreads will kick ass. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] Any reason not to put the prefix patch into RC8 for the 2.0 release?
Ilja Booij wrote: On Fri, 24 Sep 2004 22:53:25 +0200, Paul J Stevens <[EMAIL PROTECTED]> wrote: Leif, I see your point, but I feel that migration from 1.2 to 2.0 will be and remain non-trivial for all users. In fact, so much so, that I feel that dbmail-smtp/imapd/pop3d actually should refuse to run at all against a database in 1.2 state. The fact that they do run are cause for serious concern from my point of view as debian maintainer. Agreed. Perhaps we can do something simple like this: check if the physmessage table is present. If it's not -> refuse to run. That's what I was thinking about too. However, even for this I wouldn't want to hold up 2.0.0 perse. Perhaps release 2.0.0 as soon as Aaron's lmtp fix is in, initiate the switch to subversion, and release 2.0.1 in two week or so, which will then include some version check against the database. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] pthreads support?
> > > Gerrit P. Haase wrote: >> Leif Jackson wrote: >> Hello, please consider using one single process with threads, for each client one thread, instead the slower 'fork a child for every client' or Apache like pre-fork technique. >> >> >> >>> Our current 2.1 cvs head has pre-fork and I am currently working on a >>> high >>> performance version with pthreads which is coming along well but still >>> is >>> a while out. >> >> >> Hey, this is great news, I just started using lighttpd which uses >> pthreads too and it is a lot faster than Apache therefore. So I thought >> there are chances that it will speed up dbmail too. > > Threads may speed up dbmail a little, but I don't think the forking model > currently is much of a bottleneck. Combined with a redesign of the imap > codebase > however, dbmail on pthreads will kick ass. > Well going to bed now but I thought I would respond, the threading patch has been completed for the pop server and halfway through making the imap daemon thread safe... It is a major bunch of changes as well as I changed out the list.c for the GList stuff because we have glib in the requirments might as well start using it :) As for the threading code it makes about a 60% speedup on the pop3 server alone and I can't state much yet for the imap server but between the pthreads and the new list code I have finaly eradicated the last few minor memory leaks I used to see from the list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is a major re-think of some areas...etc.. Anyway a little punchy 4:36am here, been coding almost non-stop all weekend just got a bug up my $#% if you know what I mean... Later, Leif
Re: [Dbmail-dev] pthreads support?
On Mon, 27 Sep 2004 04:45:38 -0400 (EDT), Leif Jackson <[EMAIL PROTECTED]> wrote: > > > > Well going to bed now but I thought I would respond, the threading patch > has been completed for the pop server and halfway through making the imap > daemon thread safe... It is a major bunch of changes as well as I changed > out the list.c for the GList stuff because we have glib in the requirments > might as well start using it :) As for the threading code it makes about a > 60% speedup on the pop3 server alone and I can't state much yet for the > imap server but between the pthreads and the new list code I have finaly > eradicated the last few minor memory leaks I used to see from the > list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is > a major re-think of some areas...etc.. 60% speedup? Wow. I can't really imagine how using threads would speed it up so much, but it if it really works that well.. :) Can you perhaps split the patch for the GList stuff and the threads stuff. It's preferable to have somewhat smaller patches, especially for things that are not really connected. Ilja
Re: [Dbmail-dev] Any reason not to put the prefix patch into RC8 for the 2.0 release?
Ilja Booij wrote: On Fri, 24 Sep 2004 22:53:25 +0200, Paul J Stevens <[EMAIL PROTECTED]> wrote: Leif, I see your point, but I feel that migration from 1.2 to 2.0 will be and remain non-trivial for all users. In fact, so much so, that I feel that dbmail-smtp/imapd/pop3d actually should refuse to run at all against a database in 1.2 state. The fact that they do run are cause for serious concern from my point of view as debian maintainer. Agreed. Perhaps we can do something simple like this: check if the physmessage table is present. If it's not -> refuse to run. Ilja, I thought it would be easiest to provide a patch in the dbmail packages that does just this. For people who migrate from dbmail-1.2 this patch might be of use. Basically i run: "select 1=1 from dbmail_physmessage limit 1 offset 0" to check for the physmessage table, which should work in mysql and postgresql both. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl #! /bin/sh -e ## 04_version_check.dpatch by <[EMAIL PROTECTED]> ## ## All lines beginning with `## DP:' are a description of the patch. ## DP: No description. if [ $# -lt 1 ]; then echo "`basename $0`: script expects -patch|-unpatch as argument" >&2 exit 1 fi [ -f debian/patches/00patch-opts ] && . debian/patches/00patch-opts patch_opts="${patch_opts:--f --no-backup-if-mismatch} ${2:+-d $2}" case "$1" in -patch) patch -p1 ${patch_opts} < $0;; -unpatch) patch -R -p1 ${patch_opts} < $0;; *) echo "`basename $0`: script expects -patch|-unpatch as argument" >&2 exit 1;; esac exit 0 @DPATCH@ diff -urNad /usr/src/dbmail2/dbmail-2.0.0/db.c dbmail-2.0.0/db.c --- /usr/src/dbmail2/dbmail-2.0.0/db.c 2004-09-27 12:56:08.0 +0200 +++ dbmail-2.0.0/db.c 2004-09-27 12:56:51.0 +0200 @@ -131,6 +131,20 @@ */ void convert_inbox_to_uppercase(char *name); +/* + * check to make sure the database has been upgraded + */ +int db_check_version(void) +{ + snprintf(query, DEF_QUERYSIZE, "SELECT 1=1 FROM dbmail_physmessage LIMIT 1 OFFSET 0"); + if (db_query(query) == -1) { + trace(TRACE_FATAL, "%s,%s: database incompatible. You may need to run the conversion script", + __FILE__,__func__); + return -1; + } + return 0; +} + int db_begin_transaction() { snprintf(query, DEF_QUERYSIZE, diff -urNad /usr/src/dbmail2/dbmail-2.0.0/db.h dbmail-2.0.0/db.h --- /usr/src/dbmail2/dbmail-2.0.0/db.h 2004-09-27 12:56:08.0 +0200 +++ dbmail-2.0.0/db.h 2004-09-27 12:56:08.0 +0200 @@ -76,6 +76,12 @@ */ int db_connect(void); +/* + * make sure we're running against a 2.0 database layout + */ + +int db_check_version(void); + /** * \brief disconnect from database server * \return 0 diff -urNad /usr/src/dbmail2/dbmail-2.0.0/main.c dbmail-2.0.0/main.c --- /usr/src/dbmail2/dbmail-2.0.0/main.c2004-09-27 12:56:08.0 +0200 +++ dbmail-2.0.0/main.c 2004-09-27 12:56:08.0 +0200 @@ -355,6 +355,11 @@ exitcode = EX_TEMPFAIL; goto freeall; } + + if (db_check_version() != 0) { + exitcode = EX_TEMPFAIL; + goto freeall; + } /* read the whole message */ if (read_whole_message_pipe(stdin, &whole_message, diff -urNad /usr/src/dbmail2/dbmail-2.0.0/maintenance.c dbmail-2.0.0/maintenance.c --- /usr/src/dbmail2/dbmail-2.0.0/maintenance.c 2004-09-27 12:56:08.0 +0200 +++ dbmail-2.0.0/maintenance.c 2004-09-27 12:56:08.0 +0200 @@ -224,6 +224,11 @@ return -1; } + if (db_check_version() != 0) { + printf("Database version incompatible. Please upgrade your database.\n"); + return -1; + } + printf("Ok. Connected\n"); if (purge_deleted) { diff -urNad /usr/src/dbmail2/dbmail-2.0.0/serverchild.c dbmail-2.0.0/serverchild.c --- /usr/src/dbmail2/dbmail-2.0.0/serverchild.c 2004-09-27 12:56:08.0 +0200 +++ dbmail-2.0.0/serverchild.c 2004-09-27 12:56:08.0 +0200 @@ -277,6 +277,8 @@ "PerformChildTask(): could not connect to authentication"); return -1; } + if (db_check_version() != 0) + return -1; srand((int) ((int) time(NULL) + (int) getpid())); connected = 1; diff -urNad /usr/src/dbmail2/dbmail-2.0.0/sievecmd.c dbmail-2.0.0/sievecmd.c --- /usr/src/dbmail2/dbmail-2.0.0/sievecmd.c2004-09-27 12:56:08.0 +0200 +++ dbmail-2.0.0/sievecmd.c 2004-09-27 12:56:08.0 +0200 @@ -120,6 +120,12 @@ ("Failed. Could not connect to authentication (check log)\n"); r
Re: [Dbmail-dev] pthreads support?
""Leif Jackson"" <[EMAIL PROTECTED]> said: > As for the threading code it makes about a > 60% speedup on the pop3 server alone and I can't state much yet for the > imap server but between the pthreads and the new list code I have finaly > eradicated the last few minor memory leaks I used to see from the > list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is > a major re-think of some areas...etc.. Sounds like an incredible speedup, and it'll be exciting to actually be able to say, "I've done high performance multithreaded database programming" :-) I don't mean to get into a code size war with you, but did you make sure to exclude CVS and autotools from your diff? I haven't been paying close attention to whose code it is that's usually had all of autotools patched, but just a heads up on that. I have a really nice excludes file that I'll post later this week (I'm not on my dev box yet, just finished moving rooms over the weekend, and my machine is still all in boxes :-( Now that DBMail is going multithreaded, libSieve will also have to become thread safe, which depends on flex being thread safe... there will probably need to be quite a few mutexes surrounding the sieve calls in 2.1. Aaron --
Re: [Dbmail-dev] pthreads support?
Aaron Stone wrote: ""Leif Jackson"" <[EMAIL PROTECTED]> said: As for the threading code it makes about a 60% speedup on the pop3 server alone and I can't state much yet for the imap server but between the pthreads and the new list code I have finaly eradicated the last few minor memory leaks I used to see from the list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is a major re-think of some areas...etc.. Sounds like an incredible speedup, and it'll be exciting to actually be able to say, "I've done high performance multithreaded database programming" :-) I agree that it's incredible, so much so that I am skeptical. Could we get some more details on what is faster under what conditions? Given the efficiency of fork()on unix platforms, I highly doubt that a threaded implementation will make much difference at all. We are very different than a light http server that doesn't have much work to do after it starts up. Now that DBMail is going multithreaded, libSieve will also have to become thread safe, which depends on flex being thread safe... there will probably need to be quite a few mutexes surrounding the sieve calls in 2.1. When was it decided that we are "going multithreaded". All I have read is that Leif is working on a patch that includes a pthreads implementation. I have yet to see any evidence that the results are worth the effort. Remember that the additional effort is not just the initial implementation but also includes the additional maintenance since threaded code is more complicated and bugs more subtle and harder to track down. Matthew
Re: [Dbmail-dev] pthreads support?
> ""Leif Jackson"" <[EMAIL PROTECTED]> said: > >> As for the threading code it makes about a >> 60% speedup on the pop3 server alone and I can't state much yet for the >> imap server but between the pthreads and the new list code I have finaly >> eradicated the last few minor memory leaks I used to see from the >> list_nodeadd stuff that bugged me! The diff so far is almost 400k so it >> is >> a major re-think of some areas...etc.. > > Sounds like an incredible speedup, and it'll be exciting to actually be > able to say, "I've done high performance multithreaded database > programming" :-) It is much faster, but that also may be the thread inprovment of the linux kernel 2.6.8... > > I don't mean to get into a code size war with you, but did you make sure > to exclude CVS and autotools from your diff? I haven't been paying close > attention to whose code it is that's usually had all of autotools patched, > but just a heads up on that. I have a really nice excludes file that I'll > post later this week (I'm not on my dev box yet, just finished moving > rooms over the weekend, and my machine is still all in boxes :-( Hehe, nah I always exclude anything that isn't related to my changes. > > Now that DBMail is going multithreaded, libSieve will also have to become > thread safe, which depends on flex being thread safe... there will > probably need to be quite a few mutexes surrounding the sieve calls in > 2.1. Weel I said that I was working on the patch and that the pop server runs well under it, I didn't say I was anywhere near finished :) As far as the Glist patch being seperated I am not sure that will make sence as there are to many related things, I backicaly found out when I started making it threaded under the thread pool model that the current list.c code came up short or at least I felt like it did so since there is glibn in the requirments for 2.1 I figured it would make sence to use their linked list implementation which seems to also be very fast. Anyway I expect this patch will be a few more weeks off at least as I have to go out of town for a while I just wanted to let eveyone know I had good sucess with the pop server the 60 % speed up is more to do with the number of simultainus connections per second that it can sustain vs the cpu load for each connection, it is dramaticly diffrent. Thanks, Leif > > Aaron > > -- > ___ > Dbmail-dev mailing list > Dbmail-dev@dbmail.org > http://twister.fastxs.net/mailman/listinfo/dbmail-dev >
Re: [Dbmail-dev] pthreads support?
Matthew T. O'Connor wrote: When was it decided that we are "going multithreaded". No decision yet, afaik. I'm guessing Aaron's just getting exited about the idea... We're all mostly just awaiting the results from Leifs efforts, so we cn run our own performance checks. Seeing is believing, indeed. All I have read is that Leif is working on a patch that includes a pthreads implementation. I have yet to see any evidence that the results are worth the effort. Remember that the additional effort is not just the initial implementation but also includes the additional maintenance since threaded code is more complicated and bugs more subtle and harder to track down. Being somewhat intimate with the current imap codebase, I'm skeptical it can be done in a clean and maintainable manner. But given the size of the diff Leifs mentions, perhaps he has solved some of the problems I myself have been wrestling with :-) -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl