Re: [Dbmail-dev] Any reason not to put the prefix patch into RC8 for the 2.0 release?
On Fri, 24 Sep 2004 22:53:25 +0200, Paul J Stevens [EMAIL PROTECTED] wrote: 2.0 is frozen solid. Bugfixes only (so unless you want to file a bug against the dbmail_ prefix...). And about time we ship that product. I don't see anything major at the moment holding up 2.0.0-final. Yep. No more new stuff in 2.0. As much as I like the prefix patch, it came too late for 2.0.0. I'd rather focus on getting as much features as possible into 2.1. Then branch 2.1 asap, and continue working on 3.0. I think we're well on our way of getting interesting stuff in 2.1 :) My vision for 3.0: multi-master replication safe, modular sql backend, high-performance imap implementation with starttls capability (rfc3501), full ldap support. I'd also like distributed storage (delegate different users to different db-storage servers/clusters), but that may have to wait a little longer :-) Btw, Ilja -- I kind of miss the wiki. hint, hint. I'll add a Wiki. The current website is a bit of a hack. I just needed something quick to replace the olf one.. I'm thinking of using some kind of CMS to make it easier for other people to add content to the website. This might make www.dbmail.org a better way of attracting new users and developers. Leif, I see your point, but I feel that migration from 1.2 to 2.0 will be and remain non-trivial for all users. In fact, so much so, that I feel that dbmail-smtp/imapd/pop3d actually should refuse to run at all against a database in 1.2 state. The fact that they do run are cause for serious concern from my point of view as debian maintainer. Agreed. Perhaps we can do something simple like this: check if the physmessage table is present. If it's not - refuse to run. Ilja -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Any reason not to put the prefix patch into RC8 for the 2.0 release?
Ilja Booij wrote: On Fri, 24 Sep 2004 22:53:25 +0200, Paul J Stevens [EMAIL PROTECTED] wrote: Leif, I see your point, but I feel that migration from 1.2 to 2.0 will be and remain non-trivial for all users. In fact, so much so, that I feel that dbmail-smtp/imapd/pop3d actually should refuse to run at all against a database in 1.2 state. The fact that they do run are cause for serious concern from my point of view as debian maintainer. Agreed. Perhaps we can do something simple like this: check if the physmessage table is present. If it's not - refuse to run. That's what I was thinking about too. However, even for this I wouldn't want to hold up 2.0.0 perse. Perhaps release 2.0.0 as soon as Aaron's lmtp fix is in, initiate the switch to subversion, and release 2.0.1 in two week or so, which will then include some version check against the database. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] pthreads support?
Gerrit P. Haase wrote: Leif Jackson wrote: Hello, please consider using one single process with threads, for each client one thread, instead the slower 'fork a child for every client' or Apache like pre-fork technique. Our current 2.1 cvs head has pre-fork and I am currently working on a high performance version with pthreads which is coming along well but still is a while out. Hey, this is great news, I just started using lighttpd which uses pthreads too and it is a lot faster than Apache therefore. So I thought there are chances that it will speed up dbmail too. Threads may speed up dbmail a little, but I don't think the forking model currently is much of a bottleneck. Combined with a redesign of the imap codebase however, dbmail on pthreads will kick ass. Well going to bed now but I thought I would respond, the threading patch has been completed for the pop server and halfway through making the imap daemon thread safe... It is a major bunch of changes as well as I changed out the list.c for the GList stuff because we have glib in the requirments might as well start using it :) As for the threading code it makes about a 60% speedup on the pop3 server alone and I can't state much yet for the imap server but between the pthreads and the new list code I have finaly eradicated the last few minor memory leaks I used to see from the list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is a major re-think of some areas...etc.. Anyway a little punchy 4:36am here, been coding almost non-stop all weekend just got a bug up my $#% if you know what I mean... Later, Leif
Re: [Dbmail-dev] pthreads support?
On Mon, 27 Sep 2004 04:45:38 -0400 (EDT), Leif Jackson [EMAIL PROTECTED] wrote: Well going to bed now but I thought I would respond, the threading patch has been completed for the pop server and halfway through making the imap daemon thread safe... It is a major bunch of changes as well as I changed out the list.c for the GList stuff because we have glib in the requirments might as well start using it :) As for the threading code it makes about a 60% speedup on the pop3 server alone and I can't state much yet for the imap server but between the pthreads and the new list code I have finaly eradicated the last few minor memory leaks I used to see from the list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is a major re-think of some areas...etc.. 60% speedup? Wow. I can't really imagine how using threads would speed it up so much, but it if it really works that well.. :) Can you perhaps split the patch for the GList stuff and the threads stuff. It's preferable to have somewhat smaller patches, especially for things that are not really connected. Ilja
Re: [Dbmail-dev] Any reason not to put the prefix patch into RC8 for the 2.0 release?
Ilja Booij wrote:
On Fri, 24 Sep 2004 22:53:25 +0200, Paul J Stevens [EMAIL PROTECTED] wrote:
Leif, I see your point, but I feel that migration from 1.2 to 2.0 will be and
remain non-trivial for all
users. In fact, so much so, that I feel that dbmail-smtp/imapd/pop3d actually
should refuse to run at all
against a database in 1.2 state. The fact that they do run are cause for
serious concern from my point of view
as debian maintainer.
Agreed. Perhaps we can do something simple like this:
check if the physmessage table is present. If it's not - refuse to run.
Ilja, I thought it would be easiest to provide a patch in the dbmail packages
that does just this.
For people who migrate from dbmail-1.2 this patch might be of use.
Basically i run: select 1=1 from dbmail_physmessage limit 1 offset 0 to check
for the physmessage table, which should work in mysql and postgresql both.
--
Paul Stevens [EMAIL PROTECTED]
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands___www.nfg.nl
#! /bin/sh -e
## 04_version_check.dpatch by [EMAIL PROTECTED]
##
## All lines beginning with `## DP:' are a description of the patch.
## DP: No description.
if [ $# -lt 1 ]; then
echo `basename $0`: script expects -patch|-unpatch as argument 2
exit 1
fi
[ -f debian/patches/00patch-opts ] . debian/patches/00patch-opts
patch_opts=${patch_opts:--f --no-backup-if-mismatch} ${2:+-d $2}
case $1 in
-patch) patch -p1 ${patch_opts} $0;;
-unpatch) patch -R -p1 ${patch_opts} $0;;
*)
echo `basename $0`: script expects -patch|-unpatch as argument 2
exit 1;;
esac
exit 0
@DPATCH@
diff -urNad /usr/src/dbmail2/dbmail-2.0.0/db.c dbmail-2.0.0/db.c
--- /usr/src/dbmail2/dbmail-2.0.0/db.c 2004-09-27 12:56:08.0 +0200
+++ dbmail-2.0.0/db.c 2004-09-27 12:56:51.0 +0200
@@ -131,6 +131,20 @@
*/
void convert_inbox_to_uppercase(char *name);
+/*
+ * check to make sure the database has been upgraded
+ */
+int db_check_version(void)
+{
+ snprintf(query, DEF_QUERYSIZE, SELECT 1=1 FROM dbmail_physmessage
LIMIT 1 OFFSET 0);
+ if (db_query(query) == -1) {
+ trace(TRACE_FATAL, %s,%s: database incompatible. You may need
to run the conversion script,
+ __FILE__,__func__);
+ return -1;
+ }
+ return 0;
+}
+
int db_begin_transaction()
{
snprintf(query, DEF_QUERYSIZE,
diff -urNad /usr/src/dbmail2/dbmail-2.0.0/db.h dbmail-2.0.0/db.h
--- /usr/src/dbmail2/dbmail-2.0.0/db.h 2004-09-27 12:56:08.0 +0200
+++ dbmail-2.0.0/db.h 2004-09-27 12:56:08.0 +0200
@@ -76,6 +76,12 @@
*/
int db_connect(void);
+/*
+ * make sure we're running against a 2.0 database layout
+ */
+
+int db_check_version(void);
+
/**
* \brief disconnect from database server
* \return 0
diff -urNad /usr/src/dbmail2/dbmail-2.0.0/main.c dbmail-2.0.0/main.c
--- /usr/src/dbmail2/dbmail-2.0.0/main.c2004-09-27 12:56:08.0
+0200
+++ dbmail-2.0.0/main.c 2004-09-27 12:56:08.0 +0200
@@ -355,6 +355,11 @@
exitcode = EX_TEMPFAIL;
goto freeall;
}
+
+ if (db_check_version() != 0) {
+ exitcode = EX_TEMPFAIL;
+ goto freeall;
+ }
/* read the whole message */
if (read_whole_message_pipe(stdin, whole_message,
diff -urNad /usr/src/dbmail2/dbmail-2.0.0/maintenance.c
dbmail-2.0.0/maintenance.c
--- /usr/src/dbmail2/dbmail-2.0.0/maintenance.c 2004-09-27 12:56:08.0
+0200
+++ dbmail-2.0.0/maintenance.c 2004-09-27 12:56:08.0 +0200
@@ -224,6 +224,11 @@
return -1;
}
+ if (db_check_version() != 0) {
+ printf(Database version incompatible. Please upgrade your
database.\n);
+ return -1;
+ }
+
printf(Ok. Connected\n);
if (purge_deleted) {
diff -urNad /usr/src/dbmail2/dbmail-2.0.0/serverchild.c
dbmail-2.0.0/serverchild.c
--- /usr/src/dbmail2/dbmail-2.0.0/serverchild.c 2004-09-27 12:56:08.0
+0200
+++ dbmail-2.0.0/serverchild.c 2004-09-27 12:56:08.0 +0200
@@ -277,6 +277,8 @@
PerformChildTask(): could not connect to
authentication);
return -1;
}
+ if (db_check_version() != 0)
+ return -1;
srand((int) ((int) time(NULL) + (int) getpid()));
connected = 1;
diff -urNad /usr/src/dbmail2/dbmail-2.0.0/sievecmd.c dbmail-2.0.0/sievecmd.c
--- /usr/src/dbmail2/dbmail-2.0.0/sievecmd.c2004-09-27 12:56:08.0
+0200
+++ dbmail-2.0.0/sievecmd.c 2004-09-27 12:56:08.0 +0200
@@ -120,6 +120,12 @@
(Failed. Could not connect to authentication
(check log)\n);
return -1;
}
+
Re: [Dbmail-dev] pthreads support?
Aaron Stone wrote: Leif Jackson [EMAIL PROTECTED] said: As for the threading code it makes about a 60% speedup on the pop3 server alone and I can't state much yet for the imap server but between the pthreads and the new list code I have finaly eradicated the last few minor memory leaks I used to see from the list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is a major re-think of some areas...etc.. Sounds like an incredible speedup, and it'll be exciting to actually be able to say, I've done high performance multithreaded database programming :-) I agree that it's incredible, so much so that I am skeptical. Could we get some more details on what is faster under what conditions? Given the efficiency of fork()on unix platforms, I highly doubt that a threaded implementation will make much difference at all. We are very different than a light http server that doesn't have much work to do after it starts up. Now that DBMail is going multithreaded, libSieve will also have to become thread safe, which depends on flex being thread safe... there will probably need to be quite a few mutexes surrounding the sieve calls in 2.1. When was it decided that we are going multithreaded. All I have read is that Leif is working on a patch that includes a pthreads implementation. I have yet to see any evidence that the results are worth the effort. Remember that the additional effort is not just the initial implementation but also includes the additional maintenance since threaded code is more complicated and bugs more subtle and harder to track down. Matthew
Re: [Dbmail-dev] pthreads support?
Matthew T. O'Connor wrote: When was it decided that we are going multithreaded. No decision yet, afaik. I'm guessing Aaron's just getting exited about the idea... We're all mostly just awaiting the results from Leifs efforts, so we cn run our own performance checks. Seeing is believing, indeed. All I have read is that Leif is working on a patch that includes a pthreads implementation. I have yet to see any evidence that the results are worth the effort. Remember that the additional effort is not just the initial implementation but also includes the additional maintenance since threaded code is more complicated and bugs more subtle and harder to track down. Being somewhat intimate with the current imap codebase, I'm skeptical it can be done in a clean and maintainable manner. But given the size of the diff Leifs mentions, perhaps he has solved some of the problems I myself have been wrestling with :-) -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl
