[Dbmail] DB replication patch
Hi, We're using dbmail together with MySQL. Since our database is quite large and we have a lot of users we want to spread our load a little bit. We have setup our database in replication (master/slave) mode. Then what we want to do is force all read (select) activities to our slave (slaves) and all write (delete, update and insert) to our master database. I've created a patch which does this (only MySQL). What you need to do is #define the READ_DATABASE parameter in "db.h". Then there are some addition parameters to be set in "dbmail.conf" (read_db, read_user etc.). If READ_DATABASE isn't defined dbmail should work as normal, i.e use only one database. I did create a patch earlier and sent it to the list, but it's not in the new release so it's probably nothing that people are interested in? Still we are using it so I send it again... (The patch version is the cvs from 2002-12-05) Have a nice day! /Klas [EMAIL PROTECTED] db_replicate.patch Description: Binary data
[Dbmail] Process title patch
Hello, Attached is a patch that can (if you #define PROC_TITLES in config.h) give you some useful information about imap and pop clients. Instead of giving you the normal information: 20842 ?S 0:00 ./dbmail-imapd 20843 ?S 0:00 ./dbmail-imapd 20844 ?S 0:00 ./dbmail-imapd when you run 'ps ax' (or similar), you get: 21480 ?S 0:00 dbmail/imap4d : Idle 21481 ?S 0:00 dbmail/imap4d : USER root [127.0.0.1] 21482 ?S 0:00 dbmail/imap4d : USER klas [200.12.11.13] I find it quite useful when I'm tracing errors for different users. (I've patched the 2002-12-06 cvs version) Regards /Klas [EMAIL PROTECTED] process_titles.patch Description: Binary data
Re: [Dbmail] DBMAIL_USE_SAME_CONNECTION?
Hi Klas, I really wanted to have one db-connection per imap/pop process for the 1.0 release; i'm afraid the job wasn't finished for different authentication systems. Extra code will be implemented soon and the #define will be set as a compile time option. Klas Handel heeft op donderdag, 5 dec 2002 om 22:49 (Europe/Amsterdam) het volgende geschreven: Hi, Does anyone know how I should use DBMAIL_USE_SAME_CONNECTION, if I don't want to use the same connection? If I just remove the #define of DBMAIL_USE_SAME_CONNECTION things are not working. Am I supposed to write the extra code needed myself or? I can't really find any support for the use of two connections anywhere? Or am I missing something obvious? Thanks in advance! /Klas [EMAIL PROTECTED] ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] DB replication patch
Hi Klas, we are currently looking out for some kind of replication - your way of doing it is interesting. However I would prefer a system where the replication is completely transparent for dbmail mainly because of postgresql/other database support: one of the main objectives of dbmail is complete freedom for database choice as fas as dbmail is concerned. Implementing this patch would break the symmetry giving extra functionality only available with mysql. Furthermore could you be more specific on your plan of recovery if one of the machines crashes? Seems to me that you would have to rebuild your dbmail system setting it up for 1 database. Our plan for the moment is to have two databases: one used in production and the other as hot spare. When the live machine is used, the succesfull update/insert/delete queries are performed at the hot spare as well (by means of some script running in the background). At failure, all there is to do is change dbmail.conf and send a SIGHUP to the daemons. This system has one major drawback: if the update/insert/delete query fails on the hot spare but already has succeeded on the production machine. We haven't figured this out completely but then again: if the query fails probably something's really wrong with the hot spare. What do you folks think about it? Klas Handel heeft op vrijdag, 6 dec 2002 om 00:55 (Europe/Amsterdam) het volgende geschreven: Hi, We're using dbmail together with MySQL. Since our database is quite large and we have a lot of users we want to spread our load a little bit. We have setup our database in replication (master/slave) mode. Then what we want to do is force all read (select) activities to our slave (slaves) and all write (delete, update and insert) to our master database. I've created a patch which does this (only MySQL). What you need to do is #define the READ_DATABASE parameter in "db.h". Then there are some addition parameters to be set in "dbmail.conf" (read_db, read_user etc.). If READ_DATABASE isn't defined dbmail should work as normal, i.e use only one database. I did create a patch earlier and sent it to the list, but it's not in the new release so it's probably nothing that people are interested in? Still we are using it so I send it again... (The patch version is the cvs from 2002-12-05) Have a nice day! /Klas [EMAIL PROTECTED]
Re: [Dbmail] Process title patch
Hi klas, like that idea, i patched my version here but it does not show anything new in the process list. I'm running OS X - do you have any idea how portable your code is? regards roel Klas Handel heeft op vrijdag, 6 dec 2002 om 06:03 (Europe/Amsterdam) het volgende geschreven: Hello, Attached is a patch that can (if you #define PROC_TITLES in config.h) give you some useful information about imap and pop clients. Instead of giving you the normal information: 20842 ?S 0:00 ./dbmail-imapd 20843 ?S 0:00 ./dbmail-imapd 20844 ?S 0:00 ./dbmail-imapd when you run 'ps ax' (or similar), you get: 21480 ?S 0:00 dbmail/imap4d : Idle 21481 ?S 0:00 dbmail/imap4d : USER root [127.0.0.1] 21482 ?S 0:00 dbmail/imap4d : USER klas [200.12.11.13] I find it quite useful when I'm tracing errors for different users. (I've patched the 2002-12-06 cvs version) Regards /Klas [EMAIL PROTECTED]
Re: [Dbmail] Process title patch
Well i spotted the problem - if you give the '-f ' option, the filename is overwritten by the new functionality. Could you find a way to overcome this? Furthermore i was wondering if there is a sizelimit on the command line and wheter you are checking against it - wouldn't like segfaults because of a long username after all. regards roel Roel Rozendaal - IC&S heeft op vrijdag, 6 dec 2002 om 11:00 (Europe/Amsterdam) het volgende geschreven: Hi klas, like that idea, i patched my version here but it does not show anything new in the process list. I'm running OS X - do you have any idea how portable your code is? regards roel Klas Handel heeft op vrijdag, 6 dec 2002 om 06:03 (Europe/Amsterdam) het volgende geschreven: Hello, Attached is a patch that can (if you #define PROC_TITLES in config.h) give you some useful information about imap and pop clients. Instead of giving you the normal information: 20842 ?S 0:00 ./dbmail-imapd 20843 ?S 0:00 ./dbmail-imapd 20844 ?S 0:00 ./dbmail-imapd when you run 'ps ax' (or similar), you get: 21480 ?S 0:00 dbmail/imap4d : Idle 21481 ?S 0:00 dbmail/imap4d : USER root [127.0.0.1] 21482 ?S 0:00 dbmail/imap4d : USER klas [200.12.11.13] I find it quite useful when I'm tracing errors for different users. (I've patched the 2002-12-06 cvs version) Regards /Klas [EMAIL PROTECTED] ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] Database types -- and Exchange Calendar support
About an eventual plan of Exchange compatibiliy, some ideas : - eventual calendars management could follow some works of ietf group described in rfcs 2445, 2446 & 2447. Unfortunately, Microsoft stopped to work on it (one person was on the working group of those texts) and then Exchange use some proprietary protocol and fomat with MAPI. Nowadays, there is no freeze of this work, and a recent draft as been refund : http://www.ietf.org/internet-drafts/draft-ietf-calsch-cap-09.txt . But some particular initiatives had begun : MCAL used in PHP ( http://www.php.net/manual/en/ref.mcal.php ) to acces ICAP servers (for instance), libical http://www.softwarestudio.org/projects/FreeAssociation/ API for ICAP project... - well if it is not possible to be exactly compatible with exchange, dbmail can offer some equivalent services (IMAP, ICAP, LDAP...) with db storing. The most important is that clients must follow (ie. Evolution, Kalendar or iCal should have ICAP connectors). - as dbmail stores maildatas on such databases as MySQL, PostgreSQL and maybe SAPdb, we can imagine (if ever dbmail will run on win32 systems) to store datas on a MAPI database... OK, it seems to be crazy but... using dcom access on Exchange (very easy) we can read and store mail, calendar, note, contact and so on... So with a layer between dbmail and exchange, this could be. Very nice for those that want to access to calendars without THE exchange client (outlook) or the Ximian connector but with every user agent that can follow a standard protocol ! dcom calls are possible for every version of Exchange (from 5.0 to 2000) ! --8< extract of rfc3283 : 1.2 Concepts and Relationships iCalendar is the language used to describe calendar objects. iTIP describes a way to use the iCalendar language to do scheduling. iMIP describes how to do iTIP scheduling via e-mail. CAP describes a way to use the iCalendar language to access a calendar store in real- time. The relationship between calendaring protocols is similar to that between e-mail protocols. In those terms, iCalendar is analogous to RFC 2822, iTIP and iMIP are analogous to the Simple Mail Transfer Protocol (SMTP), and CAP is analogous to the Post Office Protocol (POP) or Internet Message Access Protocol (IMAP). --8< interessant, isn't it ? Pascal
Re: [Dbmail] DB replication patch
On Fri, 2002-12-06 at 03:31, Roel Rozendaal - IC&S wrote: > Hi Klas, > > Our plan for the moment is to have two databases: one used in > production and the other as hot spare. When the live machine is used, > the succesfull update/insert/delete queries are performed at the hot > spare as well (by means of some script running in the background). At > failure, all there is to do is change dbmail.conf and send a SIGHUP to > the daemons. This system has one major drawback: if the > update/insert/delete query fails on the hot spare but already has > succeeded on the production machine. We haven't figured this out > completely but then again: if the query fails probably something's > really wrong with the hot spare. What do you folks think about it? > Well, for innodb or bdb backed mysql and postgres, you could do it with transactions theoretically, transaction the update/insert/delete statements on the master, then do them on the hot spare, if it fails, roll back, if not, commit. Performance would be the question here. For MyISAM users, another method would have to be devised. -- Ryan Butler <[EMAIL PROTECTED]> ADI Internet Solutions
Re: [Dbmail] Bugfix: SMTP signal 11
On Thursday 05 December 2002 8:32 am, Andreas Richter wrote: > Hi, > I noticed that there is an extra exit(1) in serverchild.c. Will this fix > the pop3d SEGV crash? I am including a more complete list of the Trace > output that leads to and covers the crash. > Thanks, > Andy Where is this extra exit(1) at? I have not been able to use dbmail-1.0 reliably for pop3 connections. I would get random users not able to connect and running out of child processes. Is this what you are seeing as well? Thanks. -- Bret Baptist Systems and Technical Support Specialist [EMAIL PROTECTED] Internet Exposure, Inc. http://www.iexposure.com (612)676-1946 x17 Web Development-Web Marketing-ISP Services -- Today is the tomorrow you worried about yesterday.
Re: [Dbmail] Error in dbmail-imapd
See the patch below that Klas patched yesterday. It fixes this problem. The
bug occurs in quoted_string_out in imaputil.c which quotes strings. Quite a
number of email clients (like outlook express) doesn't like how dbmail was
quoting it. This meant outlook express wouldn't show them.
There is still a bug in show_address_list in imaputil.c in the line saying
fprintf(outstream,"%c",mr->value[i]); where mr-value[i] can be a "special
character" (but as we have already put a " at the beginning of this quoting
we cannot go {1}\015\012%s), But Klas is working on it..
Thanks
Craig
int quoted_string_out(FILE *outstream, const char *s)
{
int i,cnt;
for (i = 0; i < strlen(s); i++) { // check if must use literal
if (!(s[i] & 0xe0) || (s[i] & 0x80) || (s[i] == '"') || (s[i] == '\\'))
{
cnt = fprintf(outstream, "{");
cnt += fprintf(outstream, "%i", strlen(s));
cnt += fprintf(outstream, "}\015\012");
cnt += fprintf(outstream, "%s", s);
return cnt;
}
}
cnt = fprintf(outstream, "\"");
cnt += fprintf(outstream, "%s", s);
cnt += fprintf(outstream, "\"");
return cnt;
- Original Message -
From: "Sam Przyswa" <[EMAIL PROTECTED]>
To:
Sent: Friday, December 06, 2002 3:06 AM
Subject: Re: [Dbmail] Error in dbmail-imapd
> Roel Rozendaal - IC&S ([EMAIL PROTECTED]) écrivait:
> >
> >Well that's an outlook thing. Folders are imap stuff, the messages are
> >just stored and retrieved without any intervention.
>
> On Outlook Express we use the "headers only" to only store the messages
header not
> the whole message. The same mail on Courier-IMAP appear without problem in
Outlook
> Express ?!
>
> >Sam Przyswa heeft op woensdag, 4 dec 2002 om 11:50 (Europe/Amsterdam)
> >het volgende geschreven:
> >> This bug appear only with Outlook 2000, we have found an other bug
> >> with Outlook
> >> Express, if you have a mail with accentued characters in the Subject:
> >> field
> >> as "éèà" this mail don't appear in Outlook Express, we don't know why
> >> ???
>
> Sam.
>
> ___
> Dbmail mailing list
> Dbmail@dbmail.org
> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>
>
Re: [Dbmail] DB replication patch
Hi, I must say that Klas's approach seems to be the best to me. MySQL already has very acceptable support for replication and it would be foolish to rely on a script based solution that will never work half as well. PostgreSQL also has one or two replication implementations/projects if I am not mistaken. From the DBmail point of view, the only thing that changes is that we use one connection for READ operations and another for WRITE operations, this could be implemented for each db backend if necessary.. Only queries that succeed on the master server are ran on the slave. If a query that succeeded on the master fail on the slave, replication will be halted until the problem is manually resolved I think. Cedric Le 6 Décembre 2002 04:31, Roel Rozendaal - IC&S a écrit : > Hi Klas, > > we are currently looking out for some kind of replication - your way of > doing it is interesting. However I would prefer a system where the > replication is completely transparent for dbmail mainly because of > postgresql/other database support: one of the main objectives of dbmail > is complete freedom for database choice as fas as dbmail is concerned. > Implementing this patch would break the symmetry giving extra > functionality only available with mysql. > Furthermore could you be more specific on your plan of recovery if one > of the machines crashes? Seems to me that you would have to rebuild > your dbmail system setting it up for 1 database. > > Our plan for the moment is to have two databases: one used in > production and the other as hot spare. When the live machine is used, > the succesfull update/insert/delete queries are performed at the hot > spare as well (by means of some script running in the background). At > failure, all there is to do is change dbmail.conf and send a SIGHUP to > the daemons. This system has one major drawback: if the > update/insert/delete query fails on the hot spare but already has > succeeded on the production machine. We haven't figured this out > completely but then again: if the query fails probably something's > really wrong with the hot spare. What do you folks think about it? > > > Klas Handel heeft op vrijdag, 6 dec 2002 om 00:55 (Europe/Amsterdam) > > het volgende geschreven: > > Hi, > > > > We're using dbmail together with MySQL. Since our database is quite > > large and we have a lot of users we want to spread our load a little > > bit. We have setup our database in replication (master/slave) mode. > > Then what we want to do is force all read (select) activities to > > our slave (slaves) and all write (delete, update and insert) to our > > master database. > > I've created a patch which does this (only MySQL). What you need to do > > is #define the READ_DATABASE parameter in "db.h". Then there are some > > addition parameters to be set in "dbmail.conf" (read_db, read_user > > etc.). > > If READ_DATABASE isn't defined dbmail should work as normal, i.e use > > only one database. > > > > I did create a patch earlier and sent it to the list, but it's not in > > the > > new release so it's probably nothing that people are interested in? > > Still we are using it so I send it again... > > > > (The patch version is the cvs from 2002-12-05) > > > > Have a nice day! > > /Klas > > [EMAIL PROTECTED] > > > > > > ___ > Dbmail mailing list > Dbmail@dbmail.org > https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] DB replication patch
Hi Cedric, This will result into inconsistencies. I'll explain by example. Server 1 is the read-only server, server 2 is the write-only server. The imap server will do a select on a folder through server 1. The user deletes a few messagess from this folder (which will be sent as an update to server 2) and, because the imap server _will_ be faster then the replication process, the folder will be reselected with messages that are in fact deleted. After that, selecting those messages again they will result in error messages. This will only work when talking to a real database cluster. In that case you're talking to one ip address and the cluster will handle the query in the cluster, handling it's own replication. In my opinion Mysql and Postgresql are just not ready yet for real replication / clustering. We use replication to have a hot backup.. that's in my eyes the only save usage of mysql / postgresql replication. Best regards, Eelco On vrijdag, dec 6, 2002, at 21:39 Europe/Amsterdam, Cedric Veilleux wrote: Hi, I must say that Klas's approach seems to be the best to me. MySQL already has very acceptable support for replication and it would be foolish to rely on a script based solution that will never work half as well. PostgreSQL also has one or two replication implementations/projects if I am not mistaken. From the DBmail point of view, the only thing that changes is that we use one connection for READ operations and another for WRITE operations, this could be implemented for each db backend if necessary.. Only queries that succeed on the master server are ran on the slave. If a query that succeeded on the master fail on the slave, replication will be halted until the problem is manually resolved I think. Cedric Le 6 Décembre 2002 04:31, Roel Rozendaal - IC&S a écrit : Hi Klas, we are currently looking out for some kind of replication - your way of doing it is interesting. However I would prefer a system where the replication is completely transparent for dbmail mainly because of postgresql/other database support: one of the main objectives of dbmail is complete freedom for database choice as fas as dbmail is concerned. Implementing this patch would break the symmetry giving extra functionality only available with mysql. Furthermore could you be more specific on your plan of recovery if one of the machines crashes? Seems to me that you would have to rebuild your dbmail system setting it up for 1 database. Our plan for the moment is to have two databases: one used in production and the other as hot spare. When the live machine is used, the succesfull update/insert/delete queries are performed at the hot spare as well (by means of some script running in the background). At failure, all there is to do is change dbmail.conf and send a SIGHUP to the daemons. This system has one major drawback: if the update/insert/delete query fails on the hot spare but already has succeeded on the production machine. We haven't figured this out completely but then again: if the query fails probably something's really wrong with the hot spare. What do you folks think about it? Klas Handel heeft op vrijdag, 6 dec 2002 om 00:55 (Europe/Amsterdam) het volgende geschreven: Hi, We're using dbmail together with MySQL. Since our database is quite large and we have a lot of users we want to spread our load a little bit. We have setup our database in replication (master/slave) mode. Then what we want to do is force all read (select) activities to our slave (slaves) and all write (delete, update and insert) to our master database. I've created a patch which does this (only MySQL). What you need to do is #define the READ_DATABASE parameter in "db.h". Then there are some addition parameters to be set in "dbmail.conf" (read_db, read_user etc.). If READ_DATABASE isn't defined dbmail should work as normal, i.e use only one database. I did create a patch earlier and sent it to the list, but it's not in the new release so it's probably nothing that people are interested in? Still we are using it so I send it again... (The patch version is the cvs from 2002-12-05) Have a nice day! /Klas [EMAIL PROTECTED] ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail _ E.J.A. van Beek ICT Manager IC&S T: +31 30 2322878 F: +31 30 2322305 PGP-key: www.ic-s.nl/keys/eelco.txt
Re: [Dbmail] Error in dbmail-imapd
Craig Whitmore ([EMAIL PROTECTED]) écrivait: > >See the patch below that Klas patched yesterday. It fixes this problem. The >bug occurs in quoted_string_out in imaputil.c which quotes strings. Quite a >number of email clients (like outlook express) doesn't like how dbmail was >quoting it. This meant outlook express wouldn't show them. > >There is still a bug in show_address_list in imaputil.c in the line saying >fprintf(outstream,"%c",mr->value[i]); where mr-value[i] can be a "special >character" (but as we have already put a " at the beginning of this quoting >we cannot go \015\012%s), But Klas is working on it.. Yes, that's work fine now, thanks to you and to Klas ! Sam.
Re[2]: [Dbmail] DB replication patch
CV> Only queries that succeed on the master server are ran on the CV> slave. If a query that succeeded on the master fail on the slave, CV> replication will be halted until the problem is manually resolved CV> I think. Very true. Unfortunately, there are a number of "shouldn't fail" queries that will kill replication. My pet peeve with MySQL replication is that ALTER TABLE queries will NOT replicate, so the first query after one that references a changed field with shut down replication. THIS IS NOT DOCUMENTED, as far as I've found. One of my clients does this regularly; I have to figure out WHAT they changed, in the order they changed it, and propagate the changes to the two slaves. This week, he modified one table five times, deleted it, recreated it with changes, then modified it twice again... I had to restart the slave process several times, so that it could move up to the next modification query. I have a CRON job that emails me the status of replication on all servers every hour... -- Jeff Brenton President, Engineered Software Products, Inc http://espi.com Questionable web page: http://dididahdahdidit.com Liberalism grants you the freedom to advocate any idea*. * Please see http://www.dididahdahdidit.com/except.php for a current list of exceptions
