RE: PostgreSQL backend: a waste of time?
-Original Message- From: Nuno Silva [mailto:[EMAIL PROTECTED]] Sent: Saturday, November 30, 2002 2:50 AM To: Nicola Ranaldo Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: PostgreSQL backend: a waste of time? Hello! Nicola Ranaldo wrote: I cannot spread sql commands of a unique transaction over multiple pgsql connection, and a connection cannot handle parallel transactions. So if i have 1000 imapd process starting a transaction the mailbox daemon has to open 1000 pgsql connection. Reading from the DB should be trivial, right? I'm not 100% sure, but I suppose that one can virtualize the connections. What I mean is: imapd (or pop3d or lmtpd...) wants to write something - ask the daemon and the daemon will choose a free connection and commit those changes. This is the one operation simple case. Some of the DBs that cyrus maintains appear to be this simple (the mailboxes file). Other cyrus' DBs seem to require transactions (seen and delivery DB's).. This makes it harder to manage with a single daemon connecting to a RDBMS with onky a few connections. CMU people or Ken: comments? :) One solution could be: BEGIN - Allocate a new buffer to store sql commands SQL COMMANDS - add commands to buffer COMMIT - send all the buffered commands to the mailbox daemon and cleanup the buffer ABORT - cleanup the buffer Do you think this is a good solution ? I really don't know... This buffer is in the daemon? Don't you have to receive responses from the SQL DB? Or these commands are only writes (UPDATE, INSERT)? If these are only writes it seams a good ideia, but if you need to SELECT (inside the transaction) too there is the problem of different connections getting different transactions. Brasseur Valéry posted a mysql-backend patch for cyrus recently. Brasseur, do you use a mysql connection for each cyrus process too? And do you use transactions? Last time I checked mysql didn't support these. the patch is using a connection per process, but I am considering usibg mbdeamon (look at Nuno's mail from 27/11) modify to use the mysql back-end. in this case I will have a few (one for now) connection du Mysql, the mbdaemon will handle all the processes connection. for the transaction part. I use mysql 4.0 which support transaction now. But I don't use this for handling transaction in the code !!! If transactions aren't required than it's easy to have 1 connection shared amongst 100 processes, right? :) Howewer i think a pgsql connection for every master child could not be a problem, on my production server (7500 very active users, cyrus.log is 20MB/day) the average number of imapd is 15, pop3d is 30, lmtpd is 5 (under mail-bombing lmtpd process was 45). Howewer it is an AlphaServer ES45 with 4 1ghz CPU and 700Gb of raid disk and is quite fast. Wath's your experience with huge number of users or slow server ? It depends on you user base: If your system is a backend for a webmail, for instance, your users (the php or perl script) will always connect, fetch something, disconnect. In this situation you'll never see lots of simultaneous connections. If you have 50.000 users on a campus setup using IMAP you'll get 5000 concurrent connection easily. The same way if you have a company with 500 desktops all of them checking the email with IMAP you can easily get 1000 (the double) concurrent connections. As internet people says, YMMV :) Regards, Nuno Silva Bye Nicola Ranaldo IIRC someone implemented such a daemon and patched cyrus to use it. This daemon's backend was a text-file but the protocol is there. a drawing :) imapd1 imapd2 imapd3 ... imap1000 ||| | --- | daemon | --- | | | | | | | 1 2 3 4 5 6 7 1 to 7 would be postgresql connections. This number may vary... maybe 1 connection per 100 imapds? Or a user defined number. IMMV...
9800¶àÍòÈ«¹úÓÊÖ·×Ü¿â+100¶à¿îÍøÂçÓªÏúÈí¼þ¹âÅÌ,Ö»ÐèÒª50Ôª
Title: 9800¶àÍòÈ«¹úÓÊÖ·×Ü¿â 9800¶àÍòÈ«¹úÓÊÖ·×Ü¿â+100¶à¿îÓªÏúÈí¼þ¹âÅÌ Ö»ÐèÒª50Ôª ÏêÇéµã»÷ ±¾Õ¾¾¹ý½üÒ»ÄêµÄËѼ¯ÕûÀí¡¢·´¸´²âÊÔÑéÖ¤,ÏÖ¹²ÕûÀí9800¶àÍò¸ö¹úÄÚÓû§ÓʼþµØÖ·,¾×¨Òµ¹¤¾ßУÑéɸѡȥ³ýÖظ´ÎÞЧ,²¢Í¨¹ýÓʼþ¹ÜÀí¹¤¾ßÔÚÏßÑéÖ¤,²¢°´ÐÐÒµºÍµØÇø·ÖÀàºÃ.È»ºóÎÒÃÇÔÙ×Ðϸ½«ÓÊÖ·¿â×ö³É¹âÅÌ,·²¹ºÂò±¾²úÆ·Õß,¿ÉÒÔÃâ·ÑµÃµ½100¶à¿îÍøÂçÓªÏúÈí¼þ(°üÀ¨Èº·¢¡¢ËÑË÷¡¢ÑéÖ¤µÈ¹¤¾ß,¾ù¿ÉÕý³£Ê¹ÓÃ,Óиö±ðÐè×¢²á)£¬²¢ÔÚδÀ´ÊýÄêÄڵõ½±¾Õ¾µÄ¼¼ÊõÖ§³ÖºÍÐÂÓʼþµØÖ·µÄÃâ·ÑÔùËÍ¡£±¾Õ¾ÌṩµÄÓʼþµØÖ·¿â²úÆ·¾ßÓм«ºÃµÄÖÊÁ¿£¬¼«µÍµÄ¼Û¸ñ£¬ÍêÉƵÄÊÛºó·þÎñ£¬Êܵ½¿Í»§µÄÒ»ÖºÃÆÀ¡£°ÑÄ¿¹âͶÏòÎÒÃǵIJúÆ·£¬ÊÇÄúÃ÷ÖǵÄÑ¡Ôñ£¬ÎÒÃǽ«ÎªÄúÌṩ×îÓŵķþÎñ¡£ ¹âÅ̼۸ñ£º50Ôª Á¢¼´¶¨¹º ¸¶¿î·½Ê½ Èç¹ûÕâ·âÓʼþ´òÈÅÄúÁË£¬·³ÇëËæÊÖɾµô£¬²¢Çë¼ûÁ¡£ÈôÄú²»Ï£ÍûÔÙ´ÎÊÕµ½ÎÒÃǵÄÓʼþ£¬Çëµã»÷ÕâÀï
Cannot see inbox messages
Title: Message Hi! I am extremely new to cyrus and I have the following technology stack. I'm using Sendmail as an MTA and Cyrus as the POP/IMAP server. From the Pop side my messages come in perfectly. I imagine this is being done by copying to /home/$USERNAME/mbox from /var/mail/$USERNAME. I've tested the IMAP piece by setting up a new account on an IMAP compatible email client and it seems to be connecting. Ive created and deleted mailboxes and it seems to be ok. I've even installed SquirrelMail and noticed my newly created mailboxes. The only problem is I can't see anyincoming mail! There is an inbox but there doesn't appear to be a link to the /var/mail/$USERNAME or /home/$USERNAME/mbox. I've done some reasearch and noticed that there is a script out there called user2cyrus but apparently I am missing a Base 64 module. My first question is, am I missing a step in this whole process? If not, is there some step by step approach to convert my /var/mail/$USERNAMEs manually (There are only about 10)? I appreciate any help you can provide! :) Best Regards, Jason
problem with checkpoint of databases
Hi i m using the following cyrus version name : Cyrus IMAPD version: v2.1.5 2002/06/24 19:13:27 vendor : Project Cyrus support-url: http://asg.web.cmu.edu/cyrus os : Linux os-version : 2.4.18-5 environment: Cyrus SASL 2.1.5 Sleepycat Software: Berkeley DB 3.3.11: (July 12, 2001) OpenSSL 0.9.6b [engine] 9 Jul 2001 CMU Sieve 2.2 TCP Wrappers lock = flock auth = unix idle = poll mboxlist.db = db3 subs.db = flat seen.db = flat duplicate.db = db3-nosync tls.db = db3-nosync and i have from time to time big problems with checkpoint Indeed for a reason i can't determine the process ctl_cyrusdb -c never finish. As this process is launched every 30 minutes i got several dozens of that process before becoming aware of that problem. But the worse happens when i have to restart the server. Indeed as the checkpoint didn't succeed there were a lot of file named log.000xxx and the server become unavailable a long time waiting for the command ctl_cyrusdb -r to finish. it takes around one hour per day of failure of checkpointing to restart. -Is there a way to speedup that process? -will the changing the format of the database help to solve that problem? -what could cause the checkpoint to fail? Thanks in advance for any help. -- Eric Doutreleau I.N.T | Tel : +33 (0) 160764687 9 rue Charles Fourier | Fax : +33 (0) 160764321 91011 Evry France | email : [EMAIL PROTECTED]
Re: 9800XXXXXXXXXXXXXXXX+100XXXXXXXXXXXXXXXXXXXX,XXXXXX50XX
Title: 9800¶àÍòÈ«¹úÓÊÖ·×Ü¿â What kind of anti-spam is this list using? It doesn't seem to be working too well. - Original Message - From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, January 05, 2000 1:08 PM Subject: 9800+100,XX50XX 9800¶àÍòÈ«¹úÓÊÖ·×Ü¿â+100¶à¿îÓªÏúÈí¼þ¹âÅÌ Ö»ÐèÒª50Ôª ÏêÇéµã»÷
Re: Shared folders and virtual domains ?
Ken Murchison wrote: Christian Schulte wrote: Hi, I am running 2_2 cvs branch with virtual domain support turned on and everything seemd to work fine. I now wanted to move my old installation to the new one and cannot get delivery to shared folders working. If I create a shared folder with cyradm like: $cm sharedfolder I cannot do $sam sharedfolder user@domain lrswipcda and get setaclmailbox: user@domain: lrswipcda: Invalid identifier If I create a shared folder with cyradm like: $cm sharedfolder@domain I can do $sam sharedfolder@domain user@domain lrswipcda and the user can subscribe to the folder and sees it on the same level than his inbox as expected. If I now setup sendmail to send via the cyrusv2 mailer with an address like +sharedfolder@domain I get the following errors in the logs which I do not understand ! What is wrong here ? Nov 15 02:55:33 mail lmtpunix[8259]: [ID 921384 local6.debug] accepted connection Nov 15 02:55:33 mail lmtpunix[8259]: [ID 685068 local6.debug] lmtp connection preauth'd as postman Nov 15 02:55:33 mail lmtpunix[8259]: [ID 152585 local6.error] couldn't create stage directory: : No such file or directory Nov 15 02:55:33 mail lmtpunix[8259]: [ID 519036 local6.error] IOERROR: creating message file 8259-1037325333: No such file or directory Nov 15 02:55:33 mail sendmail[8262]: [ID 801593 mail.info] gAF1rq13008256: to=+sharedfolder@domain, delay=00:01:41, xdelay=00:00:00, mailer=cyrusv2, pri=210378, relay=localhost, dsn=4.2.0, stat=Deferred: 451 4.3.2 cannot create temporary file: No such file or directory Sorry for the delay, but I finally got a chance to look into this. Cyrus isn't the problem here, the problem is that the MTA is stripping the domain off of the recipient address when it gets passed to lmtpd. Try changing the cyrusv2 mailer definition to use: S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP Does not work either! I had S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP/HdrToSMTP in my cyrusv2.m4 file and changing it to S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP produces the same error! sendmail delivers correctly to lmtpd, I think: 20776 === CONNECT localhost 20776 220 LMTP Cyrus v2.2.prealpha ready 20776 LHLO 20776 250-XXX 20776 250-8BITMIME 20776 250-ENHANCEDSTATUSCODES 20776 250-PIPELINING 20776 250-SIZE 20776 250-AUTH EXTERNAL 20776 250 IGNOREQUOTA 20776 MAIL From:[EMAIL PROTECTED] SIZE=1076 BODY=8BITMIME 20776 250 2.1.0 ok 20776 RCPT To:+sharedfolder@domain 20776 DATA 20776 250 2.1.5 ok 20776 451 4.3.2 cannot create temporary file: No such file or directory 20776 QUIT 20776 221 2.0.0 bye 20776 [EOF] And the logfile states the same errors ! What makes me a bit confused is the error message itself. lmtpd is trying to create a temporary file but the error is No such file or directory. Is it a missing directory or wrong permissions on a directory ?
Re: Shared folders and virtual domains ?
On 3 Dec 2002, Christian Schulte writes: 20776 MAIL From:[EMAIL PROTECTED] SIZE=1076 BODY=8BITMIME 20776 250 2.1.0 ok 20776 RCPT To:+sharedfolder@domain 20776 DATA 20776 250 2.1.5 ok 20776 451 4.3.2 cannot create temporary file: No such file or directory And the logfile states the same errors ! What makes me a bit confused is the error message itself. lmtpd is trying to create a temporary file but the error is No such file or directory. Is it a missing directory or wrong permissions on a directory ? No such file or directory is a standard Unix error text for a single error code, ENOENT, defined in /usr/include/asm/errno.h on my Linux boxes. Don't read too much into the or directory part if the code concerned is not trying to work with a directory :-) I think this issue may be a version of something I tried to report some weeks back, which someone (Ken? I forget) thought might be hardware related, and then found a workaround for that was 'good enough' for me for the moment. The issue for me was that (based on my experiments) the default domain should not be passed to LMTP, only the other non-default ones. Any other approach generated the error message you are reporting. I suspect that had your RCPT TO: line read 20776 RCPT To:+sharedfolder it would have been accepted just fine! I do not know why that is -- I suspect some form of attempted backward compatibility? What I did to fix this here was a small, quick and slightly ugly patch to the sendmail sendmail-cf/m4/proto.m4 file, causing it to send the @domain part of the recipient address on to LMTP for all domains *except* the default one. I had every intention of going back and really figuring out what is going on well enough to report it in a clear way that makes more sense to the developers at some stage... but so far I have not found (or made?) time to do so. In case it helps, here is my patch. It helped for my situation. Jonathan --- /usr/share/sendmail-cf/m4.orig/proto.m4 Thu Jun 13 11:53:24 2002 +++ /usr/share/sendmail-cf/m4/proto.m4 Thu Oct 3 15:11:10 2002 @@ -1092,8 +1092,10 @@ dnl $H empty (but @$=w.) R $+ + $* $+ $#_LOCAL_ $: $1 + $2plussed name? R $+ $+ $#_LOCAL_ $: @ $1 nope, local address', -`R$=L @ $=w . $#_LOCAL_ $: @ $1 special local names -R$+ @ $=w . $#_LOCAL_ $: $1 regular local name') +`R$+ @ $m . $#_LOCAL_ $: $1 regular local +name default domain') +R$+ @ $j . $#_LOCAL_ $: $1 regular local name +default host') +R$=L @ $=w .$#_LOCAL_ $: @ $1`@'$2 special local names +R$+ @ $=w . $#_LOCAL_ $: $1`@'$2regular local +name') ifdef(`_MAILER_TABLE_', `dnl # not local -- try mailer table lookup -- Jonathan Marsden| Internet: [EMAIL PROTECTED] | Making electronic 1252 Judson Street | Phone: +1 (909) 795-3877 | communications work Redlands, CA 92374 | Fax: +1 (909) 795-0327 | reliably for Christian USA | http://www.xc.org/jonathan| missions worldwide
Re: Cannot see inbox messages
Hi! I am extremely new to cyrus and I have the following technology stack. I'm using Sendmail as an MTA and Cyrus as the POP/IMAP server. From the Pop side my messages come in perfectly. I imagine this is being done by copying to /home/$USERNAME/mbox from /var/mail/$USERNAME. I've tested the IMAP piece by setting up a new account on an IMAP compatible email client and it seems to be connecting. Ive created and deleted mailboxes and it seems to be ok. I've even installed SquirrelMail and noticed my newly created mailboxes. The only problem is I can't see any incoming mail! There is an inbox but there doesn't appear to be a link to the /var/mail/$USERNAME or /home/$USERNAME/mbox. I've done some reasearch and noticed that there is a script out there called user2cyrus but apparently I am missing a Base 64 module. My first question is, am I missing a step in this whole process? If not, is there some step by step approach to convert my /var/mail/$USERNAMEs manually (There are only about 10)? I appreciate any help you can provide! :) Best Regards, Jason Cyrus imapd has it's own message store with its own format. It is not interested in /var/mail/$USERNAME or /home/$USERNAME/mbox. To transfer mails, there are some tools around which you can use to transfer mails. IIRC one of them is 'formail'. Check this lists archives for more info. It has been discussed more than once. Simon
