Re: API to write to lmtp directly
Zitat von Ramprasad A Padmanabhan <[EMAIL PROTECTED]>: > > It is not that my server is getting loaded up because of the mails. > After I limt the number of concurrent lmtp connections I am able to > reduce the db lock errors to a great extent. Now I have set max lmtp > processes as 100 , But what will happen if my user mailboxes grow to > 4000 from todays 850 then the mails will be delayed forever. By the time > there will be the next batch to process. > > So I thought if there was a way of directly writing to lmtp and to all > mailboxes at the same time I will do better > You have not read my previous posts? - Reduce the concurrency of lmtp to a low number (2 - 4) because for delivering to one Storage it gains you nothing to use high concurrency. - Don´t use the "local" transport from postfix because with this you get one recipient per mail deliver. Use transport maps and virtual maps for aliase expansion to get multiple recipients per mail. Regards Andreas
Re: single message store
Zitat von Chris Picton <[EMAIL PROTECTED]>: > Hi I am running postfix with cyrus and trying to enable single message > store. It isnt working, though. What version of postfix (postconf -d | grep version) > In postfix main.cf, I have: > local_destination_concurrency_limit = 20 You should not do that. This means up to 20 concurrent connections for local submission. > default_destination_concurrency_limit = 10 > > In imapd.conf, I have: > singleinstancestore: yes > > However, I don't get messages hard linked to each other in > /var/spool/imap. > > Where could I be going wrong? > > My postfix log for a sample mail says (note that the queue ID is the > same for all, so I am assuming that postfix is sending the same message > to multiple recipients via LMTP): > > Jul 1 11:02:29 kerberos postfix/lmtp[7670]: ECC39D4772: > to=<[EMAIL PROTECTED]>, > relay=/var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp], delay=1, > status=sent (250 2.1.5 Ok) > Jul 1 11:02:29 kerberos postfix/lmtp[7472]: ECC39D4772: > to=<[EMAIL PROTECTED]>, > relay=/var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp], delay=1, > status=sent (250 2.1.5 Ok) > Jul 1 11:02:29 kerberos postfix/lmtp[7667]: ECC39D4772: > to=<[EMAIL PROTECTED]>, > relay=/var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp], delay=1, > status=sent (250 2.1.5 Ok) > Jul 1 11:02:29 kerberos postfix/lmtp[7986]: ECC39D4772: > to=<[EMAIL PROTECTED]>, > relay=/var/lib/imap/socket/lmtp[/var/lib/imap/socket/lmtp], delay=1, > status=sent (250 2.1.5 Ok) You must use a transport map entry and bypass the "local" delivery. As you can see at the process-id new ltmp clients are spawned for every deliver. Regards Andreas
Re: Too many lmtpd processes running
Zitat von Scott Adkins <[EMAIL PROTECTED]>: > --On Thursday, June 26, 2003 10:57 PM +0200 [EMAIL PROTECTED] wrote: > > > Zitat von Rob Tanner <[EMAIL PROTECTED]>: > > > >> Hi, > >> > >> I've just brought up cyrus imap v2.1.13 and I've noticed that there are > >> over lmtpd processes running. Isn't that a bit excessive? It's starting > >> to bog down the server. > >> > > > > Limit the number of lmtp clients used by your MTA (Postfix/Sendmail ...) > > to some value your server can stand. Most of the time 2 are enough. > > > > Should go to the FAQ i guess. > > I don't think I can agree with your answer :-) It really depends on the > system and how much mail you pump through it. It also depdns on how you > do your email delivery. For instance, in our sendmail configuration, we > queue all incoming mail and use cron jobs to fire off queue runners to > process the various queues we have, with some queues running more often > than others. If we have 30 sendmail queue runners processing our Cyrus > queue, then we should expect to see 30 LMTP process as well, one for each > of the queue runners that could feed them mail. Don´t know about sendmail but the point is if you have one Cyrus partition massive parallel deliver gain nothing because all the lmtp fight for the same (rare) I/O resource and you get in trouble with locking at some point. This is true for MTA local/ MDA local scenario. If you deliver from slow MTA gateways (virusscanning) over network to one fast Cyrus MDA by lmtp running a lot of "lmtpd" makes sense. > As for the original question, you didn't actually state how many LMTP > processes you are seeing. We typically don't worry about it as long as > we are under 100 processes... when we go over that though, we start > looking at the processes (with LSOF) to see if there are a lot of procs > stuck on a single file (usually a cyrus.header file of a particular user). > This is a lock problem that we have been fighting with for a long time > (we are using 2.0.16) and end up restarting the server to fix it. You can fix this by running a lower number of lmtp-clients :-) > So, my suggestion is to look at the LMTP process you have, see if a lot > of them are blocked doing nothing (i.e. probably waiting for a file to > unlock). If they all look busy, then I wouldn't worry about it. > > Scott Regards Andreas
Re: Too many lmtpd processes running
Zitat von Rob Tanner <[EMAIL PROTECTED]>: > Hi, > > I've just brought up cyrus imap v2.1.13 and I've noticed that there are > over lmtpd processes running. Isn't that a bit excessive? It's starting > to bog down the server. > Limit the number of lmtp clients used by your MTA (Postfix/Sendmail ...) to some value your server can stand. Most of the time 2 are enough. Should go to the FAQ i guess. Regards Andreas
Re: Best way of putting a single file into multiple mailboxes
Zitat von Ramprasad A Padmanabhan <[EMAIL PROTECTED]>: > > I agree with you but my situation is a little different > > I must have explained my problem more clearly, I have got around only 10 > recepients per newsletter on an average and there are around 700 such > newsletters to be delivered. These are news articles people have > subscribed to accourding to interests > > Suppose Now I increase the number of lmtp_concurrent connections and if > multiple instances of ltmp are trying to deliver to the same mailbox > then these mailboxes will get locked , so again There will be lmtp > connections waiting. I havent tried out increasing the number of > concurrent connections per user because I want to be sure before I > change anything on the live server > Best way is to limit the number of "lmtp" clients in master.cf to something around 2 if you have a single mailstore (single disk/array). because massive parallel local deliver leed to locking problems and don´t speed things up. See Postfix main.cf for notes on parallel deliver for local mail!! Use transport map to deliver direkt to lmtp to get the advantage of single instance store. Regards Andreas
Re: Best way of putting a single file into multiple mailboxes
Zitat von Rob Siemborski <[EMAIL PROTECTED]>: > On Wed, 25 Jun 2003, Ramprasad A Padmanabhan wrote: > > > I am trying to deliver newsletters to multiple people who have accounts > > on my own server. > > I am sending it as mail to postfix and delivering to all mailboxes > > The problem is there are so many lmtp connections that cyrus dies. > > Especially When I have hundreds of such newsletters generated > > Postfix should have a configuration option to increase the max # of > recipients involved in an LMTP transaction. > > I believe it is something like > > _destination_recipient_limit. > > The default is 1, which will result in pretty poor performance. > > -Rob Not the whole truth. It also depends how you feed mails to lmtp. The default destination_recipient_limit is 50. Taken from Postfix mailing-list : >> >> Can postfix 2.0.6 using the local delivery agent and a mailbox >> transport to the Cyrus2.1.12 lmtp socket take advantage of single >> message store with alias expansion? >> > >Postfix can deliver email to cyrus mailboxes after first performing alias >expansion just as you describe via mailbox_transport. The words "single >message store" suggest that you are also looking for the ability to >deliver a single stored copy of a message to multiple recipients. This you >cannot do when using the local+mailbox_transport combination. The "local" >delivery agent delivers to one user at a time. > >It is possible to use virtual_alias_maps (Postfix 2.0) and transport table >entries with a RHS of "lmtp:..." to bypass "local" entirely. In this case >multi-recipient delivery will work, and it is still possible to support >address rewriting. Finally, some of the virtual_alias_maps entries might >map to "local" domains, which provide support for command aliases, ... So the baseline is : - Use transport maps with "lmtp:" entry to bypass "local" - Raise the "destination_recipient_limit" to more then 50 - Limit the number of concurrent lmtp clients in "master.cf" to something your server can handle Regards Andreas
Re: cyrus-mysql-postfix-pam
Zitat von Bartosz Jozwiak <[EMAIL PROTECTED]>: > Hello everyone! > > I just change e-mail server to Postfix MySql Cyrus Web-Cyradm > We have 4000 users. Sometimes user can log in sometimes not. > I have noticed that this is because of MySql problem. > There is to many connections to mysql. > So after 20 minutes we had to change to old e-mail server. > Our webmail was horde/imp > Did anybody had before that problem? > Is pam not making to many connections? > Please help me to solw this problem! > Not really a Cyrus issue. Raise your limit for MySQL connects. Use Postfix 2.0.x with Proxymap to lower the need for MySQL connects. Split your MySQL for authentication from the database for mailrouting and IMP. Regards Andreas
Re: Cyrus' performance
Zitat von Mike Lohmann <[EMAIL PROTECTED]>: > Hi, > > I would like to know the performance of a cyrus imap/pop3 server. > > I've got 2 users writing about 2 mails/day, whereby one mail has approx. > > 100KB. > > What will be, in your opinion, the estimated memory- and cpu usage per user, > > when 100 of them send their mails at the same time? > > (Is there a document which tells me what cyrus 'costs'?) > > Thanks. > > Mike > If you use IMAP with a lot of users online at the same time (inside LAN) be sure to get a lot of RAM (> 1GB) and tune your OS for many open files. Be sure to chose a fast I/O system (SCSI / FC) and a filesystem capable of many small files (XFS or something similar). Split of the MTA to an other machine if the user send most of time to "extern" addresses. Regards Andreas
Re: what is that meaning in pam_mysql ??
Zitat von Temp <[EMAIL PROTECTED]>: > when after I login imap account, there is so much logs, > and how to get rid of them ?? > I seach google, but no exact answer for it. > > Mar 14 08:55:19 mx saslauthd[13604]: pam_mysql: error: sqllog set but > logmsgcolumn not set > Mar 14 08:55:19 mx saslauthd[13604]: pam_mysql: error: sqllog set but > logusercolumn not set > Mar 14 08:55:19 mx saslauthd[13604]: pam_mysql: error: sqllog set but > loghostcolumn not set > Mar 14 08:55:19 mx saslauthd[13604]: pam_mysql: error: sqllog set but > logtimecolumn not set > Mar 14 08:55:19 mx saslauthd[13606]: pam_mysql: error: sqllog set but > logtable not set > A new feature in pam_mysql 0.5 You should set sqllog=0 in your pam directive. Regards Andreas
Re: pop3d hangs on slow connections with large attachements
Zitat von Ralf Hübl <[EMAIL PROTECTED]>: > Hi List, > > I have installed cyrus 2.1.9 on a SuSE 8.1 > IMAP and POP3 works very well on the local 100MB/s network. > > But some of my Clients have a slow dial up connections. > If this clients connect to the server via pop3 the pop3d hangs during the > download of attachments larger then aprox 100KB. > The clients software (M$/or not M$) dosn't matter MS Oulook / MS Oulook > Express / Netscape / PINE / fetchmail and so on > > Before I had upgraded, I used an cyrus 2.0.16 on a SuSE 7.3. > There was the same problem and the reason for my upgrade. > Please Remarke: the problem occures only on slow connections. > Why do you expect it is Cyrus to blame?? I would say your dial-up is the problem. Search around for problems with MTU if you have some firewall/router between. This is a connection issue. Regards Andreas
Re: To build open source based scalable, stable webmail server
Zitat von Erol Ozcan <[EMAIL PROTECTED]>: > Hi! > > I am looking for a scalable, stable and cost effective webmail > server solutions. The server system > > - should be cost effective and based on open source softwares as > possible ( could be commercial ) > - should be scalable up to 200,000 users, but 150,000 a must Redundant SMTP-Gateway per MX-records More than one IMAP-Server and access by some proxy like "perdition" Redundant user base setup with LDAP > - account creation or deletion should not exceed 2 seconds With scripting depends only from speed of your hardware > - could be support concurent 100 subscribers for webmail browsing on > the system without any performance problems Lot of Memory on the WEB-Server, php-accelerator and IMP as webmail. > - should not be create real system user for mail or other purposes. > Account information stored LDAP or other databases. > - does not needed pop3 support Standard today. Everything serious can have its own user-database. > - could ( i think should ) be cluster or similiar system ( load > balance, round robin dns..., MUPDATE ) Critical point is the mailstore. > - Webmail system should be support basic webmail functionality like > read, compose, mail, delete, folders, address book, able to view > attachments... See www.horde.org. > - users quota limit could be 1 MB or even 256 MB since webmail system is > used only for special > mail traffic ( MMS content ). This is done by the mailstore (Cyrus :-) > - user accounts on the current Suse eMail server could be migratable to the > new server. ( I think > i needed write scripts ) > - could be a bit high price hardware solutions ( my customer will > provide hardwares ) > > I have looked Suse OpenExchange server, there are too many features > that we dont need such as exchange server integration, projects, > appointment etc. Its not cost effective solution any more due to > client prices. If you understand german have a look at http://www.suse.de/de/business/products/whitepapers/index.html. and http://www.suse.de/de/business/products/frameworks/highscalable_email/index.htm l Maybe SuSE provide this paper in english too somewhere on www.suse.com. Asking SuSE for support will be an option too. Regards Andreas
Re: Cyrus and Performance notes?
Zitat von Achim Altmann <[EMAIL PROTECTED]>: > Hello, > > i had read the cyrus performance-notes in online-install-guide > > Are this information enought or if is possible to find more informations > about this theme. > > I would like install Cyrus-imapd 2.1.x aon an linux-server with Hardware-raid > (level 10) with 4 HDD's > > I have about this any questions, like this > > What a Filesystem is better ext2, ext3 reiser-fs ...? > > What a inode-nr in kb is good (normal mailboxes ) ? > > What can i do to have the best performance and enlarge my system in feature > ? > > Thank's a lot for any help > > Best reagards > > Achim > > sorry, my english is bad!!! My english is not better either :-) Consider using LVM to expand your disk-space online if your filesystem and your RAID-Controller and your Hardware (SCA or FC) support it. For bigger installations you should use reiser-fs or xfs which do dynamic inode allocation so you don´t have to worry about this. Be sure to read the tuning recommendations for your OS. Regards Andreas
RE: Massive Problems (lmtp, db corruption) - Ahh!
At 10:43 31.10.2002 -0500, Su Li wrote: >It won't limit other processes. I only limited the number of processes for lmtpd. Other processes like imapd can still go 10, 20, 30 or more. > >I agree that unlimited can have a big number. But to be able to config will be very very helpful. The reseason, I change the code is, if there are too many lmtpd is very harmful. Lmptd will fight for db, the db lock will go up, and new lmptd will be created but still can not get db. So the number of lmtpd should be limited to a reseanable number according to the machine where you run Cyrus IMAP. > Why not limit the case for too many lmtpd? In postfix master.cf set the limit for lmtp-client to 1-4 because more concurrency on your local disk don´t help anyway. With sendmail there should be a equal setting. Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
Re: blocking logins at server
At 12:55 30.10.2002 -0500, twk wrote: >We are running both 1.5.x and 2.1.x versions of Cyrus. I have a $#%$^#$%^ user >that has his email client set to check every folder ever few seconds. I can't >remove him from the password file, as we use Kerberos for authentication. I >don't want to disable him in kerberos, just block his logins to the server. Is >there any relatively straightforward way to do this? Don´t know about 1.5.x but have a look at "popminpoll". Good point to start "man 5 imapd.conf". Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
Re: Is using Berkeley DB 4.0.14 recommended?
At 13:27 17.10.2002 +0200, Simon Matter wrote: >Mark Keasling schrieb: >> >> Hi, >> >> I'm trying to use Berkeley DB 4.0.14 and am getting the dreaded >> DBERROR db4: 4 lockers >> messages even when only starting up the master process. From >> what I've seen reported, the lockers number will only increase and >> cyrus will eventually start failing. >> >> Is it recommended to: >> revert to a previous/different version of Berkeley DB? >> use database types other then db3 (ie skiplist, flat) >> >> If a different version of the database should be used, what is the >> latest version, supported by Cyrus imap 2.1.9, which doesn't exhibit >> this problem (does anyone know)? > >It's an annoying problem nobody seems to know about, or care about, or >care how to fix it or know who is responsible to fix it. > >I'm also really intersted in this. I wonder if this is only with Cyrus 2.1.x. If have stress-tested Cyrus 2.0.16 with DB 4.0.14 with some 100.000 messages and never get this error. Anyone seen this with 2.0.x versions? Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
Re: weird cyradm
At 11:34 24.09.2002 +0800, Hafiz Malik wrote: > hello there, >when i want to run cyradm, the usual "Please enter your >password:" doesnt show up... but when i just put in the password the blank >spaces.. thing back to normal.. but.. totally failed when i want to use >cyradm -file (script) >> %cyradm localhost > lm >> Please enter your password: % >> wrong password %cyradm localhost >application-specific initialization failed: authentication failed >Please enter your password: % any one know why? Maybe you have "allowanonymous" so login without username works for the first step but you are not able to do admin work. Try "cyradm localhost -u cyrus" instead. If your authentication fails check the imapd.conf. Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
RE: how many users can Cyrus server support? -- is 5.5 msg/sec slow?
At 09:49 19.09.2002 -0400, Su Li wrote:
>Thanks Andreas,
>
>I tried using sendmail 11 and Cyrus IMAP4 v2.1.8. I can get 5-6 msg/sec. If
>I queue up the mails by stopping Cyrus IMAP, after I got 1000 mails in
>sendmail mail store then start Cyrus IMAP. I can get 17 msg/sec. I don't
>know why there is such a big difference.
Hello Su Li
If you have incoming mail it stresses your I/O system additionally
(create,read,move).
If you transfer from spool you only read the files and send them to Cyrus.
BTW : What hardware you are testing on?
>I tried using postfix 1.11, I can get 2-3 msg/sec. I didn't change any
>config after I installed postfix using rpm. For Cyrus IMAP my configure are
>as the following:
2-3 msg/sec for small mails is incredible slow. I get this on a crappy
P-133 with 2 Gig UW-SCSI disk.
Have a look for the following :
- Reduce logging as far as usable
- Turn on async-logging (see "man syslog.conf")
- Try to deliver mails by "lmtp" not with "deliver" command
- Dont't use filtering if you are looking for highest throughput
- Use a fast I/O system (SCSI-RAID with battery-backed WB-cache for servers)
- If you are stuck with IDE turn on DMA ("man hdparm")
- Don´t watch the logs with "tail -f" during test :-)
- Inspect your logs for all sort of error/warning/timeouts
>./configure --with-auth=unix --without-krb4 --without-ucdsnmp
>--with-bdb-incdir=/usr/local/BerkeleyDB.4.0/include--with-bdb-libdir=/usr/lo
>cal/BerkeleyDB.4.0/lib --with-dblib=berkeley
>--with-dbpath=/usr/local/BerkeleyDB.4.0 --with-lock=fcntl
>--with-mboxlist-db=skiplist --with-seen-db=skiplis --without-notify
>--disable-sieve
>
>Can you let me know what have you done to improve the performance? I also
>not quiet understand about your through put 8-40 msg/sec, in what case will
>you have 8 msg/sec and at what case you have 40?
The 8 msg/sec was with /var/spool on ReiserFS, the 40 msg/sec with XFS or
ext2 (async, no chattr +S). I guess the later two cheat a little bit in
syncing the disk.
We tested with injecting 1000 mails 1K size each always to the same
recipient as fast as possible and examine the logs how long the run was.
While the inject is running, nearly no mail flow from postfix -> cyrus
because the system wasn´t able to catch up. But this was at a rate of more
then 70 msg/sec injecting on a stock IDE-PC.
Not a real usage pattern but a rough estimate-number.
We used Cyrus 2.0.16 with BerkeleyDB 4.0.14 on a Linux 2.4.18.
Regards
--
Andreas Hödle
Kühn & Weyh Software GmbH
WWW.KWSOFT.DE
Re: Problems with cyrus-imap
At 09:45 19.09.2002 +0200, Víctor Guerra wrote: >That doesn't resolve my problem, the mailboxes are still in >/var/spool/imap/user and "hashimapspool" makes my server cyrus don't >authenticate users > Afaik only new created mailboxes will have this feature furtheron. I guess you must export/import your user/mails if you want this feature for accounts which already exist. Also be sure your version supports it (e.g. 2.0.16 should be okay). Have a look in the logfiles if it starts with this parameter set. Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
Re: how many users can Cyrus server support? -- is 5.5 msg/sec slow?
At 13:40 18.09.2002 -0400, Su Li wrote: >Hi, > >I have been trying my best to get better delivery speed of Cyrus server. Now >I am stop at 5.5 mesg/sec. I wonder, can any body tell me how many users can >your cyrus mail server support? --no matter what system or configuration you >are using. Also if you can tell me how many emails through put you can get >from your mail server? -- that will be the best. > >Right now, I don't know is 5.5 msg/sec slow or not. > >5.5 * 3600 * 24 = 475200 msg/day > >If each user send 4 mails per hour, 4 * 24 =96 msg/day > >475200 / 96 = 4950 users > >That means my cyrus mail server can only support 4950 users. That is not >much for a big organization. > >So can you let me know the max number of users you can support on your >system or max though put msg/sec? If 5.5 msg/sec if the right number, I just >stop trying :) > Okay, for the records. We have recently done a short test with Postfix 1.11 -> lmtp -> Cyrus 2.0.16 and get a speed from 8 .. 40 msgs /sec (1k each) on a single IDE harddisk with 2x 266 PII and 512 MB RAM. If you get a fast SCSI-Array you can beat this rate by far. Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
RE: Cyrus IMAP Presentation
At 09:20 17.09.2002 -0500, Schmehl, Paul L wrote: >What is noatime? "# man -k noatime" shows nothing on RH 7.2. > "man mount" is the right place to look :-) Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
Re: Problems with cyrus-imap
At 13:14 17.09.2002 +0200, Víctor Guerra wrote: >Hello everybody, i am a new subscriber of that list and i have a question. I >have installed a Cyrus-IMAP server in a Red Hat 7.2 and i have a problem: i >have 19970 mailboxes in my server, those mailboxes are in >/var/spool/imap/user, some time ago i have had a problem when the number of >mailboxes riches a high number, and the system crashes because Linux do not >support more than 32768 directories at the same level (isn't it). I have >found that in /var/imap/user there are 27 folders... each one called as a >letter of the ABC and i want that cyrus delivers all the mails in those >folders, for example, if a user called [EMAIL PROTECTED] receives an >e-mail... that e-mail must be delivered in /var/imap/user/j/ , if a user >called [EMAIL PROTECTED] receives an e-mail... that must be delivered in >/var/imap/user/a/. Does anybody can to help me? , thanks for all. > I guess "hashimapspool" is what you are looking for. "man imapd.conf" Regards -- Andreas Hödle Kühn & Weyh Software GmbH WWW.KWSOFT.DE
