Re: hanging imapd processes (was Re: Runaway processes ?)
Craig Sanders wrote: On Sun, Nov 28, 2004 at 09:26:34PM +0200, Uffe Pensar wrote: I'm in the process of separating the webmail from the imap and I have have installed xinetd with max_load and other limits. But still I can't understand that i get those hanging imapd processes ? # ps axu|grep imapd|grep root root 22700 0.0 0.0 4064 1308 ?SNov25 0:00 [imapd] root 11359 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] root 6473 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] root 3801 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] root 6194 0.0 0.0 1752 732 pts/7S21:11 0:00 grep imapd woody and uw-imap-ssl (those hanging connections are not coming from webmail) there could be clients still connected from Nov 25 & Nov 27. impossible to tell just from a ps listing. you can use netstat and lsof to see which processes have active connections (or are listening) on the imap port. see the man pages for details. yes from lsof I can that it isn't a webmail but imps-clients (perhaps broken ?). I was wondering why they never time out. but i wouldn't bother. uw-imapd is junk, and its problems are pretty much unfixable. that explains a lot. i'd just replace it with something sane. BTW, you chose the more difficult path. instead of just replacing uw-imapd with dovecot, which would have been a simple action with one isolated effect (changing the imap daemon), you chose to replace inetd with xinetd, which affects dozens or possibly hundreds of unrelated inet daemons. why? I visited the dovecot homepages and got the impression that dovecot was kind of fresh and that it would not take a mail-directory as an argument (our users have their mail in a mail-directory under their home-directory) And then I know of a bigger site then ours that use uw-imap (but with suse and perhaps a newer version then the one in woody stable) so I thought let's see if we can get it stable with a minimum of changes (xinetd, separating webmail and imap and kernel 2.4.28). craig - uffe -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: hanging imapd processes (was Re: Runaway processes ?)
On Sun, Nov 28, 2004 at 09:26:34PM +0200, Uffe Pensar wrote: > I'm in the process of separating the webmail from the imap and I have > have installed xinetd with > max_load and other limits. > But still I can't understand that i get those hanging imapd processes ? > > # ps axu|grep imapd|grep root > root 22700 0.0 0.0 4064 1308 ?SNov25 0:00 [imapd] > root 11359 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] > root 6473 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] > root 3801 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] > root 6194 0.0 0.0 1752 732 pts/7S21:11 0:00 grep imapd > > woody and uw-imap-ssl (those hanging connections are not coming from > webmail) there could be clients still connected from Nov 25 & Nov 27. impossible to tell just from a ps listing. you can use netstat and lsof to see which processes have active connections (or are listening) on the imap port. see the man pages for details. but i wouldn't bother. uw-imapd is junk, and its problems are pretty much unfixable. that explains a lot. i'd just replace it with something sane. BTW, you chose the more difficult path. instead of just replacing uw-imapd with dovecot, which would have been a simple action with one isolated effect (changing the imap daemon), you chose to replace inetd with xinetd, which affects dozens or possibly hundreds of unrelated inet daemons. why? craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
hanging imapd processes (was Re: Runaway processes ?)
Craig Sanders wrote: On Wed, Nov 24, 2004 at 07:15:43PM +0200, Uffe Pensar wrote: [141 lines of quoted material deleted. please learn to trim your quotes ] ok thanks for all the good advices, I will install postfix and have a look att the dovecot and xinetd packages if you're running dovecot, you don't need xineted. it is a standalone daemon, which has it's own (configurable) limits on maximum simultaneous connections. but as a quick fix its seems that authenticating from a local server (instead of radius) and restricting the number of webmailsessions has helped for the moment. But I suppose we have to buy more servers in the near future. yes, separating the tasks of 1) sending & receiving mail and 2) storing it & providing imap/pop/webmail/etc access is very useful. remember that mail is an I/O bound system. i.e. most of the time the processor is sitting idle waiting for disk I/O to complete. upgrading the CPU will do little or no good here. to improve performance, you need to improve the I/O speed - you can do this with faster disks, hardware raid card with large non-volatile cache, and by adding more RAM to the system. or by spreading the I/O load over more disks and/or more servers. I'm in the process of separating the webmail from the imap and I have have installed xinetd with max_load and other limits. But still I can't understand that i get those hanging imapd processes ? # ps axu|grep imapd|grep root root 22700 0.0 0.0 4064 1308 ?SNov25 0:00 [imapd] root 11359 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] root 6473 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] root 3801 0.0 0.0 4064 1400 ?SNov27 0:00 [imapd] root 6194 0.0 0.0 1752 732 pts/7S21:11 0:00 grep imapd woody and uw-imap-ssl (those hanging connections are not coming from webmail) - uffe -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Runaway processes ?
On Wed, Nov 24, 2004 at 07:15:43PM +0200, Uffe Pensar wrote: > > [141 lines of quoted material deleted. please learn to trim your quotes ] > > ok thanks for all the good advices, I will install postfix and have a > look att the dovecot and xinetd packages if you're running dovecot, you don't need xineted. it is a standalone daemon, which has it's own (configurable) limits on maximum simultaneous connections. > but as a quick fix its seems that authenticating from a local server > (instead of radius) and restricting the number of webmailsessions has > helped for the moment. But I suppose we have to buy more servers in > the near future. yes, separating the tasks of 1) sending & receiving mail and 2) storing it & providing imap/pop/webmail/etc access is very useful. remember that mail is an I/O bound system. i.e. most of the time the processor is sitting idle waiting for disk I/O to complete. upgrading the CPU will do little or no good here. to improve performance, you need to improve the I/O speed - you can do this with faster disks, hardware raid card with large non-volatile cache, and by adding more RAM to the system. or by spreading the I/O load over more disks and/or more servers. a good starting point for a design is to have multiple small & cheap machines which are configured to accept all incoming mail (i.e. the MX records point at them), filter spam and viruses, and then either forward it on to the backend mail store server, or write it directly into NFS-mounted mail spool directories. if you use nfs, then you MUST use an NFS-safe mailbox store, like Maildir. trying to use mbox over NFS will almost certainly lead to mailbox corruption (although on debian it should be safe because *all* mail handling programs should use the *same* NFS-safe locking method. i wouldn't count on it though, especially if you compile your own mail programs rather than use the packaged ones) note that these MX boxes *MUST* have access to a list of valid recipients for all domains that it accepts mail for. this allows it to reject mail for unknown users during the smtp session rather than accept-and-bounce, so it doesn't generate backscatter and get bogged down with virus bounces and undeliverable spam bounces. the backend mail store could be either one very large and expensive server which stores the mail(*) and handles all the imap clients, or one medium-sized file-server (which stores the mail) and several small and cheap imap boxes which handle the imap connections, with NFS-mounted mail store (see comments above about NFS-safe mailboxes). when building the mail store, I/O performance is your key design criteria. don't worry about CPU, all the CPU-heavy tasks (like spam and virus filtering) are dont by the MX boxes. you may optionally want another box to handle all outgoing mail (i.e. the one that your clients use to send mail through, the mail relay). this one should also be optimised for fast I/O. one good way of doing this is to use a solid-state-disk (SSD - essentially a large battery-backed ramdisk that looks like a scsi or ide drive) for the mail queue. e.g. mount /var/spool/postfix (or /var/spool/mqueue if using sendmail) on the SSD device.SSDs are typically small and expensive, 1 or 2GB is likely to be the limit of affordabilitybut that's more than enough for a mail queue partition. in my experience, though, all but the largest ISPs receive dozens or hundreds of times more mail than they send, even when taking mailing lists into account. the mail store machine can double as the outbound mail relay - giving it an SSD device for the mail queue is a good idea. (*) e.g. on multiple 15000 rpm hard disks on a hardware raid-5 controller with at least 128MB of non-volatile cache ram. or whatever else it takes to optimise this box for extremely fast I/O. craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Runaway processes ?
Craig Sanders wrote: On Tue, Nov 23, 2004 at 02:28:43PM +0200, Ulf Pensar wrote: We have an emailserver that we had to reboot the hard way a couple of times a week.Now its a couple of time a day (perhaps because the number of users have been growing) [...] the inetd generates root owned processes and it doesnt stop before inetd is being killed. Then we have to reboot the server to go on working. if you really must run your imap server from inetd, then consider using xinetd which allows limits on the number of simultaneous connections. or get rid of uw-imapd junk and replace it with something better (see below) I guess it is the webmail that is creating those imap-processes but I'm not sure (could be imaps-clients of course). Have you seen anything like that and what could be done? Facts: dell power edge 600 Sc, intel celeron 1,7 GHz, 2 GB ECC memory running: - woody (kernel compiled from debian package kernel-source-2.4.18) fine so far. fortunately, there's a lot you can do to help performance. - uw-imap-ssl (starting from inetd) replace with something sane. dovecot or courier-imapd for example. dovecot works with mbox and Maildir mail boxes, courier-imapd only with Maildir. you probably have mbox if you're running an old sendmail machine. it's a trivial upgrade - "apt-get install dovecot". btw, both dovecot and courier support both SSL encrypted and unencrypted versions of the protocols. dovecot does pop & imap in the one package, while courier has separate packages: courier-imapd and courier-popd. - sendmail (with milter and clam) replace with postfix. watch your mail transport related load vanish instantly. - bind, dhcp, mysql, radiusd-cistron (latest woody packages) it kind of makes sense to have radius on your mail server, IF you are authenticating against /etc/passwd. dhcp, and mysql could be moved to other machines. bind probably can't be moved without a lot of pain, if you have domains delegated to this IP address. if you're just using it as a caching resolver, then consider replacing it with something lighter - perhaps maradns or djbdns. - webmail (the latest stable imp/horde) - php-4.3.9 - imapproxy, just for the webmail (the latest) imp doesn't have to run on the mail server. consider moving these to another machine, perhaps your web server. or consider using a different webmail program. imp is pretty heavy on resources like memory. courier-sqwebmail is fairly light and integrates well with the other programs in the courier suite, courier-maildrop, courier-imap, etc. there are other lightweight ones around too, if you don't like sqwebmail. other things you can do: 1. encourage people to delete mail from the server rather than leaving it on there. you can do this by implementing quotas. start by setting a quota which is several megabytes *above* the largest mailbox (i.e. the "unofficial, temporary quota"). announce that you are setting a quota of what your eventual target is (i.e. the "official quota"). gradually reduce the quota every week until target is reached. don't tell your users how much leeway they have at any given moment because they will abuse that knowledge. if they ask just tell them the official quota and mention that some *unspecified* leeway is given for a short *unspecified* time. if you don't want to compile the kernel for quota support and install the quota package, you can crudely simulate it for mailboxes with postfix's 'mailbox_size_limit' parameter. this is per mailbox file. imap users can get around the quota by saving messages to different mailbox files. alternatively, if you must allow users to have huge mailboxes, then: 2. switch to Maildir rather than mbox. craig ok thanks for all the good advices, I will install postfix and have a look att the dovecot and xinetd packages but as a quick fix its seems that authenticating from a local server (instead of radius) and restricting the number of webmailsessions has helped for the moment. But I suppose we have to buy more servers in the near future. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
switching to dovecot from uw-imapd (was: Re: Runaway processes ?)
Craig Sanders wrote: - uw-imap-ssl (starting from inetd) replace with something sane. dovecot or courier-imapd for example. dovecot works with mbox and Maildir mail boxes, courier-imapd only with Maildir. you probably have mbox if you're running an old sendmail machine. it's a trivial upgrade - "apt-get install dovecot". btw, both dovecot and courier support both SSL encrypted and unencrypted versions of the protocols. dovecot does pop & imap in the one package, while courier has separate packages: courier-imapd and courier-popd. Just a side-note here, I attempted to switch an old but lightly used server from uw-imapd that was an old Potato machine that had been continually upgraded to Woody to dovecot via this method and it didn't work. The reason appeared to be that wy back when mail directories were in /home//~Mail vs. ~mail and other things, hacks were added to uw-imapd's pre-inst scripts to deal with all the changes as they came and went... dovecot's packages didn't seem to have this level of intelligence in their pre-inst scripts. I just mention it because I never had time to screw around with it on that box, I just reinstalled uw-imapd and forgot about it... it was more an exercise in getting all the boxes I admin on the same toolset, but in the case of that box, it just wasn't worth messing around with that day. Nate -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Runaway processes ?
On Tue, Nov 23, 2004 at 02:28:43PM +0200, Ulf Pensar wrote: > We have an emailserver that we had to reboot the hard way a couple > of times a week.Now its a couple of time a day (perhaps because > the number of users have been growing) > [...] > the inetd generates root owned processes and > it doesnt stop before inetd is being killed. Then we have to reboot > the server to go on working. if you really must run your imap server from inetd, then consider using xinetd which allows limits on the number of simultaneous connections. or get rid of uw-imapd junk and replace it with something better (see below) > I guess it is the webmail that is creating those imap-processes but > I'm not sure (could be imaps-clients of course). > > Have you seen anything like that and what could be done? > > Facts: > dell power edge 600 Sc, intel celeron 1,7 GHz, 2 GB ECC memory > running: > - woody (kernel compiled from debian package kernel-source-2.4.18) fine so far. fortunately, there's a lot you can do to help performance. > - uw-imap-ssl (starting from inetd) replace with something sane. dovecot or courier-imapd for example. dovecot works with mbox and Maildir mail boxes, courier-imapd only with Maildir. you probably have mbox if you're running an old sendmail machine. it's a trivial upgrade - "apt-get install dovecot". btw, both dovecot and courier support both SSL encrypted and unencrypted versions of the protocols. dovecot does pop & imap in the one package, while courier has separate packages: courier-imapd and courier-popd. > - sendmail (with milter and clam) replace with postfix. watch your mail transport related load vanish instantly. > - bind, dhcp, mysql, radiusd-cistron (latest woody packages) it kind of makes sense to have radius on your mail server, IF you are authenticating against /etc/passwd. dhcp, and mysql could be moved to other machines. bind probably can't be moved without a lot of pain, if you have domains delegated to this IP address. if you're just using it as a caching resolver, then consider replacing it with something lighter - perhaps maradns or djbdns. > - webmail (the latest stable imp/horde) > - php-4.3.9 > - imapproxy, just for the webmail (the latest) imp doesn't have to run on the mail server. consider moving these to another machine, perhaps your web server. or consider using a different webmail program. imp is pretty heavy on resources like memory. courier-sqwebmail is fairly light and integrates well with the other programs in the courier suite, courier-maildrop, courier-imap, etc. there are other lightweight ones around too, if you don't like sqwebmail. other things you can do: 1. encourage people to delete mail from the server rather than leaving it on there. you can do this by implementing quotas. start by setting a quota which is several megabytes *above* the largest mailbox (i.e. the "unofficial, temporary quota"). announce that you are setting a quota of what your eventual target is (i.e. the "official quota"). gradually reduce the quota every week until target is reached. don't tell your users how much leeway they have at any given moment because they will abuse that knowledge. if they ask just tell them the official quota and mention that some *unspecified* leeway is given for a short *unspecified* time. if you don't want to compile the kernel for quota support and install the quota package, you can crudely simulate it for mailboxes with postfix's 'mailbox_size_limit' parameter. this is per mailbox file. imap users can get around the quota by saving messages to different mailbox files. alternatively, if you must allow users to have huge mailboxes, then: 2. switch to Maildir rather than mbox. craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Runaway processes ?
Hi, We have an emailserver that we had to reboot the hard way a couple of times a week.Now its a couple of time a day (perhaps because the number of users have been growing) From the logs we can't see anything, it just stops logging. From a process-logging-script (runs 'ps axu' every minute) there is neither anything unusual except some old (several days) root owned imap-processess that, as I understand it, shouldn't be there. The load goes up very quickly (normally it is something between 1-4 with peaks up to 10). From a nagios-server we can see that the load and the number of processes are rising fast (up to 50, 70 and 100 and then there is nothing to do). Twice I have been able to see what happens: the inetd generates root owned processes and it doesnt stop before inetd is being killed. Then we have to reboot the server to go on working. I guess it is the webmail that is creating those imap-processes but I'm not sure (could be imaps-clients of course). Have you seen anything like that and what could be done? Facts: dell power edge 600 Sc, intel celeron 1,7 GHz, 2 GB ECC memory running: - woody (kernel compiled from debian package kernel-source-2.4.18) - uw-imap-ssl (starting from inetd) - webmail (the latest stable imp/horde) - php-4.3.9 - sendmail (with milter and clam) - bind, dhcp, mysql, radiusd-cistron (latest woody packages) - imapproxy, just for the webmail (the latest) -uffe -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
