Re: [rt-users] Can't fork at Mailer.pm
On 5.10.2012 19:05, Thomas Sibley wrote: On 10/05/2012 02:31 AM, Martin Drasar wrote: after upgrading to RT 4.0.6 I am having problems with mail sending. This line starts to appear in log: Scrip Commit 6 died. - Can't fork at /usr/share/perl5/Mail/Mailer.pm line 145 and no mail can be sent until I restart the Apache. Do you have any idea why it is happening? And more importantly - how can I solve this so it does not happen again? Can't fork means your operating system is dangerously out of memory. You should tune Apache and anything else on the box (your database server?) to coexist peacefully. What MailCommand are you using in RT? Hi Thomas, I am using sendmail as a mail command, but you are most likely right about insufficient memory. I have checked the logs and Apache processes were dying of memory starvation. I have doubled the memory (it was only 1 gig) and will check if it keeps happening. Thanks, Martin Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
Re: [rt-users] Can't fork at Mailer.pm
On 8 Oct 2012, at 09:42, Martin Drasar dra...@ics.muni.cz wrote: On 5.10.2012 19:05, Thomas Sibley wrote: On 10/05/2012 02:31 AM, Martin Drasar wrote: after upgrading to RT 4.0.6 I am having problems with mail sending. This line starts to appear in log: Scrip Commit 6 died. - Can't fork at /usr/share/perl5/Mail/Mailer.pm line 145 and no mail can be sent until I restart the Apache. Do you have any idea why it is happening? And more importantly - how can I solve this so it does not happen again? Can't fork means your operating system is dangerously out of memory. You should tune Apache and anything else on the box (your database server?) to coexist peacefully. What MailCommand are you using in RT? Hi Thomas, I am using sendmail as a mail command, but you are most likely right about insufficient memory. I have checked the logs and Apache processes were dying of memory starvation. I have doubled the memory (it was only 1 gig) and will check if it keeps happening. Make sure you have plenty of swap allocated; you may not need the memory as physical. I run the RT web server on a machine with only 2GB of RAM, and it's fine. I don't run the database on the same machine, though. You have to remember that when Apache/perl forks to exec the sendmail binary, temporarily you double the virtual memory requirement of the Apache/perl setup. You either have to have virtual memory overcommit turned on (which I think is the default on most Linux systems these days) or to have enough swap allocated to cope. vm.overcommit_memory = 0 in /etc/sysctl.conf is probably what you want. You probably actively don't want it to be set to 2. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
Re: [rt-users] Can't fork at Mailer.pm
On 8.10.2012 10:53, Tim Cutts wrote: Make sure you have plenty of swap allocated; you may not need the memory as physical. I run the RT web server on a machine with only 2GB of RAM, and it's fine. I don't run the database on the same machine, though. You have to remember that when Apache/perl forks to exec the sendmail binary, temporarily you double the virtual memory requirement of the Apache/perl setup. You either have to have virtual memory overcommit turned on (which I think is the default on most Linux systems these days) or to have enough swap allocated to cope. vm.overcommit_memory = 0 in /etc/sysctl.conf is probably what you want. You probably actively don't want it to be set to 2. Tim Hi Tim, thank you for the tips, I was not aware about doubling the virtual memory requirements. I will look into it. Martin Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
Re: [rt-users] Can't fork at Mailer.pm
On 8 Oct 2012, at 10:03, Martin Drasar dra...@ics.muni.cz wrote: thank you for the tips, I was not aware about doubling the virtual memory requirements. I will look into it. It's only for a split second, and the memory isn't used. UNIX doesn't just start an application in one go, it uses two steps: 1) fork This duplicates the current process, and both processes continue executing. UNIX does this with so-called copy-on-write, so the pages aren't actually duplicated until they're modified, but nevertheless, the potential exists for both copies to be completely required, so if you have vm.overcommit_memory set to 2, you need all that virtual memory to be available just in case. 2) exec the newly forked process usually immediately executes the exec() system call, which replaces the current process' virtual memory image with the desired program. At this point, the virtual memory requirements go back down again (assuming the new process is something small, which in the case of sendmail, it is). Apologies if there are errors and oversimplifications in the above, I'm not exactly a total UNIX beardy type. We've seen this bite us on some of our HPC clusters at work - there, we do have vm.overcommit_memory set to 2, because we want programs allocating too much memory to die immediately, without risk of the kernel's out of memory killer zapping the wrong thing. A common problem on such machines is very large computational jobs trying to start sub-processes - as soon as they do, their virtual memory requirement doubles and the fork() fails with an out of memory error. It just seemed to me that it might be the case with your system. Try the output of: cat /proc/sys/vm/overcommit_memory if it's 2, then this is probably your problem. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
Re: [rt-users] Can't fork at Mailer.pm
On 8.10.2012 11:29, Tim Cutts wrote: It's only for a split second, and the memory isn't used. UNIX doesn't just start an application in one go, it uses two steps: 1) fork This duplicates the current process, and both processes continue executing. UNIX does this with so-called copy-on-write, so the pages aren't actually duplicated until they're modified, but nevertheless, the potential exists for both copies to be completely required, so if you have vm.overcommit_memory set to 2, you need all that virtual memory to be available just in case. 2) exec the newly forked process usually immediately executes the exec() system call, which replaces the current process' virtual memory image with the desired program. At this point, the virtual memory requirements go back down again (assuming the new process is something small, which in the case of sendmail, it is). Apologies if there are errors and oversimplifications in the above, I'm not exactly a total UNIX beardy type. We've seen this bite us on some of our HPC clusters at work - there, we do have vm.overcommit_memory set to 2, because we want programs allocating too much memory to die immediately, without risk of the kernel's out of memory killer zapping the wrong thing. A common problem on such machines is very large computational jobs trying to start sub-processes - as soon as they do, their virtual memory requirement doubles and the fork() fails with an out of memory error. It just seemed to me that it might be the case with your system. Try the output of: cat /proc/sys/vm/overcommit_memory if it's 2, then this is probably your problem. Tim I have checked the overcommit settings and it is zero. It seems that the problem was purely in RT and postgres running with mostly default settings on one machine with not enough memory. It was good enough for RT 3.8.7, but it is not for 4.0.6. Although now I remember a few times when processes were killed because of insufficient memory back then. But it happened so little that we did not pay attention. I will not be making the same mistake again... I will ask some greybeard to check the settings for me or throw memory at it, until the problem goes away :-) Anyway, thanks a lot for your help. Martin Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
[rt-users] Can't fork at Mailer.pm
Hi, after upgrading to RT 4.0.6 I am having problems with mail sending. This line starts to appear in log: Scrip Commit 6 died. - Can't fork at /usr/share/perl5/Mail/Mailer.pm line 145 and no mail can be sent until I restart the Apache. Scrip 6 is On Correspond Notify Requestors and Ccs Do you have any idea why it is happening? And more importantly - how can I solve this so it does not happen again? Thank you for your insights. Martin Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
Re: [rt-users] Can't fork at Mailer.pm
On 10/05/2012 02:31 AM, Martin Drasar wrote: after upgrading to RT 4.0.6 I am having problems with mail sending. This line starts to appear in log: Scrip Commit 6 died. - Can't fork at /usr/share/perl5/Mail/Mailer.pm line 145 and no mail can be sent until I restart the Apache. Do you have any idea why it is happening? And more importantly - how can I solve this so it does not happen again? Can't fork means your operating system is dangerously out of memory. You should tune Apache and anything else on the box (your database server?) to coexist peacefully. What MailCommand are you using in RT? Final RT training for 2012 in Atlanta, GA - October 23 24 http://bestpractical.com/training We're hiring! http://bestpractical.com/jobs
