[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Please disregard my last comment. Those packages did not actually have the patch applied due to some nuances with dpatch in the build. As a result those tests are invalid. I have since cleanly applied the patch, built test packages, and tested. These test have succeeded, with 100,000 SSL request being issued and completed successfully, without loss of SSL connection, and without out- of-memory errors. I'm submitting the attached debdiff as the fix for this issue, and Chuck will be bringing for the SRU. The patch is the one suggested by Apache ( http://svn.apache.org/viewvc?view=rev&revision=654119 ) :-Dustin ** Attachment added: "224945_fix_ssl_mem_leak.debdiff" http://launchpadlibrarian.net/14494690/224945_fix_ssl_mem_leak.debdiff -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
The bug is in Apache, not OpenSSL. ** Changed in: openssl (Ubuntu) Importance: High => Undecided Assignee: Dustin Kirkland (kirkland) => (unassigned) Status: Triaged => Invalid -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Patch submitted. ** Changed in: apache2 (Ubuntu) Status: Confirmed => In Progress Target: None => ubuntu-8.04.1 -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Upstream bug filed with Apache: https://issues.apache.org/bugzilla/show_bug.cgi?id=44975 They suggested applying this patch: http://svn.apache.org/viewvc?view=rev&revision=654119 Test packages available in my PPA. This patch fixes half of the problem. It solves the out-of-memory errors, however, SSL connections are simply dropped when the server is flooded. Note that Feisty's apache2+mod_ssl is able to handle the flood on both the client and server without errors. :-Dustin -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
FWIW, I have also reproduced this problem in Gutsy as well, which uses openssl-0.9.8e-5ubuntu3. However, Dapper-Feisty do not exhibit this problem. Feisty uses openssl-0.9.8c-4ubuntu0.2. It seems that the regression was introduced sometime between openssl-0.9.8c and 0.9.8e. :-Dustin ** Changed in: openssl (Ubuntu) Importance: Undecided => High Assignee: (unassigned) => Dustin Kirkland (kirkland) Status: New => Triaged -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Léobaillard- I cannot reproduce this with SSL disabled. Can you give very detailed steps for reproducing this problem without SSL? :-Dustin ** Also affects: openssl (Ubuntu) Importance: Undecided Status: New -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
The only way to reboot the machine safely is to use the Magic SysRq keys. For instance, I used : AltGr + SysRq + s, AltGr + SysRq + u and AltGr + SysRq + b which syncs the mounted file systems, remounts all the mounted file systems in Read Only mode and finally reboots the computer. But I still can't prevent Apache from crashing the system even with SSL disabled. -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Chuck- SSL configuration: NameVirtualHost *:443 SSLEngine On SSLCertificateFile /etc/ssl/certs/ssl.pem DocumentRoot /var/www Options Indexes :-Dustin -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
I've reproduced this in a Hardy VM with 128MB of memory. Beyond the default server install, I installed: apache2-mpm-worker ssl-cert I enabled ssl with a2enmod. I created a very simple ssl config for apache and enabled it with a2ensite. Then, from the VM host, I ran: time ab -n 10 -c 20 -f tls1 https://192.168.122.74:443/ It locked up the VM. I could see apache2 consuming all virtual memory, and the load pegged at 20.87. Interestingly, I couldn't shutdown or ctrl-alt-delete the VM to gracefully reset it to a sane state. :-Dustin ** Changed in: apache2 (Ubuntu) Assignee: (unassigned) => Dustin Kirkland (kirkland) Status: Incomplete => Confirmed -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Hello, Can you provide your ssl configuration so I can reproduce this? Please include the output of dpkg -l | grep apache. Thanks chuck ** Changed in: apache2 (Ubuntu) Status: New => Incomplete -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
BTW, here's some debug information from /var/log/kern.log : May 3 19:51:22 yoda kernel: [10632.509889] Out of memory: kill process 9047 (apache2) score 19040 or a child May 3 19:51:22 yoda kernel: [10632.510045] Killed process 9047 (apache2) May 3 19:51:22 yoda kernel: [10634.310259] named invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 May 3 19:51:22 yoda kernel: [10634.310291] Pid: 4539, comm: named Not tainted 2.6.24-16-server #1 May 3 19:51:22 yoda kernel: [10634.310370] [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120 May 3 19:51:22 yoda kernel: [10634.310460] [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0 May 3 19:51:22 yoda kernel: [10634.310510] [agpgart:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380 May 3 19:51:22 yoda kernel: [10634.310573] [filemap_fault+0x10a/0x420] filemap_fault+0x10a/0x420 May 3 19:51:22 yoda kernel: [10634.310638] [__do_fault+0x83/0x4c0] __do_fault+0x83/0x4c0 May 3 19:51:22 yoda kernel: [10634.310747] [handle_mm_fault+0x21b/0xb80] handle_mm_fault+0x21b/0xb80 May 3 19:51:22 yoda kernel: [10634.310778] [] default_wake_function+0x0/0x10 May 3 19:51:22 yoda kernel: [10634.310875] [sched_clock+0x13/0x30] sched_clock+0x13/0x30 May 3 19:51:22 yoda kernel: [10634.310914] [snd_pcm:getnstimeofday+0x34/0x9790] getnstimeofday+0x34/0xf0 May 3 19:51:22 yoda kernel: [10634.310966] [snd_pcm:ktime_get_ts+0x1e/0x4f0] ktime_get_ts+0x1e/0x60 May 3 19:51:22 yoda kernel: [10634.310996] [do_page_fault+0x143/0x900] do_page_fault+0x143/0x900 May 3 19:51:22 yoda kernel: [10634.311025] [sys_futex+0x98/0x120] sys_futex+0x98/0x120 May 3 19:51:22 yoda kernel: [10634.311054] [snd_seq:do_gettimeofday+0x34/0xa450] do_gettimeofday+0x34/0xe0 May 3 19:51:22 yoda kernel: [10634.311090] [sys_gettimeofday+0x28/0x80] sys_gettimeofday+0x28/0x80 May 3 19:51:22 yoda kernel: [10634.38] [do_page_fault+0x0/0x900] do_page_fault+0x0/0x900 May 3 19:51:22 yoda kernel: [10634.311134] [error_code+0x72/0x78] error_code+0x72/0x78 May 3 19:51:22 yoda kernel: [10634.311173] [unix_create1+0x50/0x120] unix_create1+0x50/0x120 May 3 19:51:22 yoda kernel: [10634.311229] === May 3 19:51:22 yoda kernel: [10634.311236] Mem-info: May 3 19:51:22 yoda kernel: [10634.311243] DMA per-cpu: May 3 19:51:22 yoda kernel: [10634.311253] CPU0: Hot: hi:0, btch: 1 usd: 0 Cold: hi:0, btch: 1 usd: 0 May 3 19:51:22 yoda kernel: [10634.311264] Normal per-cpu: May 3 19:51:22 yoda kernel: [10634.311273] CPU0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 43 May 3 19:51:22 yoda kernel: [10634.311291] Active:87819 inactive:295 dirty:0 writeback:0 unstable:0 May 3 19:51:22 yoda kernel: [10634.311297] free:976 slab:3176 mapped:2 pagetables:1570 bounce:0 May 3 19:51:22 yoda kernel: [10634.311314] DMA free:1552kB min:104kB low:128kB high:156kB active:10208kB inactive:16kB present:16256kB pages_scanned:127731 all_unreclaimable? yes May 3 19:51:22 yoda kernel: [10634.311327] lowmem_reserve[]: 0 364 364 364 May 3 19:51:22 yoda kernel: [10634.311347] Normal free:2352kB min:2384kB low:2980kB high:3576kB active:341068kB inactive:1164kB present:372812kB pages_scanned:706813 all_unreclaimable? yes May 3 19:51:22 yoda kernel: [10634.311361] lowmem_reserve[]: 0 0 0 0 May 3 19:51:22 yoda kernel: [10634.311375] DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1552kB May 3 19:51:22 yoda kernel: [10634.311407] Normal: 18*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2336kB May 3 19:51:22 yoda kernel: [10634.311443] Swap cache: add 0, delete 0, find 0/0, race 0+0 May 3 19:51:22 yoda kernel: [10634.311452] Free swap = 0kB May 3 19:51:22 yoda kernel: [10634.311458] Total swap = 0kB May 3 19:51:22 yoda kernel: [10634.311464] Free swap:0kB May 3 19:51:22 yoda kernel: [10634.338503] 98032 pages of RAM May 3 19:51:22 yoda kernel: [10634.338519] 0 pages of HIGHMEM May 3 19:51:22 yoda kernel: [10634.338525] 1953 reserved pages May 3 19:51:22 yoda kernel: [10634.338531] 246726 pages shared May 3 19:51:22 yoda kernel: [10634.338537] 0 pages swap cached May 3 19:51:22 yoda kernel: [10634.338543] 0 pages dirty May 3 19:51:22 yoda kernel: [10634.338548] 0 pages writeback May 3 19:51:22 yoda kernel: [10634.338554] 2 pages mapped May 3 19:51:22 yoda kernel: [10634.338560] 3176 pages slab May 3 19:51:22 yoda kernel: [10634.338565] 1570 pages pagetables -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 224945] Re: memory leaks in apache2 when running mod_ssl
Hi, I've got the same bug. I've tried disabling mod_ssl without any success, the memory leak is still happening... Do you have an idea ? Maybe it's not/not only SSL after all ? -- memory leaks in apache2 when running mod_ssl https://bugs.launchpad.net/bugs/224945 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
