Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > Simon Josefsson wrote: >> You could install memcached and modify >> /etc/apache2/mods-available/gnutls.conf to use it instead of a dbm file. > > I tried that but I get an error: > > # /etc/init.d/apache2 restart > Restarting web server: apache2Syntax error on line 6 of > /etc/apache2/mods-enabled/gnutls.conf: > Invalid Type for GnuTLSCache! > failed! > > The contents of my file: > > > > GnuTLSCache memcache "127.0.0.1" > # GnuTLSCache dbm /var/cache/apache2/gnutls_cache > > > > Is mod_gnutls in Debian Lenny built without memcache support? Alas, yes. It appears as if you updated #497097, thanks. It would be useful to prepare back-ported mod-gnutls packages with memcached support for lenny though. /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Simon Josefsson wrote: > You could install memcached and modify > /etc/apache2/mods-available/gnutls.conf to use it instead of a dbm file. I tried that but I get an error: # /etc/init.d/apache2 restart Restarting web server: apache2Syntax error on line 6 of /etc/apache2/mods-enabled/gnutls.conf: Invalid Type for GnuTLSCache! failed! The contents of my file: GnuTLSCache memcache "127.0.0.1" # GnuTLSCache dbm /var/cache/apache2/gnutls_cache Is mod_gnutls in Debian Lenny built without memcache support? -- Sander Marechal ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > Simon Josefsson wrote: >> Maybe libdb people are interested in your cache file, to debug this, >> although it does contain sensitive information so be careful about >> sending it. > > In that case the libdb people are out of luck. If it was just my > information in there then it wouldn't matter so much, but there is a lot > of other people's information in there as well. > >> You could install memcached and modify >> /etc/apache2/mods-available/gnutls.conf to use it instead of a dbm file. > > Sounds good. Thanks for the advice. If memcached works better, it would be useful to know. You could also try again with dbm, and see for how long it works unless you run into the same problem. But I've seen it before, so probably there is a real bug in there. >> Btw, your MTA refuses direct e-mails: >> >> We're sorry, but the user account you are trying to reach has exceeded its >> size limit. As a result, we were unable to deliver this message to the >> intended recipient. Please try sending this message again at a later time. >> >> Reporting-MTA: DNS; mycingular.com >> Received-From-MTA: DNS; [204.9.89.153] >> >> Final-Recipient: RFC822; cla...@mycingular.com > > That's not me. I run my own mailserver on mail.jejik.com and have no > limits (except for the size of the hard drive :-) I've never heared of > mycingular.com > > Are you sure it's not somebody else subscribed to this mailinglist? Ah, you are right. /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Simon Josefsson wrote: > Maybe libdb people are interested in your cache file, to debug this, > although it does contain sensitive information so be careful about > sending it. In that case the libdb people are out of luck. If it was just my information in there then it wouldn't matter so much, but there is a lot of other people's information in there as well. > You could install memcached and modify > /etc/apache2/mods-available/gnutls.conf to use it instead of a dbm file. Sounds good. Thanks for the advice. > Btw, your MTA refuses direct e-mails: > > We're sorry, but the user account you are trying to reach has exceeded its > size limit. As a result, we were unable to deliver this message to the > intended recipient. Please try sending this message again at a later time. > > Reporting-MTA: DNS; mycingular.com > Received-From-MTA: DNS; [204.9.89.153] > > Final-Recipient: RFC822; cla...@mycingular.com That's not me. I run my own mailserver on mail.jejik.com and have no limits (except for the size of the hard drive :-) I've never heared of mycingular.com Are you sure it's not somebody else subscribed to this mailinglist? -- Sander Marechal ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > Simon Josefsson wrote: >> I recall something like that, it happened if the cache was corrupt. >> Maybe you could stop apache, copy away /var/cache/apache2/gnutls_cache, >> and start apache again, to see if it solves the problem? Save the cache >> file so we can try to debug why this happened. > > That worked! Good. > I have a copy of the cache. Before I publish this on the mailinglist > here, what is in the cache? No private information like private keys or > anything? I don't think it contains private keys, but it definitely contains secrets for the TLS sessions you have had live. Maybe you could get dbm people to help you debug the file for them, there are probably some dbm tools to inspect dbm files. Maybe Nikos can answer this better, I'm not a mod_gnutls developer (yet :)). /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > Simon Josefsson wrote: >> Next step would be to >> run 'gdb /usr/sbin/apache2 PID' or similar and then run 'bt'. >> Installing apache2-dbg may help, if you are on debian. > > I have a backtrace, see the attachment. The last bit is this: Thanks. Definitely looks like a dbm problem, it busy-loops in libdb: > #0 0x7f50f6eb7bc9 in __memp_fget () from /usr/lib/libdb-4.6.so > #1 0x7f50f6e86827 in __db_doff () from /usr/lib/libdb-4.6.so > #2 0x7f50f6e12dbc in __ham_del_pair () from /usr/lib/libdb-4.6.so > #3 0x7f50f6e09f02 in __ham_quick_delete () from /usr/lib/libdb-4.6.so > #4 0x7f50f6e6da3c in __db_del () from /usr/lib/libdb-4.6.so > #5 0x7f50f6e7fbfc in __db_del_pp () from /usr/lib/libdb-4.6.so > #6 0x7f50f7d0e377 in ?? () from /usr/lib/libaprutil-1.so.0 > #7 0x7f50f046368e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so > #8 0x7f50f046398e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so > #9 0x7f50f53f484d in _gnutls_store_session () from > /usr/lib/libgnutls.so.26 Maybe libdb people are interested in your cache file, to debug this, although it does contain sensitive information so be careful about sending it. You could install memcached and modify /etc/apache2/mods-available/gnutls.conf to use it instead of a dbm file. Btw, your MTA refuses direct e-mails: We're sorry, but the user account you are trying to reach has exceeded its size limit. As a result, we were unable to deliver this message to the intended recipient. Please try sending this message again at a later time. Reporting-MTA: DNS; mycingular.com Received-From-MTA: DNS; [204.9.89.153] Final-Recipient: RFC822; cla...@mycingular.com Action: failed Status: 5.2.2 Remote-MTA: DNS; mycingular.com Diagnostic-Code: 550 s...user over quota /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Simon Josefsson wrote: > I recall something like that, it happened if the cache was corrupt. > Maybe you could stop apache, copy away /var/cache/apache2/gnutls_cache, > and start apache again, to see if it solves the problem? Save the cache > file so we can try to debug why this happened. That worked! I have a copy of the cache. Before I publish this on the mailinglist here, what is in the cache? No private information like private keys or anything? -- Sander Marechal ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Simon Josefsson wrote: > Next step would be to > run 'gdb /usr/sbin/apache2 PID' or similar and then run 'bt'. > Installing apache2-dbg may help, if you are on debian. I have a backtrace, see the attachment. The last bit is this: #0 0x7f50f6eb7bc9 in __memp_fget () from /usr/lib/libdb-4.6.so #1 0x7f50f6e86827 in __db_doff () from /usr/lib/libdb-4.6.so #2 0x7f50f6e12dbc in __ham_del_pair () from /usr/lib/libdb-4.6.so #3 0x7f50f6e09f02 in __ham_quick_delete () from /usr/lib/libdb-4.6.so #4 0x7f50f6e6da3c in __db_del () from /usr/lib/libdb-4.6.so #5 0x7f50f6e7fbfc in __db_del_pp () from /usr/lib/libdb-4.6.so #6 0x7f50f7d0e377 in ?? () from /usr/lib/libaprutil-1.so.0 #7 0x7f50f046368e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so #8 0x7f50f046398e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so #9 0x7f50f53f484d in _gnutls_store_session () from /usr/lib/libgnutls.so.26 -- Sander Marechal (gdb) bt #0 0x7f50f6eb7bc9 in __memp_fget () from /usr/lib/libdb-4.6.so #1 0x7f50f6e86827 in __db_doff () from /usr/lib/libdb-4.6.so #2 0x7f50f6e12dbc in __ham_del_pair () from /usr/lib/libdb-4.6.so #3 0x7f50f6e09f02 in __ham_quick_delete () from /usr/lib/libdb-4.6.so #4 0x7f50f6e6da3c in __db_del () from /usr/lib/libdb-4.6.so #5 0x7f50f6e7fbfc in __db_del_pp () from /usr/lib/libdb-4.6.so #6 0x7f50f7d0e377 in ?? () from /usr/lib/libaprutil-1.so.0 #7 0x7f50f046368e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so #8 0x7f50f046398e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so #9 0x7f50f53f484d in _gnutls_store_session () from /usr/lib/libgnutls.so.26 #10 0x7f50f53f4ab4 in _gnutls_server_register_current_session () from /usr/lib/libgnutls.so.26 #11 0x7f50f53ee208 in _gnutls_handshake_common () from /usr/lib/libgnutls.so.26 #12 0x7f50f53ee2a2 in gnutls_handshake () from /usr/lib/libgnutls.so.26 #13 0x7f50f046249e in ?? () from /usr/lib/apache2/modules/mod_gnutls.so #14 0x7f50f0462837 in mgs_filter_input () from /usr/lib/apache2/modules/mod_gnutls.so #15 0x0042cbcb in ap_rgetline_core (s=0x17f3378, n=8192, read=0x7fff00362c00, r=0x17f3348, fold=0, bb=0x17f4ac0) at /build/buildd/apache2-2.2.9/server/protocol.c:231 #16 0x0042d520 in ap_read_request (conn=0x1a32d08) at /build/buildd/apache2-2.2.9/server/protocol.c:596 #17 0x004466d0 in ap_process_http_connection (c=0x1a32d08) at /build/buildd/apache2-2.2.9/modules/http/http_core.c:183 #18 0x004403d3 in ap_run_process_connection (c=0x1a32d08) at /build/buildd/apache2-2.2.9/server/connection.c:43 #19 0x0044dc20 in child_main (child_num_arg=) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:680 #20 0x0044df74 in make_child (s=0x1575968, slot=1) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:777 #21 0x0044ebb6 in ap_mpm_run (_pconf=, plog=, s=) at /build/buildd/apache2-2.2.9/server/mpm/prefork/prefork.c:912 #22 0x00425be5 in main (argc=3, argv=0x7fff003630f8) at /build/buildd/apache2-2.2.9/server/main.c:732 ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > Simon Josefsson wrote: >> If strace doesn't show anything, it means it isn't doing any syscalls, >> which can happen if it is stuck in a busy loop. Next step would be to >> run 'gdb /usr/sbin/apache2 PID' or similar and then run 'bt'. >> Installing apache2-dbg may help, if you are on debian. > > I'm on Debian. > > Meanwhile, I managed to get an strace by reducing the number of forks > that Apache makes and attaching an strace to them all. See the > attachment (I hope your mailinglist accepts attachments). At the end of > this trace nothing happens anymore but the process still uses 100% CPU. > > I'll get a GDB backtrace next. I recall something like that, it happened if the cache was corrupt. Maybe you could stop apache, copy away /var/cache/apache2/gnutls_cache, and start apache again, to see if it solves the problem? Save the cache file so we can try to debug why this happened. /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Simon Josefsson wrote:
> If strace doesn't show anything, it means it isn't doing any syscalls,
> which can happen if it is stuck in a busy loop. Next step would be to
> run 'gdb /usr/sbin/apache2 PID' or similar and then run 'bt'.
> Installing apache2-dbg may help, if you are on debian.
I'm on Debian.
Meanwhile, I managed to get an strace by reducing the number of forks
that Apache makes and attaching an strace to them all. See the
attachment (I hope your mailinglist accepts attachments). At the end of
this trace nothing happens anymore but the process still uses 100% CPU.
I'll get a GDB backtrace next.
--
Sander Marechal
Process 5310 attached - interrupt to quit
semop(3604498, 0x7f50f7cfda60, 1) = 0
epoll_wait(18, {{EPOLLIN, {u32=27462808, u64=27462808}}}, 2, 4294967295) = 1
accept(7, {sa_family=AF_INET6, sin6_port=htons(49176), inet_pton(AF_INET6,
":::85.113.252.144", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
[68719476764]) = 19
semop(3604498, 0x7f50f7cfda66, 1) = 0
getsockname(19, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6,
":::192.168.1.5", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
[68719476764]) = 0
fcntl(19, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(19, F_SETFL, O_RDWR|O_NONBLOCK) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 128008}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=12, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718629966
getrusage(RUSAGE_SELF, {ru_utime={0, 128008}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=12, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718629966
read(19,
"\26\3\1\0g\1\0\0c\3\1I\361\2045\210\315\230,\203\32\335\366\376V\236\267\2\307\255\245\356A"...,
8000) = 108
getrusage(RUSAGE_SELF, {ru_utime={0, 128008}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=12, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718629966
writev(19,
[{"\26\3\1\0J\2\0\0F\3\1I\361\205|\247\253h\353\250\365\351\357y\374\4\0\177\363\335!C!"...,
79}], 1) = 79
writev(19,
[{"\26\3\1\4q\v\0\4m\0\4j\0\4g0\202\4c0\202\2K\240\3\2\1\2\2\3\6o\25"...,
1142}], 1) = 1142
getrusage(RUSAGE_SELF, {ru_utime={0, 128008}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=12, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718629966
getrusage(RUSAGE_SELF, {ru_utime={0, 148009}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=14, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718629968
getrusage(RUSAGE_SELF, {ru_utime={0, 148009}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=14, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718629968
writev(19,
[{"\26\3\1\2\215\f\0\2\211\1\0\254k\333A2J\232\233\361f\336^\23\211X/\257r\266e\31\207"...,
658}], 1) = 658
writev(19, [{"\26\3\1\0\t\r\0\0\5\2\1\2\0\0"..., 14}], 1) = 14
writev(19, [{"\26\3\1\0\4\16\0\0\0"..., 9}], 1) = 9
poll([{fd=19, events=POLLIN}], 1, 30) = 1 ([{fd=19, revents=POLLIN}])
read(19, "\25\3\1\0\2\0020"..., 8000) = 7
write(2, "[Fri Apr 24 11:25:16 2009] [error"..., 131) = 131
writev(19, [{"\25\3\1\0\2\2G"..., 7}], 1) = 7
poll([{fd=19, events=POLLIN}], 1, 30) = 1 ([{fd=19, revents=POLLIN}])
read(19, ""..., 8000) = 0
shutdown(19, 1 /* send */) = 0
poll([{fd=19, events=POLLIN}], 1, 2000) = 1 ([{fd=19, revents=POLLIN|POLLHUP}])
read(19, ""..., 512)= 0
close(19) = 0
read(9, 0x7fff00362cc7, 1) = -1 EAGAIN (Resource temporarily
unavailable)
semop(3604498, 0x7f50f7cfda60, 1) = 0
epoll_wait(18, {{EPOLLIN, {u32=27462808, u64=27462808}}}, 2, 4294967295) = 1
accept(7, {sa_family=AF_INET6, sin6_port=htons(56853), inet_pton(AF_INET6,
":::192.168.1.2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
[68719476764]) = 19
semop(3604498, 0x7f50f7cfda66, 1) = 0
getsockname(19, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6,
":::192.168.1.5", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
[68719476764]) = 0
fcntl(19, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(19, F_SETFL, O_RDWR|O_NONBLOCK) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 152009}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=15, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718630341
getrusage(RUSAGE_SELF, {ru_utime={0, 152009}, ru_stime={0, 24001}, ...}) = 0
times({tms_utime=15, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 1718630341
read(19,
"\26\3\0\0o\1\0\0k\3\0I\361\205+-5\342\252~\35\301aM\10#\16#pS{Y\207"..., 8000)
= 116
stat("/var/cache/apache2/gnutls_cache", {st_mode=S_IFREG|0644, st_size=147456,
...}) = 0
open("/var/cache/apache2/gnutls_cache", O_RDWR) = 20
fcntl(20, F_SETFD, FD_CLOEXEC) = 0
read(20,
"\0\0\0\0\1\0\0\0\0\0\0\0a\25\6\0\t\0\0\0\0\20\0\0\0\10\0\0\34\0\0\0&"..., 512)
= 512
close(20) = 0
open("DB_CONFIG", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/var/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
open("/proc/stat", O_RDONLY)= 20
fstat(20, {st_mode=S_IFREG|044
Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > Simon Josefsson wrote: >> Sander Marechal writes: >> >>> How can I debug this? Here's a typical configuration for one of my domains: >> >> What does 'strace -p PID' for the PIDs of the apache daemon indicate? > > Nothing, but I'm probably not doing it right. When I run the strace on > the process of the request I am making then it is showing nothing, but > at that point the process is already running and using 100% CPU. > > I think that to get strace output I need to run it as soon as it starts. > But how do I do that? I don't know the pid in advance and I can only run > strace when I know the pid and the process is already running. > > Is there any way to do this automatically when the next Apache process > starts or something? If strace doesn't show anything, it means it isn't doing any syscalls, which can happen if it is stuck in a busy loop. Next step would be to run 'gdb /usr/sbin/apache2 PID' or similar and then run 'bt'. Installing apache2-dbg may help, if you are on debian. /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Simon Josefsson wrote: > Sander Marechal writes: > >> How can I debug this? Here's a typical configuration for one of my domains: > > What does 'strace -p PID' for the PIDs of the apache daemon indicate? Nothing, but I'm probably not doing it right. When I run the strace on the process of the request I am making then it is showing nothing, but at that point the process is already running and using 100% CPU. I think that to get strace output I need to run it as soon as it starts. But how do I do that? I don't know the pid in advance and I can only run strace when I know the pid and the process is already running. Is there any way to do this automatically when the next Apache process starts or something? -- Sander ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
Re: [Modules] mod_gnutls making Apache use 100% CPU
Sander Marechal writes: > How can I debug this? Here's a typical configuration for one of my domains: What does 'strace -p PID' for the PIDs of the apache daemon indicate? /Simon ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
[Modules] mod_gnutls making Apache use 100% CPU
Hello, I have a problem with mod_gnutls. It makes Apache use 100% CPU. When I visit one on my domains on the server with a browser it just keeps on "connecting..." forever (this is Firefox 3 on Linux, it has SNI support). There is nothing in the logfiles and loglevel is set to debug. I am sure it is caused by mod_gnutls. All domains that do not use mod_gnutls work fine. Requests made to domains that do use mod_gnutls never get anywhere. I ran my PHP debugger (xdebug) and it doesn't show up, meaning that the request never even makes it to PHP. It gets stuck before that. Everything was working fine up to 5 AM this morning (as indicated by the logfiles). Nothing changed on the server. I tried restarting Apache and even rebooting the server. Didn't help. How can I debug this? Here's a typical configuration for one of my domains: DocumentRoot /path/to/docroot ServerName example.org:443 # SSL using GnuTLS GnuTLSEnable On GnuTLSPriorities PERFORMANCE:%COMPAT GnuTLSCertificateFile /etc/apache2/ssl/example.org.cert GnuTLSKeyFile /root/certs/example.org.key GnuTLSClientVerify require GnuTLSClientCAFile /etc/ssl/certs/cacert.org.pem ErrorLog /var/log/apache2/error.log LogLevel debug CustomLog /var/log/apache2/access.log combined ServerSignature On -- Sander Marechal ___ Modules mailing list Modules@lists.outoforder.cc http://lists.outoforder.cc/mailman/listinfo/modules
