[users@httpd] help interpreting an Apache 2.2 segmentation fault backtrace. [I]
Classification: For internal use only Hi guys, I wonder if you could me some clues about how to interpret the following, frequently observed, backtrace from an Apache 2.2 segmentation fault (Apache 2.2.29, 32-bit, Linux, kernel 2.6.32.59-0.7-default SLES11 SP2). For this particular request, we've got two modules that are involved. a) RSA (now EMC) Cleartrust Access Manager agent (4.7.0.66/Linux/32-bit) (libct_apache22_agent) handling SSO authentication/authorization b) Oracle Weblogic Webserver Connector Plugin (mod_wl) handling proxying requests to an Weblogic Servlet Container backend. To me it looks like some kind of APR pool corruption or a corruption of the pointer for the APR pool, and it looks like the mod_wl handler is called first and then parts of the libct module are getting called and finally segfaulting inside the APR routines. Anyone care to improve that interpretation? GNU gdb 6.6 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i586-suse-linux... Using host libthread_db library /usr/lib/libthread_db.so.1. Core was generated by `/apache22/bin/httpd -f /apache22/conf/dynamic/hsm_apache22/httpd.conf -Dnick=uk'. Program terminated with signal 11, Segmentation fault. #0 0xf7612ac1 in allocator_alloc () from /versions/apache-2.2.29/lib/libapr-1.so.0 (gdb) bt #0 0xf7612ac1 in allocator_alloc () from /versions/apache-2.2.29/lib/libapr-1.so.0 #1 0xf6f12f40 in apr_sdbm_rdonly () from /apache22/modules/libct_apache22_agent.so #2 0x0010 in ?? () #3 0xf7026ea8 in apr_sdbm_rdonly () from /apache22/modules/libct_apache22_agent.so #4 0x0007 in ?? () #5 0x001f in ?? () #6 0xffceaa08 in ?? () #7 0xf6e931a3 in ct_malloc () from /apache22/modules/libct_apache22_agent.so #8 0xf76134e1 in apr_pool_create_ex () from /versions/apache-2.2.29/lib/libapr-1.so.0 #9 0x0b97af00 in ?? () #10 0x1fe8 in ?? () #11 0xffceaa4c in ?? () #12 0xf70f74d7 in operator new () from /usr/lib/libstdc++.so.6 #13 0xf71410bb in _ZN6ReaderC9EP10apr_pool_tiP10LogContext () from /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so #14 0xf7141212 in Reader::Reader () from /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so #15 0xf71587b6 in URL::connect () from /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so #16 0xf7153bd8 in ApacheProxy::getConnection () from /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so #17 0xf713cd63 in request_handler () from /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so #18 0x080aaa47 in ap_run_handler () #19 0x0b9a9298 in ?? () #20 0x in ?? () (gdb) Regards, Mark --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] help interpreting an Apache 2.2 segmentation fault backtrace. [I]
Hi Mark, Zitat von Mark Blackman mark.black...@db.com: Classification: For internal use only Hi guys, I wonder if you could me some clues about how to interpret the following, frequently observed, backtrace from an Apache 2.2 segmentation fault (Apache 2.2.29, 32-bit, Linux, kernel 2.6.32.59-0.7-default SLES11 SP2). [...] To me it looks like some kind of APR pool corruption or a corruption of the pointer for the APR pool, and it looks like the mod_wl handler is called first and then parts of the libct module are getting called and finally segfaulting inside the APR routines. Anyone care to improve that interpretation? it might be as simple as a heap corruption. Does apache22 use glibc malloc() routines? Then you might give setting MALLOC_CHECK_ to a non-zero value a try. OTOH, this may significantly reduce the performance of your httpd. [...] Program terminated with signal 11, Segmentation fault. #0 0xf7612ac1 in allocator_alloc () from /versions/apache-2.2.29/lib/libapr-1.so.0 (gdb) bt #0 0xf7612ac1 in allocator_alloc () from /versions/apache-2.2.29/lib/libapr-1.so.0 #1 0xf6f12f40 in apr_sdbm_rdonly () from /apache22/modules/libct_apache22_agent.so #2 0x0010 in ?? () #3 0xf7026ea8 in apr_sdbm_rdonly () from /apache22/modules/libct_apache22_agent.so #4 0x0007 in ?? () #5 0x001f in ?? () #6 0xffceaa08 in ?? () #7 0xf6e931a3 in ct_malloc () from /apache22/modules/libct_apache22_agent.so #8 0xf76134e1 in apr_pool_create_ex () from /versions/apache-2.2.29/lib/libapr-1.so.0 #9 0x0b97af00 in ?? () #10 0x1fe8 in ?? () #11 0xffceaa4c in ?? () #12 0xf70f74d7 in operator new () from /usr/lib/libstdc++.so.6 #13 0xf71410bb in _ZN6ReaderC9EP10apr_pool_tiP10LogContext () from /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so [...] Regards, Jens - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] help interpreting an Apache 2.2 segmentation fault backtrace. [I]
Classification: For internal use only Thanks, it is almost certainly heap corruption, but I would need something to point the finger at. Considering that httpd 2.2.29 is so mature, I'd be surprised if it's any httpd code. I'm personally speculating that one of either those two modules is interfering with the allocations of the other, but I'd like to confirm that and track it to one or the other. We do sometimes get Abort-triggered core dumps where glibc is triggering an Abort on a 'double free or corruption', suggesting that some of the MALLOC_CHECK_s checks are being invoked, although the environment variable hasn't be set. I need to remedy this, so I need more than just a statement of heap corruption ideally, but a specific cause and remedy. Abort-triggered dump here: *** glibc detected *** /apache22/bin/httpd: double free or corruption (out): 0x0bbf9568 *** === Backtrace: = /usr/lib/libc.so.6[0xf757e432] /usr/lib/libc.so.6[0xf757fc3f] /usr/lib/libc.so.6(__libc_free+0x84)[0xf757ff94] /versions/apache-2.2.29/lib/libapr-1.so.0[0xf76bee3f] /versions/apache-2.2.29/lib/libapr-1.so.0(apr_pool_destroy+0x114)[0xf76bf45f] /versions/wlsplugins_generic_11.1.1.7.0/lib/mod_wl.so(_ZN6Reader5resetEv+0x3c)[0xf71ed9d4] === Memory map: 08048000-0827c000 r-xp fd:04 2671007 /versions/apache-2.2.29/bin/httpd 0827c000-0828a000 rwxp 00233000 fd:04 2671007 /versions/apache-2.2.29/bin/httpd 0828a000-0bd21000 rwxp 00:00 0 [heap] I am talking to Oracle about the Weblogic proxying module (WLS Plugin), but they've not come up much of a diagnosis yet. This is all prefork, so it's not one thread interfering with another. How hard/common is it for one module to screw up the pool allocation (for all the modules) for a request? -Original Message- From: Jens-U. Mozdzen [mailto:jmozd...@nde.ag] Sent: 06 August 2015 13:54 To: users@httpd.apache.org Subject: Re: [users@httpd] help interpreting an Apache 2.2 segmentation fault backtrace. [I] ... it might be as simple as a heap corruption. Does apache22 use glibc malloc() routines? Then you might give setting MALLOC_CHECK_ to a non-zero value a try. OTOH, this may significantly reduce the performance of your httpd. --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm for information about privacy. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
