Hi Massimo!
04.09.2015 12:56, Massimo Manghi пишет:
Are you loading some binary extensions in that application?
My first attempt was to eliminate all binary extensions. There was a
few. I temporarily commented out all of them. Except for Itcl, which is
a base. Apps cannot run without it.
I can make gdb backtraces now.
Backtraces aren't the same all the times.
* Here's #1:
# gdb /usr/sbin/apache2 /tmp/core.1
...
[New LWP 4726]
[New LWP 4725]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fa43ec992b3 in select () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007fa43ec992b3 in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fa43a29a435 in NotifierThreadProc () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#2 0x00007fa43ef6b0a4 in start_thread (arg=0x7fa438060700) at
pthread_create.c:309
#3 0x00007fa43eca004d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Here's #2:
(gdb) bt
#0 0x00007fa43ebef107 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fa43ebf04e8 in __GI_abort () at abort.c:89
#2 0x00007fa43a25f11e in Tcl_PanicVA () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#3 0x00007fa43a25f1ad in Tcl_Panic () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#4 0x00007fa43a2716a6 in Tcl_AppendStringsToObjVA () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#5 0x00007fa43a27177d in Tcl_AppendStringsToObj () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#6 0x00007fa43713d2fa in ItclMapMethodNameProc () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#7 0x00007fa43a29c851 in TclOOObjectCmdCore () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#8 0x00007fa43a19da77 in TclNRRunCallbacks () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#9 0x00007fa43713c724 in ItclObjectCmd () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#10 0x00007fa437137cb8 in CallItclObjectCmd () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#11 0x00007fa43a19da77 in TclNRRunCallbacks () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#12 0x00007fa437137e38 in Itcl_EvalMemberCode () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#13 0x00007fa437138b24 in Itcl_InvokeMethodIfExists () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#14 0x00007fa43713a8bb in ItclDestructBase () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#15 0x00007fa43713a865 in ItclDestructBase () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#16 0x00007fa43713a941 in CallDestructBase () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#17 0x00007fa43a19da77 in TclNRRunCallbacks () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#18 0x00007fa43713acc4 in ObjectRenamedTrace () from
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
#19 0x00007fa43a19aeaf in CallCommandTraces () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#20 0x00007fa43a19b041 in Tcl_DeleteCommandFromToken () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#21 0x00007fa43a256a9d in TclTeardownNamespace () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#22 0x00007fa43a19c54c in DeleteInterpProc () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#23 0x00007fa43a19b0ae in Tcl_DeleteCommandFromToken () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#24 0x00007fa43a256a9d in TclTeardownNamespace () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#25 0x00007fa43a19c54c in DeleteInterpProc () from
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
#26 0x00007fa43a3ff3b5 in Rivet_ProcessorCleanup (data=0x7fa43fb690a0)
at mod_rivet/mod_rivet.c:501
#27 0x00007fa439426f88 in Rivet_MPM_Finalize (data=<optimized out>) at
mod_rivet/rivet_prefork_mpm.c:52
#28 0x00007fa43f19c9be in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#29 0x00007fa43bad120e in ?? () from
/usr/lib/apache2/modules/mod_mpm_prefork.so
#30 0x00007fa43bad1696 in ?? () from
/usr/lib/apache2/modules/mod_mpm_prefork.so
#31 0x00007fa43bad1a01 in ?? () from
/usr/lib/apache2/modules/mod_mpm_prefork.so
#32 0x00007fa43bad2667 in ?? () from
/usr/lib/apache2/modules/mod_mpm_prefork.so
#33 0x00007fa43faace7e in ap_run_mpm ()
#34 0x00007fa43faa63c3 in main ()
(gdb)
Here is shared library info:
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
No linux-vdso.so.1
0x00007fa43f5df710 0x00007fa43f62e0c5 Yes (*)
/lib/x86_64-linux-gnu/libpcre.so.3
0x00007fa43f3be7e0 0x00007fa43f3d3f7f Yes (*)
/usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
0x00007fa43f18ce90 0x00007fa43f1a87cf Yes (*)
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
0x00007fa43ef689f0 0x00007fa43ef74731 Yes
/lib/x86_64-linux-gnu/libpthread.so.0
0x00007fa43ebd94a0 0x00007fa43ed036a3 Yes
/lib/x86_64-linux-gnu/libc.so.6
0x00007fa43e9b6620 0x00007fa43e9b7ce7 Yes (*)
/lib/x86_64-linux-gnu/libuuid.so.1
0x00007fa43e7af350 0x00007fa43e7b206c Yes
/lib/x86_64-linux-gnu/librt.so.1
0x00007fa43e576cc0 0x00007fa43e57b8b4 Yes
/lib/x86_64-linux-gnu/libcrypt.so.1
0x00007fa43e372ed0 0x00007fa43e37397e Yes
/lib/x86_64-linux-gnu/libdl.so.2
0x00007fa43e14cb90 0x00007fa43e164ff9 Yes (*)
/lib/x86_64-linux-gnu/libexpat.so.1
0x00007fa43f84cae0 0x00007fa43f865170 Yes
/lib64/ld-linux-x86-64.so.2
0x00007fa439c583b0 0x00007fa439c5cb4e Yes
/lib/x86_64-linux-gnu/libnss_compat.so.2
0x00007fa439a43160 0x00007fa439a4e693 Yes
/lib/x86_64-linux-gnu/libnsl.so.1
0x00007fa4398361a0 0x00007fa43983c1bc Yes
/lib/x86_64-linux-gnu/libnss_nis.so.2
0x00007fa43962a2a0 0x00007fa439630803 Yes
/lib/x86_64-linux-gnu/libnss_files.so.2
0x00007fa4385668c0 0x00007fa4385683bb Yes
/usr/local/lib/rivet/librivetlib.so
0x00007fa43df46cf0 0x00007fa43df47603 Yes (*)
/usr/lib/apache2/modules/mod_access_compat.so
0x00007fa43dd42120 0x00007fa43dd430f3 Yes (*)
/usr/lib/apache2/modules/mod_alias.so
0x00007fa43db3dfe0 0x00007fa43db3f05e Yes (*)
/usr/lib/apache2/modules/mod_auth_basic.so
0x00007fa43d939f70 0x00007fa43d93a712 Yes (*)
/usr/lib/apache2/modules/mod_authn_core.so
0x00007fa43d736ae0 0x00007fa43d73703a Yes (*)
/usr/lib/apache2/modules/mod_authn_file.so
0x00007fa43d5305d0 0x00007fa43d53220a Yes (*)
/usr/lib/apache2/modules/mod_authz_core.so
0x00007fa43d32bfd0 0x00007fa43d32c83a Yes (*)
/usr/lib/apache2/modules/mod_authz_groupfile.so
0x00007fa43d128de0 0x00007fa43d1294fc Yes (*)
/usr/lib/apache2/modules/mod_authz_host.so
0x00007fa43cf25980 0x00007fa43cf25daa Yes (*)
/usr/lib/apache2/modules/mod_authz_user.so
0x00007fa43cd1d420 0x00007fa43cd2187a Yes (*)
/usr/lib/apache2/modules/mod_autoindex.so
0x00007fa43cb16270 0x00007fa43cb185b8 Yes (*)
/usr/lib/apache2/modules/mod_cgi.so
0x00007fa43c90b9c0 0x00007fa43c90fd9a Yes (*)
/usr/lib/apache2/modules/mod_deflate.so
0x00007fa43c6f1220 0x00007fa43c702469 Yes (*)
/lib/x86_64-linux-gnu/libz.so.1
0x00007fa43c4ebf10 0x00007fa43c4ec95f Yes (*)
/usr/lib/apache2/modules/mod_dir.so
0x00007fa43c2e8b30 0x00007fa43c2e8eab Yes (*)
/usr/lib/apache2/modules/mod_env.so
0x00007fa43c0e4160 0x00007fa43c0e5f12 Yes (*)
/usr/lib/apache2/modules/mod_filter.so
0x00007fa43bede860 0x00007fa43bee037d Yes (*)
/usr/lib/apache2/modules/mod_headers.so
0x00007fa43bcd8700 0x00007fa43bcd9df1 Yes (*)
/usr/lib/apache2/modules/mod_mime.so
0x00007fa43bad07f0 0x00007fa43bad2de0 Yes (*)
/usr/lib/apache2/modules/mod_mpm_prefork.so
0x00007fa43b8c6ed0 0x00007fa43b8cb263 Yes (*)
/usr/lib/apache2/modules/mod_negotiation.so
0x00007fa43b6c1f10 0x00007fa43b6c3035 Yes (*)
/usr/lib/apache2/modules/mod_reqtimeout.so
0x00007fa43b4b3350 0x00007fa43b4bbf1a Yes (*)
/usr/lib/apache2/modules/mod_rewrite.so
0x00007fa43b2ad110 0x00007fa43b2ae002 Yes (*)
/usr/lib/apache2/modules/mod_setenvif.so
0x00007fa43b0a5db0 0x00007fa43b0a8571 Yes (*)
/usr/lib/apache2/modules/mod_socache_shmcb.so
---Type <return> to continue, or q <return> to quit---
0x00007fa43ae7aa40 0x00007fa43ae93839 Yes (*)
/usr/lib/apache2/modules/mod_ssl.so
0x00007fa43ac22db0 0x00007fa43ac573ee Yes (*)
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
0x00007fa43a886200 0x00007fa43a97bcdf Yes (*)
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
0x00007fa43a60f270 0x00007fa43a611147 Yes (*)
/usr/lib/apache2/modules/mod_status.so
0x00007fa43a3fe9e0 0x00007fa43a408dfc Yes
/usr/lib/apache2/modules/mod_rivet.so
0x00007fa43a18c1f0 0x00007fa43a2b1fb8 Yes (*)
/opt/ActiveTcl-8.6/lib/libtcl8.6.so
0x00007fa439e64580 0x00007fa439ecfd96 Yes
/lib/x86_64-linux-gnu/libm.so.6
0x00007fa439426c70 0x00007fa439427061 Yes
/usr/local/lib/rivet/mpm/rivet_prefork_mpm.so
0x00007fa437116250 0x00007fa43714b208 Yes (*)
/opt/ActiveTcl-8.6/lib/itcl4.0.3/libitcl4.0.3.so
0x00007fa436ffde50 0x00007fa437007698 Yes (*)
/opt/ActiveTcl-8.6/lib/teapot/package/linux-glibc2.3-x86_64/lib/Oratcl4.5/libOratcl4.5.so
0x00007fa434de6250 0x00007fa435739a88 Yes
/opt/oracle/home1/lib/libclntsh.so
0x00007fa434631060 0x00007fa4346618a8 Yes (*)
/opt/oracle/home1/lib/libnnz11.so
0x00007fa4343de630 0x00007fa4343de872 Yes (*)
/lib/x86_64-linux-gnu/libaio.so.1
0x00007fa434213470 0x00007fa434217c98 Yes (*)
/opt/oracle/home1/lib/libnque11.so
0x00007fa433bec570 0x00007fa433c00b68 Yes (*)
/opt/ActiveTcl-8.6/lib/teapot/package/linux-glibc2.3-x86_64/lib/Trf2.1.4/libTrf2.1.4.so
0x00007fa433921300 0x00007fa433a39258 Yes (*)
/opt/ActiveTcl-8.6/lib/teapot/package/linux-glibc2.3-x86_64/lib/tls1.6.5/libtls1.6.5.so
(*): Shared library is missing debugging information.
(gdb)
--
Best regards, Anton.
-- Massimo
On 09/04/2015 08:45 AM, Anton Osennikov wrote:
Here's complete block of lines from error.log around last "signal
Aborted", for apache child pid 4789. "create interpreter" and "run
interpreter" lines are from my custom rivet-childinit.tcl. They are
about creating and running slave interpreters for apps in virtual hosts.
'nemoweb' application causes the problem.
2015-09-04 11:28:38 create interpreter 'nemoweb' [4789]
2015-09-04 11:28:39 run interpreter 'appsvc' [4680]
2015-09-04 11:28:44 run interpreter 'nemoweb' [4789]
2015-09-04 11:28:46 create interpreter 'cabinet/rodnsvyaz' [5060]
[Fri Sep 04 11:28:46.660949 2015] [:debug] [pid 4660]
mod_rivet/rivet_prefork_mpm.c(49): (20014)Internal error: mod_rivet:
Running prefork bridge finalize method
[Fri Sep 04 11:28:46.660974 2015] [:debug] [pid 4660]
mod_rivet/mod_rivet.c(464): Thread exiting after 0 requests served (18
vhosts)
2015-09-04 11:28:47 create interpreter 'appsvc' [5068]
2015-09-04 11:28:47 run interpreter 'appsvc' [5068]
2015-09-04 11:28:47 run interpreter 'cabinet/rodnsvyaz' [5060]
[Fri Sep 04 11:28:49.663374 2015] [:debug] [pid 4835]
mod_rivet/rivet_prefork_mpm.c(49): (20014)Internal error: mod_rivet:
Running prefork bridge finalize method
[Fri Sep 04 11:28:49.663396 2015] [:debug] [pid 4835]
mod_rivet/mod_rivet.c(464): Thread exiting after 0 requests served (18
vhosts)
2015-09-04 11:28:50 run interpreter 'appsvc' [4680]
2015-09-04 11:28:53 create interpreter 'appsvc' [4792]
2015-09-04 11:28:53 run interpreter 'appsvc' [4792]
2015-09-04 11:28:56 run interpreter 'appsvc' [4763]
[Fri Sep 04 11:28:57.673112 2015] [:debug] [pid 4789]
mod_rivet/rivet_prefork_mpm.c(49): (20014)Internal error: mod_rivet:
Running prefork bridge finalize method
[Fri Sep 04 11:28:57.673138 2015] [:debug] [pid 4789]
mod_rivet/mod_rivet.c(464): Thread exiting after 0 requests served (18
vhosts)
Tcl_AppendStringsToObj called with shared object
2015-09-04 11:28:58 create interpreter 'appsvc' [5058]
2015-09-04 11:28:58 run interpreter 'appsvc' [5058]
2015-09-04 11:28:59 run interpreter 'cabinet/rodnsvyaz' [4680]
2015-09-04 11:28:59 run interpreter 'cabinet/rodnsvyaz' [5060]
[Fri Sep 04 11:28:59.674532 2015] [core:notice] [pid 1330] AH00051:
child pid 4789 exit signal Aborted (6), possible coredump in /tmp
2015-09-04 11:29:02 run interpreter 'appsvc' [5068]
2015-09-04 11:29:04 run interpreter 'appsvc' [4763]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
