I haven't seen anything, but it also looks like you're using a Transformation Plugin, if you'd be willing to share code we could possibly help.
Brian On Fri, Oct 4, 2013 at 8:17 AM, James Peach <[email protected]> wrote: > On Oct 3, 2013, at 10:52 PM, Vladyslav Bachynskyi <[email protected]> > wrote: > > > Hi All, > > > > One of my ats servers was crashed :( > > My system information: > > > > Centos 6.4 - 2.6.32-358.18.1.el6.x86_64 > > Apache Traffic Server - traffic_server - 4.0.1 - (build # 8913 on Sep 9 > > 2013 at 13:16:57) > > > > traffic.out output: > > > > NOTE: Traffic Server received Sig 11: Segmentation fault > > /opt/trafficserver-4.0.1/bin/traffic_server - STACK TRACE: > > /lib64/libpthread.so.0(+0x3b3300f500)[0x2abf3e03a500] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN10HttpTunnel12producer_runEP18HttpTunnelProducer+0x25f)[0x56b8ef] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN10HttpTunnel10tunnel_runEP18HttpTunnelProducer+0xd9)[0x56ca49] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM23setup_internal_transferEMS_FiiPvE+0x633)[0x534ce3] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x3e1)[0x5353c1] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x547)[0x532e87] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x3b7)[0x52b637] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x8b)[0x5314fb] > > > /opt/trafficserver-4.0.1/bin/traffic_server(TSHttpTxnReenable+0x404)[0x4b8694] > > > /usr/local/lib/libatscppapi.so.1(_ZN9atscppapi5utils8internal20invokePluginForEventEPNS_17TransactionPluginEP13tsapi_httptxn7TSEvent+0x195)[0x2ac01c4ded45] > > It looks like you are using the LinkedIn C++ API in a transform plugin? > Brian, does this ring any bells? > > > /usr/local/lib/libatscppapi.so.1(+0xfa8e)[0x2ac01c4e0a8e] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x116)[0x52b396] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x114)[0x532a54] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x527a0d] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x531308] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d6)[0x4e7d46] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a2fcf] > > > /opt/trafficserver-4.0.1/bin/traffic_server(_ZN7EThread7executeEv+0x683)[0x6a3b93] > > /opt/trafficserver-4.0.1/bin/traffic_server[0x6a1e6a] > > /lib64/libpthread.so.0(+0x3b33007851)[0x2abf3e032851] > > /lib64/libc.so.6(clone+0x6d)[0x3b32ce894d] > > > > Linux syslog output (it's not a full log output, as messages repeats. I > > can provide full log output if needed): > > > > Oct 3 23:01:53 ats-node01 kernel: [ET_NET 22][2719]: segfault at 10 ip > > 000000000056b8ef sp 00002abf459156e0 error 4 in > > traffic_server[400000+350000] > > Oct 3 23:01:53 ats-node01 abrt[9515]: > > /opt/trafficserver-4.0.1/core.2696 fd(-1) is not a regular file with > > link count 1: Permission denied > > Oct 3 23:04:22 ats-node01 kernel: INFO: task [ET_NET 0]:2696 blocked > > for more than 120 seconds. > > Oct 3 23:04:22 ats-node01 kernel: "echo 0 > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > Oct 3 23:04:22 ats-node01 kernel: [ET_NET 0] D 0000000000000009 > > 0 2696 2648 0x00000080 > > Oct 3 23:04:22 ats-node01 kernel: ffff8804ba037c98 0000000000000082 > > ffff8804ba037c60 ffff8804ba037c5c > > Oct 3 23:04:22 ats-node01 kernel: ffff8804ba037c68 ffff88033fc25400 > > ffff880028256700 00000000000009fd > > Oct 3 23:04:22 ats-node01 kernel: ffff8804bb8c45f8 ffff8804ba037fd8 > > 000000000000fb88 ffff8804bb8c45f8 > > Oct 3 23:04:22 ats-node01 kernel: Call Trace: > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81073095>] > exit_mm+0x95/0x180 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff810734df>] > do_exit+0x15f/0x870 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8150e130>] ? > > thread_return+0x4e/0x76e > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8109b2c1>] ? > > lock_hrtimer_base+0x31/0x60 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81073c48>] > > do_group_exit+0x58/0xd0 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81088f36>] > > get_signal_to_deliver+0x1f6/0x460 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8100a265>] > do_signal+0x75/0x800 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff811c7ac6>] ? > > ep_poll+0x306/0x330 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81063410>] ? > > default_wake_function+0x0/0x20 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8121bed6>] ? > > security_file_permission+0x16/0x20 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8100aa80>] > > do_notify_resume+0x90/0xc0 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8100b341>] > int_signal+0x12/0x17 > > Oct 3 23:04:22 ats-node01 kernel: INFO: task traffic_server:2697 > > blocked for more than 120 seconds. > > Oct 3 23:04:22 ats-node01 kernel: "echo 0 > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > Oct 3 23:04:22 ats-node01 kernel: traffic_serve D 0000000000000000 > > 0 2697 2648 0x00000080 > > Oct 3 23:04:22 ats-node01 kernel: ffff8804bb8d9c98 0000000000000082 > > ffff8804bb8d9cb8 ffffffff8112f9bc > > Oct 3 23:04:22 ats-node01 kernel: ffff8804bb8d9c68 ffffffff810aa5e0 > > ffff8804bb8d9c28 ffffffff81277679 > > Oct 3 23:04:22 ats-node01 kernel: ffff8804ba4a6638 ffff8804bb8d9fd8 > > 000000000000fb88 ffff8804ba4a6638 > > Oct 3 23:04:22 ats-node01 kernel: Call Trace: > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8112f9bc>] ? > > release_pages+0x21c/0x250 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff810aa5e0>] ? > > exit_robust_list+0x90/0x160 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81277679>] ? > > cpumask_next_and+0x29/0x50 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81073095>] > exit_mm+0x95/0x180 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff810734df>] > do_exit+0x15f/0x870 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81073c48>] > > do_group_exit+0x58/0xd0 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff81088f36>] > > get_signal_to_deliver+0x1f6/0x460 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8100a265>] > do_signal+0x75/0x800 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8109c0f7>] ? > > hrtimer_nanosleep+0xe7/0x180 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8109af20>] ? > > hrtimer_wakeup+0x0/0x30 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8100aa80>] > > do_notify_resume+0x90/0xc0 > > Oct 3 23:04:22 ats-node01 kernel: [<ffffffff8100b341>] > int_signal+0x12/0x17 > > > > =========== [ SKIP ] =============== > > > > Oct 3 23:04:47 ats-node01 abrt[9515]: Saved core dump of pid 2696 > > (/opt/trafficserver-4.0.1/bin/traffic_server) to > > /var/spool/abrt/ccpp-2013-10-03-23:01:53-2696 (12643663872 bytes) > > Oct 3 23:04:47 ats-node01 abrtd: Directory > > 'ccpp-2013-10-03-23:01:53-2696' creation detected > > Oct 3 23:04:47 ats-node01 abrtd: Executable > > '/opt/trafficserver-4.0.1/bin/traffic_server' doesn't belong to any > package > > Oct 3 23:04:47 ats-node01 abrtd: 'post-create' on > > '/var/spool/abrt/ccpp-2013-10-03-23:01:53-2696' exited with 1 > > Oct 3 23:04:47 ats-node01 abrtd: Corrupted or bad directory > > '/var/spool/abrt/ccpp-2013-10-03-23:01:53-2696', deleting > > Oct 3 23:04:48 ats-node01 traffic_cop[2639]: (test) read failed [104 > > 'Connection reset by peer'] > > Oct 3 23:04:48 ats-node01 traffic_cop[2639]: server heartbeat failed [1] > > Oct 3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0} > > FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > > Oct 3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0} > > FATAL: (last system error 104: Connection reset by peer) > > Oct 3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0} > > ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message > > Oct 3 23:04:48 ats-node01 traffic_manager[2648]: {0x7ff4a50ce7e0} > > ERROR: (last system error 32: Broken pipe) > > Oct 3 23:04:48 ats-node01 traffic_cop[2639]: cop received child status > > signal [2648 256] > > Oct 3 23:04:48 ats-node01 traffic_cop[2639]: traffic_manager not > > running, making sure traffic_server is dead > > Oct 3 23:04:48 ats-node01 traffic_cop[2639]: spawning traffic_manager > > Oct 3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: --- Manager > > Starting --- > > Oct 3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: Manager Version: > > Apache Traffic Server - traffic_manager - 4.0.1 - (build # 8913 on Sep > > 9 2013 at 13:15:00) > > Oct 3 23:04:48 ats-node01 traffic_manager[9533]: NOTE: > > RLIMIT_NOFILE(7):cur(30000),max(30000) > > Oct 3 23:04:50 ats-node01 traffic_server[9578]: NOTE: --- Server > > Starting --- > > Oct 3 23:04:50 ats-node01 traffic_server[9578]: NOTE: Server Version: > > Apache Traffic Server - traffic_server - 4.0.1 - (build # 8913 on Sep 9 > > 2013 at 13:16:57) > > Oct 3 23:04:58 ats-node01 traffic_cop[2639]: server heartbeat succeeded > > > > > > > > Any suggestions will be very appreciated! > >
