[ https://issues.apache.org/jira/browse/TS-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-3943: ------------------------------ Fix Version/s: (was: 6.1.0) 6.2.0 > ATS 6 crash over and over again on restart under load > ----------------------------------------------------- > > Key: TS-3943 > URL: https://issues.apache.org/jira/browse/TS-3943 > Project: Traffic Server > Issue Type: Bug > Components: ICP > Affects Versions: 6.0.0 > Reporter: Luca Rea > Assignee: Leif Hedstrom > Labels: regresion > Fix For: 6.2.0 > > > Hi, > when ICP is enabled and ATS is under huge load a "service trafficserver stop" > followed by a "service trafficserver start" cause a crashing loop (ATS crash, > restart, crash again, restart, crash again... ) > {code} > [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local' > traffic_server: using root directory '/usr/local' > traffic_server: Segmentation fault (Address not mapped to object > [(nil)])traffic_server - STACK TRACE: > traffic_server: Segmentation fault (Address not mapped to object > [(nil)])traffic_server - STACK TRACE: > /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809] > /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809] > /lib64/libc.so.6[0x31866326a0] > /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16] > /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe] > /lib64/libc.so.6[0x31866326a0] > /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16] > /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] > /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xbde)[0x59cd4e] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] > /usr/local/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0x15c)[0x59a3fc] > /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8] > /usr/local/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x152)[0x5760a2] > /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x2b1)[0x6e26c1] > /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0xbd)[0x57574d] > /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x142)[0x5831d2] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7b6)[0x59c926] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x229)[0x59c399] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] > /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x3dc)[0x59371c] > /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8] > /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2b2)[0x595b22] > /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x75f)[0x59d8cf] > /usr/local/bin/traffic_server(_ZN17HttpClientSession15new_transactionEv+0xec)[0x5769dc] > /usr/local/bin/traffic_server(_ZN18ProxyClientSession14do_api_calloutE12TSHttpHookID+0x14d)[0x4e585d] > /usr/local/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionP9MIOBufferP14IOBufferReaderb+0x224)[0x578654] > /usr/local/bin/traffic_server(_ZN17HttpSessionAccept6acceptEP14NetVConnectionP9MIOBufferP14IOBufferReader+0x1ea)[0x5729ea] > /usr/local/bin/traffic_server(_ZN23ProtocolProbeTrampoline17ioCompletionEventEiPv+0x483)[0x4e5343] > /usr/local/bin/traffic_server[0x71eaf9] > /usr/local/bin/traffic_server[0x723c0e] > /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x7151d2] > /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x125)[0x742d05] > /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4c9)[0x743649] > /usr/local/bin/traffic_server[0x74217a] > /lib64/libpthread.so.0(+0x3186a079d1)[0x2b864a19f9d1] > /lib64/libc.so.6(clone+0x6d)[0x31866e88fd] > /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] > /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a] > {code} > {code} > Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: --- Manager Starting > --- > Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: Manager Version: > Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 > 2015 at 15:19:27) > Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: --- traffic_server > Starting --- > Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: traffic_server Version: > Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 > 2015 at 15:18:47) > Sep 23 16:12:07 redis-01 kernel: [ET_NET 4][22681]: segfault at 0 ip > 00000000004aca16 sp 00002b3bad5ceb00 error 4 in traffic_server[400000+415000] > Sep 23 16:12:10 redis-01 abrt[22993]: Saved core dump of pid 22676 > (/usr/local/bin/traffic_server) to > /var/spool/abrt/ccpp-2015-09-23-16:12:07-22676 (913047552 bytes) > Sep 23 16:12:10 redis-01 abrtd: Directory 'ccpp-2015-09-23-16:12:07-22676' > creation detected > Sep 23 16:12:10 redis-01 abrtd: Executable '/usr/local/bin/traffic_server' > doesn't belong to any package and ProcessUnpackaged is set to 'no' > Sep 23 16:12:10 redis-01 abrtd: 'post-create' on > '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676' exited with 1 > Sep 23 16:12:10 redis-01 abrtd: Deleting problem directory > '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676' > Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: > <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe) > Sep 23 16:12:10 redis-01 traffic_cop[22665]: cop received child status signal > [22667 256] > Sep 23 16:12:10 redis-01 traffic_cop[22665]: traffic_manager not running, > making sure traffic_server is dead > Sep 23 16:12:10 redis-01 traffic_cop[22665]: spawning traffic_manager > Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: --- Manager Starting > --- > Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: Manager Version: > Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 > 2015 at 15:19:27) > Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: --- traffic_server > Starting --- > Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: traffic_server Version: > Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 > 2015 at 15:18:47) > Sep 23 16:12:15 redis-01 kernel: [ET_NET 255][23280]: segfault at 0 ip > 00000000004aca16 sp 00002b7d2aee6b00 error 4 in traffic_server[400000+415000] > Sep 23 16:12:15 redis-01 abrt[23339]: Not saving repeating crash in > '/usr/local/bin/traffic_server' > Sep 23 16:12:17 redis-01 abrt[23339]: Saved core dump of pid 23024 to > core.23024 (906752000 bytes) > Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: > <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe) > Sep 23 16:12:17 redis-01 traffic_cop[22665]: cop received child status signal > [23012 256] > Sep 23 16:12:17 redis-01 traffic_cop[22665]: traffic_manager not running, > making sure traffic_server is dead > Sep 23 16:12:17 redis-01 traffic_cop[22665]: spawning traffic_manager > Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: --- Manager Starting > --- > Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: Manager Version: > Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 > 2015 at 15:19:27) > Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: --- traffic_server > Starting --- > Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: traffic_server Version: > Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 > 2015 at 15:18:47) > Sep 23 16:12:22 redis-01 kernel: [ET_NET 193][23562]: segfault at 0 ip > 00000000004aca16 sp 00002b866aaa2b00 error 4 in traffic_server[400000+415000] > Sep 23 16:12:22 redis-01 abrt[23684]: Not saving repeating crash in > '/usr/local/bin/traffic_server' > Sep 23 16:12:24 redis-01 abrt[23684]: Saved core dump of pid 23368 to > core.23368 (910950400 bytes) > Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: > <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe) > Sep 23 16:12:25 redis-01 traffic_cop[22665]: cop received child status signal > [23356 256] > Sep 23 16:12:25 redis-01 traffic_cop[22665]: traffic_manager not running, > making sure traffic_server is dead > Sep 23 16:12:25 redis-01 traffic_cop[22665]: unable to start traffic_manager, > retrying in 60 second(s) > {code} > Disabling ICP solve the issue. > I have never used and tested ICP before (older releases). -- This message was sent by Atlassian JIRA (v6.3.4#6332)