c-taylor opened a new issue #7521: URL: https://github.com/apache/trafficserver/issues/7521
I will try to repro this in a controlled environment, but sharing the stack trace early... ATS 9.0.1 Crash/trace shortly after per_server.connection.max triggered WARNING messages. This appeared after 1-2h of load: ``` /some/path/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x99)[0x559787130dc9] /lib64/libpthread.so.0(+0xf630)[0x7f4e0247b630] /some/path/bin/traffic_server(_ZN17OutboundConnTrack8TxnState12Warn_BlockedEPKNS_9TxnConfigEliPK8sockaddrPKc+0x41)[0x55978718cd11] /some/path/bin/traffic_server(_ZN6HttpSM19do_http_server_openEb+0x1455)[0x55978719a745] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x32e)[0x5597871b089e] /some/path/bin/traffic_server(_ZN6HttpSM30send_origin_throttled_responseEv+0x10e)[0x5597871a2d2e] /some/path/bin/traffic_server(_ZN6HttpSM19do_http_server_openEb+0x12d8)[0x55978719a5c8] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x32e)[0x5597871b089e] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM22state_cache_open_writeEiPv+0x3f5)[0x5597871a68a5] /some/path/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x1b1)[0x5597871937b1] /some/path/bin/traffic_server(_ZN11HttpCacheSM22state_cache_open_writeEiPv+0x48a)[0x5597871ee44a] /some/path/bin/traffic_server(_ZN7CacheVC8callcontEi+0x7b)[0x5597872d4b9b] /some/path/bin/traffic_server(_ZN5Cache10open_writeEP12ContinuationPKN3ats10CryptoHashEP8HTTPInfolS5_13CacheFragTypePKci+0x40e)[0x5597872de97e] /some/path/bin/traffic_server(_ZN14CacheProcessor10open_writeEP12ContinuationiPK12HttpCacheKeyP7HTTPHdrP8HTTPInfol13CacheFragType+0x3a)[0x5597872ba35a] /some/path/bin/traffic_server(_ZN11HttpCacheSM10open_writeEPK12HttpCacheKeyP3URLP7HTTPHdrP8HTTPInfolbb+0x116)[0x5597871ee786] /some/path/bin/traffic_server(_ZN6HttpSM23do_cache_prepare_actionEP11HttpCacheSMP8HTTPInfobb+0x218)[0x5597871ad6d8] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5bb)[0x5597871b0b2b] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x6a3)[0x5597871abf43] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x1191)[0x5597871b1701] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x287)[0x55978719c707] /some/path/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1f7)[0x55978719c427] /some/path/bin/traffic_server(TSHttpTxnReenable+0x1d3)[0x55978714ad93] /var/run/trafficserver/5e386dbb-1989-4f9c-9417-c92047a73a0d/some/path/lib/trafficserver/plugins/cache_promote.so(+0x55bc)[0x7f4d83bf05bc] /some/path/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xde)[0x55978713c9ae] /some/path/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xe0)[0x55978713d960] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xc5d)[0x55978719d0dd] /some/path/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1f7)[0x55978719c427] /some/path/bin/traffic_server(TSHttpTxnReenable+0x1d3)[0x55978714ad93] /some/path/lib/trafficserver/plugins/xdebug.so(+0x63b8)[0x7f4ded1f33b8] /some/path/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xde)[0x55978713c9ae] /some/path/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xe0)[0x55978713d960] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xc5d)[0x55978719d0dd] /some/path/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0x46b)[0x5597871a6e8b] /some/path/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x1b1)[0x5597871937b1] /some/path/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x322)[0x5597871edc92] /some/path/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrPK27OverridableHttpConfigParams13CacheFragTypePKci+0xca)[0x5597872d091a] /some/path/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationPK12HttpCacheKeyP7HTTPHdrPK27OverridableHttpConfigParamsl13CacheFragType+0x27)[0x5597872ba317] /some/path/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrPK27OverridableHttpConfigParamsl+0xbd)[0x5597871ee88d] /some/path/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x210)[0x5597871ad300] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x243)[0x5597871b07b3] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x287)[0x55978719c707] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe84)[0x5597871b13f4] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x11d2)[0x5597871b1742] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x287)[0x55978719c707] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe84)[0x5597871b13f4] /some/path/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x10c)[0x559787196bfc] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x287)[0x55978719c707] /some/path/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1f7)[0x55978719c427] /some/path/bin/traffic_server(TSHttpTxnReenable+0x1d3)[0x55978714ad93] /some/path/lib/trafficserver/plugins/xdebug.so(+0x6dad)[0x7f4ded1f3dad] /some/path/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xde)[0x55978713c9ae] /some/path/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xe0)[0x55978713d960] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xc5d)[0x55978719d0dd] /some/path/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1f7)[0x55978719c427] /some/path/bin/traffic_server(TSHttpTxnReenable+0x1d3)[0x55978714ad93] /some/path/lib/trafficserver/plugins/healthchecks.so(+0x220c)[0x7f4dee55a20c] /some/path/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xde)[0x55978713c9ae] /some/path/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xe0)[0x55978713d960] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xc5d)[0x55978719d0dd] /some/path/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1f7)[0x55978719c427] /some/path/bin/traffic_server(TSHttpTxnReenable+0x1d3)[0x55978714ad93] /some/path/lib/trafficserver/plugins/header_rewrite.so(+0x1ed2b)[0x7f4deebe3d2b] /some/path/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xde)[0x55978713c9ae] /some/path/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xe0)[0x55978713d960] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xc5d)[0x55978719d0dd] /some/path/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1f7)[0x55978719c427] /some/path/bin/traffic_server(TSHttpTxnReenable+0x1d3)[0x55978714ad93] /some/path/lib/trafficserver/plugins/stats_over_http.so(+0x1b8b)[0x7f4deedfbb8b] /some/path/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xde)[0x55978713c9ae] /some/path/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xe0)[0x55978713d960] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xc5d)[0x55978719d0dd] /some/path/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe84)[0x5597871b13f4] /some/path/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0xe28)[0x559787195728] /some/path/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x1b1)[0x5597871937b1] /some/path/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x287)[0x55978719c707] /some/path/bin/traffic_server(_ZN6HttpSM17state_add_to_listEiPv+0x49)[0x559787193889] /some/path/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP16ProxyTransactionP14IOBufferReader+0x8c2)[0x559787194882] /some/path/bin/traffic_server(_ZN16ProxyTransaction15new_transactionEb+0x1db)[0x55978731917b] /some/path/bin/traffic_server(_ZN18Http1ClientSession16state_keep_aliveEiPv+0x151)[0x559787180411] /some/path/bin/traffic_server(_ZN8PluginVC17process_read_sideEb+0x419)[0x559787313de9] /some/path/bin/traffic_server(_ZN8PluginVC18process_write_sideEb+0x570)[0x559787314670] /some/path/bin/traffic_server(_ZN8PluginVC12main_handlerEiPv+0x4ca)[0x559787312e0a] /some/path/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x154)[0x55978737ff64] /some/path/bin/traffic_server(_ZN7EThread13process_queueEP5QueueI5EventNS1_9Link_linkEEPiS5_+0xef)[0x55978738023f] /some/path/bin/traffic_server(_ZN7EThread15execute_regularEv+0x1ab)[0x55978738062b] /some/path/bin/traffic_server(_ZN7EThread7executeEv+0x10a)[0x559787380aca] /some/path/bin/traffic_server(+0x3406cb)[0x55978737f6cb] /lib64/libpthread.so.0(+0x7ea5)[0x7f4e02473ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4e018858dd] traffic_manager ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message traffic_manager ERROR: last system error 104: Connection reset by peer Alarm Message: The TS-TM connection is broken for some reason. Either restart TS and TM or correct this error for TM to display TS statistics correctly traffic_manager NOTE: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault traffic_manager NOTE: [Alarms::signalAlarm] Server Process was reset ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
