[ https://issues.apache.org/jira/browse/TS-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698224#comment-13698224 ]
Tommy Lee commented on TS-1956: ------------------------------- Leif, thanks for the efforts. In some conditions the traffic_server restarts after some days with one strange crash. No problem with this, as we know that traffic_cop and traffic_manager takes care of that. But with this crash, ATS get stucked. I can reproduce easily here simulating traffic server crash by killing the process. Do you need more debug infos ? Please let me know and I will fill the ticket. Thanks again. > Under Heavy Load TS 3.3.4-dev can't (re)start > --------------------------------------------- > > Key: TS-1956 > URL: https://issues.apache.org/jira/browse/TS-1956 > Project: Traffic Server > Issue Type: Bug > Reporter: Tommy Lee > Priority: Blocker > Fix For: 3.3.6 > > Attachments: backtrace.log > > > Hi, > We run TS in forward mode, under REALLY HEAVY load. Currently we run version > 3.3.2-dev without problems. > But today, we tried to upgrade to version 3.3.4-dev without lucky. > We've noticed that, if TS restarts, it enters in this Segfault Loop. > Below are traffic.out logs with debug .* > I'll try to debug with GDB too, but I cannot stop this server for too long, > because of our operations. > Thanks in advance. > ---------------------------- > {code} > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (iocore_net_processor) > NetProcessor::main_accept - port 3128,recv_bufsize 0, send_bufsize 262144, > sockopt 0x0 > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (http_tproxy) Marking > accept server 0x20fd9e0 on port 3128 as inbound transparent > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (http_tproxy) Listen > port inbound transparency enabled. > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (iocore_net_accept) > Created accept thread #1 for port 3128 > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (iocore_net_processor) > NetProcessor::main_accept - port 3128,recv_bufsize 0, send_bufsize 262144, > sockopt 0x0 > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (http_tproxy) Marking > accept server 0x20fdd00 on port 3128 as inbound transparent > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (http_tproxy) Listen > port inbound transparency enabled. > [Jun 14 11:54:14.563] Server {0x2b069d226500} DEBUG: (iocore_net_accept) > Created accept thread #1 for port 3128 > [Jun 14 11:54:14.563] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.202.81.5:46089 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.202.101.4:41361 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.156.38:59164 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b06a5413700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c000000] accepted connection from > 10.202.81.5:46089 transport type = 0 > [Jun 14 11:54:14.564] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.35.9:51533 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.201.20:10964 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b06a5615700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c015a80] accepted connection from > 10.200.156.38:59164 transport type = 0 > [Jun 14 11:54:14.564] Server {0x2b069d226500} DEBUG: (iocore_net_processor) > NetProcessor::main_accept - port 3128,recv_bufsize 0, send_bufsize 262144, > sockopt 0x0 > [Jun 14 11:54:14.564] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.202.148.2:44203 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b069d226500} DEBUG: (http_tproxy) Marking > accept server 0x20fe020 on port 3128 as inbound transparent > [Jun 14 11:54:14.564] Server {0x2b06a5514700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c015d20] accepted connection from > 10.202.101.4:41361 transport type = 0 > [Jun 14 11:54:14.564] Server {0x2b06a5817700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c015540] accepted connection from > 10.200.201.20:10964 transport type = 0NOTE: Traffic Server received Sig 11: > Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > [Jun 14 11:54:14.564] Server {0x2b069d226500} DEBUG: (http_tproxy) Listen > port inbound transparency enabled. > [Jun 14 11:54:14.564] Server {0x2b069d226500} DEBUG: (iocore_net_accept) > Created accept thread #1 for port 3128 > NOTE: Traffic Server received Sig 11: Segmentation fault > [Jun 14 11:54:14.564] Server {0x2b06a5716700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c0157e0] accepted connection from > 10.200.35.9:51533 transport type = 0/usr/local/cache-3.3.4/bin/traffic_server > - STACK TRACE: > [Jun 14 11:54:14.564] Server {0x2b085c120700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.131.24:65434 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.564] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.157.26:52514 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.23.7:64431 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b06a5918700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c0152a0] accepted connection from > 10.202.148.2:44203 transport type = 0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.156.60:46017 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b06a5a19700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c015000] accepted connection from > 10.200.157.26:52514 transport type = 0 > [Jun 14 11:54:14.565] Server {0x2b085c120700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.131.24:65441 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.202.232.4:51438 -> *Not IP address [0]*:0 > NOTE: Traffic Server received Sig 11: Segmentation fault > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > [Jun 14 11:54:14.565] Server {0x2b06a5b1a700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c014d60] accepted connection from > 10.200.23.7:64431 transport type = 0 > [Jun 14 11:54:14.565] Server {0x2b06a5c1b700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c014ac0] accepted connection from > 10.200.156.60:46017 transport type = 0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.63.37:49727 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.16.39:60477 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.181.8:20561 -> *Not IP address [0]*:0 > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > [Jun 14 11:54:14.565] Server {0x2b06a5e1d700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c175d20] accepted connection from > 10.200.131.24:65441 transport type = 0 > [Jun 14 11:54:14.565] Server {0x2b085c120700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.101.16:49423 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b085c120700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.201.174.20:62929 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.565] Server {0x2b06a5d1c700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c160000] accepted connection from > 10.200.131.24:65434 transport type = 0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionb+0x1cc)[0x507c4c] > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > [Jun 14 11:54:14.565] Server {0x2b06a5f1e700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c014820] accepted connection from > 10.202.232.4:51438 transport type = 0 > [Jun 14 11:54:14.565] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.181.8:20558 -> *Not IP address [0]*:0 > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /usr/local/cache-3.3.4/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionb+0x1cc)[0x507c4c] > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /usr/local/cache-3.3.4/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionb+0x1cc)[0x507c4c] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN10HttpAccept9mainEventEiPv+0x2fd)[0x50211d] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN10HttpAccept9mainEventEiPv+0x2fd)[0x50211d] > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /usr/local/cache-3.3.4/bin/traffic_server(_ZN18UnixNetVConnection11acceptEventEiP5Event+0x41a)[0x67a83a] > [Jun 14 11:54:14.565] Server {0x2b06a6221700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c014040] accepted connection from > 10.200.181.8:20561 transport type = 0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN10HttpAccept9mainEventEiPv+0x2fd)[0x50211d] > [Jun 14 11:54:14.565] Server {0x2b06a6322700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c175a80] accepted connection from > 10.200.101.16:49423 transport type = 0 > NOTE: Traffic Server received Sig 11: Segmentation fault > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /usr/local/cache-3.3.4/bin/traffic_server(_ZN18UnixNetVConnection11acceptEventEiP5Event+0x41a)[0x67a83a] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x91)[0x69b791] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionb+0x1cc)[0x507c4c] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN10HttpAccept9mainEventEiPv+0x2fd)[0x50211d] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN18UnixNetVConnection11acceptEventEiP5Event+0x41a)[0x67a83a] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x91)[0x69b791] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > [Jun 14 11:54:14.565] Server {0x2b06a6120700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c0142e0] accepted connection from > 10.200.16.39:60477 transport type = 0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionb+0x1cc)[0x507c4c] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN7EThread7executeEv+0x6e3)[0x69c443] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN10HttpAccept9mainEventEiPv+0x2fd)[0x50211d] > [Jun 14 11:54:14.566] Server {0x2b085c120700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.150.1.23:63892 -> *Not IP address [0]*:0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN18UnixNetVConnection11acceptEventEiP5Event+0x41a)[0x67a83a] > [Jun 14 11:54:14.567] Server {0x2b06a6524700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c013da0] accepted connection from > 10.200.181.8:20558 transport type = 0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x91)[0x69b791] > NOTE: Traffic Server received Sig 11: Segmentation fault > [Jun 14 11:54:14.566] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.181.8:20559 -> *Not IP address [0]*:0 > /usr/local/cache-3.3.4/bin/traffic_server - STACK TRACE: > /usr/local/cache-3.3.4/bin/traffic_server(_ZN18UnixNetVConnection11acceptEventEiP5Event+0x41a)[0x67a83a] > [Jun 14 11:54:14.567] Server {0x2b085c4a9700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.200.149.11:61021 -> *Not IP address [0]*:0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x91)[0x69b791] > /usr/local/cache-3.3.4/bin/traffic_server(_ZN7EThread7executeEv+0x6e3)[0x69c443] > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > [Jun 14 11:54:14.567] Server {0x2b06a601f700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c014580] accepted connection from > 10.200.63.37:49727 transport type = 0 > /usr/local/cache-3.3.4/bin/traffic_server[0x69a56a] > /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e)[0x2b069c01bf8e] > /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2b069cd4ee1d] > [Jun 14 11:54:14.568] Server {0x2b0850514700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.150.29.41:59402 -> *Not IP address [0]*:0 > /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0)[0x2b069c023bd0] > [Jun 14 11:54:14.566] Server {0x2b06a6423700} DEBUG: (http_seq) > [HttpAccept:mainEvent 0x2b085c1757e0] accepted connection from > 10.201.174.20:62929 transport type = 0 > /usr/local/cache-3.3.4/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionb+0x1cc)[0x507c4c] > [Jun 14 11:54:14.567] Server {0x2b085c120700} DEBUG: (iocore_net_server) > Connection accepted [Server]. 10.203.51.4:55910 -> *Not IP address [0]*:0 > [Jun 14 11:54:14.753] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] NO EVENTS TO PROCESS > [Jun 14 11:54:14.753] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] add fd 9 to select set > [Jun 14 11:54:15.260] Manager {0x7f17559d3740} DEBUG: (StatProInit) > [StatProcessor] Processing Statistics.... > [Jun 14 11:54:15.754] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] NO EVENTS TO PROCESS > [Jun 14 11:54:15.754] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] add fd 9 to select set > [Jun 14 11:54:16.262] Manager {0x7f17559d3740} DEBUG: (StatProInit) > [StatProcessor] Processing Statistics.... > [Jun 14 11:54:16.755] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] NO EVENTS TO PROCESS > [Jun 14 11:54:16.755] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] add fd 9 to select set > [Jun 14 11:54:17.263] Manager {0x7f17559d3740} DEBUG: (StatProInit) > [StatProcessor] Processing Statistics.... > [Jun 14 11:54:17.263] Manager {0x7f17559d3740} DEBUG: (RecOp) [varType] > proxy.node.http.user_agent_total_response_bytes is of type 1 > [Jun 14 11:54:17.756] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] NO EVENTS TO PROCESS > [Jun 14 11:54:17.757] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] add fd 9 to select set > [Jun 14 11:54:18.482] Manager {0x7f17559d3740} DEBUG: (StatProInit) > [StatProcessor] Processing Statistics.... > [Jun 14 11:54:18.758] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] NO EVENTS TO PROCESS > [Jun 14 11:54:18.758] Manager {0x7f174bfff700} DEBUG: (event) > [event_callback_main] add fd 9 to select set > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira