[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949149#comment-13949149 ] Timur Irmatov commented on TS-2669: --- Do you use Linux native AIO ? If so, there is a confirmed bug in 4.1.2 with Linux AIO that causes clearing of cache on ATS restart. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949152#comment-13949152 ] AnDao commented on TS-2669: --- Dear Timur Irmatov, Thanks for your reply, I'm build ATS with no special option, not enable linux_native_aio when configure the build. How can I check if my ATS is using linux_native_aio or not? Thanks, AnDQT. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949178#comment-13949178 ] Timur Irmatov commented on TS-2669: --- Well, I cannot tell for sure, but I see two ways: * Re-run configure and look for "checking whether to enable Linux native AIO"; * run 'ldd /path/to/traffic_server' and look if it is linked with libaio.so. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949187#comment-13949187 ] AnDao commented on TS-2669: --- Hi Timur Irmatov, I had check, I didn't include option build native aio (--use_linux_native_aio); and "ldd traffic_server|grep -i aio" not show up libaio.so How can we troubleshoot to avoid other restart and lost cache? I see that ATS dump a core file when crash, do it help any? Thanks. AnDQT. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949189#comment-13949189 ] Timur Irmatov commented on TS-2669: --- Well then I would think that you are not using Linux native AIO. Then may be it is another bug. I am just a user of ATS, so cannot help any further. May be some of the developers will look into your issue. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949458#comment-13949458 ] Leif Hedstrom commented on TS-2669: --- This definitely sounds the same as either the Linux native AIO or the "interim" cache. They would be enabled with {code} --enable-linux-native-aio {code} and {code} --enable-interim-cache {code} Neither works with properly (afaik) with that version you have. Unless you really have a need for either, don't use them (IMO). > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950238#comment-13950238 ] AnDao commented on TS-2669: --- Thanks Leif Hedstrom, but my ATS doesn't enable them. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950437#comment-13950437 ] Zhao Yongming commented on TS-2669: --- well, maybe something that need more check, please findout the diags.log lines like: {code} [Mar 27 20:31:25.948] {0x2b02748fde00} STATUS: opened /var/log/trafficserver/diags.log [Mar 27 20:31:25.948] {0x2b02748fde00} NOTE: updated diags config [Mar 27 20:31:25.954] Server {0x2b02748fde00} NOTE: cache clustering disabled [Mar 27 20:31:25.964] Server {0x2b02748fde00} NOTE: ip_allow.config updated, reloading [Mar 27 20:31:25.969] Server {0x2b02748fde00} NOTE: loading SSL certificate configuration from /etc/trafficserver/ssl_multicert.config [Mar 27 20:31:25.976] Server {0x2b02748fde00} NOTE: cache clustering disabled [Mar 27 20:31:25.977] Server {0x2b02748fde00} NOTE: logging initialized[15], logging_mode = 3 [Mar 27 20:31:25.978] Server {0x2b02748fde00} NOTE: loading plugin '/usr/lib64/trafficserver/plugins/libloader.so' [Mar 27 20:31:25.982] Server {0x2b02748fde00} NOTE: loading plugin '/usr/local/ironbee/libexec/ts_ironbee.so' [Mar 27 20:31:25.983] Server {0x2b02748fde00} NOTE: Rolling interval adjusted from 0 sec to 300 sec for /var/log/trafficserver/ts-ironbee.log [Mar 27 20:31:25.992] Server {0x2b02748fde00} NOTE: traffic server running [Mar 27 20:31:26.077] Server {0x2b0275d8e700} NOTE: cache enabled {code} you may find out that ' traffic server running' indicates that ATS is running, and 'cache enabled' show that cache is working. due to your system is crash and cache is not enabled, I will suspect that your ATS does not have that 'cache enabled' line. this is often caused by something like privilege issue, that ATS server process does not have the write privilege on the disk block device files etc. or anything else you may findout in the diags.log or even system logs. the interim cache & aio bugs may cause you lose the saved data, interim cache may cause you lose the data while server process restart, and AIO may get all data clear. but all those bugs fixed in the v4.1.0 release. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_he
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950457#comment-13950457 ] AnDao commented on TS-2669: --- Dear Zhao Yongming, Sure that my ATS cache enabled. You can see my attached images. I'm very worry if it can be happen again, that will impact much to our business. Thanks. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > Attachments: cachobjecs.png, storage.png > > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950510#comment-13950510 ] Zhao Yongming commented on TS-2669: --- please attach the server start log in diags.log file, as I attached. and please tell us how is your storage is configured. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > Attachments: cachobjecs.png, storage.png > > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950554#comment-13950554 ] AnDao commented on TS-2669: --- Hi, Here is diags.log after ATS restart: . [Mar 27 12:31:15.227] Server {0x2b3b56ee8700} WARNING: cache directory overflow on '/data/trafficserver-4.x/cache.db' segment 23, purging... [Mar 27 12:33:43.270] Server {0x2b3b56be5700} WARNING: cache directory overflow on '/data/trafficserver-4.x/cache.db' segment 36, purging... [Mar 27 12:48:17.352] Server {0x2b3b56067420} WARNING: cache directory overflow on '/data/trafficserver-4.x/cache.db' segment 35, purging... [Mar 27 12:57:15.648] {0x2ba3c9258420} STATUS: opened /zserver/trafficserver-4.1.2/var/log/trafficserver/diags.log [Mar 27 12:57:15.648] {0x2ba3c9258420} NOTE: updated diags config [Mar 27 12:57:15.661] Server {0x2ba3c9258420} NOTE: cache clustering disabled [Mar 27 12:57:15.682] Server {0x2ba3c9258420} NOTE: ip_allow.config updated, reloading [Mar 27 12:57:15.704] Server {0x2ba3c9258420} NOTE: cache clustering disabled [Mar 27 12:57:15.706] Server {0x2ba3c9258420} NOTE: logging initialized[15], logging_mode = 0 [Mar 27 12:57:15.707] Server {0x2ba3c9258420} NOTE: loading plugin '/zserver/trafficserver-4.1.2/libexec/trafficserver/cacheurl.so' [Mar 27 12:57:15.749] Server {0x2ba3c9258420} NOTE: traffic server running [Mar 27 12:57:15.931] Server {0x2ba3c9ed7700} NOTE: no good directory, clearing '/data/trafficserver-4.x/cache.db 57344:28835833' [Mar 27 12:57:18.325] Server {0x2ba3c9cd5700} NOTE: cache enabled ##3 > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > Attachments: cachobjecs.png, storage.png > > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950563#comment-13950563 ] AnDao commented on TS-2669: --- And the storage configure: ]#cat storage.config /data/trafficserver-4.x/ 220G > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > Attachments: cachobjecs.png, storage.png > > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2669) ATS crash, then restart with all cached objects cleared
[ https://issues.apache.org/jira/browse/TS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950778#comment-13950778 ] Leif Hedstrom commented on TS-2669: --- That is not a recommended configuration. In fact, I can not stress enough how important it is to use the raw devices. My recommendation is that you reconfigure your disk, and make a 220GB partition, and assign that partition to ATS. Running on file system cache like this is intended for testing / development. That much said, I don't know why it has to purge your segments. Maybe [~amc] does, but not sure. > ATS crash, then restart with all cached objects cleared > --- > > Key: TS-2669 > URL: https://issues.apache.org/jira/browse/TS-2669 > Project: Traffic Server > Issue Type: Bug >Reporter: AnDao > Attachments: cachobjecs.png, storage.png > > > Hi all, > I'm using ATS 4.1.2, my ATS is just crashed and restart and clean all the > cached objects, cause my backend servers overload. Why ATS do clean all the > cached objects when crash and restart? > The log is: > * manager.log > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} FATAL: > [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::mgmtShutdown] Executing shutdown request. > [Mar 27 12:57:13.022] Manager {0x7f597e3477e0} NOTE: > [LocalManager::processShutdown] Executing process shutdown request. > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: > [LocalManager::sendMgmtMsgToProcesses] Error writing message > [Mar 27 12:57:13.028] Manager {0x7f597e3477e0} ERROR: (last system error 32: > Broken pipe) > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} STATUS: opened > /zserver/log/trafficserver/manager.log > [Mar 27 12:57:13.174] {0x7ffaeec7e7e0} NOTE: updated diags config > [Mar 27 12:57:13.520] Manager {0x7ffaeec7e7e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-358.6.2.el6.x86_64' > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 80 > [Mar 27 12:57:13.550] Manager {0x7ffaeec7e7e0} NOTE: [TrafficManager] Setup > complete > [Mar 27 12:57:14.618] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::startProxy] Launching ts process > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: > [LocalManager::pollMgmtProcessServer] New process connecting fd '15' > [Mar 27 12:57:14.632] Manager {0x7ffaeec7e7e0} NOTE: [Alarms::signalAlarm] > Server Process born > *** traffic.out *** > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 15: Terminated > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' > NOTE: Traffic Server received Sig 11: Segmentation fault > /zserver/trafficserver-4.1.2/bin/traffic_server - STACK TRACE: > /lib64/libpthread.so.0(+0x35a360f500)[0x2b3b55819500] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact47change_response_header_because_of_range_requestEPNS_5StateEP7HTTPHdr+0x240)[0x54b8a0] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact28handle_content_length_headerEPNS_5StateEP7HTTPHdrS3_+0x2c8)[0x54bc38] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3e3)[0x54c0c3] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN12HttpTransact22handle_transform_readyEPNS_5StateE+0x70)[0x54ca40] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x28)[0x51b418] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM38state_response_wait_for_transform_readEiPv+0xed)[0x52988d] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x533178] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN17TransformTerminus12handle_eventEiPv+0x1d2)[0x4e8c62] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a5a0f] > /zserver/trafficserver-4.1.2/bin/traffic_server(_ZN7EThread7executeEv+0x63b)[0x6a658b] > /zserver/trafficserver-4.1.2/bin/traffic_server[0x6a48aa] > /lib64/libpthread.so.0(+0x35a3607851)[0x2b3b55811851] > /lib64/libc.so.6(clone+0x6d)[0x35a32e890d] > [E. Mgmt] log ==> [TrafficManager] using root directory > '/zserver/trafficserver-4.1.2' > [TrafficServer] using root directory '/zserver/trafficserver-4.1.2' -- This message was sent by Atlassian JIRA (v6.2#6252)