[
https://issues.apache.org/jira/browse/TS-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolai Gorchilov updated TS-2970:
----------------------------------
Description:
ATS 5.0.1 Running under Linux 3+ kernel in TPROXY mode.
Same configuration runs without a single crash for weeks, but in the moment I
enable tr-pass (8081:ip-in=127.0.0.1:tr-ful:tr-pass) on server_ports I get many
crashes in matter of seconds to minutes after starting the ATS:
{noformat}
FATAL: HttpSM.cc:2714: failed assert `event == HTTP_TUNNEL_EVENT_DONE`
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(ink_fatal_die+0x0)[0x2b52382e8067]
/z/lib/libtsutil.so.5(ink_get_rand()+0x0)[0x2b52382e6b28]
/z/bin/traffic_server(HttpSM::tunnel_handler(int, void*)+0x148)[0x5cf9aa]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0x333)[0x5cebeb]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(CheckConnect::handle_connect(int, Event*)+0x2f0)[0x74313c]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server[0x743c82]
/z/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x389)[0x7449c3]
/z/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x79)[0x744638]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x6f3)[0x73dc21]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x11e)[0x7664ca]
/z/bin/traffic_server(EThread::execute()+0x45b)[0x766aa1]
/z/bin/traffic_server[0x76598f]
/lib64/libpthread.so.0(+0x7034)[0x2b52398f4034]
/lib64/libc.so.6(clone+0x6d)[0x2b523a63eb5d]
{noformat}
{noformat}
FATAL: HttpSM.cc:1687: failed assert `0`
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6545caa837]
/z/lib/libtsutil.so.5(+0x1d51f)[0x2b6545ca951f]
/z/bin/traffic_server(HttpSM::state_http_server_open(int, void*)+0xfd)[0x5a13cd]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ac88]
/z/bin/traffic_server[0x65313a]
/z/bin/traffic_server(HostDBContinuation::probeEvent(int,
Event*)+0x27a)[0x65967a]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736071]
/z/bin/traffic_server(EThread::execute()+0x2fb)[0x73692b]
/z/bin/traffic_server[0x73530a]
/lib64/libpthread.so.0(+0x7034)[0x2b65472b0034]
/lib64/libc.so.6(clone+0x6d)[0x2b6547ffab5d]
{noformat}
{noformat}
FATAL: HttpTunnel.cc:554: failed assert `active == false`
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b679373e837]
/z/lib/libtsutil.so.5(+0x1d51f)[0x2b679373d51f]
/z/bin/traffic_server[0x5d660f]
/z/bin/traffic_server(HttpSM::kill_this()+0x674)[0x59a974]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0x148)[0x59acf8]
/z/bin/traffic_server(CheckConnect::handle_connect(int, Event*)+0x119)[0x70d659]
/z/bin/traffic_server(UnixNetVConnection::mainEvent(int,
Event*)+0x3f0)[0x716640]
/z/bin/traffic_server(InactivityCop::check_inactivity(int,
Event*)+0x275)[0x708045]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736071]
/z/bin/traffic_server(EThread::execute()+0x2fb)[0x73692b]
/z/bin/traffic_server[0x73530a]
/lib64/libpthread.so.0(+0x7034)[0x2b6794d44034]
/lib64/libc.so.6(clone+0x6d)[0x2b6795a8eb5d]
{noformat}
My configure options other than layout-related are:
{noformat}
--with-group=proxy \
--with-xml=libxml2 \
--disable-static \
--disable-static-libts \
--disable-spdy \
--enable-interim-cache \
--enable-tproxy \
--enable-hwloc \
--enable-experimental-plugins \
--enable-example-plugins
{noformat}
P.S. Report updated due to some --enable-debug related crashes, that weren't
connected to tr-pass at all
was:
ATS 5.0.1 Running under Linux 3+ kernel in TPROXY mode.
Same configuration runs without a single crash for weeks, but in the moment I
enable tr-pass (8081:ip-in=127.0.0.1:tr-ful:tr-pass) on server_ports I get many
crashes in matter of seconds to minutes after starting the ATS:
* No plugins, no interim storage enabled
{noformat}
FATAL: HttpSM.cc:2714: failed assert `event == HTTP_TUNNEL_EVENT_DONE`
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(ink_fatal_die+0x0)[0x2b52382e8067]
/z/lib/libtsutil.so.5(ink_get_rand()+0x0)[0x2b52382e6b28]
/z/bin/traffic_server(HttpSM::tunnel_handler(int, void*)+0x148)[0x5cf9aa]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0x333)[0x5cebeb]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(CheckConnect::handle_connect(int, Event*)+0x2f0)[0x74313c]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server[0x743c82]
/z/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x389)[0x7449c3]
/z/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x79)[0x744638]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x6f3)[0x73dc21]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x11e)[0x7664ca]
/z/bin/traffic_server(EThread::execute()+0x45b)[0x766aa1]
/z/bin/traffic_server[0x76598f]
/lib64/libpthread.so.0(+0x7034)[0x2b52398f4034]
/lib64/libc.so.6(clone+0x6d)[0x2b523a63eb5d]
{noformat}
* With enabled interim storage: without traffic while indexing the storage
{noformat}
NOTE: Traffic Server received Sig 11: Segmentation fault
/z/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0xf280)[0x2b24483d2280]
/z/bin/traffic_server(dir_clean_range_interimvol(long, long,
InterimCacheVol*)+0x7f)[0x6f8b9a]
/z/bin/traffic_server(InterimCacheVol::handle_recover_from_data(int,
void*)+0xaba)[0x6e9da2]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(AIOCallbackInternal::io_complete(int,
void*)+0x3f)[0x6f210f]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x11e)[0x7664ca]
/z/bin/traffic_server(EThread::execute()+0xc9)[0x76670f]
/z/bin/traffic_server[0x76598f]
/lib64/libpthread.so.0(+0x7034)[0x2b24483ca034]
/lib64/libc.so.6(clone+0x6d)[0x2b2449114b5d]
{noformat}
* with some plugins enabled
{noformat}
FATAL: HttpSM.cc:3840: failed assert `pending_action == NULL`
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(ink_fatal_die+0x0)[0x2b8f03a97067]
/z/lib/libtsutil.so.5(ink_get_rand()+0x0)[0x2b8f03a95b28]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x81)[0x5d3a91]
/z/bin/traffic_server(HttpSM::set_next_state()+0xa02)[0x5df0cc]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x1ba)[0x5de6c2]
/z/bin/traffic_server(HttpSM::handle_api_return()+0x138)[0x5cb200]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x883)[0x5cafb9]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x1a5)[0x5ca6bf]
/z/bin/traffic_server(TSHttpTxnReenable+0x13e)[0x511d6e]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b8f10532508]
/z/bin/traffic_server(INKContInternal::handle_event(int, void*)+0xaa)[0x506404]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(APIHook::invoke(int, void*)+0x6a)[0x506d22]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x612)[0x5cad48]
/z/bin/traffic_server(HttpSM::do_api_callout_internal()+0x1b7)[0x5d7823]
/z/bin/traffic_server(HttpSM::do_api_callout()+0x26)[0x5e5144]
/z/bin/traffic_server(HttpSM::set_next_state()+0x6d)[0x5de737]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x1ba)[0x5de6c2]
/z/bin/traffic_server(HttpSM::handle_api_return()+0x138)[0x5cb200]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x883)[0x5cafb9]
/z/bin/traffic_server(HttpSM::do_api_callout_internal()+0x1b7)[0x5d7823]
/z/bin/traffic_server(HttpSM::do_api_callout()+0x26)[0x5e5144]
/z/bin/traffic_server(HttpSM::set_next_state()+0x6d)[0x5de737]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x1ba)[0x5de6c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x1b3)[0x5de87d]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x1ba)[0x5de6c2]
/z/bin/traffic_server(HttpSM::handle_api_return()+0x138)[0x5cb200]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x883)[0x5cafb9]
/z/bin/traffic_server(HttpSM::do_api_callout_internal()+0x1b7)[0x5d7823]
/z/bin/traffic_server(HttpSM::do_api_callout()+0x26)[0x5e5144]
/z/bin/traffic_server(HttpSM::set_next_state()+0x6d)[0x5de737]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x1ba)[0x5de6c2]
/z/bin/traffic_server(HttpSM::handle_api_return()+0x138)[0x5cb200]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x883)[0x5cafb9]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x1a5)[0x5ca6bf]
/z/bin/traffic_server(TSHttpTxnReenable+0x13e)[0x511d6e]
/z/lib/plugins/cacheurl.so(+0x418e)[0x2b8f10f6f18e]
/z/bin/traffic_server(INKContInternal::handle_event(int, void*)+0xaa)[0x506404]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(APIHook::invoke(int, void*)+0x6a)[0x506d22]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x612)[0x5cad48]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x1a5)[0x5ca6bf]
/z/bin/traffic_server(TSHttpTxnReenable+0x13e)[0x511d6e]
/z/lib/plugins/tslua.so(+0x6265)[0x2b8f1073b265]
/z/bin/traffic_server(INKContInternal::handle_event(int, void*)+0xaa)[0x506404]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(APIHook::invoke(int, void*)+0x6a)[0x506d22]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x612)[0x5cad48]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x1a5)[0x5ca6bf]
/z/bin/traffic_server(TSHttpTxnReenable+0x13e)[0x511d6e]
/z/lib/plugins/stats_over_http.so(+0x1b08)[0x2b8f1032eb08]
/z/bin/traffic_server(INKContInternal::handle_event(int, void*)+0xaa)[0x506404]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(APIHook::invoke(int, void*)+0x6a)[0x506d22]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x612)[0x5cad48]
/z/bin/traffic_server(HttpSM::do_api_callout_internal()+0x1b7)[0x5d7823]
/z/bin/traffic_server(HttpSM::do_api_callout()+0x26)[0x5e5144]
/z/bin/traffic_server(HttpSM::set_next_state()+0x6d)[0x5de737]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x1ba)[0x5de6c2]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0xb57)[0x5c8ccf]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0x272)[0x5ceb2a]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server[0x743b96]
/z/bin/traffic_server[0x7444d9]
/z/bin/traffic_server(UnixNetVConnection::net_read_io(NetHandler*,
EThread*)+0x2b)[0x746229]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x623)[0x73db51]
/z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x11e)[0x7664ca]
/z/bin/traffic_server(EThread::execute()+0x45b)[0x766aa1]
/z/bin/traffic_server[0x76598f]
/lib64/libpthread.so.0(+0x7034)[0x2b8f050a3034]
/lib64/libc.so.6(clone+0x6d)[0x2b8f05dedb5d]
{noformat}
My configure options other than layout-related are:
{noformat}
--with-group=proxy \
--with-xml=libxml2 \
--disable-static \
--disable-static-libts \
--disable-spdy \
--enable-interim-cache \
--enable-tproxy \
--enable-hwloc \
--enable-experimental-plugins \
--enable-example-plugins \
--enable-debug
{noformat}
> Different crashes when using tr-pass
> ------------------------------------
>
> Key: TS-2970
> URL: https://issues.apache.org/jira/browse/TS-2970
> Project: Traffic Server
> Issue Type: Bug
> Components: Core, HTTP
> Affects Versions: 5.0.0, 5.0.1
> Reporter: Nikolai Gorchilov
>
> ATS 5.0.1 Running under Linux 3+ kernel in TPROXY mode.
> Same configuration runs without a single crash for weeks, but in the moment I
> enable tr-pass (8081:ip-in=127.0.0.1:tr-ful:tr-pass) on server_ports I get
> many crashes in matter of seconds to minutes after starting the ATS:
> {noformat}
> FATAL: HttpSM.cc:2714: failed assert `event == HTTP_TUNNEL_EVENT_DONE`
> /z/bin/traffic_server - STACK TRACE:
> /z/lib/libtsutil.so.5(ink_fatal_die+0x0)[0x2b52382e8067]
> /z/lib/libtsutil.so.5(ink_get_rand()+0x0)[0x2b52382e6b28]
> /z/bin/traffic_server(HttpSM::tunnel_handler(int, void*)+0x148)[0x5cf9aa]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0x333)[0x5cebeb]
> /z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
> /z/bin/traffic_server(CheckConnect::handle_connect(int,
> Event*)+0x2f0)[0x74313c]
> /z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
> /z/bin/traffic_server[0x743c82]
> /z/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
> EThread*)+0x389)[0x7449c3]
> /z/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
> EThread*)+0x79)[0x744638]
> /z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x6f3)[0x73dc21]
> /z/bin/traffic_server(Continuation::handleEvent(int, void*)+0x68)[0x4f089a]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x11e)[0x7664ca]
> /z/bin/traffic_server(EThread::execute()+0x45b)[0x766aa1]
> /z/bin/traffic_server[0x76598f]
> /lib64/libpthread.so.0(+0x7034)[0x2b52398f4034]
> /lib64/libc.so.6(clone+0x6d)[0x2b523a63eb5d]
> {noformat}
> {noformat}
> FATAL: HttpSM.cc:1687: failed assert `0`
> /z/bin/traffic_server - STACK TRACE:
> /z/lib/libtsutil.so.5(+0x1e837)[0x2b6545caa837]
> /z/lib/libtsutil.so.5(+0x1d51f)[0x2b6545ca951f]
> /z/bin/traffic_server(HttpSM::state_http_server_open(int,
> void*)+0xfd)[0x5a13cd]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ac88]
> /z/bin/traffic_server[0x65313a]
> /z/bin/traffic_server(HostDBContinuation::probeEvent(int,
> Event*)+0x27a)[0x65967a]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736071]
> /z/bin/traffic_server(EThread::execute()+0x2fb)[0x73692b]
> /z/bin/traffic_server[0x73530a]
> /lib64/libpthread.so.0(+0x7034)[0x2b65472b0034]
> /lib64/libc.so.6(clone+0x6d)[0x2b6547ffab5d]
> {noformat}
> {noformat}
> FATAL: HttpTunnel.cc:554: failed assert `active == false`
> /z/bin/traffic_server - STACK TRACE:
> /z/lib/libtsutil.so.5(+0x1e837)[0x2b679373e837]
> /z/lib/libtsutil.so.5(+0x1d51f)[0x2b679373d51f]
> /z/bin/traffic_server[0x5d660f]
> /z/bin/traffic_server(HttpSM::kill_this()+0x674)[0x59a974]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0x148)[0x59acf8]
> /z/bin/traffic_server(CheckConnect::handle_connect(int,
> Event*)+0x119)[0x70d659]
> /z/bin/traffic_server(UnixNetVConnection::mainEvent(int,
> Event*)+0x3f0)[0x716640]
> /z/bin/traffic_server(InactivityCop::check_inactivity(int,
> Event*)+0x275)[0x708045]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736071]
> /z/bin/traffic_server(EThread::execute()+0x2fb)[0x73692b]
> /z/bin/traffic_server[0x73530a]
> /lib64/libpthread.so.0(+0x7034)[0x2b6794d44034]
> /lib64/libc.so.6(clone+0x6d)[0x2b6795a8eb5d]
> {noformat}
> My configure options other than layout-related are:
> {noformat}
> --with-group=proxy \
> --with-xml=libxml2 \
> --disable-static \
> --disable-static-libts \
> --disable-spdy \
> --enable-interim-cache \
> --enable-tproxy \
> --enable-hwloc \
> --enable-experimental-plugins \
> --enable-example-plugins
> {noformat}
> P.S. Report updated due to some --enable-debug related crashes, that weren't
> connected to tr-pass at all
--
This message was sent by Atlassian JIRA
(v6.2#6252)
