sudheerv opened a new issue #6932:
URL: https://github.com/apache/trafficserver/issues/6932


   After a week of working through and resolving more frequent crashes in prod, 
we ended up with hopefully one of the last remaining slow burn crashes. This 
crash would happen once or twice a day out-of-nowhere after seemingly running 
fine until then indicating that this is related to some sort of unique request 
or some other asynchronous/background activity. Digging deeper, I've isolated 
that to a spike in the metric `proxy.process.hostdb.cache.total_inserts`  - 
Each of those odd crashes (we've ats9 running in a box in each of our DCs) 
always seem to follow a noticeable spike in hostdb inserts a minute before the 
crash. 
   
   That led me to look through all the recent HostDB changes which may be 
causing the issue and to isolate further, I've reverted PR #5952 (and the 
follow-up PR #6686 ) which seems to have fixed the issue. Will need to run this 
through the weekend to confirm but with those two PRs reverted, haven't seen 
any crash since yesterday. 
   
   Below are some example stack traces that were seen. In most cases, the stack 
traces don't seem to give much to go on, as they are only symptoms of what 
appears to be some sort of memory corruption. 
   
   
   ```
   (gdb) bt
   #0  0x00002b1c1476106f in _Unwind_IteratePhdrCallback (info=<optimized out>, 
size=<optimized out>, ptr=0x2b1c21901980) at 
../.././libgcc/unwind-dw2-fde-dip.c:398
   #1  0x00002b1c14aa242c in dl_iterate_phdr () from /lib64/libc.so.6
   #2  0x00002b1c14761501 in _Unwind_Find_FDE (pc=0x2b1c16d38e81 
<lixer::LixPlugin::handleReadRequestHeadersPreRemap(atscppapi::Transaction&)+845>,
 bases=bases@entry=0x2b1c21901b08) at ../.././libgcc/unwind-dw2-fde-dip.c:469
   #3  0x00002b1c1475da43 in uw_frame_state_for 
(context=context@entry=0x2b1c21901a60, fs=fs@entry=0x2b1c21901b50) at 
../.././libgcc/unwind-dw2.c:1249
   #4  0x00002b1c1475f988 in _Unwind_Backtrace (trace=0x2b1c14a79da0 
<backtrace_helper>, trace_argument=0x2b1c21901d10) at 
../.././libgcc/unwind.inc:290
   #5  0x00002b1c14a79f16 in backtrace () from /lib64/libc.so.6
   #6  0x00002b1c12064a43 in ink_stack_trace_dump () at ink_stack_trace.cc:63
   #7  0x00002b1c120798b3 in signal_crash_handler (signo=signo@entry=11) at 
signals.cc:180
   #8  0x00000000004c42de in crash_logger_invoke (signo=11, 
info=0x2b1c219022b0, ctx=0x2b1c21902180) at traffic_server/Crash.cc:173
   #9  <signal handler called>
   #10 0x00000000004cf935 in ~Ptr (this=0x2b1c21902e70, __in_chrg=<optimized 
out>) at ../include/tscore/Ptr.h:215
   #11 MutexTryLock::~MutexTryLock (this=0x2b1c21902e70, __in_chrg=<optimized 
out>) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Lock.h:550
   #12 0x00000000004ecd02 in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6097
   #13 0x00002b1c2f4df5ad in UreGenerator::handleReadRequestHeadersPreRemap 
(this=0x2b1c15fffa80, transaction=...) at 
/export/content/data/multiproduct-post-commit/i001/workspace/ats-plugin-userrequestevent_78f261cbcb2e5016edb9902f1accbb4e89878e6f/ats-plugin-userrequestevent/main/src/UreGenerator.cc:311
   #14 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents 
(cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b1ccb9c0490) at 
GlobalPlugin.cc:65
   #15 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b1c17f53ba0, event=60016, edata=0x2b1ccb9c0490) at 
traffic_server/InkAPI.cc:1096
   #16 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60016, 
this=0x2b1c17f53ba0) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #17 APIHook::invoke (this=this@entry=0x2b1c15d75ec0, event=60016, 
edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
   #18 0x000000000054ef77 in HttpSM::state_api_callout 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1500
   #19 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #20 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #21 0x00002b1c16d38e82 in lixer::LixPlugin::handleReadRequestHeadersPreRemap 
(this=0x2b1c17bcafc0, transaction=...) at lixer.cc:127
   #22 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents 
(cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b1ccb9c0490) at 
GlobalPlugin.cc:65
   #23 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b1c17f53e20, event=60016, edata=0x2b1ccb9c0490) at 
traffic_server/InkAPI.cc:1096
   #24 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60016, 
this=0x2b1c17f53e20) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #25 APIHook::invoke (this=this@entry=0x2b1c15d75ee0, event=60016, 
edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
   #26 0x000000000054ef77 in HttpSM::state_api_callout (this=0x2b1ccb9c0490, 
event=event@entry=0, data=data@entry=0x0) at HttpSM.cc:1500
   #27 0x000000000054f8df in HttpSM::do_api_callout_internal (this=<optimized 
out>) at HttpSM.cc:5335
   #28 0x000000000055e4b5 in HttpSM::do_api_callout 
(this=this@entry=0x2b1ccb9c0490) at HttpSM.cc:365
   #29 0x00000000005588b3 in HttpSM::set_next_state (this=0x2b1ccb9c0490) at 
HttpSM.cc:7408
   #30 0x00000000005433f2 in HttpSM::call_transact_and_set_next_state 
(this=this@entry=0x2b1ccb9c0490, f=f@entry=0x0) at HttpSM.cc:7374
   #31 0x000000000055392a in HttpSM::handle_api_return (this=0x2b1ccb9c0490) at 
HttpSM.cc:1635
   #32 0x000000000054f4ae in HttpSM::state_api_callout 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1567
   #33 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #34 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #35 0x00002b1c417f72b6 in setSceLixValue (contp=0x2b1c56111560, 
event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at 
HttpDebugger.cc:109
   #36 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b1c56111560, event=60002, edata=0x2b1ccb9c0490) at 
traffic_server/InkAPI.cc:1096
   #37 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, 
this=0x2b1c56111560) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #38 APIHook::invoke (this=this@entry=0x2b1c15d75ac0, event=60002, 
edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
   #39 0x000000000054ef77 in HttpSM::state_api_callout 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1500
   #40 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #41 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #42 0x00002b1c3ac2d3b2 in cont_rewrite_headers (contp=0x2b1c17f52f20, 
event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at 
header_rewrite.cc:332
   #43 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b1c17f52f20, event=60002, edata=0x2b1ccb9c0490) at 
traffic_server/InkAPI.cc:1096
   #44 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, 
this=0x2b1c17f52f20) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #45 APIHook::invoke (this=this@entry=0x2b1c15d75ce0, event=60002, 
edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
   #46 0x000000000054ef77 in HttpSM::state_api_callout 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1500
   #47 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #48 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #49 0x00002b1c3a977ff1 in 
dc_selection::EntityRoutingPlugin::handleReadRequestHeaders 
(this=0x2b1c15c210c0, transaction=...) at EntityRoutingPlugin.cc:342
   #50 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents 
(cont=<optimized out>, event=TS_EVENT_HTTP_READ_REQUEST_HDR, 
edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
   #51 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b1c17f53060, event=60002, edata=0x2b1ccb9c0490) at 
traffic_server/InkAPI.cc:1096
   #52 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, 
this=0x2b1c17f53060) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #53 APIHook::invoke (this=this@entry=0x2b1c15d75d00, event=60002, 
edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
   #54 0x000000000054ef77 in HttpSM::state_api_callout 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1500
   #55 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #56 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #57 0x00002b1c340a9bf9 in TraceHeadersPlugin::handleReadRequestHeaders 
(this=0x2b1c15c06bc0, transaction=...) at trace-headers.cc:875
   #58 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents 
(cont=<optimized out>, event=TS_EVENT_HTTP_READ_REQUEST_HDR, 
edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
   #59 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b1c17f53a60, event=60002, edata=0x2b1ccb9c0490) at 
traffic_server/InkAPI.cc:1096
   ---Type <return> to continue, or q <return> to quit---q
   
   ```
   
   
   ```
   #0  ssl_callback_session_ticket (ssl=0x2b588f91e000, keyname=0x2b57d2f043f0 
"\260D\360\322W+", iv=0x2b57d2f043e0 "", cipher_ctx=0x2b585887cd40, 
hctx=0x2b582398e700, enc=1) at SSLSessionTicket.cc:67
   67      SSLSessionTicket.cc: No such file or directory.
   (gdb) bt
   #0  ssl_callback_session_ticket (ssl=0x2b588f91e000, keyname=0x2b57d2f043f0 
"\260D\360\322W+", iv=0x2b57d2f043e0 "", cipher_ctx=0x2b585887cd40, 
hctx=0x2b582398e700, enc=1) at SSLSessionTicket.cc:67
   #1  0x00002b57c96a1718 in tls_construct_new_session_ticket () from 
/lib/libssl.so.1.1
   #2  0x00002b57c96933f7 in state_machine () from /lib/libssl.so.1.1
   #3  0x00002b57c9668dea in ssl3_read_bytes () from /lib/libssl.so.1.1
   #4  0x00002b57c96708fa in ssl3_read () from /lib/libssl.so.1.1
   #5  0x00002b57c967c07d in ssl_read_internal () from /lib/libssl.so.1.1
   #6  0x00002b57c967c253 in SSL_read () from /lib/libssl.so.1.1
   #7  0x0000000000754315 in SSLReadBuffer (ssl=0x2b588f91e000, 
buf=0x2b589ce95000, nbytes=nbytes@entry=4096, nread=@0x2b57d2f04968: 0) at 
SSLUtils.cc:1851
   #8  0x00000000007439fa in ssl_read_from_net (ret=<synthetic pointer>, 
lthread=<optimized out>, sslvc=0x2b58dcf86330) at SSLNetVConnection.cc:275
   #9  SSLNetVConnection::net_read_io (this=0x2b58dcf86330, nh=0x2b57ce223d80, 
lthread=<optimized out>) at SSLNetVConnection.cc:665
   #10 0x0000000000762498 in NetHandler::process_ready_list 
(this=this@entry=0x2b57ce223d80) at UnixNet.cc:412
   #11 0x000000000076278d in NetHandler::waitForActivity (this=0x2b57ce223d80, 
timeout=<optimized out>) at UnixNet.cc:547
   #12 0x00000000007c6a9a in EThread::execute_regular 
(this=this@entry=0x2b57ce220000) at UnixEThread.cc:266
   #13 0x00000000007c6d62 in EThread::execute (this=0x2b57ce220000) at 
UnixEThread.cc:327
   #14 0x00000000007c5109 in spawn_thread_internal (a=0x2b57cc56de80) at 
Thread.cc:92
   #15 0x00002b57ca315dd5 in start_thread () from /lib64/libpthread.so.0
   #16 0x00002b57cb0c6ead in clone () from /lib64/libc.so.6
   (gdb) info thr
     Id   Target Id         Frame
   ```
   
   
   ```
   #9  <signal handler called>
   #10 std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, 
__x=0x2b4c234e0fa0, __p=<optimized out>, __header=...) at 
../../../.././libstdc++-v3/src/c++98/tree.cc:237
   #11 0x00002b4c2c3528aa in std::_Rb_tree<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >, 
std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >, 
std::_Select1st<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > >, 
AtsPluginUtils::CaseInsensitiveStringComparator, 
std::allocator<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > > 
>::_M_insert_<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >, 
std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >, 
std::_Select1st<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > >, 
AtsPluginUtils::CaseInsensitiveStringComparator, 
std::allocator<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > > 
>::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, 
std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >&&, 
std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >, 
std::_Select1st<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > >, 
AtsPluginUtils::CaseInsensitiveStringComparator, 
std::allocator<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > > >::_Alloc_node&) 
(this=0x2b4c2350bb20, __x=0x0, __p=0x2b4c224e0d00, __v=<unknown type in 
/lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x730ae>, 
__node_gen=...) at /usr/include/c++/7.3.0/bits/stl_tree.h:1755
   #12 0x00002b4c2c350df2 in std::_Rb_tree<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >, 
std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >, 
std::_Select1st<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > >, 
AtsPluginUtils::CaseInsensitiveStringComparator, 
std::allocator<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > > 
>::_M_insert_unique<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > 
>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >&&) (this=0x2b4c2350bb20, 
__v=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 
0x756e9>) at /usr/include/c++/7.3.0/bits/stl_tree.h:2096
   #13 0x00002b4c2c34f124 in std::map<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > >, 
AtsPluginUtils::CaseInsensitiveStringComparator, 
std::allocator<std::pair<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > > > 
>::insert(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const, 
std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > >&&) (this=0x2b4c2350bb20, 
__x=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 
0x77e21>) at /usr/include/c++/7.3.0/bits/stl_map.h:803
   #14 0x00002b4c2c34b442 in 
AtsPluginUtils::ClientRequestImpl::appendHeaderField (this=0x2b4c2350ba00, 
name=..., value=...) at ClientRequestImpl.cc:565
   #15 0x00002b4c2c34b310 in AtsPluginUtils::ClientRequestImpl::setHeaderField 
(this=0x2b4c2350ba00, name=..., value=...) at ClientRequestImpl.cc:555
   #16 0x00002b4c2c3474f9 in AtsPluginUtils::ClientRequest::setHeaderField 
(this=0x2b4c235fe8e0, name=..., value=...) at ClientRequest.cc:171
   #17 0x00002b4c2f84c5a3 in PluginControlInternals::addPluginToSignalHeader 
(pluginName=..., clientRequest=..., pLogger=0x2b4c2422c100) at 
PluginHeaderSignal.cc:137
   #18 0x00002b4c2f83f7f6 in PluginControls::canIrun (this=0x2b4c20503350) at 
PluginControls.cc:564
   #19 0x00002b4c2f30f2ee in UreGenerator::handleReadRequestHeadersPreRemap 
(this=0x2b4c15fffb80, transaction=...) at 
/export/content/data/multiproduct-post-commit/i001/workspace/ats-plugin-userrequestevent_78f261cbcb2e5016edb9902f1accbb4e89878e6f/ats-plugin-userrequestevent/main/src/UreGenerator.cc:280
   #20 0x00002b4c2ba44c07 in (anonymous namespace)::handleGlobalPluginEvents 
(cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b4c25107950) at 
GlobalPlugin.cc:65
   #21 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b4c17b7bba0, event=60016, edata=0x2b4c25107950) at 
traffic_server/InkAPI.cc:1096
   #22 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60016, 
this=0x2b4c17b7bba0) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #23 APIHook::invoke (this=this@entry=0x2b4c15d75ec0, event=60016, 
edata=edata@entry=0x2b4c25107950) at traffic_server/InkAPI.cc:1333
   #24 0x000000000054ef77 in HttpSM::state_api_callout 
(this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1500
   #25 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #26 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b4c25107950, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #27 0x00002b4c2b81de82 in lixer::LixPlugin::handleReadRequestHeadersPreRemap 
(this=0x2b4c16ff2fc0, transaction=...) at lixer.cc:127
   #28 0x00002b4c2ba44c07 in (anonymous namespace)::handleGlobalPluginEvents 
(cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b4c25107950) at 
GlobalPlugin.cc:65
   #29 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b4c17b7be20, event=60016, edata=0x2b4c25107950) at 
traffic_server/InkAPI.cc:1096
   #30 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60016, 
this=0x2b4c17b7be20) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   #31 APIHook::invoke (this=this@entry=0x2b4c15d75ee0, event=60016, 
edata=edata@entry=0x2b4c25107950) at traffic_server/InkAPI.cc:1333
   #32 0x000000000054ef77 in HttpSM::state_api_callout (this=0x2b4c25107950, 
event=event@entry=0, data=data@entry=0x0) at HttpSM.cc:1500
   #33 0x000000000054f8df in HttpSM::do_api_callout_internal (this=<optimized 
out>) at HttpSM.cc:5335
   #34 0x000000000055e4b5 in HttpSM::do_api_callout 
(this=this@entry=0x2b4c25107950) at HttpSM.cc:365
   #35 0x00000000005588b3 in HttpSM::set_next_state (this=0x2b4c25107950) at 
HttpSM.cc:7408
   #36 0x00000000005433f2 in HttpSM::call_transact_and_set_next_state 
(this=this@entry=0x2b4c25107950, f=f@entry=0x0) at HttpSM.cc:7374
   #37 0x000000000055392a in HttpSM::handle_api_return (this=0x2b4c25107950) at 
HttpSM.cc:1635
   38 0x000000000054f4ae in HttpSM::state_api_callout 
(this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1567
   #39 0x0000000000552734 in HttpSM::state_api_callback 
(this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) 
at HttpSM.cc:1367
   #40 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b4c25107950, 
event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
   #41 0x00002b4c41bf72b6 in setSceLixValue (contp=0x2b4c54d3a560, 
event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b4c25107950) at 
HttpDebugger.cc:109
   #42 0x00000000004d85a1 in INKContInternal::handle_event 
(this=0x2b4c54d3a560, event=60002, edata=0x2b4c25107950) at 
traffic_server/InkAPI.cc:1096
   #43 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60002, 
this=0x2b4c54d3a560) at 
/home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
   ---Type <return> to continue, or q <return> to quit---q
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to