Opened a new issue... https://github.com/apache/trafficserver/issues/2327
On Wed, Aug 2, 2017 at 4:25 PM, Dk Jack <[email protected]> wrote: > call_event = core_lock_retry event.. > > bt full output below... I'll go ahead and file a GitHub issue. Thanks. > > > #5 0x00002b8a417d3c35 in _ink_assert ( > expression=expression@entry=0x7f13e0 "call_event == > core_lock_retry_event", > file=file@entry=0x7f1413 "PluginVC.cc", line=line@entry=216) at > ink_assert.cc:37 > > > (gdb) bt full > #0 0x00002b8a43807428 in __GI_raise (sig=sig@entry=6) > at ../sysdeps/unix/sysv/linux/raise.c:54 > resultvar = 0 > pid = 72 > selftid = 136 > #1 0x00002b8a43809187 in __GI_abort () at abort.c:118 > act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = > 0x0}, > sa_mask = {__val = {18446744073709551615 <repeats 16 times>}}, > sa_flags = 0, sa_restorer = 0x0} > sigs = {__val = {32, 0 <repeats 15 times>}} > #2 0x00002b8a417d5adf in ink_die_die_die () at ink_error.cc:43 > No locals. > #3 ink_fatal_va (fmt=0x2b8a417e29ff "%s:%d: failed assert `%s`", > ap=ap@entry=0x2b8a4d311cf0) at ink_error.cc:65 > msg = "FATAL: PluginVC.cc:216: failed assert `call_event == > core_lock_retry_event`", '\000' <repeats 948 times> > #4 0x00002b8a417d5b79 in ink_fatal ( > message_format=message_format@entry=0x2b8a417e29ff "%s:%d: failed > assert `%s`") at ink_error.cc:73 > ap = <error reading variable ap (Attempt to dereference a generic > pointer.)> > #5 0x00002b8a417d3c35 in _ink_assert ( > expression=expression@entry=0x7f13e0 "call_event == > core_lock_retry_event", ---Type <return> to continue, or q <return> to > quit--- > file=file@entry=0x7f1413 "PluginVC.cc", line=line@entry=216) > at ink_assert.cc:37 > No locals. > #6 0x00000000004fc68c in PluginVC::main_handler (this=0x2b8ad8807048, > event=<optimized out>, data=0x2c31fa0) at PluginVC.cc:216 > __FUNCTION__ = "main_handler" > call_event = 0x2c31fa0 > read_mutex_held = <optimized out> > write_mutex_held = false > read_side_mutex = {m_ptr = 0x2b8ad40071c0} > write_side_mutex = {m_ptr = 0x0} > #7 0x00000000007e0752 in Continuation::handleEvent (data=0x2c31fa0, > event=1, > this=<optimized out>) at I_Continuation.h:153 > No locals. > #8 EThread::process_event (this=this@entry=0x2b8a4c707010, > e=e@entry=0x2c31fa0, calling_code=1) at UnixEThread.cc:148 > c_temp = <optimized out> > lock = {m = {m_ptr = 0x2b8ad9002d20}, lock_acquired = true} > #9 0x00000000007e15d8 in EThread::execute (this=0x2b8a4c707010) > at UnixEThread.cc:202 > done_one = <optimized out> > e = 0x2c31fa0 > NegativeQueue = {<DLL<Event, Event::Link_link>> = {head = > 0x2816700}, > ---Type <return> to continue, or q <return> to quit--- > tail = 0x2816700} > next_time = 1501263000357130435 > #10 0x00000000007e01d5 in spawn_thread_internal (a=0x27a24d0) at > Thread.cc:86 > p = 0x27a24d0 > #11 0x00002b8a42b176ba in start_thread (arg=0x2b8a4d312700) > at pthread_create.c:333 > __res = <optimized out> > pd = 0x2b8a4d312700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {47873000548096, > 126801912536511865, 0, 140735034297183, 47873000548800, 0, > 6257443582078367097, 6257462270246139257}, > mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, > data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > __PRETTY_FUNCTION__ = "start_thread" > #12 0x00002b8a438d93dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 > No locals. > (gdb) > > > On Wed, Aug 2, 2017 at 3:18 PM, Phil Sorber <[email protected]> wrote: > >> It seems pretty straightforward that the code expects call_event to be >> sm_lock_retry_event or core_lock_retry_event if it enters that code block. >> Can you print out the value of call_event from that stack frame (#5)? >> Maybe >> also do a `bt full` and see if anything interesting falls out of that. >> >> Thanks. >> >> On Wed, Aug 2, 2017 at 1:38 PM Dk Jack <[email protected]> wrote: >> >> > Will do. >> > >> > Even though this happened twice in our environment, I don't know the >> exact >> > process to reproduce it. Not sure if it makes sense to open an issue >> > without proper steps to reproduce it. I was hoping to get some guidance >> on >> > how to reproduce it from someone who is more familiar with this area of >> the >> > code. >> > >> > Bhasker. >> > >> > > On Aug 2, 2017, at 6:26 AM, Phil Sorber <[email protected]> wrote: >> > > >> > > Dk, >> > > >> > > Can you open a github issue on this and then also test with the new >> 6.2.2 >> > > RC0 I just put up for a vote? >> > > >> > > If you can't mark the issue for the 6.2.3 milestone, I can do that for >> > you. >> > > If this is still broken in the 6.2.2 RC0 we can stop the vote and >> figure >> > > out a fix depending on the severity. >> > > >> > > Thanks. >> > > >> > >> On Wed, Aug 2, 2017 at 12:11 AM Dk Jack <[email protected]> wrote: >> > >> >> > >> Btw, this crash happened on 6.2.1. >> > >> >> > >> Please let me know if this is not the correct forum for these sort of >> > >> questions. Thanks. >> > >> >> > >> >> > >> >> > >>> On Jul 31, 2017, at 11:15 AM, Dk Jack <[email protected]> wrote: >> > >>> >> > >>> Hi, >> > >>> I am seeing an ATS crash. The stack trace and the crash log info for >> > the >> > >> crash are pasted below. Seems to be happening in >> PluginVC::main_handler >> > >> line 216. I would appreciate it if someone can comment on possible >> > reasons >> > >> for this crash? Thanks. >> > >>> >> > >>> D. >> > >>> >> > >>> >> > >>> [Thread debugging using libthread_db enabled] >> > >>> Using host libthread_db library >> > >> "/lib/x86_64-linux-gnu/libthread_db.so.1". >> > >>> Core was generated by `/opt/trafficserver/bin/traffic_server -M >> > >> --bind_stdout /opt/trafficserver/var/l'. >> > >>> Program terminated with signal SIGABRT, Aborted. >> > >>> #0 0x00002b8a43807428 in __GI_raise (sig=sig@entry=6) at >> > >> ../sysdeps/unix/sysv/linux/raise.c:54 >> > >>> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or >> directory. >> > >>> [Current thread is 1 (Thread 0x2b8a4d312700 (LWP 136))] >> > >>> (gdb) bt >> > >>> #0 0x00002b8a43807428 in __GI_raise (sig=sig@entry=6) at >> > >> ../sysdeps/unix/sysv/linux/raise.c:54 >> > >>> #1 0x00002b8a43809187 in __GI_abort () at abort.c:118 >> > >>> #2 0x00002b8a417d5adf in ink_die_die_die () at ink_error.cc:43 >> > >>> #3 ink_fatal_va (fmt=0x2b8a417e29ff "%s:%d: failed assert `%s`", >> > >> ap=ap@entry=0x2b8a4d311cf0) >> > >>> at ink_error.cc:65 >> > >>> #4 0x00002b8a417d5b79 in ink_fatal ( >> > >>> message_format=message_format@entry=0x2b8a417e29ff "%s:%d: >> failed >> > >> assert `%s`") >> > >>> at ink_error.cc:73 >> > >>> #5 0x00002b8a417d3c35 in _ink_assert ( >> > >>> expression=expression@entry=0x7f13e0 "call_event == >> > >> core_lock_retry_event", >> > >>> file=file@entry=0x7f1413 "PluginVC.cc", line=line@entry=216) at >> > >> ink_assert.cc:37 >> > >>> #6 0x00000000004fc68c in PluginVC::main_handler >> (this=0x2b8ad8807048, >> > >> event=<optimized out>, >> > >>> data=0x2c31fa0) at PluginVC.cc:216 >> > >>> #7 0x00000000007e0752 in Continuation::handleEvent (data=0x2c31fa0, >> > >> event=1, this=<optimized out>) >> > >>> at I_Continuation.h:153 >> > >>> #8 EThread::process_event (this=this@entry=0x2b8a4c707010, >> e=e@entry >> > =0x2c31fa0, >> > >> calling_code=1) >> > >>> at UnixEThread.cc:148 >> > >>> #9 0x00000000007e15d8 in EThread::execute (this=0x2b8a4c707010) at >> > >> UnixEThread.cc:202 >> > >>> #10 0x00000000007e01d5 in spawn_thread_internal (a=0x27a24d0) at >> > >> Thread.cc:86 >> > >>> #11 0x00002b8a42b176ba in start_thread (arg=0x2b8a4d312700) at >> > >> pthread_create.c:333 >> > >>> #12 0x00002b8a438d93dd in clone () at >> > >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 >> > >>> (gdb) >> > >>> >> > >>> 212 } else { >> > >>> 213 if (call_event == sm_lock_retry_event) { >> > >>> 214 sm_lock_retry_event = NULL; >> > >>> 215 } else { >> > >>> 216 ink_release_assert(call_event == core_lock_retry_event); >> > >>> 217 core_lock_retry_event = NULL; >> > >>> 218 } >> > >>> 219 >> > >>> >> > >>> Process: [ET_NET 0] [72] >> > >>> File: /opt/trafficserver/bin/traffic_server >> > >>> Version: Traffic Server 6.2.1 >> > >>> System Version: Linux x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 >> > >> 3.10.0-514.16.1.el7.x86_64 >> > >>> Date: Sat, 29 Jul 2017 08:15:01 +0000 >> > >>> >> > >>> No target signal information >> > >>> >> > >>> No target CPU registers >> > >>> >> > >>> Thread 136, [ET_NET 9]: >> > >>> 0 0x00000000004afb02 crash_logger_invoke(int, siginfo_t*, void*) >> + >> > >> 0x82 >> > >>> 1 0x00002b8a42b21390 __restore_rt + (nil) >> > >>> 2 0x00002b8a43807428 gsignal + 0x38 >> > >>> 3 0x00002b8a4380902a abort + 0x16a >> > >>> 4 0x00002b8a417d5adf ink_fatal_va(char const*, __va_list_tag*) + >> > 0x9f >> > >>> 5 0x00002b8a417d5b79 ink_fatal(char const*, ...) + 0x99 >> > >>> 6 0x00002b8a417d3c35 _ink_assert + 0x15 >> > >>> 7 0x00000000004fc68c PluginVC::main_handler(int, void*) + 0x5ac >> > >>> 8 0x00000000007e0752 EThread::process_event(Event*, int) + 0x92 >> > >>> 9 0x00000000007e15d8 EThread::execute() + 0x808 >> > >>> 10 0x00000000007e01d5 spawn_thread_internal(void*) + 0x55 >> > >>> 11 0x00002b8a42b176ba start_thread + 0xca >> > >>> 12 0x00002b8a438d93dd clone + 0x6d >> > >>> 13 0x0000000000000000 0x0 + 0x6d >> > >>> >> > >> >> > >> > >
