call_event = core_lock_retry event..
bt full output below... I'll go ahead and file a GitHub issue. Thanks.
#5 0x00002b8a417d3c35 in _ink_assert (
expression=expression@entry=0x7f13e0 "call_event ==
core_lock_retry_event",
file=file@entry=0x7f1413 "PluginVC.cc", line=line@entry=216) at
ink_assert.cc:37
(gdb) bt full
#0 0x00002b8a43807428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54
resultvar = 0
pid = 72
selftid = 136
#1 0x00002b8a43809187 in __GI_abort () at abort.c:118
act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0},
sa_mask = {__val = {18446744073709551615 <repeats 16 times>}},
sa_flags = 0, sa_restorer = 0x0}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00002b8a417d5adf in ink_die_die_die () at ink_error.cc:43
No locals.
#3 ink_fatal_va (fmt=0x2b8a417e29ff "%s:%d: failed assert `%s`",
ap=ap@entry=0x2b8a4d311cf0) at ink_error.cc:65
msg = "FATAL: PluginVC.cc:216: failed assert `call_event ==
core_lock_retry_event`", '\000' <repeats 948 times>
#4 0x00002b8a417d5b79 in ink_fatal (
message_format=message_format@entry=0x2b8a417e29ff "%s:%d: failed
assert `%s`") at ink_error.cc:73
ap = <error reading variable ap (Attempt to dereference a generic
pointer.)>
#5 0x00002b8a417d3c35 in _ink_assert (
expression=expression@entry=0x7f13e0 "call_event ==
core_lock_retry_event", ---Type <return> to continue, or q <return> to
quit---
file=file@entry=0x7f1413 "PluginVC.cc", line=line@entry=216)
at ink_assert.cc:37
No locals.
#6 0x00000000004fc68c in PluginVC::main_handler (this=0x2b8ad8807048,
event=<optimized out>, data=0x2c31fa0) at PluginVC.cc:216
__FUNCTION__ = "main_handler"
call_event = 0x2c31fa0
read_mutex_held = <optimized out>
write_mutex_held = false
read_side_mutex = {m_ptr = 0x2b8ad40071c0}
write_side_mutex = {m_ptr = 0x0}
#7 0x00000000007e0752 in Continuation::handleEvent (data=0x2c31fa0,
event=1,
this=<optimized out>) at I_Continuation.h:153
No locals.
#8 EThread::process_event (this=this@entry=0x2b8a4c707010,
e=e@entry=0x2c31fa0, calling_code=1) at UnixEThread.cc:148
c_temp = <optimized out>
lock = {m = {m_ptr = 0x2b8ad9002d20}, lock_acquired = true}
#9 0x00000000007e15d8 in EThread::execute (this=0x2b8a4c707010)
at UnixEThread.cc:202
done_one = <optimized out>
e = 0x2c31fa0
NegativeQueue = {<DLL<Event, Event::Link_link>> = {head =
0x2816700},
---Type <return> to continue, or q <return> to quit---
tail = 0x2816700}
next_time = 1501263000357130435
#10 0x00000000007e01d5 in spawn_thread_internal (a=0x27a24d0) at
Thread.cc:86
p = 0x27a24d0
#11 0x00002b8a42b176ba in start_thread (arg=0x2b8a4d312700)
at pthread_create.c:333
__res = <optimized out>
pd = 0x2b8a4d312700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {47873000548096,
126801912536511865, 0, 140735034297183, 47873000548800, 0,
6257443582078367097, 6257462270246139257},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0},
data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#12 0x00002b8a438d93dd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
(gdb)
On Wed, Aug 2, 2017 at 3:18 PM, Phil Sorber <[email protected]> wrote:
> It seems pretty straightforward that the code expects call_event to be
> sm_lock_retry_event or core_lock_retry_event if it enters that code block.
> Can you print out the value of call_event from that stack frame (#5)? Maybe
> also do a `bt full` and see if anything interesting falls out of that.
>
> Thanks.
>
> On Wed, Aug 2, 2017 at 1:38 PM Dk Jack <[email protected]> wrote:
>
> > Will do.
> >
> > Even though this happened twice in our environment, I don't know the
> exact
> > process to reproduce it. Not sure if it makes sense to open an issue
> > without proper steps to reproduce it. I was hoping to get some guidance
> on
> > how to reproduce it from someone who is more familiar with this area of
> the
> > code.
> >
> > Bhasker.
> >
> > > On Aug 2, 2017, at 6:26 AM, Phil Sorber <[email protected]> wrote:
> > >
> > > Dk,
> > >
> > > Can you open a github issue on this and then also test with the new
> 6.2.2
> > > RC0 I just put up for a vote?
> > >
> > > If you can't mark the issue for the 6.2.3 milestone, I can do that for
> > you.
> > > If this is still broken in the 6.2.2 RC0 we can stop the vote and
> figure
> > > out a fix depending on the severity.
> > >
> > > Thanks.
> > >
> > >> On Wed, Aug 2, 2017 at 12:11 AM Dk Jack <[email protected]> wrote:
> > >>
> > >> Btw, this crash happened on 6.2.1.
> > >>
> > >> Please let me know if this is not the correct forum for these sort of
> > >> questions. Thanks.
> > >>
> > >>
> > >>
> > >>> On Jul 31, 2017, at 11:15 AM, Dk Jack <[email protected]> wrote:
> > >>>
> > >>> Hi,
> > >>> I am seeing an ATS crash. The stack trace and the crash log info for
> > the
> > >> crash are pasted below. Seems to be happening in
> PluginVC::main_handler
> > >> line 216. I would appreciate it if someone can comment on possible
> > reasons
> > >> for this crash? Thanks.
> > >>>
> > >>> D.
> > >>>
> > >>>
> > >>> [Thread debugging using libthread_db enabled]
> > >>> Using host libthread_db library
> > >> "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > >>> Core was generated by `/opt/trafficserver/bin/traffic_server -M
> > >> --bind_stdout /opt/trafficserver/var/l'.
> > >>> Program terminated with signal SIGABRT, Aborted.
> > >>> #0 0x00002b8a43807428 in __GI_raise (sig=sig@entry=6) at
> > >> ../sysdeps/unix/sysv/linux/raise.c:54
> > >>> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> > >>> [Current thread is 1 (Thread 0x2b8a4d312700 (LWP 136))]
> > >>> (gdb) bt
> > >>> #0 0x00002b8a43807428 in __GI_raise (sig=sig@entry=6) at
> > >> ../sysdeps/unix/sysv/linux/raise.c:54
> > >>> #1 0x00002b8a43809187 in __GI_abort () at abort.c:118
> > >>> #2 0x00002b8a417d5adf in ink_die_die_die () at ink_error.cc:43
> > >>> #3 ink_fatal_va (fmt=0x2b8a417e29ff "%s:%d: failed assert `%s`",
> > >> ap=ap@entry=0x2b8a4d311cf0)
> > >>> at ink_error.cc:65
> > >>> #4 0x00002b8a417d5b79 in ink_fatal (
> > >>> message_format=message_format@entry=0x2b8a417e29ff "%s:%d: failed
> > >> assert `%s`")
> > >>> at ink_error.cc:73
> > >>> #5 0x00002b8a417d3c35 in _ink_assert (
> > >>> expression=expression@entry=0x7f13e0 "call_event ==
> > >> core_lock_retry_event",
> > >>> file=file@entry=0x7f1413 "PluginVC.cc", line=line@entry=216) at
> > >> ink_assert.cc:37
> > >>> #6 0x00000000004fc68c in PluginVC::main_handler
> (this=0x2b8ad8807048,
> > >> event=<optimized out>,
> > >>> data=0x2c31fa0) at PluginVC.cc:216
> > >>> #7 0x00000000007e0752 in Continuation::handleEvent (data=0x2c31fa0,
> > >> event=1, this=<optimized out>)
> > >>> at I_Continuation.h:153
> > >>> #8 EThread::process_event (this=this@entry=0x2b8a4c707010,
> e=e@entry
> > =0x2c31fa0,
> > >> calling_code=1)
> > >>> at UnixEThread.cc:148
> > >>> #9 0x00000000007e15d8 in EThread::execute (this=0x2b8a4c707010) at
> > >> UnixEThread.cc:202
> > >>> #10 0x00000000007e01d5 in spawn_thread_internal (a=0x27a24d0) at
> > >> Thread.cc:86
> > >>> #11 0x00002b8a42b176ba in start_thread (arg=0x2b8a4d312700) at
> > >> pthread_create.c:333
> > >>> #12 0x00002b8a438d93dd in clone () at
> > >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> > >>> (gdb)
> > >>>
> > >>> 212 } else {
> > >>> 213 if (call_event == sm_lock_retry_event) {
> > >>> 214 sm_lock_retry_event = NULL;
> > >>> 215 } else {
> > >>> 216 ink_release_assert(call_event == core_lock_retry_event);
> > >>> 217 core_lock_retry_event = NULL;
> > >>> 218 }
> > >>> 219
> > >>>
> > >>> Process: [ET_NET 0] [72]
> > >>> File: /opt/trafficserver/bin/traffic_server
> > >>> Version: Traffic Server 6.2.1
> > >>> System Version: Linux x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017
> > >> 3.10.0-514.16.1.el7.x86_64
> > >>> Date: Sat, 29 Jul 2017 08:15:01 +0000
> > >>>
> > >>> No target signal information
> > >>>
> > >>> No target CPU registers
> > >>>
> > >>> Thread 136, [ET_NET 9]:
> > >>> 0 0x00000000004afb02 crash_logger_invoke(int, siginfo_t*, void*) +
> > >> 0x82
> > >>> 1 0x00002b8a42b21390 __restore_rt + (nil)
> > >>> 2 0x00002b8a43807428 gsignal + 0x38
> > >>> 3 0x00002b8a4380902a abort + 0x16a
> > >>> 4 0x00002b8a417d5adf ink_fatal_va(char const*, __va_list_tag*) +
> > 0x9f
> > >>> 5 0x00002b8a417d5b79 ink_fatal(char const*, ...) + 0x99
> > >>> 6 0x00002b8a417d3c35 _ink_assert + 0x15
> > >>> 7 0x00000000004fc68c PluginVC::main_handler(int, void*) + 0x5ac
> > >>> 8 0x00000000007e0752 EThread::process_event(Event*, int) + 0x92
> > >>> 9 0x00000000007e15d8 EThread::execute() + 0x808
> > >>> 10 0x00000000007e01d5 spawn_thread_internal(void*) + 0x55
> > >>> 11 0x00002b8a42b176ba start_thread + 0xca
> > >>> 12 0x00002b8a438d93dd clone + 0x6d
> > >>> 13 0x0000000000000000 0x0 + 0x6d
> > >>>
> > >>
> >
>
