2008/4/22 Keisuke MORI <[EMAIL PROTECTED]>: > Hi, > > > > "Andrew Beekhof" <[EMAIL PROTECTED]> writes: > > > On Wed, Apr 16, 2008 at 1:31 PM, HIDEO YAMAUCHI > > <[EMAIL PROTECTED]> wrote: > >> Hi Andrew, > >> > >> > >> > I asked for the right function but the wrong frame number - I should > >> > have asked for frame 2. Sorry :( > >> > >> (gdb) frame 2 > >> #2 0x0000000000416c74 in stop_recurring_action_by_rsc (key=0x755f60, > value=0x755f40, > >> user_data=0x545a10) at lrm.c:1442 > >> 1442 if(op->interval != 0 && safe_str_eq(op->rsc_id, > rsc->id)) { > >> (gdb) print *rsc > >> Variable "rsc" is not available. > >> (gdb) print *op > >> No symbol "op" in current context. > >> > >> Is what or my operation a mistake? > > > > Looks like gcc is being too clever for it's own good (by optimizing > > away some of the variables) :-( > > > > Can you try the following patch please? > > > > diff -r be12cb83cd2d crmd/lrm.c > > --- a/crmd/lrm.c Wed Apr 16 10:46:59 2008 +0200 > > +++ b/crmd/lrm.c Wed Apr 16 15:02:16 2008 +0200 > > @@ -1451,7 +1451,7 @@ stop_recurring_action_by_rsc(gpointer ke > > { > > lrm_rsc_t *rsc = user_data; > > struct recurring_op_s *op = (struct recurring_op_s*)value; > > - > > + crm_info("op->rsc=%s (%p), rsc=%s (%p)", crm_str(op->rsc_id), > > op->rsc_id, crm_str(rsc->id), rsc->id); > > if(op->interval != 0 && safe_str_eq(op->rsc_id, rsc->id)) { > > cancel_op(rsc, key, op->call_id, FALSE); > > } > > > > I think I found the cause of this issue. > I attached the additional log with your patch (a bit different though) > and the stacktrace. > > Here's my observation: > > - An element of pending_ops is removed at lrm.c:L497 > - It is called inside from g_has_table_foreach() at L1475 > - This is violating the usage of g_has_table_foreach() according > to the glib manual. > - Therefore the iteration can not proceed correctly and would > try to refer to a removed element.
Turns out that the Stateful resource in CTS was never getting promoted. Once I fixed this, I was able to trigger the bug too (in the last few minutes). Thanks for your diagnosis and the patch, you've certainly saved me some time :-) > > http://hg.linux-ha.org/lha-2.1/annotate/333aef5bd4ed/crm/crmd/lrm.c > (...) > 946 /* not doing this will block the node from shutting down */ > 947 g_hash_table_remove(pending_ops, key); > (...) > 1475 g_hash_table_foreach(pending_ops, > stop_recurring_action_by_rsc, rsc); > > > http://library.gnome.org/devel/glib/stable/glib-Hash-Tables.html#g-hash-table-foreach > (...) > The hash table may not be modified while iterating over it (you can't > add/remove items). > > > I also attached my suggested patch, although I can not guarantee > the correctness but just to show you the idea. > > Thanks, > > -- > Keisuke MORI > NTT DATA Intellilink Corporation > > > _______________________________________________________ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ > > _______________________________________________________ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/