Hi Andrew, all,
as I wrote before, I caught two paths where stonithd (c72f5ca) dumps core.
Here are gdb backtraces for them (sorry for posting them inline, I was
requested to do that ASAP and I hope it is not yet too late for 1.1.8 ;)
). Some vars are optimized out, but I hope that doesn't matter. If some
more information is needed please just request it.
First one is:
...
Core was generated by `/usr/libexec/pacemaker/stonithd'.
Program terminated with signal 11, Segmentation fault.
...
(gdb) bt
#0 0x00007f4aec6cdb51 in __strlen_sse2 () from /lib64/libc.so.6
#1 0x00007f4aec6cd866 in strdup () from /lib64/libc.so.6
#2 0x000000000040c6f6 in create_remote_stonith_op (client=0x1871120
"2194f1b8-5722-49c3-bed1-c8fecc78ca02", request=0x1884840, peer=<value
optimized out>)
at remote.c:313
#3 0x000000000040cf40 in initiate_remote_stonith_op (client=<value
optimized out>, request=0x1884840, manual_ack=0) at remote.c:336
#4 0x000000000040a2be in stonith_command (client=0x1870a80, id=<value
optimized out>, flags=<value optimized out>, request=0x1884840, remote=0x0)
at commands.c:1380
#5 0x0000000000403252 in st_ipc_dispatch (c=0x18838d0, data=<value
optimized out>, size=329) at main.c:142
#6 0x00007f4aebaf8d64 in ?? () from /usr/lib64/libqb.so.0
#7 0x00007f4aebaf908e in qb_ipcs_dispatch_connection_request () from
/usr/lib64/libqb.so.0
#8 0x00007f4aee26fda5 in gio_read_socket (gio=<value optimized out>,
condition=G_IO_IN, data=0x18732f0) at mainloop.c:353
#9 0x00007f4aebf8ef0e in g_main_context_dispatch () from
/lib64/libglib-2.0.so.0
#10 0x00007f4aebf92938 in ?? () from /lib64/libglib-2.0.so.0
#11 0x00007f4aebf92d55 in g_main_loop_run () from /lib64/libglib-2.0.so.0
#12 0x0000000000403a98 in main (argc=<value optimized out>,
argv=0x7fffa3443148) at main.c:890
(gdb) bt full
#0 0x00007f4aec6cdb51 in __strlen_sse2 () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f4aec6cd866 in strdup () from /lib64/libc.so.6
No symbol table info available.
#2 0x000000000040c6f6 in create_remote_stonith_op (client=0x1871120
"2194f1b8-5722-49c3-bed1-c8fecc78ca02", request=0x1884840, peer=<value
optimized out>)
at remote.c:313
nodeid = <value optimized out>
node = 0x1871790
op = 0x187e2e0
dev = <value optimized out>
__func__ = "create_remote_stonith_op"
__PRETTY_FUNCTION__ = "create_remote_stonith_op"
#3 0x000000000040cf40 in initiate_remote_stonith_op (client=<value
optimized out>, request=0x1884840, manual_ack=0) at remote.c:336
query = 0x0
client_id = 0x1871120 "2194f1b8-5722-49c3-bed1-c8fecc78ca02"
op = 0x0
__func__ = "initiate_remote_stonith_op"
__PRETTY_FUNCTION__ = "initiate_remote_stonith_op"
#4 0x000000000040a2be in stonith_command (client=0x1870a80, id=<value
optimized out>, flags=<value optimized out>, request=0x1884840, remote=0x0)
at commands.c:1380
alternate_host = <value optimized out>
dev = <value optimized out>
target = 0x1883f40 "1074005258"
call_options = 4610
rc = -95
is_reply = 0
always_reply = 0
reply = 0x0
data = 0x0
op = 0x187e550 "st_fence"
client_id = 0x1874cb0 "2194f1b8-5722-49c3-bed1-c8fecc78ca02"
__func__ = "stonith_command"
__PRETTY_FUNCTION__ = "stonith_command"
__FUNCTION__ = "stonith_command"
#5 0x0000000000403252 in st_ipc_dispatch (c=0x18838d0, data=<value
optimized out>, size=329) at main.c:142
id = 4
flags = 1
request = 0x1884840
client = 0x1870a80
__FUNCTION__ = "st_ipc_dispatch"
__func__ = "st_ipc_dispatch"
__PRETTY_FUNCTION__ = "st_ipc_dispatch"
#6 0x00007f4aebaf8d64 in ?? () from /usr/lib64/libqb.so.0
No symbol table info available.
#7 0x00007f4aebaf908e in qb_ipcs_dispatch_connection_request () from
/usr/lib64/libqb.so.0
No symbol table info available.
#8 0x00007f4aee26fda5 in gio_read_socket (gio=<value optimized out>,
condition=G_IO_IN, data=0x18732f0) at mainloop.c:353
adaptor = 0x18732f0
fd = 15
__func__ = "gio_read_socket"
#9 0x00007f4aebf8ef0e in g_main_context_dispatch () from
/lib64/libglib-2.0.so.0
No symbol table info available.
#10 0x00007f4aebf92938 in ?? () from /lib64/libglib-2.0.so.0
No symbol table info available.
#11 0x00007f4aebf92d55 in g_main_loop_run () from /lib64/libglib-2.0.so.0
No symbol table info available.
#12 0x0000000000403a98 in main (argc=<value optimized out>,
argv=0x7fffa3443148) at main.c:890
flag = <value optimized out>
lpc = 0
---Type <return> to continue, or q <return> to quit---
argerr = 0
option_index = 0
cluster = {uuid = 0x176da90 "1090782474", uname = 0x176dac0
"vd01-b", nodeid = 1090782474, cs_dispatch = 0x404050
<stonith_peer_ais_callback>,
destroy = 0x404230 <stonith_peer_ais_destroy>}
actions = {0x40e3fb "reboot", 0x40e402 "off", 0x40ea75 "list",
0x40e406 "monitor", 0x40e40e "status"}
__func__ = "main"
Second is (segfault in CRM_ASSERT()):
...
Core was generated by `/usr/libexec/pacemaker/stonithd'.
Program terminated with signal 11, Segmentation fault.
#0 stonith_command (client=0x0, id=0, flags=0, request=0xb342f0,
remote=0xb39cf0 "vd01-d") at commands.c:1258
1258 commands.c: No such file or directory.
in commands.c
...
(gdb) bt
#0 stonith_command (client=0x0, id=0, flags=0, request=0xb342f0,
remote=0xb39cf0 "vd01-d") at commands.c:1258
#1 0x00000000004040e4 in stonith_peer_callback (kind=<value optimized
out>, from=<value optimized out>,
data=0x7fffa5327cc8 "<st-reply
st_origin=\"stonith_construct_async_reply\" t=\"stonith-ng\"
st_op=\"st_notify\" st_device_id=\"manual_ack\"
st_remote_op=\"25917710-8972-4c40-b783-9648749396a4\"
st_clientid=\"936ea671-61ba-4258-8e12-"...) at main.c:218
#2 stonith_peer_ais_callback (kind=<value optimized out>, from=<value
optimized out>,
data=0x7fffa5327cc8 "<st-reply
st_origin=\"stonith_construct_async_reply\" t=\"stonith-ng\"
st_op=\"st_notify\" st_device_id=\"manual_ack\"
st_remote_op=\"25917710-8972-4c40-b783-9648749396a4\"
st_clientid=\"936ea671-61ba-4258-8e12-"...) at main.c:254
#3 0x00007f92ded376ca in ais_dispatch_message (handle=<value optimized
out>, groupName=<value optimized out>, nodeid=<value optimized out>,
pid=<value optimized out>, msg=0x7fffa5327a78, msg_len=<value
optimized out>) at corosync.c:551
#4 pcmk_cpg_deliver (handle=<value optimized out>, groupName=<value
optimized out>, nodeid=<value optimized out>, pid=<value optimized out>,
msg=0x7fffa5327a78, msg_len=<value optimized out>) at corosync.c:619
#5 0x00007f92de91ceaf in cpg_dispatch (handle=7749363892505018368,
dispatch_types=<value optimized out>) at cpg.c:412
#6 0x00007f92ded34a42 in pcmk_cpg_dispatch (user_data=<value optimized
out>) at corosync.c:577
#7 0x00007f92def61d27 in mainloop_gio_callback (gio=<value optimized
out>, condition=G_IO_IN, data=0xb2d400) at mainloop.c:535
#8 0x00007f92dcc7ff0e in g_main_context_dispatch () from
/lib64/libglib-2.0.so.0
#9 0x00007f92dcc83938 in ?? () from /lib64/libglib-2.0.so.0
#10 0x00007f92dcc83d55 in g_main_loop_run () from /lib64/libglib-2.0.so.0
#11 0x0000000000403a98 in main (argc=<value optimized out>,
argv=0x7fffa5427de8) at main.c:890
(gdb) bt full
#0 stonith_command (client=0x0, id=0, flags=0, request=0xb342f0,
remote=0xb39cf0 "vd01-d") at commands.c:1258
call_options = 4104
rc = -95
is_reply = 1
always_reply = 0
reply = 0x0
data = 0x0
op = 0xb34370 "st_notify"
client_id = 0xb3ddc0 "936ea671-61ba-4258-8e12-98542a541b23"
__func__ = "stonith_command"
__PRETTY_FUNCTION__ = "stonith_command"
__FUNCTION__ = "stonith_command"
#1 0x00000000004040e4 in stonith_peer_callback (kind=<value optimized
out>, from=<value optimized out>,
data=0x7fffa5327cc8 "<st-reply
st_origin=\"stonith_construct_async_reply\" t=\"stonith-ng\"
st_op=\"st_notify\" st_device_id=\"manual_ack\"
st_remote_op=\"25917710-8972-4c40-b783-9648749396a4\"
st_clientid=\"936ea671-61ba-4258-8e12-"...) at main.c:218
remote = 0xb39cf0 "vd01-d"
#2 stonith_peer_ais_callback (kind=<value optimized out>, from=<value
optimized out>,
data=0x7fffa5327cc8 "<st-reply
st_origin=\"stonith_construct_async_reply\" t=\"stonith-ng\"
st_op=\"st_notify\" st_device_id=\"manual_ack\"
st_remote_op=\"25917710-8972-4c40-b783-9648749396a4\"
st_clientid=\"936ea671-61ba-4258-8e12-"...) at main.c:254
xml = 0xb342f0
__func__ = "stonith_peer_ais_callback"
#3 0x00007f92ded376ca in ais_dispatch_message (handle=<value optimized
out>, groupName=<value optimized out>, nodeid=<value optimized out>,
pid=<value optimized out>, msg=0x7fffa5327a78, msg_len=<value
optimized out>) at corosync.c:551
data = 0x7fffa5327cc8 "<st-reply
st_origin=\"stonith_construct_async_reply\" t=\"stonith-ng\"
st_op=\"st_notify\" st_device_id=\"manual_ack\"
st_remote_op=\"25917710-8972-4c40-b783-9648749396a4\"
st_clientid=\"936ea671-61ba-4258-8e12-"...
uncompressed = 0x0
xml = 0x0
#4 pcmk_cpg_deliver (handle=<value optimized out>, groupName=<value
optimized out>, nodeid=<value optimized out>, pid=<value optimized out>,
msg=0x7fffa5327a78, msg_len=<value optimized out>) at corosync.c:619
ais_msg = 0x7fffa5327a78
__func__ = "pcmk_cpg_deliver"
#5 0x00007f92de91ceaf in cpg_dispatch (handle=7749363892505018368,
dispatch_types=<value optimized out>) at cpg.c:412
timeout = 0
error = <value optimized out>
cpg_inst = 0xb2cd90
res_cpg_confchg_callback = <value optimized out>
res_cpg_deliver_callback = 0x7fffa53279c0
res_cpg_totem_confchg_callback = <value optimized out>
cpg_inst_copy = {c = 0xb2cdf0, finalize = 0, context = 0x0,
{model_data = {model = CPG_MODEL_V1}, model_v1_data = {model =
CPG_MODEL_V1,
cpg_deliver_fn = 0x7f92ded37300 <pcmk_cpg_deliver>,
cpg_confchg_fn = 0x7f92ded33fb0 <pcmk_cpg_membership>,
cpg_totem_confchg_fn = 0,
flags = 0}}, iteration_list_head = {next = 0xb2cdd0, prev
= 0xb2cdd0}}
dispatch_data = 0x7fffa53279c0
member_list = {{nodeid = 1090782474, pid = 4965, reason = 0},
{nodeid = 1107559690, pid = 3544, reason = 0}, {nodeid = 1124336906, pid
= 4487,
reason = 3544}, {nodeid = 0, pid = 0, reason = 0} <repeats
125 times>}
left_list = {{nodeid = 0, pid = 0, reason = 0} <repeats 128 times>}
joined_list = {{nodeid = 1107559690, pid = 3544, reason = 1},
{nodeid = 0, pid = 0, reason = 0} <repeats 127 times>}
group_name = {length = 11, value = "stonith-ng", '\000' <repeats
117 times>}
left_list_start = <value optimized out>
joined_list_start = <value optimized out>
i = <value optimized out>
ring_id = {nodeid = 0, seq = 0}
totem_member_list = {0 <repeats 128 times>}
errno_res = <value optimized out>
dispatch_buf =
"\005\000\000\000\000\000\000\000W\004\000\000\000\000\000\000\240\224\327)\204\177\000\000\v\000\000\000\000\000\000\000stonith-ng",
'\000' <repeats 118 times>"\237,
\003\000\000\000\000\000\000\n\005\004C\000\000\000\000\207\021\000\000\204\177\000\000\000\000\000\000\000\000\000\000\237\003\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001",
'\000' <repeats 19 times>, "\t", '\000' <repeats 263 times>,
"\n\005\004C\207\021\000\000\000\000\000\000\t\000\000\000\006\000\000\000vd01-d",
'\000' <repeats 250 times>, "O\001\000\000\000\000\000\000<st-reply
st_origin=\"stonith_construct_asyn"...
#6 0x00007f92ded34a42 in pcmk_cpg_dispatch (user_data=<value optimized
out>) at corosync.c:577
rc = 0
---Type <return> to continue, or q <return> to quit---
__func__ = "pcmk_cpg_dispatch"
#7 0x00007f92def61d27 in mainloop_gio_callback (gio=<value optimized
out>, condition=G_IO_IN, data=0xb2d400) at mainloop.c:535
keep = 1
client = 0xb2d400
__func__ = "mainloop_gio_callback"
#8 0x00007f92dcc7ff0e in g_main_context_dispatch () from
/lib64/libglib-2.0.so.0
No symbol table info available.
#9 0x00007f92dcc83938 in ?? () from /lib64/libglib-2.0.so.0
No symbol table info available.
#10 0x00007f92dcc83d55 in g_main_loop_run () from /lib64/libglib-2.0.so.0
No symbol table info available.
#11 0x0000000000403a98 in main (argc=<value optimized out>,
argv=0x7fffa5427de8) at main.c:890
flag = <value optimized out>
lpc = 0
argerr = 0
option_index = 0
cluster = {uuid = 0xb2da90 "1090782474", uname = 0xb2dac0
"vd01-b", nodeid = 1090782474, cs_dispatch = 0x404050
<stonith_peer_ais_callback>,
destroy = 0x404230 <stonith_peer_ais_destroy>}
actions = {0x40e3fb "reboot", 0x40e402 "off", 0x40ea75 "list",
0x40e406 "monitor", 0x40e40e "status"}
__func__ = "main"
Best,
Vladislav
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
