No recent network changes. Will check for abnormal traffic using wireshark.
I also notice that the XML lines are partial (no ending '>', closing " and sometimes partial words) in logs. Any lines > 472 characters are truncated to 472 characters. Wondering is it due to anyother limitations. I can post some line tomorrow when i am back to work. On Wed, Jun 8, 2016 at 8:00 PM, Ken Gaillot <kgail...@redhat.com> wrote: > On 06/08/2016 06:14 AM, Narayanamoorthy Srinivasan wrote: > > I have a pacemaker cluster with two pacemaker remote nodes. Recently the > > remote nodes started throwing below errors and SDB started self-fencing. > > Appreciate if someone throws light on what could be the issue and the > fix. > > > > OS - SLES 12 SP1 > > Pacemaker Remote version - pacemaker-remote-1.1.13-14.7.x86_64 > > > > 2016-06-08T14:11:46.009073+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser > > error : AttValue: ' expected > > 2016-06-08T14:11:46.009314+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > key="neutron-ha-tool_monitor_0" operation="monitor" > > crm-debug-origin="do_update_ > > 2016-06-08T14:11:46.009443+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > ^ > > 2016-06-08T14:11:46.009567+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser > > error : attributes construct error > > 2016-06-08T14:11:46.009697+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > key="neutron-ha-tool_monitor_0" operation="monitor" > > crm-debug-origin="do_update_ > > 2016-06-08T14:11:46.009824+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > ^ > > 2016-06-08T14:11:46.009948+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser > > error : Couldn't find end of Start Tag lrm_rsc_op line 1 > > 2016-06-08T14:11:46.010070+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > key="neutron-ha-tool_monitor_0" operation="monitor" > > crm-debug-origin="do_update_ > > 2016-06-08T14:11:46.010191+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > ^ > > 2016-06-08T14:11:46.010460+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser > > error : Premature end of data in tag lrm_resource line 1 > > 2016-06-08T14:11:46.010718+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > key="neutron-ha-tool_monitor_0" operation="monitor" > > crm-debug-origin="do_update_ > > 2016-06-08T14:11:46.010977+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: > > ^ > > 2016-06-08T14:11:46.011234+05:30 d18-fb-7b-18-f1-8e > > pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser > > error : Premature end of data in tag lrm_resources line 1 > > > > > > -- > > Thanks & Regards > > Moorthy > > This sounds like the network traffic between the cluster nodes and the > remote nodes is being corrupted. Have there been any network changes > lately? Switch/firewall/etc. equipment/settings? MTU? > > You could try using a packet sniffer such as wireshark to see if the > traffic looks abnormal in some way. The payload is XML so it should be > more or less readable. > > > _______________________________________________ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Thanks & Regards Moorthy
_______________________________________________ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org