On 3 May 2014, at 6:20 am, Radoslaw Garbacz <radoslaw.garb...@xtremedatainc.com> wrote:
> Hi, > > I have a strange situation, which I would like to ask about, whether it is a > bug, misconfiguration or an intended behavior. Sort version: Thats not a valid test Medium version: Thats not a valid test and there are updates available for pacemaker in el6 Long version: Using iptables in this way not only stops the cluster from seeing its peer, but also stops the cluster from talking to itself on the same node. At which point nothing will work. Did you configure fencing? > > A disconnected node does not detect it is lost, and does not perform any > actions to stop, even though resource agents report errors when monitored, > just the number of processes (of some hanged resource agents) keeps growing. > > Seems like pacemaker ignores timeouts when trying to update CIB. > > The situation is caused by corosync not detecting lost quorum due to firewall > blocking lo. As far as I checked this prevents corosync from detecting > problems with the cluster, and when lo access is restored everything should > be fine, but shouldn't pacemaker detect lost CIB service and do something > about it? Maybe there is a configuration parameter to control this? > > Technical details: > > 1) > 1.1) machine: Amazon Linux: Linux ... 3.10.35-43.137.amzn1.x86_64 #1 SMP Wed > Apr 2 09:36:59 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > 1.2) Pacemaker: Pacemaker 1.1.9-1512.el6 > 1.3) corosync: Corosync Cluster Engine, version '2.3.2' > > > 2) Net: basic: ethx, lo > > 3) iptables: > *filter > :INPUT ACCEPT [0:0] > :FORWARD ACCEPT [0:0] > :OUTPUT ACCEPT [0:0] > -A INPUT -p tcp -m tcp -s <my_machine> --dport 22 -j ACCEPT > -A INPUT -j DROP > -A OUTPUT -p tcp -m tcp -d <my_machine> --sport 22 -j ACCEPT > -A OUTPUT -j DROP > COMMIT > > 4) crm config: > <crm_config> > <cluster_property_set id="cib-bootstrap-options"> > <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" > value="false"/> > <nvpair id="cib-bootstrap-options-no-quorum-policy" > name="no-quorum-policy" value="stop"/> > <nvpair id="cib-bootstrap-options-stop-orphan-resources" > name="stop-orphan-resources" value="true"/> > <nvpair id="cib-bootstrap-options-start-failure-is-fatal" > name="start-failure-is-fatal" value="true"/> > <nvpair id="cib-bootstrap-options-expected-quorum-votes" > name="expected-quorum-votes" value="3"/> > <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" > value="1.1.9-1512.el6-2a917dd"/> > <nvpair id="cib-bootstrap-options-cluster-infrastructure" > name="cluster-infrastructure" value="corosync"/> > </cluster_property_set> > </crm_config> > > > 5) Example resource config: > <primitive class="ocf" id="dbx_ready_nodes" provider="dbxcl" > type="ready.ocf.sh"> > <instance_attributes id="dbx_ready_nodes-instance_attributes"> > <nvpair id="dbx_ready_nodes-instance_attributes-dbxclrole" > name="dbxclrole" value="''"/> > </instance_attributes> > <operations> > <op id="dbx_ready_nodes-start-timeout-1min-on-fail-stop" > interval="0s" name="start" on-fail="stop" timeout="1min"/> > <op id="dbx_ready_nodes-stop-timeout-8min" interval="0s" name="stop" > timeout="8min"/> > <op id="dbx_ready_nodes-monitor-interval-83s" interval="83s" > name="monitor" on-fail="stop" timeout="60s"/> > <op id="dbx_ready_nodes-validate-all-interval-29s" interval="29s" > name="validate-all" on-fail="stop" timeout="60s"/> > </operations> > </primitive> > > > 6) Logs: > Below a resource "dbx_ready_nodes" monitor action returns error, but nothing > happens, the resource is not being requested to stop (even though it should, > as can be seen above) > > May 02 20:04:13 [16191] ip-10-116-169-85 lrmd: debug: > operation_finished: dbx_ready_nodes_monitor_83000:8669 - exited with rc=1 > May 02 20:04:13 [16191] ip-10-116-169-85 lrmd: debug: log_finished: > finished - rsc:dbx_ready_nodes action:monitor call_id:142 pid:8669 > exit-code:1 exec-time:0ms queue-time:0ms > May 02 20:04:13 [16154] ip-10-116-169-85 corosync debug [TOTEM ] > sendmsg(mcast) failed (non-critical): Operation not permitted (1) > May 02 20:04:13 [16154] ip-10-116-169-85 corosync debug [TOTEM ] > sendmsg(mcast) failed (non-critical): Operation not permitted (1) > May 02 20:04:13 [16154] ip-10-116-169-85 corosync debug [TOTEM ] > sendmsg(mcast) failed (non-critical): Operation not permitted (1) > May 02 20:04:13 [16154] ip-10-116-169-85 corosync debug [TOTEM ] > sendmsg(mcast) failed (non-critical): Operation not permitted (1) > May 02 20:04:13 [16154] ip-10-116-169-85 corosync debug [TOTEM ] > sendmsg(mcast) failed (non-critical): Operation not permitted (1) > May 02 20:04:13 [16154] ip-10-116-169-85 corosync debug [TOTEM ] > sendmsg(mcast) failed (non-critical): Operation not permitted (1) > May 02 20:04:13 [16154] ip-10-116-169-85 corosync warning [MAIN ] Totem is > unable to form a cluster because of an operating system or network fault. The > most common cause of this message is that th > e local firewall is configured improperly. > > > Thanks in advance > > -- > Best Regards, > > Radoslaw Garbacz > XtremeData Incorporation > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org