I have been using a 2-node cluster with a quorum disk successfully for
about 2 years. Beginning today, the cluster will not boot correctly.
The RHCS services start, but fencing fails with:
dlm: no local IP address has been set
dlm: cannot start dlm lowcomms -107
This seems to be a symtpom of the fact that the cluster votes do not include
votes from the quorum
device:
# clustat
Cluster Status for example-infra @ Tue Mar 15 00:02:35 2011
Member Status: Inquorate
Member Name ID Status
------ ---- ---- ------
example-infr2-admin.domain.com 1 Online,
Local
example-infr1-admin.domain.com 2 Offline
/dev/mpath/quorum 0 Offline
[root@example-infr2 ~]# cman_tool status
Version: 6.2.0
Config Version: 239
Cluster Name: example-infra
Cluster Id: 42813
Cluster Member: Yes
Cluster Generation: 676844
Membership state: Cluster-Member
Nodes: 1
Expected votes: 2
Total votes: 1
Quorum: 2 Activity blocked
Active subsystems: 7
Flags:
Ports Bound: 0
Node name: example-infr2-admin.domain.com
Node ID: 1
Multicast addresses: 239.192.167.228
Node addresses: 192.168.110.3
The shared-SAN-disk quorum device is readable from each node. Testing
with "mkqdisk -L" and "dd if=/dev/mpath/quorum of=/dev/quorum.dump"
both succeed from each node.
When run in the foreground, "qdisk -d -f" gives messages that seem to indicate
that it is successful:
# qdiskd -d -f
[22568] debug: Loading configuration information
[22568] debug: Heuristic: '/bin/ping -c3 -W1 -t2 192.168.110.10' score=1
interval=2 tko=9
[22568] debug: 1 heuristics loaded
[22568] debug: Quorum Daemon: 1 heuristics, 3 interval, 15 tko, 1 votes
[22568] debug: Run Flags: 00000035
[22568] info: Quorum Daemon Initializing
[22568] debug: I/O Size: 512 Page Size: 4096
[22569] info: Heuristic: '/bin/ping -c3 -W1 -t2 192.168.110.10' UP
[22568] debug: Node 3 is UP
[22568] info: Node 3 is the master
[22568] info: Initial score 1/1
[22568] info: Initialization complete
[22568] notice: Score sufficient for master operation (1/1;
required=1); upgrading
Any suggestions?
Thanks,
Mark
------------Versions-----------------
Linux example-infr2.domain.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25
lvm2-cluster-2.02.56-7.el5_5.4
cman-2.0.115-34.el5_5.4
system-config-cluster-1.0.57-3.el5_5.1
rgmanager-2.0.52-6.el5.centos.8
----------excerpt from cluster.conf----------------
<?xml version="1.0"?>
<cluster alias="example-infra" config_version="239" name="example-infra">
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="30"/>
<clusternodes>
<clusternode name="example-infr2-admin.domain.com" nodeid="1"
votes="1">
<fence>
<method name="1">
<device modulename=""
name="example-infr2-drac"/>
</method>
<method name="2"/>
</fence>
</clusternode>
<clusternode name="example-infr1-admin.domain.com" nodeid="2"
votes="1">
<fence>
<method name="1">
<device modulename=""
name="example-infr1-drac"/>
</method>
<method name="2"/>
<method name="3"/>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="2" two_node="0"/>
<quorumd device="/dev/mpath/quorum" interval="3" tko="15" votes="1">
<heuristic interval="2" program="/bin/ping -c3 -W1 -t2
192.168.110.10" score="1" tko="9"/>
</quorumd>
<totem token="54000"/>
</cluster>
--------------------------------------------------------------------------
--
Linux-cluster mailing list
Linux-cluster@redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
