Hi all,
I am new to corosync/pacemaker and I have a 2 node "production" cluster
with (corosync+pacemaker+drbd)
Node1 = lws1h1.mydomain.com
Node2 = lws1h2.mydomain.com
they are in online/online failover setup ..... services are only running
where DRBD resides... the other node stays online to take over if Node1
fails
this is the SW versions:
corosync-2.3.0-1.el6.x86_64
drbd84-utils-8.4.2-1.el6.elrepo.x86_64
pacemaker-1.1.8-1.el6.x86_64
OS: CentOS 6.4 x64bit
the cluster is configured with Quorum (not sure what that is)
few days ago I placed one of the nodes in maintenance mode "after" services
where going bad due to a problem .... I dont remember the details of how I
moved/migrated the resources but I usually use LCMC GUI tool .... also I
did some restart for corosync / pacemaker in a random ways :$
after that.. Node1 became slave and Node2 became master!
services are now sticking on Node2 and I cant migrate them even by force to
Node1 (tried command line tools and LCMC tool)
more details/outputs:
*####################### Start ###############################*
[aalishe@lws1h1 ~]$ sudo crm_mon -Afro
Last updated: Sun Apr 6 15:25:52 2014
Last change: Sun Apr 6 14:16:15 2014 via crm_resource on
lws1h2.mydomain.com
Stack: corosync
Current DC: lws1h2.mydomain.com (2) - partition with quorum
Version: 1.1.8-1.el6-394e906
2 Nodes configured, unknown expected votes
10 Resources configured.
Online: [ lws1h1.mydomain.com lws1h2.mydomain.com ]
Full list of resources:
Resource Group: SuperMetaService
SuperFloatIP (ocf::heartbeat:IPaddr2): Started
lws1h2.mydomain.com
SuperFs1 (ocf::heartbeat:Filesystem): Started lws1h2.mydomain.com
SuperFs2 (ocf::heartbeat:Filesystem): Started lws1h2.mydomain.com
SuperFs3 (ocf::heartbeat:Filesystem): Started lws1h2.mydomain.com
SuperFs4 (ocf::heartbeat:Filesystem): Started lws1h2.mydomain.com
Master/Slave Set: SuperDataClone [SuperData]
Masters: [ lws1h2.mydomain.com ]
Slaves: [ lws1h1.mydomain.com ]
SuperMetaSQL (ocf::mydomain:pgsql): Started lws1h2.mydomain.com
SuperGTS (ocf::mydomain:mmon): Started lws1h2.mydomain.com
SuperCQP (ocf::mydomain:mmon): Started lws1h2.mydomain.com
Node Attributes:
* Node lws1h1.mydomain.com:
* Node lws1h2.mydomain.com:
+ master-SuperData : 10000
Operations:
* Node lws1h2.mydomain.com:
SuperFs1: migration-threshold=1000000
+ (1241) start: rc=0 (ok)
SuperMetaSQL: migration-threshold=1000000
+ (1254) start: rc=0 (ok)
+ (1257) monitor: interval=30000ms rc=0 (ok)
SuperFloatIP: migration-threshold=1000000
+ (1236) start: rc=0 (ok) + (1239) monitor: interval=30000ms rc=0 (ok)
SuperData:0: migration-threshold=1000000 + (957) probe: rc=0 (ok) +
(1230) promot
*########################### End ###########################*
CRM Configuration
*####################### Start ###############################*
[aalishe@lws1h1 ~]$ sudo crm configure show
node $id="1" lws1h1.mydomain.com \
attributes standby="off"
node $id="2" lws1h2.mydomain.com \
attributes standby="off"
primitive SuperCQP ocf:mydomain:mmon \
params mmond="/opt/mydomain/platform/bin/"
cfgfile="/opt/mydomain/platform/etc/mmon_mydomain_cqp.xml"
pidfile="/opt/mydomain/platform/var/run/mmon_mydomain_cqp.pid"
user="mydomainsvc" db="bigdata" dbport="5434" \
operations $id="SuperCQP-operations" \
op start interval="0" timeout="120" \
op stop interval="0" timeout="120" \
op monitor interval="120" timeout="120" start-delay="0" \
meta target-role="started" is-managed="true"
primitive SuperData ocf:linbit:drbd \
params drbd_resource="r0" \
op monitor interval="60s" \
meta target-role="started"
primitive SuperFloatIP ocf:heartbeat:IPaddr2 \
params ip="10.100.0.225" cidr_netmask="24" \
op monitor interval="30s" \
meta target-role="started"
primitive SuperFs1 ocf:heartbeat:Filesystem \
params device="/dev/drbd1" directory="/mnt/drbd1" fstype="ext4" \
meta target-role="started"
primitive SuperFs2 ocf:heartbeat:Filesystem \
params device="/dev/drbd2" directory="/mnt/drbd2" fstype="ext4" \
meta target-role="started"
primitive SuperFs3 ocf:heartbeat:Filesystem \
params device="/dev/drbd3" directory="/mnt/drbd3" fstype="ext4"
primitive SuperFs4 ocf:heartbeat:Filesystem \
params device="/dev/drbd4" directory="/mnt/drbd4" fstype="ext4" \
meta target-role="started"
primitive SuperGTS ocf:mydomain:mmon \
params mmond="/opt/mydomain/platform/bin/"
cfgfile="/opt/mydomain/platform/etc/mmon_mydomain_gts.xml"
pidfile="/opt/mydomain/platform/var/run/mmon_mydomain_gts.pid" user=
"mydomainsvc" db="bigdata" \
operations $id="SuperGTS-operations" \
op start interval="0" timeout="120" \
op stop interval="0" timeout="120" \
op monitor interval="120" timeout="120" start-delay="0" \
meta target-role="started" is-managed="true"
primitive SuperMetaSQL ocf:mydomain:pgsql \
op monitor interval="30" timeout="30" depth="0" \
op start interval="0" timeout="120" \
op stop interval="0" timeout="120" \
params pgdata="/mnt/drbd1/pgsql/data" pgdb="bigdata" \
meta target-role="Started" is-managed="true"
group SuperMetaService SuperFloatIP SuperFs1 SuperFs2 SuperFs3 SuperFs4 \
meta target-role="Started"
ms SuperDataClone SuperData \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true" target-role="Started"
colocation CqpOnGts inf: SuperCQP SuperGTS
colocation GtsOnMeta inf: SuperGTS SuperMetaSQL
colocation MetaSQLonData inf: SuperMetaSQL SuperDataClone:Master
colocation ServiceOnDrbd inf: SuperMetaService SuperDataClone:Master
order CqpAfterGts inf: SuperGTS:start SuperCQP
order GtsAfterMeta inf: SuperMetaSQL:start SuperGTS
order MetaAfterService inf: SuperMetaService:start SuperMetaSQL
order ServiceAfterDrbd inf: SuperDataClone:promote SuperMetaService:start
property $id="cib-bootstrap-options" \
dc-version="1.1.8-1.el6-394e906" \
cluster-infrastructure="corosync" \
stonith-enabled="false" \
no-quorum-policy="ignore" \
last-lrm-refresh="1396817730" \
maintenance-mode="false"
*######################## End ###########################*
messages I saw the first time I migrated / or maybe unimgrated services
*############################Start ######################*
/usr/sbin/crm_resource -r SuperMetaService --migrate
WARNING: Creating rsc_location constraint 'cli-standby-SuperMetaService'
with a score of -INFINITY for resource SuperMetaService on
lws1h2.mydomain.com.
This will prevent SuperMetaService from running on lws1h2.mydomain.com
until the constraint is removed using the 'crm_resource -U' command or
manually with cibadmin
This will be the case even if lws1h2.mydomain.com is the last node in
the
cluster
This message can be disabled with -Q
[aali...@lws1h2.mydomain.com:~#] /usr/sbin/cibadmin --obj_type constraints
-C -X '<rsc_location id="cli-standby-SuperDataClone"
rsc="SuperDataClone"><rule id="cli-standby-SuperDataClone-rule"
score="-INFINITY" role="Master"><expression attribute="#uname"
id="cli-standby-SuperDataClone-expression" operation="eq"
value="lws1h2.mydomain.com"/></rule></rsc_location>'
[aali...@lws1h2.mydomain.com:~#] /usr/sbin/crm_resource -r SuperMetaSQL
--migrate
Resource SuperMetaSQL not moved: not-active and no preferred location
specified.
Error performing operation: Invalid argument
*###############################End#############################*
Errors from System Logs
*############################# Start ###############################*
[aalishe@lws1h1 ~]$ sudo tail -n 300 /var/log/messages | grep -E
"error|warning"
Apr 6 14:16:14 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:14 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:15 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:15 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:15 lws1h1 kernel: crm_simulate[29878]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fffd9cff2b0 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:15 lws1h1 kernel: crm_simulate[29886]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fff7f89d140 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:15 lws1h1 kernel: crm_simulate[29893]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fffef934290 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:15 lws1h1 kernel: crm_simulate[29900]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fff7d4e0d90 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:51 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:51 lws1h1 kernel: crm_simulate[31902]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fff36787850 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:51 lws1h1 kernel: crm_simulate[31909]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fff45ed4830 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:52 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:52 lws1h1 kernel: crm_simulate[31917]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fffe68127a0 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:52 lws1h1 kernel: crm_simulate[31924]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fff80548b20 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:55 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 14:16:56 lws1h1 kernel: crm_simulate[31958]: segfault at 1d4c0 ip
0000003a2284812c sp 00007ffff103d430 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:56 lws1h1 kernel: crm_simulate[31965]: segfault at 1d4c0 ip
0000003a2284812c sp 00007fffcb795e30 error 4 in
libc-2.12.so[3a22800000+18b000]
Apr 6 14:16:57 lws1h1 cibmon[28024]: error: crm_element_value: Couldn't
find src in NULL
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: crm_xml_err: XML
Error: I/O warning : failed to load external entity
"/tmp/lcmc-test-387474fb-b2c3-4c7c-96ed-f207635579e2.xml"
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: filename2xml: Parsing
failed (domain=8, level=1, code=1549): failed to load external entity
"/tmp/lcmc-test-387474fb-b2c3-4c7c-96ed-f207635579e2.xml"
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: filename2xml: Couldn't
parse /tmp/lcmc-test-387474fb-b2c3-4c7c-96ed-f207635579e2.xml
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: crm_abort:
xpath_search: Triggered assert at xml.c:2742 : xml_top != NULL
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: crm_element_value:
Couldn't find validate-with in NULL
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: crm_abort:
update_validation: Triggered assert at xml.c:2586 : *xml_blob != NULL
Apr 6 16:12:37 lws1h1 crm_simulate[29796]: error: crm_element_value:
Couldn't find validate-with in NULL
Apr 6 16:21:24 lws1h1 attrd[32458]: warning: attrd_cib_callback: Update
fail-count-SuperCQP=(null) failed: Transport endpoint is not connected
Apr 6 16:21:24 lws1h1 attrd[32458]: warning: attrd_cib_callback: Update
last-failure-SuperCQP=(null) failed: Transport endpoint is not connected
Apr 6 16:26:11 lws1h1 attrd[32458]: warning: attrd_cib_callback: Update
fail-count-SuperCQP=(null) failed: Transport endpoint is not connected
Apr 6 16:26:11 lws1h1 crmd[15595]: warning: decode_transition_key: Bad
UUID (crm-resource-2879) in sscanf result (3) for 0:0:crm-resource-2879
Apr 6 16:26:11 lws1h1 crmd[15595]: error: send_msg_via_ipc: Unknown
Sub-system (2879_crm_resource)... discarding message.
Apr 6 16:38:59 lws1h1 attrd[32458]: warning: attrd_cib_callback: Update
fail-count-SuperGTS=(null) failed: Transport endpoint is not connected
Apr 6 16:38:59 lws1h1 attrd[32458]: warning: attrd_cib_callback: Update
last-failure-SuperGTS=(null) failed: Transport endpoint is not connected
*#####################################End###########################*
Thanks inadvance for your help
** Please note that I will reward whomever "First" helps me "solve" this
issue **
--
View this message in context:
http://linux-ha.996297.n3.nabble.com/Master-Became-Slave-Cluster-unstable-tp15583.html
Sent from the Linux-HA mailing list archive at Nabble.com.
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
