13.11.2013, 03:22, "Andrew Beekhof" <and...@beekhof.net>: > On 12 Nov 2013, at 4:42 pm, Andrey Groshev <gre...@yandex.ru> wrote: > >> 11.11.2013, 03:44, "Andrew Beekhof" <and...@beekhof.net>: >>> On 8 Nov 2013, at 7:49 am, Andrey Groshev <gre...@yandex.ru> wrote: >>>> Hi, PPL! >>>> I need help. I do not understand... Why has stopped working. >>>> This configuration work on other cluster, but on corosync1. >>>> >>>> So... cluster postgres with master/slave. >>>> Classic config as in wiki. >>>> I build cluster, start, he is working. >>>> Next I kill postgres on Master with 6 signal, as if "disk space left" >>>> >>>> # pkill -6 postgres >>>> # ps axuww|grep postgres >>>> root 9032 0.0 0.1 103236 860 pts/0 S+ 00:37 0:00 grep >>>> postgres >>>> >>>> PostgreSQL die, But crm_mon shows that the master is still running. >>>> >>>> Last updated: Fri Nov 8 00:42:08 2013 >>>> Last change: Fri Nov 8 00:37:05 2013 via crm_attribute on >>>> dev-cluster2-node4 >>>> Stack: corosync >>>> Current DC: dev-cluster2-node4 (172793107) - partition with quorum >>>> Version: 1.1.10-1.el6-368c726 >>>> 3 Nodes configured >>>> 7 Resources configured >>>> >>>> Node dev-cluster2-node2 (172793105): online >>>> pingCheck (ocf::pacemaker:ping): Started >>>> pgsql (ocf::heartbeat:pgsql): Started >>>> Node dev-cluster2-node3 (172793106): online >>>> pingCheck (ocf::pacemaker:ping): Started >>>> pgsql (ocf::heartbeat:pgsql): Started >>>> Node dev-cluster2-node4 (172793107): online >>>> pgsql (ocf::heartbeat:pgsql): Master >>>> pingCheck (ocf::pacemaker:ping): Started >>>> VirtualIP (ocf::heartbeat:IPaddr2): Started >>>> >>>> Node Attributes: >>>> * Node dev-cluster2-node2: >>>> + default_ping_set : 100 >>>> + master-pgsql : -INFINITY >>>> + pgsql-data-status : STREAMING|ASYNC >>>> + pgsql-status : HS:async >>>> * Node dev-cluster2-node3: >>>> + default_ping_set : 100 >>>> + master-pgsql : -INFINITY >>>> + pgsql-data-status : STREAMING|ASYNC >>>> + pgsql-status : HS:async >>>> * Node dev-cluster2-node4: >>>> + default_ping_set : 100 >>>> + master-pgsql : 1000 >>>> + pgsql-data-status : LATEST >>>> + pgsql-master-baseline : 0000000002000078 >>>> + pgsql-status : PRI >>>> >>>> Migration summary: >>>> * Node dev-cluster2-node4: >>>> * Node dev-cluster2-node2: >>>> * Node dev-cluster2-node3: >>>> >>>> Tickets: >>>> >>>> CONFIG: >>>> node $id="172793105" dev-cluster2-node2. \ >>>> attributes pgsql-data-status="STREAMING|ASYNC" standby="false" >>>> node $id="172793106" dev-cluster2-node3. \ >>>> attributes pgsql-data-status="STREAMING|ASYNC" standby="false" >>>> node $id="172793107" dev-cluster2-node4. \ >>>> attributes pgsql-data-status="LATEST" >>>> primitive VirtualIP ocf:heartbeat:IPaddr2 \ >>>> params ip="10.76.157.194" \ >>>> op start interval="0" timeout="60s" on-fail="stop" \ >>>> op monitor interval="10s" timeout="60s" on-fail="restart" \ >>>> op stop interval="0" timeout="60s" on-fail="block" >>>> primitive pgsql ocf:heartbeat:pgsql \ >>>> params pgctl="/usr/pgsql-9.1/bin/pg_ctl" >>>> psql="/usr/pgsql-9.1/bin/psql" pgdata="/var/lib/pgsql/9.1/data" >>>> tmpdir="/tmp/pg" start_opt="-p 5432" >>>> logfile="/var/lib/pgsql/9.1//pgstartup.log" rep_mode="async" node_list=" >>>> dev-cluster2-node2. dev-cluster2-node3. dev-cluster2-node4. " >>>> restore_command="gzip -cd >>>> /var/backup/pitr/dev-cluster2-master#5432/xlog/%f.gz > %p" >>>> primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 >>>> keepalives_count=5" master_ip="10.76.157.194" \ >>>> op start interval="0" timeout="60s" on-fail="restart" \ >>>> op monitor interval="5s" timeout="61s" on-fail="restart" \ >>>> op monitor interval="1s" role="Master" timeout="62s" >>>> on-fail="restart" \ >>>> op promote interval="0" timeout="63s" on-fail="restart" \ >>>> op demote interval="0" timeout="64s" on-fail="stop" \ >>>> op stop interval="0" timeout="65s" on-fail="block" \ >>>> op notify interval="0" timeout="66s" >>>> primitive pingCheck ocf:pacemaker:ping \ >>>> params name="default_ping_set" host_list="10.76.156.1" >>>> multiplier="100" \ >>>> op start interval="0" timeout="60s" on-fail="restart" \ >>>> op monitor interval="10s" timeout="60s" on-fail="restart" \ >>>> op stop interval="0" timeout="60s" on-fail="ignore" >>>> ms msPostgresql pgsql \ >>>> meta master-max="1" master-node-max="1" clone-node-max="1" >>>> notify="true" target-role="Master" clone-max="3" >>>> clone clnPingCheck pingCheck \ >>>> meta clone-max="3" >>>> location l0_DontRunPgIfNotPingGW msPostgresql \ >>>> rule $id="l0_DontRunPgIfNotPingGW-rule" -inf: not_defined >>>> default_ping_set or default_ping_set lt 100 >>>> colocation r0_StartPgIfPingGW inf: msPostgresql clnPingCheck >>>> colocation r1_MastersGroup inf: VirtualIP msPostgresql:Master >>>> order rsc_order-1 0: clnPingCheck msPostgresql >>>> order rsc_order-2 0: msPostgresql:promote VirtualIP:start >>>> symmetrical=false >>>> order rsc_order-3 0: msPostgresql:demote VirtualIP:stop symmetrical=false >>>> property $id="cib-bootstrap-options" \ >>>> dc-version="1.1.10-1.el6-368c726" \ >>>> cluster-infrastructure="corosync" \ >>>> stonith-enabled="false" \ >>>> no-quorum-policy="stop" >>>> rsc_defaults $id="rsc-options" \ >>>> resource-stickiness="INFINITY" \ >>>> migration-threshold="1" >>>> >>>> Tell me where to look - why does pacemaker not work? >>> You might want to follow some of the steps at: >>> >>> http://blog.clusterlabs.org/blog/2013/debugging-pacemaker/ >>> >>> under the heading "Resource-level failures". >> Yes. Thank you. >> I've seen this article and now I study it in more detail. >> A lot of information in the logs, so it is difficult to determine where the >> error is, and where the consequence of error. >> Now I'm trying to figure it out. >> >> BUT... >> While I can say with certainty that the RA with monitor in the MS(pgsql) is >> called ONLY on the node on which the last was launched PACEMAKER. > > It looks like you're hitting > https://github.com/beekhof/pacemaker/commit/58962338 > Since you appear to be on rhel6 (or a clone of rhel6), can I suggest you use > the 1.1.10 packages that come with 6.4? > They include the above patch.
I already use (builded from source two weeks ago) * pacemaker 1.1.10 * resource-agents 3.9.5 * corosync 2.3.2 * libqb 0.16 & CentOS 6.4 The same config work on pacemaker 1.1.9/corosync 1.4.5 Not ideal, but no such problem. At first idea - I thought I should move the target-role=Master from MS to primitive pgsql. And so even working. But after a crash killing the main PostgreSQL process - started the same. Today's experiments showed that this behavior starts after I add in MS "notify=true". But primitive pgsql not properly work without "notify" messages. While I in frustration :( > Also, just to be sure. Are you expecting monitor operations to detect when > you started a resource manually? > If so, you'll need a monitor operation with role=Stopped. We don't do that by > default. I expect that the resource monitoring on all the time, otherwise how to control them? Or I do not quite understand the question. >>> 'crm_mon -o' might be a good source of information too. >> Therefore, I see that my resources allegedly functioning normally. >> >> # crm_mon -o1 >> Last updated: Tue Nov 12 09:27:16 2013 >> Last change: Tue Nov 12 00:08:35 2013 via crm_attribute on >> dev-cluster2-node2 >> Stack: corosync >> Current DC: dev-cluster2-node2 (172793105) - partition with quorum >> Version: 1.1.10-1.el6-368c726 >> 3 Nodes configured >> 337 Resources configured >> >> Online: [ dev-cluster2-node2 dev-cluster2-node3 dev-cluster2-node4 ] >> >> Clone Set: clonePing [pingCheck] >> Started: [ dev-cluster2-node2 dev-cluster2-node3 dev-cluster2-node4 ] >> Master/Slave Set: msPgsql [pgsql] >> Masters: [ dev-cluster2-node2 ] >> Slaves: [ dev-cluster2-node3 dev-cluster2-node4 ] >> VirtualIP (ocf::heartbeat:IPaddr2): Started dev-cluster2-node2 >> >> Operations: >> * Node dev-cluster2-node2: >> pingCheck: migration-threshold=1 >> + (20) start: rc=0 (ok) >> + (23) monitor: interval=10000ms rc=0 (ok) >> pgsql: migration-threshold=1 >> + (41) promote: rc=0 (ok) >> + (87) monitor: interval=1000ms rc=8 (master) >> VirtualIP: migration-threshold=1 >> + (49) start: rc=0 (ok) >> + (52) monitor: interval=10000ms rc=0 (ok) >> * Node dev-cluster2-node3: >> pingCheck: migration-threshold=1 >> + (20) start: rc=0 (ok) >> + (23) monitor: interval=10000ms rc=0 (ok) >> pgsql: migration-threshold=1 >> + (26) start: rc=0 (ok) >> + (32) monitor: interval=10000ms rc=0 (ok) >> * Node dev-cluster2-node4: >> pingCheck: migration-threshold=1 >> + (20) start: rc=0 (ok) >> + (23) monitor: interval=10000ms rc=0 (ok) >> pgsql: migration-threshold=1 >> + (26) start: rc=0 (ok) >> + (32) monitor: interval=10000ms rc=0 (ok) >> >> In reality now killed (signal 4|6) the PG master and the penultimate slave >> PG. >> IMHO, even if I have something configured incorrectly, the inability to >> monitor the resource must cause a fatal error. >> Or is there a reason not to do so? >> >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org > > , > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org