date:20180530

Re: [ClusterLabs] PAF not starting resource successfully after node reboot (was: How to set up fencing/stonith)

2018-05-30 Thread Andrei Borzenkov

31.05.2018 01:30, Casey & Gina пишет:
>> In this case, the agent is returning "master (failed)", which does not
>> mean that it previously failed when it was master -- it means it is
>> currently running as master, in a failed condition.
> 
> Well, it surely is NOT running.  So the likely problem is the way it's doing 
> this check?  I see a lot of people here using PAF - I'd be surprised if such 
> a bug weren't discovered already...
> 
>> Stopping an already stopped service does not return an error -- here,
>> the agent is saying it was unable to demote or stop a running instance.
> 
> I still don't understand.  There are *NO* postgres processes running on the 
> node, no additions to the log file.  Nothing whatsoever that supports the 
> notion that it's a running instance.
> 
>> Unfortunately clustering has some inherent complexity that gives it a
>> steep learning curve. On top of that, logging/troubleshooting
>> improvements are definitely an area of ongoing need in pacemaker. The
>> good news is that once a cluster is running successfully, it's usually
>> smooth sailing after that.
> 
> I hope so...  I just don't see what I'm doing that's outside of the standard 
> box.  I've set up PAF following it's instructions.  I see that others here 
> are using it.  Hasn't anybody else gotten such a setup working already?  I 
> would think this is a pretty standard failure case that anybody would test if 
> they've set up a cluster...  In any case, I'll keep persisting as long as I 
> can...on to debugging...
> 
>> You can debug like this:
>>
>> 1. Unmanage the resource in pacemaker, so you can mess with it
>> manually.
>>
>> 2. Cause the desired failure for testing. Pacemaker should detect the
>> failure, but not do anything about it.
> 
> I executed `pcs resource unmanage postgresql-ha`, and then powered off the 
> master node.

There is no "master node" in pacemaker. There is master/slave resource
so at the best it is "node on which specific resource has master role".
And we have no way to know which on which node you resource had master
role when you did it. Please be more specific, otherwise it is hard to
impossible to follow.

>  The fencing kicked in and restarted the node.  After the node rebooted, I 
> issued a `pcs cluster start` on it as the crm_resource command complained 
> about the CIB without doing that.
> 
> I then ended up seeing this:
> 
> --
>  vfencing   (stonith:external/vcenter): Started d-gp2-dbpg0-1
>  postgresql-master-vip  (ocf::heartbeat:IPaddr2):   Started d-gp2-dbpg0-2
>  Master/Slave Set: postgresql-ha [postgresql-10-main] (unmanaged)
>  postgresql-10-main (ocf::heartbeat:pgsqlms):   Started d-gp2-dbpg0-3 
> (unmanaged)
>  postgresql-10-main (ocf::heartbeat:pgsqlms):   Slave d-gp2-dbpg0-1 
> (unmanaged)

Not specifically related to your problem but I wonder what is the
difference. For all I know for master/slave "Started" == "Slave" so I'm
surprised to see two different states listed here.


>  postgresql-10-main (ocf::heartbeat:pgsqlms):   FAILED Master 
> d-gp2-dbpg0-2 (unmanaged)
> 
> Failed Actions:
> * postgresql-10-main_monitor_0 on d-gp2-dbpg0-2 'master (failed)' (9): 
> call=14, status=complete, exitreason='Instance "postgresql-10-main" 
> controldata indicates a running primary instance, the instance has probably 
> crashed',

Well, apparently resource agent does not like crashed instance. It is
quite possible, I have been working with another replicated database
where it was necessary to manually fix configuration after failover,
*outside* of pacemaker. Pacemaker simply failed to start resource which
had unexpected state.

This needs someone familiar with this RA and application to answer.

Note that it is not quite normal use case. You explicitly disabled any
handling by RA, thus effectively not using pacemaker high availability
at all. Does it fail over master if you do not unmanage resource and
kill node where resource has master role?

> last-rc-change='Wed May 30 22:18:16 2018', queued=0ms, exec=190ms
> * postgresql-10-main_monitor_15000 on d-gp2-dbpg0-2 'master (failed)' (9): 
> call=16, status=complete, exitreason='Instance "postgresql-10-main" 
> controldata indicates a running primary instance, the instance has probably 
> crashed',
> last-rc-change='Wed May 30 22:18:16 2018', queued=0ms, exec=138ms
> --
> 
>> 3. Run crm_resource with the -VV option and --force-* with whatever
>> action you want to attempt (in this case, demote or stop). The -VV (aka
>> --verbose --verbose) will turn on OCF_TRACE_RA. The --force-* command
>> will read the resource configuration and do the same thing pacemaker
>> would do to execute the command.
> 
> I thought that I would want to see what the "check" is doing to do the check, 
> since you're telling me that it thinks the service is running when it's 
> definitely not.  I tried the following command which didn't work (am I doing 
> something wrong?):
> 
> --
>

Re: [ClusterLabs] PAF not starting resource successfully after node reboot (was: How to set up fencing/stonith)

2018-05-30 Thread Casey & Gina

> In this case, the agent is returning "master (failed)", which does not
> mean that it previously failed when it was master -- it means it is
> currently running as master, in a failed condition.

Well, it surely is NOT running.  So the likely problem is the way it's doing 
this check?  I see a lot of people here using PAF - I'd be surprised if such a 
bug weren't discovered already...

> Stopping an already stopped service does not return an error -- here,
> the agent is saying it was unable to demote or stop a running instance.

I still don't understand.  There are *NO* postgres processes running on the 
node, no additions to the log file.  Nothing whatsoever that supports the 
notion that it's a running instance.

> Unfortunately clustering has some inherent complexity that gives it a
> steep learning curve. On top of that, logging/troubleshooting
> improvements are definitely an area of ongoing need in pacemaker. The
> good news is that once a cluster is running successfully, it's usually
> smooth sailing after that.

I hope so...  I just don't see what I'm doing that's outside of the standard 
box.  I've set up PAF following it's instructions.  I see that others here are 
using it.  Hasn't anybody else gotten such a setup working already?  I would 
think this is a pretty standard failure case that anybody would test if they've 
set up a cluster...  In any case, I'll keep persisting as long as I can...on to 
debugging...

> You can debug like this:
> 
> 1. Unmanage the resource in pacemaker, so you can mess with it
> manually.
> 
> 2. Cause the desired failure for testing. Pacemaker should detect the
> failure, but not do anything about it.

I executed `pcs resource unmanage postgresql-ha`, and then powered off the 
master node.  The fencing kicked in and restarted the node.  After the node 
rebooted, I issued a `pcs cluster start` on it as the crm_resource command 
complained about the CIB without doing that.

I then ended up seeing this:

--
 vfencing   (stonith:external/vcenter): Started d-gp2-dbpg0-1
 postgresql-master-vip  (ocf::heartbeat:IPaddr2):   Started d-gp2-dbpg0-2
 Master/Slave Set: postgresql-ha [postgresql-10-main] (unmanaged)
 postgresql-10-main (ocf::heartbeat:pgsqlms):   Started d-gp2-dbpg0-3 
(unmanaged)
 postgresql-10-main (ocf::heartbeat:pgsqlms):   Slave d-gp2-dbpg0-1 
(unmanaged)
 postgresql-10-main (ocf::heartbeat:pgsqlms):   FAILED Master 
d-gp2-dbpg0-2 (unmanaged)

Failed Actions:
* postgresql-10-main_monitor_0 on d-gp2-dbpg0-2 'master (failed)' (9): call=14, 
status=complete, exitreason='Instance "postgresql-10-main" controldata 
indicates a running primary instance, the instance has probably crashed',
last-rc-change='Wed May 30 22:18:16 2018', queued=0ms, exec=190ms
* postgresql-10-main_monitor_15000 on d-gp2-dbpg0-2 'master (failed)' (9): 
call=16, status=complete, exitreason='Instance "postgresql-10-main" controldata 
indicates a running primary instance, the instance has probably crashed',
last-rc-change='Wed May 30 22:18:16 2018', queued=0ms, exec=138ms
--

> 3. Run crm_resource with the -VV option and --force-* with whatever
> action you want to attempt (in this case, demote or stop). The -VV (aka
> --verbose --verbose) will turn on OCF_TRACE_RA. The --force-* command
> will read the resource configuration and do the same thing pacemaker
> would do to execute the command.

I thought that I would want to see what the "check" is doing to do the check, 
since you're telling me that it thinks the service is running when it's 
definitely not.  I tried the following command which didn't work (am I doing 
something wrong?):

--
root@d-gp2-dbpg0-2:/var# crm_resource -r postgresql-ha -VV --force-check 
 warning: unpack_rsc_op_failure:Processing failed op monitor for 
postgresql-10-main:2 on d-gp2-dbpg0-2: master (failed) (9)
 warning: unpack_rsc_op_failure:Processing failed op monitor for 
postgresql-10-main:2 on d-gp2-dbpg0-2: master (failed) (9)
Operation monitor for postgresql-10-main:0 (ocf:heartbeat:pgsqlms) returned 5
 >  stderr: Use of uninitialized value $OCF_Functions::ARG[0] in pattern match 
 > (m//) at 
 > /usr/lib/ocf/resource.d/heartbeat/../../lib/heartbeat/OCF_Functions.pm line 
 > 392.
 >  stderr: ocf-exit-reason:PAF v2.2.0 is compatible with Pacemaker 1.1.13 and 
 > greater
Error performing operation: Input/output error
--

Attempting to force-demote didn't work either:

--
root@d-gp2-dbpg0-2:/var# crm_resource -r postgresql-ha -VV --force-demote
 warning: unpack_rsc_op_failure:Processing failed op monitor for 
postgresql-10-main:2 on d-gp2-dbpg0-2: master (failed) (9)
 warning: unpack_rsc_op_failure:Processing failed op monitor for 
postgresql-10-main:2 on d-gp2-dbpg0-2: master (failed) (9)
resource postgresql-ha is running on: d-gp2-dbpg0-3 
resource postgresql-ha is running on: d-gp2-dbpg0-1 
resource postgresql-ha is running on: d-gp2-dbpg0-2 Master
It is not

Re: [ClusterLabs] Pacemaker PostgreSQL cluster

2018-05-30 Thread Ken Gaillot

On Wed, 2018-05-30 at 09:31 +0200, Salvatore D'angelo wrote:
> Hi,
> 
> Last question. In order to migrate pacemaker with minimum downtime
> the option I see are Rolling (node by node) and Disconnect Reattach
> http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemak
> er_Explained/ap-upgrade.html
> 
> What I want to do is first migrate pacemaker manually and then
> automate it with some scripts.
> 
> According to what Ken Gaillot said:
> 
> "Rolling upgrades are always supported within the same major number
> line
> (i.e. 1.anything to 1.anything). With the major number change,
> rolling
> upgrades will not always be supported. In the case of 2.0.0, we are
> supporting rolling upgrades from 1.1.11 or later on top of corosync 2
> or later. You should be fine whichever you choose.”
> 
> if I implement a set of scripts based on Rolling upgrade to migrate
> from 1.1.14 to 1.1.18/2.0.0 the risk is that in the future if there
> will be an upgrade with major number change I should rewrite my
> automation script to support another type of migration (probably
> Detach and Reattach). My question is, if I want to avoid this extra
> work in the future, is the Detach Reattach procedure more adaptable
> to whatever version migration? 

Each approach has its advantages and disadvantages. Detach+reattach is
indeed more adaptable to version changes and even underlying cluster
layer changes. Its downsides are that you can't do it with Pacemaker
Remote nodes, and there is no HA during the upgrade.

BTW while it's hard to predict, I suspect there won't be another major
bump for another decade.

> My understanding is that with this procedure PostgreSQL will always
> be up and running and I only need detach pacemaker from them on the
> three nodes, migrate them and then Reattach. During this period What
> happen if PostgreSQL master goes down?

As you probably suspected, there will be no automatic recovery.

> Thanks again for support.
> 
> > On 30 May 2018, at 04:04, Ken Gaillot  wrote:
> > 
> > On Tue, 2018-05-29 at 22:25 +0200, Salvatore D'angelo wrote:
> > > Hi,
> > > 
> > > Regarding last question about pacemaker dependencies for Ubuntu I
> > > found this for 1.1.18:
> > > https://launchpad.net/ubuntu/+source/pacemaker/1.1.18-0ubuntu2/+b
> > > uild
> > > /14818856
> > > 
> > > It’s not clear to me why pacemaker 1.1.18 is available on
> > > launchpad.net and not on the official Ubuntu Search Packages
> > > website.
> > > However, can I assume 1.1.19 and 2.2.0 have the same dependencies
> > > list (considering they have only removed deprecated function and
> > > applied some bug fixes)?
> > 
> > Yes, the dependencies should be the same (when corosync 2 is used)
> > 
> > > Thanks again for answers
> > > 
> > > 
> > > > On 29 May 2018, at 17:41, Jehan-Guillaume de Rorthais  > > > ibo.
> > > > com> wrote:
> > > > 
> > > > On Tue, 29 May 2018 14:23:31 +0200
> > > > Salvatore D'angelo  wrote:
> > > > ...
> > > > > 2. I read some documentation about upgrade and since we want
> > > > > 0 ms
> > > > > downtime I
> > > > > think the Rolling Upgrade (node by node) is the better
> > > > > approach.
> > > > 
> > > > The 0ms upgrade is almost impossible. At some point, you will
> > > > have
> > > > to move the
> > > > master somewhere else.
> > > > 
> > > > Unless you have some session management that are able to wait
> > > > for
> > > > the
> > > > current sessions to finish, then hold the incoming sessions
> > > > while
> > > > you are
> > > > moving the master, you will have downtime and/or xact rollback.
> > > > 
> > > > Good luck anyway :)
> > > > 
> > > > -- 
> > > > Jehan-Guillaume de Rorthais
> > > > Dalibo
> > > 
> > > ___
> > > Users mailing list: Users@clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > > tch.
> > > pdf
> > > Bugs: http://bugs.clusterlabs.org
> > -- 
> > Ken Gaillot 
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker PostgreSQL cluster

2018-05-30 Thread Salvatore D'angelo

Hi,

Last question. In order to migrate pacemaker with minimum downtime the option I 
see are Rolling (node by node) and Disconnect Reattach
http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ap-upgrade.html

What I want to do is first migrate pacemaker manually and then automate it with 
some scripts.

According to what Ken Gaillot said:

"Rolling upgrades are always supported within the same major number line
(i.e. 1.anything to 1.anything). With the major number change, rolling
upgrades will not always be supported. In the case of 2.0.0, we are
supporting rolling upgrades from 1.1.11 or later on top of corosync 2
or later. You should be fine whichever you choose.”

if I implement a set of scripts based on Rolling upgrade to migrate from 1.1.14 
to 1.1.18/2.0.0 the risk is that in the future if there will be an upgrade with 
major number change I should rewrite my automation script to support another 
type of migration (probably Detach and Reattach). My question is, if I want to 
avoid this extra work in the future, is the Detach Reattach procedure more 
adaptable to whatever version migration? 
My understanding is that with this procedure PostgreSQL will always be up and 
running and I only need detach pacemaker from them on the three nodes, migrate 
them and then Reattach. During this period What happen if PostgreSQL master 
goes down?
Thanks again for support.

> On 30 May 2018, at 04:04, Ken Gaillot  wrote:
> 
> On Tue, 2018-05-29 at 22:25 +0200, Salvatore D'angelo wrote:
>> Hi,
>> 
>> Regarding last question about pacemaker dependencies for Ubuntu I
>> found this for 1.1.18:
>> https://launchpad.net/ubuntu/+source/pacemaker/1.1.18-0ubuntu2/+build 
>> 
>> /14818856
>> 
>> It’s not clear to me why pacemaker 1.1.18 is available on
>> launchpad.net  and not on the official Ubuntu Search 
>> Packages website.
>> However, can I assume 1.1.19 and 2.2.0 have the same dependencies
>> list (considering they have only removed deprecated function and
>> applied some bug fixes)?
> 
> Yes, the dependencies should be the same (when corosync 2 is used)
> 
>> Thanks again for answers
>> 
>> 
>>> On 29 May 2018, at 17:41, Jehan-Guillaume de Rorthais >> com> wrote:
>>> 
>>> On Tue, 29 May 2018 14:23:31 +0200
>>> Salvatore D'angelo  wrote:
>>> ...
 2. I read some documentation about upgrade and since we want 0 ms
 downtime I
 think the Rolling Upgrade (node by node) is the better approach.
>>> 
>>> The 0ms upgrade is almost impossible. At some point, you will have
>>> to move the
>>> master somewhere else.
>>> 
>>> Unless you have some session management that are able to wait for
>>> the
>>> current sessions to finish, then hold the incoming sessions while
>>> you are
>>> moving the master, you will have downtime and/or xact rollback.
>>> 
>>> Good luck anyway :)
>>> 
>>> -- 
>>> Jehan-Guillaume de Rorthais
>>> Dalibo
>> 
>> ___
>> Users mailing list: Users@clusterlabs.org 
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> 
>> Project Home: http://www.clusterlabs.org 
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch 
>> .
>> pdf
>> Bugs: http://bugs.clusterlabs.org 
> -- 
> Ken Gaillot mailto:kgail...@redhat.com>>
> ___
> Users mailing list: Users@clusterlabs.org 
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> 
> Project Home: http://www.clusterlabs.org 
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
> 
> Bugs: http://bugs.clusterlabs.org 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] the kernel information when pacemaker restarts the PAF resource because of monitor timeout

2018-05-30 Thread Klecho


Hi,

That's related to a thing I'm fighting for.

An option to skip X lost monitoring attempts is planned, but not 
implemented yet, as far as I know.


Regards,

Klecho


On 30/05/18 06:08, 范国腾 wrote:

Hi,

The cluster uses the PAF to manage the postgres db, and it use the GFS2 to 
manage the shared storage. The configuration is as attachment.

When we are doing the performance test, the CPU is very high. We set the op 
monitor timeout 100 seconds. PAF call pg_isready to monitor the db. When the 
call load becoming higher, the pg_isready response time increase. When it has 
no response after 100 seconds, the pacemaker restarts the PAF resource. Then 
there is many kernel log and then the PAF resource start fails.

So my question is:
1. When the monitor operation is timeout, there is many kernel log printed in 
/var/log/messages, could you please help check if this log shows the cluster 
has anything wrong? It seems like the share disk storage error prevents the 
database to start.
2.. When the cluster runs as product, it could not avoid the call load become 
high for some time and the monitor will become timeout. Then the PAF resource 
will be restarted. Is there any way to avoid the resource to restart when the 
system is busy?

Thanks
Steven




___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


--
Klecho

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] PAF not starting resource successfully after node reboot (was: How to set up fencing/stonith)

Re: [ClusterLabs] PAF not starting resource successfully after node reboot (was: How to set up fencing/stonith)

Re: [ClusterLabs] Pacemaker PostgreSQL cluster

Re: [ClusterLabs] Pacemaker PostgreSQL cluster

Re: [ClusterLabs] the kernel information when pacemaker restarts the PAF resource because of monitor timeout

5 matches

Site Navigation

Mail list logo

Footer information