Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-29 Thread Andrew Beekhof 
On Thu, Apr 19, 2012 at 7:06 PM, Vladislav Bogdanov
 wrote:
> 19.04.2012 11:24, Andreas Kurz wrote:
>> On 04/18/2012 11:46 PM, ruslan usifov wrote:
>>>
>>>
>>> 2012/4/18 Andreas Kurz mailto:andr...@hastexo.com>>
>>>
>>>     On 04/17/2012 09:31 PM, ruslan usifov wrote:
>>>     >
>>>     >
>>>     > 2012/4/17 Proskurin Kirill >>     
>>>     > >>
>>>     >
>>>     >     On 04/17/2012 03:46 PM, ruslan usifov wrote:
>>>     >
>>>     >         2012/4/17 Andreas Kurz >>     
>>>     >         >
>>>     
>>>     >         >>     >
>>>     >
>>>     >            On 04/14/2012 11:14 PM, ruslan usifov wrote:
>>>     >             > Hello
>>>     >             >
>>>     >             > I remove 2 nodes from cluster, with follow sequence:
>>>     >             >
>>>     >             > crm_node --force -R 
>>>     >             > crm_node --force -R 
>>>     >             > cibadmin --delete --obj_type nodes --crm_xml '>>     >         uname="node1"/>'
>>>     >             > cibadmin --delete --obj_type status --crm_xml
>>>     '>>     >            uname="node1"/>'
>>>     >             > cibadmin --delete --obj_type nodes --crm_xml '>>     >         uname="node2"/>'
>>>     >             > cibadmin --delete --obj_type status --crm_xml
>>>     '>>     >            uname="node2"/>'
>>>     >             >
>>>     >             >
>>>     >             > Nodes after this deleted, but if for example i restart
>>>     >         (reboot)
>>>     >            one of
>>>     >             > existent nodes in working cluster, this deleted nodes
>>>     >         appear again in
>>>     >             > OFFLINE state
>>>     >
>>>     >
>>>     >     I have this problem some time ago.
>>>     >     I "solved" it something like that:
>>>     >
>>>     >     crm node delete NODENAME
>>>     >     crm_node --force --remove NODENAME
>>>     >     cibadmin --delete --obj_type nodes --crm_xml '>>     uname="NODENAME"/>'
>>>     >     cibadmin --delete --obj_type status --crm_xml '>>     >     uname="NODENAME"/>'
>>>     >
>>>     >     --
>>>     >
>>>     >
>>>     > I do the same, but some times after cluster reconfiguration (node
>>>     failed
>>>     > due power supply failure) removed nodes appear again, and this happens
>>>     > 3-4 times
>>>
>>>     And the same behavior if you switch your cluster into maintenance-mode
>>>     (to avoid service downtime) and stop/start pacemaker and corosync
>>>     completely?
>>>
>>>
>>> We will have maintenance window at this Friday (20.04.2012) so after
>>> that i can report more info.
>>
>> Of course, that is the safest option ... though you won't have a service
>> downtime if you enable maintenance-mode prior to cluster restart.
>
> Unless you are using DLM (CLVM, GFS2, OCFS2). Then you should not stop
> corosync - dlm_controld uses CPG.
>
> And, DLM may use pacemaker parts for fencing (cib, attrd, stonith,
> depending on version).
>
>>
>>>
>>> PS: I had similar situation on other cluster some times ago, and there i
>>> fully restart cluster and problem reproduced. But after some time(about
>>> 1-2 week) not existent nodes have ceased to appear
>>
>> Now that is really strange ... if that happens again, the
>> corosync/pacemaker log files would be really interesting to have a look at.
>
> I recall that is a known issue for a rather long time.
> One need to do a full (not rolling) restart to make node fully disappear.
> I checked this again not so long ago, and yes, node deletion does not
> work with current master branch (or very close to it) - it appears again
> after pacemaker restart on any other node.

Not really enough info do anything about.

>
> May be it is because of lrmd cache, like with failed actions? It looks
> very similar to that.

Nope. The cache is for the local node, if the node is gone so is its cache.

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-19 Thread David Vossel 
- Original Message -
> From: "Vladislav Bogdanov" 
> To: pacemaker@oss.clusterlabs.org
> Sent: Thursday, April 19, 2012 4:06:33 AM
> Subject: Re: [Pacemaker] Periodically appear non-existent nodes
> 
> 19.04.2012 11:24, Andreas Kurz wrote:
> > On 04/18/2012 11:46 PM, ruslan usifov wrote:
> >>
> >>
> >> 2012/4/18 Andreas Kurz  >> <mailto:andr...@hastexo.com>>
> >>
> >> On 04/17/2012 09:31 PM, ruslan usifov wrote:
> >> >
> >> >
> >> > 2012/4/17 Proskurin Kirill  >> <mailto:k.prosku...@corp.mail.ru>
> >> > <mailto:k.prosku...@corp.mail.ru
> >> > <mailto:k.prosku...@corp.mail.ru>>>
> >> >
> >> > On 04/17/2012 03:46 PM, ruslan usifov wrote:
> >> >
> >> > 2012/4/17 Andreas Kurz  >> <mailto:andr...@hastexo.com>
> >> > <mailto:andr...@hastexo.com
> >> > <mailto:andr...@hastexo.com>>
> >> <mailto:andr...@hastexo.com <mailto:andr...@hastexo.com>
> >> > <mailto:andr...@hastexo.com
> >> > <mailto:andr...@hastexo.com>>>>
> >> >
> >> >
> >> >On 04/14/2012 11:14 PM, ruslan usifov wrote:
> >> > > Hello
> >> > >
> >> > > I remove 2 nodes from cluster, with follow
> >> > > sequence:
> >> > >
> >> > > crm_node --force -R 
> >> > > crm_node --force -R 
> >> > > cibadmin --delete --obj_type nodes --crm_xml
> >> > > ' >> > uname="node1"/>'
> >> > > cibadmin --delete --obj_type status --crm_xml
> >> ' >> >uname="node1"/>'
> >> > > cibadmin --delete --obj_type nodes --crm_xml
> >> > > ' >> > uname="node2"/>'
> >> > > cibadmin --delete --obj_type status --crm_xml
> >> ' >> >uname="node2"/>'
> >> > >
> >> > >
> >> > > Nodes after this deleted, but if for example i
> >> > > restart
> >> > (reboot)
> >> >one of
> >> > > existent nodes in working cluster, this
> >> > > deleted nodes
> >> > appear again in
> >> > > OFFLINE state
> >> >
> >> >
> >> > I have this problem some time ago.
> >> > I "solved" it something like that:
> >> >
> >> > crm node delete NODENAME
> >> > crm_node --force --remove NODENAME
> >> > cibadmin --delete --obj_type nodes --crm_xml ' >> uname="NODENAME"/>'
> >> > cibadmin --delete --obj_type status --crm_xml
> >> > ' >> > uname="NODENAME"/>'
> >> >
> >> > --
> >> >
> >> >
> >> > I do the same, but some times after cluster reconfiguration
> >> > (node
> >> failed
> >> > due power supply failure) removed nodes appear again, and
> >> > this happens
> >> > 3-4 times
> >>
> >> And the same behavior if you switch your cluster into
> >> maintenance-mode
> >> (to avoid service downtime) and stop/start pacemaker and
> >> corosync
> >> completely?
> >>
> >>
> >> We will have maintenance window at this Friday (20.04.2012) so
> >> after
> >> that i can report more info.
> > 
> > Of course, that is the safest option ... though you won't have a
> > service
> > downtime if you enable maintenance-mode prior to cluster restart.
> 
> Unless you are using DLM (CLVM, GFS2, OCFS2). Then you should not
> stop
> corosync - dlm_controld uses CPG.
> 
> And, DLM may use pacemaker parts for fencing (cib, attrd, stonith,
> depending on version).
> 
> > 
> >>
> >> PS: I had similar situation on other cluster some times ago, and
> >> there i
> >> fully restart cluster and problem reproduced. But after some
> >> time(about
> >> 1-2 week) not existent nodes have ceased to appear
> > 
> > Now that is really strange ... if that happens again, the
> > corosync/pacemaker log files would be really interesting to have a
> > look at.
> 
> I recall that is a known issue for a rather long time.
> One need to do a full (not rolling) restart to make node fully
> disappear.
> I checked this again not so long ago, and yes, node deletion does not
> work with current master branch (or very close to it) - it appears
> again
> after pacemaker restart on any other node.
> 
> May be it is because of lrmd cache, like with failed actions? It
> looks
> very similar to that.

Looks similar, but it shouldn't be related.

-- Vossel

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-19 Thread David Vossel 
- Original Message -
> From: "ruslan usifov" 
> To: "The Pacemaker cluster resource manager" 
> Sent: Tuesday, April 17, 2012 6:46:00 AM
> Subject: Re: [Pacemaker] Periodically appear non-existent nodes
> 
> 
> 2012/4/17 Andreas Kurz < andr...@hastexo.com >
> 
> 
> 
> 
> On 04/14/2012 11:14 PM, ruslan usifov wrote:
> > Hello
> > 
> > I remove 2 nodes from cluster, with follow sequence:
> > 
> > crm_node --force -R 
> > crm_node --force -R 
> > cibadmin --delete --obj_type nodes --crm_xml ' > uname="node1"/>'
> > cibadmin --delete --obj_type status --crm_xml ' > uname="node1"/>'
> > cibadmin --delete --obj_type nodes --crm_xml ' > uname="node2"/>'
> > cibadmin --delete --obj_type status --crm_xml ' > uname="node2"/>'
> > 
> > 
> > Nodes after this deleted, but if for example i restart (reboot) one
> > of
> > existent nodes in working cluster, this deleted nodes appear again
> > in
> > OFFLINE state
> 
> Just to double check ... corosync was already stopped (on these
> to-be-deleted nodes) prior to the deletion and it's still stopped on
> the
> removed nodes? ... and no cman involved?
> 
> 
> This nodes doesn't present physically:-)) (we remove this from
> network), so no corosync no cman not anything else

I don't know if this is what you are experiencing, but here is one explanation 
that I can easily reproduce.

If you remove the node from the CIB then disconnect the node from the network 
while corosync is running on the node,  a loss of membership will be detected 
by corosync on the remaining nodes.  Pacemaker on the other nodes will get a 
message from corosync saying node membership changed with the id of the node 
that left the cluster.  Pacemaker then says, hey we know about this node that 
isn't online which will re-populate some of the fields in the CIB you just 
deleted.

-- Vossel

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-19 Thread Andreas Kurz 
On 04/19/2012 11:06 AM, Vladislav Bogdanov wrote:
> 19.04.2012 11:24, Andreas Kurz wrote:
>> On 04/18/2012 11:46 PM, ruslan usifov wrote:
>>>
>>>
>>> 2012/4/18 Andreas Kurz mailto:andr...@hastexo.com>>
>>>
>>> On 04/17/2012 09:31 PM, ruslan usifov wrote:
>>> >
>>> >
>>> > 2012/4/17 Proskurin Kirill >> 
>>> > >>
>>> >
>>> > On 04/17/2012 03:46 PM, ruslan usifov wrote:
>>> >
>>> > 2012/4/17 Andreas Kurz >> 
>>> > >
>>> 
>>> > >> >
>>> >
>>> >On 04/14/2012 11:14 PM, ruslan usifov wrote:
>>> > > Hello
>>> > >
>>> > > I remove 2 nodes from cluster, with follow sequence:
>>> > >
>>> > > crm_node --force -R 
>>> > > crm_node --force -R 
>>> > > cibadmin --delete --obj_type nodes --crm_xml '>> > uname="node1"/>'
>>> > > cibadmin --delete --obj_type status --crm_xml
>>> '>> >uname="node1"/>'
>>> > > cibadmin --delete --obj_type nodes --crm_xml '>> > uname="node2"/>'
>>> > > cibadmin --delete --obj_type status --crm_xml
>>> '>> >uname="node2"/>'
>>> > >
>>> > >
>>> > > Nodes after this deleted, but if for example i restart
>>> > (reboot)
>>> >one of
>>> > > existent nodes in working cluster, this deleted nodes
>>> > appear again in
>>> > > OFFLINE state
>>> >
>>> >
>>> > I have this problem some time ago.
>>> > I "solved" it something like that:
>>> >
>>> > crm node delete NODENAME
>>> > crm_node --force --remove NODENAME
>>> > cibadmin --delete --obj_type nodes --crm_xml '>> uname="NODENAME"/>'
>>> > cibadmin --delete --obj_type status --crm_xml '>> > uname="NODENAME"/>'
>>> >
>>> > --
>>> >
>>> >
>>> > I do the same, but some times after cluster reconfiguration (node
>>> failed
>>> > due power supply failure) removed nodes appear again, and this happens
>>> > 3-4 times
>>>
>>> And the same behavior if you switch your cluster into maintenance-mode
>>> (to avoid service downtime) and stop/start pacemaker and corosync
>>> completely?
>>>
>>>
>>> We will have maintenance window at this Friday (20.04.2012) so after
>>> that i can report more info.
>>
>> Of course, that is the safest option ... though you won't have a service
>> downtime if you enable maintenance-mode prior to cluster restart.
> 
> Unless you are using DLM (CLVM, GFS2, OCFS2). Then you should not stop
> corosync - dlm_controld uses CPG.
> 
> And, DLM may use pacemaker parts for fencing (cib, attrd, stonith,
> depending on version).

Yes, of course ... that won't work if you are using dlm. Thanks for
pointing that out explicitly, Vladislav ... and to have it now here in
the ml archive for the records ;-)

Regards,
Andreas

-- 
Need help with Pacemaker?
http://www.hastexo.com/now

> 
>>
>>>
>>> PS: I had similar situation on other cluster some times ago, and there i
>>> fully restart cluster and problem reproduced. But after some time(about
>>> 1-2 week) not existent nodes have ceased to appear
>>
>> Now that is really strange ... if that happens again, the
>> corosync/pacemaker log files would be really interesting to have a look at.
> 
> I recall that is a known issue for a rather long time.
> One need to do a full (not rolling) restart to make node fully disappear.
> I checked this again not so long ago, and yes, node deletion does not
> work with current master branch (or very close to it) - it appears again
> after pacemaker restart on any other node.
> 
> May be it is because of lrmd cache, like with failed actions? It looks
> very similar to that.
> 
> Andrew, David?
> 
> Best,
> Vladislav
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org






signature.asc
Description: OpenPGP digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-19 Thread Vladislav Bogdanov 
19.04.2012 11:24, Andreas Kurz wrote:
> On 04/18/2012 11:46 PM, ruslan usifov wrote:
>>
>>
>> 2012/4/18 Andreas Kurz mailto:andr...@hastexo.com>>
>>
>> On 04/17/2012 09:31 PM, ruslan usifov wrote:
>> >
>> >
>> > 2012/4/17 Proskurin Kirill > 
>> > >>
>> >
>> > On 04/17/2012 03:46 PM, ruslan usifov wrote:
>> >
>> > 2012/4/17 Andreas Kurz > 
>> > >
>> 
>> > > >
>> >
>> >On 04/14/2012 11:14 PM, ruslan usifov wrote:
>> > > Hello
>> > >
>> > > I remove 2 nodes from cluster, with follow sequence:
>> > >
>> > > crm_node --force -R 
>> > > crm_node --force -R 
>> > > cibadmin --delete --obj_type nodes --crm_xml '> > uname="node1"/>'
>> > > cibadmin --delete --obj_type status --crm_xml
>> '> >uname="node1"/>'
>> > > cibadmin --delete --obj_type nodes --crm_xml '> > uname="node2"/>'
>> > > cibadmin --delete --obj_type status --crm_xml
>> '> >uname="node2"/>'
>> > >
>> > >
>> > > Nodes after this deleted, but if for example i restart
>> > (reboot)
>> >one of
>> > > existent nodes in working cluster, this deleted nodes
>> > appear again in
>> > > OFFLINE state
>> >
>> >
>> > I have this problem some time ago.
>> > I "solved" it something like that:
>> >
>> > crm node delete NODENAME
>> > crm_node --force --remove NODENAME
>> > cibadmin --delete --obj_type nodes --crm_xml '> uname="NODENAME"/>'
>> > cibadmin --delete --obj_type status --crm_xml '> > uname="NODENAME"/>'
>> >
>> > --
>> >
>> >
>> > I do the same, but some times after cluster reconfiguration (node
>> failed
>> > due power supply failure) removed nodes appear again, and this happens
>> > 3-4 times
>>
>> And the same behavior if you switch your cluster into maintenance-mode
>> (to avoid service downtime) and stop/start pacemaker and corosync
>> completely?
>>
>>
>> We will have maintenance window at this Friday (20.04.2012) so after
>> that i can report more info.
> 
> Of course, that is the safest option ... though you won't have a service
> downtime if you enable maintenance-mode prior to cluster restart.

Unless you are using DLM (CLVM, GFS2, OCFS2). Then you should not stop
corosync - dlm_controld uses CPG.

And, DLM may use pacemaker parts for fencing (cib, attrd, stonith,
depending on version).

> 
>>
>> PS: I had similar situation on other cluster some times ago, and there i
>> fully restart cluster and problem reproduced. But after some time(about
>> 1-2 week) not existent nodes have ceased to appear
> 
> Now that is really strange ... if that happens again, the
> corosync/pacemaker log files would be really interesting to have a look at.

I recall that is a known issue for a rather long time.
One need to do a full (not rolling) restart to make node fully disappear.
I checked this again not so long ago, and yes, node deletion does not
work with current master branch (or very close to it) - it appears again
after pacemaker restart on any other node.

May be it is because of lrmd cache, like with failed actions? It looks
very similar to that.

Andrew, David?

Best,
Vladislav

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-19 Thread Andreas Kurz 
On 04/18/2012 11:46 PM, ruslan usifov wrote:
> 
> 
> 2012/4/18 Andreas Kurz mailto:andr...@hastexo.com>>
> 
> On 04/17/2012 09:31 PM, ruslan usifov wrote:
> >
> >
> > 2012/4/17 Proskurin Kirill  
> > >>
> >
> > On 04/17/2012 03:46 PM, ruslan usifov wrote:
> >
> > 2012/4/17 Andreas Kurz  
> > >
> 
> >  >
> >
> >On 04/14/2012 11:14 PM, ruslan usifov wrote:
> > > Hello
> > >
> > > I remove 2 nodes from cluster, with follow sequence:
> > >
> > > crm_node --force -R 
> > > crm_node --force -R 
> > > cibadmin --delete --obj_type nodes --crm_xml ' > uname="node1"/>'
> > > cibadmin --delete --obj_type status --crm_xml
> ' >uname="node1"/>'
> > > cibadmin --delete --obj_type nodes --crm_xml ' > uname="node2"/>'
> > > cibadmin --delete --obj_type status --crm_xml
> ' >uname="node2"/>'
> > >
> > >
> > > Nodes after this deleted, but if for example i restart
> > (reboot)
> >one of
> > > existent nodes in working cluster, this deleted nodes
> > appear again in
> > > OFFLINE state
> >
> >
> > I have this problem some time ago.
> > I "solved" it something like that:
> >
> > crm node delete NODENAME
> > crm_node --force --remove NODENAME
> > cibadmin --delete --obj_type nodes --crm_xml ' uname="NODENAME"/>'
> > cibadmin --delete --obj_type status --crm_xml ' > uname="NODENAME"/>'
> >
> > --
> >
> >
> > I do the same, but some times after cluster reconfiguration (node
> failed
> > due power supply failure) removed nodes appear again, and this happens
> > 3-4 times
> 
> And the same behavior if you switch your cluster into maintenance-mode
> (to avoid service downtime) and stop/start pacemaker and corosync
> completely?
> 
> 
> We will have maintenance window at this Friday (20.04.2012) so after
> that i can report more info.

Of course, that is the safest option ... though you won't have a service
downtime if you enable maintenance-mode prior to cluster restart.

> 
> PS: I had similar situation on other cluster some times ago, and there i
> fully restart cluster and problem reproduced. But after some time(about
> 1-2 week) not existent nodes have ceased to appear

Now that is really strange ... if that happens again, the
corosync/pacemaker log files would be really interesting to have a look at.

Regards,
Andreas

-- 
Need help with Pacemaker?
http://www.hastexo.com/services/remote

> 
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org




signature.asc
Description: OpenPGP digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-18 Thread ruslan usifov 
2012/4/18 Andreas Kurz 

> On 04/17/2012 09:31 PM, ruslan usifov wrote:
> >
> >
> > 2012/4/17 Proskurin Kirill  > >
> >
> > On 04/17/2012 03:46 PM, ruslan usifov wrote:
> >
> > 2012/4/17 Andreas Kurz  >   > >>
> >
> >
> >On 04/14/2012 11:14 PM, ruslan usifov wrote:
> > > Hello
> > >
> > > I remove 2 nodes from cluster, with follow sequence:
> > >
> > > crm_node --force -R 
> > > crm_node --force -R 
> > > cibadmin --delete --obj_type nodes --crm_xml ' > uname="node1"/>'
> > > cibadmin --delete --obj_type status --crm_xml ' >uname="node1"/>'
> > > cibadmin --delete --obj_type nodes --crm_xml ' > uname="node2"/>'
> > > cibadmin --delete --obj_type status --crm_xml ' >uname="node2"/>'
> > >
> > >
> > > Nodes after this deleted, but if for example i restart
> > (reboot)
> >one of
> > > existent nodes in working cluster, this deleted nodes
> > appear again in
> > > OFFLINE state
> >
> >
> > I have this problem some time ago.
> > I "solved" it something like that:
> >
> > crm node delete NODENAME
> > crm_node --force --remove NODENAME
> > cibadmin --delete --obj_type nodes --crm_xml ' uname="NODENAME"/>'
> > cibadmin --delete --obj_type status --crm_xml ' > uname="NODENAME"/>'
> >
> > --
> >
> >
> > I do the same, but some times after cluster reconfiguration (node failed
> > due power supply failure) removed nodes appear again, and this happens
> > 3-4 times
>
> And the same behavior if you switch your cluster into maintenance-mode
> (to avoid service downtime) and stop/start pacemaker and corosync
> completely?
>
>
We will have maintenance window at this Friday (20.04.2012) so after that i
can report more info.

PS: I had similar situation on other cluster some times ago, and there i
fully restart cluster and problem reproduced. But after some time(about 1-2
week) not existent nodes have ceased to appear
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-18 Thread Andreas Kurz 
On 04/17/2012 09:31 PM, ruslan usifov wrote:
> 
> 
> 2012/4/17 Proskurin Kirill  >
> 
> On 04/17/2012 03:46 PM, ruslan usifov wrote:
> 
> 2012/4/17 Andreas Kurz    >>
> 
> 
>On 04/14/2012 11:14 PM, ruslan usifov wrote:
> > Hello
> >
> > I remove 2 nodes from cluster, with follow sequence:
> >
> > crm_node --force -R 
> > crm_node --force -R 
> > cibadmin --delete --obj_type nodes --crm_xml ' uname="node1"/>'
> > cibadmin --delete --obj_type status --crm_xml 'uname="node1"/>'
> > cibadmin --delete --obj_type nodes --crm_xml ' uname="node2"/>'
> > cibadmin --delete --obj_type status --crm_xml 'uname="node2"/>'
> >
> >
> > Nodes after this deleted, but if for example i restart
> (reboot)
>one of
> > existent nodes in working cluster, this deleted nodes
> appear again in
> > OFFLINE state
> 
> 
> I have this problem some time ago.
> I "solved" it something like that:
> 
> crm node delete NODENAME
> crm_node --force --remove NODENAME
> cibadmin --delete --obj_type nodes --crm_xml ''
> cibadmin --delete --obj_type status --crm_xml ' uname="NODENAME"/>'
> 
> -- 
> 
> 
> I do the same, but some times after cluster reconfiguration (node failed
> due power supply failure) removed nodes appear again, and this happens
> 3-4 times

And the same behavior if you switch your cluster into maintenance-mode
(to avoid service downtime) and stop/start pacemaker and corosync
completely?

Regards,
Andreas

-- 
Need help with Pacemaker?
http://www.hastexo.com/now

> 
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org






signature.asc
Description: OpenPGP digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-18 Thread Dejan Muhamedagic 
Hi,

On Tue, Apr 17, 2012 at 03:55:21PM +0400, Proskurin Kirill wrote:
> On 04/17/2012 03:46 PM, ruslan usifov wrote:
>> 2012/4/17 Andreas Kurz mailto:andr...@hastexo.com>>
>>
>> On 04/14/2012 11:14 PM, ruslan usifov wrote:
>>  > Hello
>>  >
>>  > I remove 2 nodes from cluster, with follow sequence:
>>  >
>>  > crm_node --force -R 
>>  > crm_node --force -R 
>>  > cibadmin --delete --obj_type nodes --crm_xml ''
>>  > cibadmin --delete --obj_type status --crm_xml '> uname="node1"/>'
>>  > cibadmin --delete --obj_type nodes --crm_xml ''
>>  > cibadmin --delete --obj_type status --crm_xml '> uname="node2"/>'
>>  >
>>  >
>>  > Nodes after this deleted, but if for example i restart (reboot)
>> one of
>>  > existent nodes in working cluster, this deleted nodes appear again in
>>  > OFFLINE state
>
> I have this problem some time ago.
> I "solved" it something like that:
>
> crm node delete NODENAME
> crm_node --force --remove NODENAME
> cibadmin --delete --obj_type nodes --crm_xml ''
> cibadmin --delete --obj_type status --crm_xml ' uname="NODENAME"/>'

The last three commands is what the first one does. No more and
no less.

Thanks,

Dejan

>
> -- 
> Best regards,
> Proskurin Kirill
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-17 Thread ruslan usifov 
2012/4/17 Proskurin Kirill 

> On 04/17/2012 03:46 PM, ruslan usifov wrote:
>
>> 2012/4/17 Andreas Kurz mailto:andr...@hastexo.com>>
>>
>>
>>On 04/14/2012 11:14 PM, ruslan usifov wrote:
>> > Hello
>> >
>> > I remove 2 nodes from cluster, with follow sequence:
>> >
>> > crm_node --force -R 
>> > crm_node --force -R 
>> > cibadmin --delete --obj_type nodes --crm_xml ''
>> > cibadmin --delete --obj_type status --crm_xml '>uname="node1"/>'
>> > cibadmin --delete --obj_type nodes --crm_xml ''
>> > cibadmin --delete --obj_type status --crm_xml '>uname="node2"/>'
>> >
>> >
>> > Nodes after this deleted, but if for example i restart (reboot)
>>one of
>> > existent nodes in working cluster, this deleted nodes appear again
>> in
>> > OFFLINE state
>>
>
> I have this problem some time ago.
> I "solved" it something like that:
>
> crm node delete NODENAME
> crm_node --force --remove NODENAME
> cibadmin --delete --obj_type nodes --crm_xml ''
> cibadmin --delete --obj_type status --crm_xml ' uname="NODENAME"/>'
>
> --
>

I do the same, but some times after cluster reconfiguration (node failed
due power supply failure) removed nodes appear again, and this happens 3-4
times
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-17 Thread Proskurin Kirill 

On 04/17/2012 03:46 PM, ruslan usifov wrote:

2012/4/17 Andreas Kurz mailto:andr...@hastexo.com>>

On 04/14/2012 11:14 PM, ruslan usifov wrote:
 > Hello
 >
 > I remove 2 nodes from cluster, with follow sequence:
 >
 > crm_node --force -R 
 > crm_node --force -R 
 > cibadmin --delete --obj_type nodes --crm_xml ''
 > cibadmin --delete --obj_type status --crm_xml ''
 > cibadmin --delete --obj_type nodes --crm_xml ''
 > cibadmin --delete --obj_type status --crm_xml ''
 >
 >
 > Nodes after this deleted, but if for example i restart (reboot)
one of
 > existent nodes in working cluster, this deleted nodes appear again in
 > OFFLINE state


I have this problem some time ago.
I "solved" it something like that:

crm node delete NODENAME
crm_node --force --remove NODENAME
cibadmin --delete --obj_type nodes --crm_xml ''
cibadmin --delete --obj_type status --crm_xml 'uname="NODENAME"/>'


--
Best regards,
Proskurin Kirill

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-17 Thread ruslan usifov 
2012/4/17 Andreas Kurz 

> On 04/14/2012 11:14 PM, ruslan usifov wrote:
> > Hello
> >
> > I remove 2 nodes from cluster, with follow sequence:
> >
> > crm_node --force -R 
> > crm_node --force -R 
> > cibadmin --delete --obj_type nodes --crm_xml ''
> > cibadmin --delete --obj_type status --crm_xml ' uname="node1"/>'
> > cibadmin --delete --obj_type nodes --crm_xml ''
> > cibadmin --delete --obj_type status --crm_xml ' uname="node2"/>'
> >
> >
> > Nodes after this deleted, but if for example i restart (reboot) one of
> > existent nodes in working cluster, this deleted nodes appear again in
> > OFFLINE state
>
> Just to double check ... corosync was already stopped (on these
> to-be-deleted nodes) prior to the deletion and it's still stopped on the
> removed nodes? ... and no cman involved?
>
>
This nodes doesn't present physically:-)) (we remove this from network), so
no corosync no cman not anything else
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Periodically appear non-existent nodes

2012-04-17 Thread Andreas Kurz 
On 04/14/2012 11:14 PM, ruslan usifov wrote:
> Hello
> 
> I remove 2 nodes from cluster, with follow sequence:
> 
> crm_node --force -R 
> crm_node --force -R 
> cibadmin --delete --obj_type nodes --crm_xml ''
> cibadmin --delete --obj_type status --crm_xml ''
> cibadmin --delete --obj_type nodes --crm_xml ''
> cibadmin --delete --obj_type status --crm_xml ''
> 
> 
> Nodes after this deleted, but if for example i restart (reboot) one of
> existent nodes in working cluster, this deleted nodes appear again in
> OFFLINE state

Just to double check ... corosync was already stopped (on these
to-be-deleted nodes) prior to the deletion and it's still stopped on the
removed nodes? ... and no cman involved?

Regards,
Andreas

-- 
Need help with Pacemaker?
http://www.hastexo.com/now

> 
> PS:
>OS ubuntu 10.0.4(2.6.32-40)
>pacemaker 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
>corosync 1.4.2
> 
> 
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org






signature.asc
Description: OpenPGP digital signature
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Periodically appear non-existent nodes

2012-04-14 Thread ruslan usifov 
Hello

I remove 2 nodes from cluster, with follow sequence:

crm_node --force -R 
crm_node --force -R 
cibadmin --delete --obj_type nodes --crm_xml ''
cibadmin --delete --obj_type status --crm_xml ''
cibadmin --delete --obj_type nodes --crm_xml ''
cibadmin --delete --obj_type status --crm_xml ''


Nodes after this deleted, but if for example i restart (reboot) one of
existent nodes in working cluster, this deleted nodes appear again in
OFFLINE state

PS:
   OS ubuntu 10.0.4(2.6.32-40)
   pacemaker 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
   corosync 1.4.2
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org