Re: [ClusterLabs] Pacemaker with Zookeeper??

[Andrew Beekhof]

tl;dr - dont port pacemaker, use pacemaker-remote instead

On Wed, May 18, 2016 at 5:20 PM, Jan Friesse  wrote:
> Ken Gaillot napsal(a):
>
>> On 05/17/2016 09:54 AM, Digimer wrote:
>>>
>>> On 16/05/16 04:35 AM, Bogdan Dobrelya wrote:

 On 05/16/2016 09:23 AM, Jan Friesse wrote:
>>
>> Hi,
>>
>> I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
>> it possible?
>> Is there any examination about that?


 Indeed, would be *great* to have a Pacemaker based control plane on top
 of other "pluggable" distributed KVS & messaging systems, for example
 etcd as well :)
 I'm looking forward to joining any dev efforts around that, although I'm
 not a Java or Go developer.
>>>
>>>
>>> Part of open source is the freedom to do whatever you want, of course.
>>>
>>> Let me ask though; What problems would zookeeper, etcd or other systems
>>> solve that can't be solved in corosync?
>>>
>>> I ask because the HA community just finished a multi-year effort to
>>> merge different projects into one common HA stack. This has a lot of
>>> benefits to the user base, not least of which is lack of confusion.
>>>
>>> Strikes me that the significant time investment in supporting a new
>>> comms layer would be much more beneficially spent on improving the
>>> existing stack.
>>>
>>> Again, anyone is free to do whatever they want... I just don't see the
>>> motivator personally.
>>>
>>> digimer
>>
>>
>> I see one big difference that is both a strength and a weakness: these
>> other packages have a much wider user base beyond the HA cluster use
>> case. The strength is that there will be many more developers working to
>> fix bugs, add features, etc. The weakness is that most of those
>
>
> This is exactly what I was thinking about during 2.x developement. If
> replacement of Corosync wouldn't make more sense than continue developing of
> Corosync. I was able to accept implementing some features. Sadly, there was
> exactly ONE project which would be able to replace corosync (Spread toolkit)
> which is even less widespread than Corosync.
>
> From my point of view, replacement of corosync must be (at least) able to:
> - Work without quorum

Agreed. It's a non-starter if messaging stops when quorum is lost.

> - Support 2 node clusters
> - Allow multiple links (something like RRP)

Doesn't bonding (and the imminent arrival of knet) make this somewhat optional?

> - Don't include SPOF (so nothing like configuration stored on one node only
> and/or different machine on network)

There can be subtle variations on this.  The pattern in OpenStack is
to have a "management node".
Which sounds like a SPOF but they also require that the service be
able to function without it.  So its a grey area.

> - Provide EVS/VS

Pacemaker could live without this.  Heartbeat didn't provide it either.

> - Provide something like qdevice

Or the ability to create it.  In fairness, Pacemaker has gotten by for
a long long time without it :-)


It would be nice to be considered for the kinds of scaled deployments
that kubernetes and etcd are built for because that's where all the
excitement and mindshare is.  Zookeeper was one of the options I
thought of too, however realistically Pacemaker is not what those
folks are looking for.  At those scales our stack's smarts take a back
seat to the idea that there are so many copies that dozens can die and
the only recovery you need is to maybe start some more copies (because
with so many, there is always a master around somewhere).


For those of us with a need to scale _and_ an appreciation of "real"
resource orchestration, I would argue that a better architecture is a
small traditional cluster managing a much larger pool of
pacemaker-remote nodes.  Putting effort into making that really shine
(especially since its pretty solid already) is likely to have a better
payoff than porting to another messaging layer.


>
> Both zookeeper and etcd builds on top of quite simple to understand
> membership mechanism (zookeeper = elected master, something like amoeba,
> etcd = raft), what's nice, because it means more contributors. Sadly bare
> metal HA must work even in situations where "simple" quorum is not enough.
>
>
>
>> developers are ignorant of HA clustering and could easily cause more
>> problems for the HA use case than they fix.
>>
>> Another potential benefit is the familiarity factor -- people are more
>> comfortable with things they recognize from somewhere else. So it might
>> help Pacemaker adoption, especially in the communities that already use
>> these packages.
>>
>> I'm not aware of any technical advantages, and I wouldn't expect any,
>> given corosync's long HA focus.
>>
>  From my point of view (and yes, I'm biased), biggest problem of
> Zookeper
> is need to have quorum
>
> (https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing).
> Direct consequence is inability to 

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Jan Friesse]

Ken Gaillot napsal(a):

On 05/17/2016 09:54 AM, Digimer wrote:

On 16/05/16 04:35 AM, Bogdan Dobrelya wrote:

On 05/16/2016 09:23 AM, Jan Friesse wrote:

Hi,

I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
it possible?
Is there any examination about that?


Indeed, would be *great* to have a Pacemaker based control plane on top
of other "pluggable" distributed KVS & messaging systems, for example
etcd as well :)
I'm looking forward to joining any dev efforts around that, although I'm
not a Java or Go developer.


Part of open source is the freedom to do whatever you want, of course.

Let me ask though; What problems would zookeeper, etcd or other systems
solve that can't be solved in corosync?

I ask because the HA community just finished a multi-year effort to
merge different projects into one common HA stack. This has a lot of
benefits to the user base, not least of which is lack of confusion.

Strikes me that the significant time investment in supporting a new
comms layer would be much more beneficially spent on improving the
existing stack.

Again, anyone is free to do whatever they want... I just don't see the
motivator personally.

digimer


I see one big difference that is both a strength and a weakness: these
other packages have a much wider user base beyond the HA cluster use
case. The strength is that there will be many more developers working to
fix bugs, add features, etc. The weakness is that most of those


This is exactly what I was thinking about during 2.x developement. If 
replacement of Corosync wouldn't make more sense than continue 
developing of Corosync. I was able to accept implementing some features. 
Sadly, there was exactly ONE project which would be able to replace 
corosync (Spread toolkit) which is even less widespread than Corosync.


From my point of view, replacement of corosync must be (at least) able to:
- Work without quorum
- Support 2 node clusters
- Allow multiple links (something like RRP)
- Don't include SPOF (so nothing like configuration stored on one node 
only and/or different machine on network)

- Provide EVS/VS
- Provide something like qdevice

Both zookeeper and etcd builds on top of quite simple to understand 
membership mechanism (zookeeper = elected master, something like amoeba, 
etcd = raft), what's nice, because it means more contributors. Sadly 
bare metal HA must work even in situations where "simple" quorum is not 
enough.




developers are ignorant of HA clustering and could easily cause more
problems for the HA use case than they fix.

Another potential benefit is the familiarity factor -- people are more
comfortable with things they recognize from somewhere else. So it might
help Pacemaker adoption, especially in the communities that already use
these packages.

I'm not aware of any technical advantages, and I wouldn't expect any,
given corosync's long HA focus.


 From my point of view (and yes, I'm biased), biggest problem of Zookeper
is need to have quorum
(https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing).
Direct consequence is inability to tolerate one node failure in 2 node
cluster -> no 2 node clusters (and such deployment is extremely
popular). Also Corosync can operate completely without quorum.

Regards,
   Honza



Thanks for your help!
Hai Nguyen


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Ken Gaillot]

On 05/17/2016 06:50 AM, Bogdan Dobrelya wrote:
> On 05/17/2016 01:17 PM, Adam Spiers wrote:
>> Bogdan Dobrelya  wrote:
>>> On 05/16/2016 09:23 AM, Jan Friesse wrote:
> Hi,
>
> I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
> it possible?
> Is there any examination about that?
>>>
>>> Indeed, would be *great* to have a Pacemaker based control plane on top
>>> of other "pluggable" distributed KVS & messaging systems, for example
>>> etcd as well :)
>>> I'm looking forward to joining any dev efforts around that, although I'm
>>> not a Java or Go developer.
>>
>> Pacemaker and corosync are written in C, so I suspect it would be
>> pretty awkward to replace corosync with something which isn't.
>>
>> ___
>> Users mailing list: Users@clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
> 
> Would be nice to see a clear picture for what is the coupling between
> those separate services making the Idea impossible? Is it libqb with its
> C-only API (wild guessing, sorry!), anything else? Theoretically, if
> there were no loose coupling, one could replace storage or messaging
> components or both and still leverage a Pacemaker CP :)

The only language dependency is that Pacemaker is written in C and needs
to hook into the messaging layer, so the messaging layer needs some C
API. Since most languages offer some way for library functions to be
called by a C program, it shouldn't be a deal-breaker.

Two issues make this challenging:

First, all the places in Pacemaker code that currently can call either
heartbeat or corosync functions would need equivalents for the new
layer, which would require a large investment of developer time. Search
the code for "SUPPORT_COROSYNC" to get an idea.

Second, corosync has been optimized over many years for the HA cluster
use case, and any replacement would have a lot of catching up to do, to
be truly equivalent. Communication layers that haven't been forged by
experience with HA clusters are highly unlikely to handle the known
corner cases and tricky situations. See the corosync and votequorum man
pages for just the user-tunable parameters that have evolved.

It is possible, just difficult.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Adam Spiers]

Bogdan Dobrelya  wrote:
> On 05/16/2016 09:23 AM, Jan Friesse wrote:
> >> Hi,
> >>
> >> I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
> >> it possible?
> >> Is there any examination about that?
> 
> Indeed, would be *great* to have a Pacemaker based control plane on top
> of other "pluggable" distributed KVS & messaging systems, for example
> etcd as well :)
> I'm looking forward to joining any dev efforts around that, although I'm
> not a Java or Go developer.

Pacemaker and corosync are written in C, so I suspect it would be
pretty awkward to replace corosync with something which isn't.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Bogdan Dobrelya]

On 05/16/2016 09:23 AM, Jan Friesse wrote:
>> Hi,
>>
>> I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
>> it possible?
>> Is there any examination about that?

Indeed, would be *great* to have a Pacemaker based control plane on top
of other "pluggable" distributed KVS & messaging systems, for example
etcd as well :)
I'm looking forward to joining any dev efforts around that, although I'm
not a Java or Go developer.

> 
> From my point of view (and yes, I'm biased), biggest problem of Zookeper
> is need to have quorum
> (https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing).
> Direct consequence is inability to tolerate one node failure in 2 node
> cluster -> no 2 node clusters (and such deployment is extremely
> popular). Also Corosync can operate completely without quorum.
> 
> Regards,
>   Honza
> 
>>
>> Thanks for your help!
>> Hai Nguyen
>>
>>
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Jan Friesse]

Hi,

I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
it possible?
Is there any examination about that?


From my point of view (and yes, I'm biased), biggest problem of 
Zookeper is need to have quorum 
(https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing). 
Direct consequence is inability to tolerate one node failure in 2 node 
cluster -> no 2 node clusters (and such deployment is extremely 
popular). Also Corosync can operate completely without quorum.


Regards,
  Honza



Thanks for your help!
Hai Nguyen





___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org