Re: [ClusterLabs] corosync/pacemaker on ~100 nodes cluser

2016-08-25 Thread Kai Dupke 
On 08/23/2016 06:26 PM, Radoslaw Garbacz wrote:
> I would like to ask for settings (and hardware requirements) to have
> corosync/pacemaker running on about 100 nodes cluster.

Not bad ...

For the SUSE HA offering we limit support to 32 nodes because after this
the timing gets so critical that we can't make a support statement anymore.

We do have customers with special testing / setups running ~40 nodes,
but that really depends on the workload and resources you have to configure.

Looks like you've found this, too.

pacemaker_remote might be a chance to get more nodes, depending on the
services you're running.

I wonder if not a cloud solution (with some HA embedded) or a break up
in smaller chunks is beneficial.

It's not only about the technical capabilities but the mental
capabilities of the admins limiting the reliability of such a huge cluster.

Just in case, there are HPC clusters with many more nodes, but these
address a total different use case.

regards,
Kai Dupke
Senior Product Manager, HA & HPC
-- 
Sell not virtue to purchase wealth, nor liberty to purchase power.
Phone:  +49-(0)5102-9310828 Mail: kdu...@suse.com
Mobile: +49-(0)173-5876766  WWW:  www.suse.com

SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg)

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync/pacemaker on ~100 nodes cluser

2016-08-24 Thread Radoslaw Garbacz 
Hi,

Thank you for the advice. Indeed, seems like Pacemaker Remote will solve my
big cluster problem.

With regard to your questions about my current solution, I scale corosync
parameters based on the number of nodes, additionally modifying some of the
kernel network parameters. Tests I did let me select certain corosync
settings, which works, but are possibly not the best (cluster is quite slow
when reacting to some quorum related events).

The problem seems to be only related to cluster start, once running, any
operations such as node lost/reconnect, agents creation/start/stop work
well. Memory and network seems important with regard to the hardware.

Below are settings I used for my latest test (the largest working cluster I
tried):
* latest pacemaker/corosync
* 55 c3.4xlarge nodes (amazon cloud)
* 55 active nodes, 552 resources in a cluster
* kernel settings:
net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 5000

* corosync settings:
token: 12000
consensus: 16000
join: 1500
send_join: 80
merge: 2000
downcheck: 2000
max_network_delay: 150 # for azure

Best regards,


On Tue, Aug 23, 2016 at 12:00 PM, Ken Gaillot  wrote:

> On 08/23/2016 11:46 AM, Klaus Wenninger wrote:
> > On 08/23/2016 06:26 PM, Radoslaw Garbacz wrote:
> >> Hi,
> >>
> >> I would like to ask for settings (and hardware requirements) to have
> >> corosync/pacemaker running on about 100 nodes cluster.
> > Actually I had thought that 16 would be the limit for full
> > pacemaker-cluster-nodes.
> > For larger deployments pacemaker-remote should be the way to go. Were
> > you speaking of a cluster with remote-nodes?
> >
> > Regards,
> > Klaus
> >>
> >> For now some nodes get totally frozen (high CPU, high network usage),
> >> so that even login is not possible. By manipulating
> >> corosync/pacemaker/kernel parameters I managed to run it on ~40 nodes
> >> cluster, but I am not sure which parameters are critical, how to make
> >> it more responsive and how to make the number of nodes even bigger.
>
> 16 is a practical limit without special hardware and tuning, so that's
> often what companies that offer support for clusters will accept.
>
> I know people have gone well higher than 16 with a lot of optimization,
> but I think somewhere between 32 and 64 corosync can't keep up with the
> messages. Your 40 nodes sounds about right. I'd be curious to hear what
> you had to do (with hardware, OS tuning, and corosync tuning) to get
> that far.
>
> As Klaus mentioned, Pacemaker Remote is the preferred way to go beyond
> that currently:
>
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-
> single/Pacemaker_Remote/index.html
>
> >> Thanks,
> >>
> >> --
> >> Best Regards,
> >>
> >> Radoslaw Garbacz
> >> XtremeData Incorporation
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>



-- 
Best Regards,

Radoslaw Garbacz
XtremeData Incorporation
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync/pacemaker on ~100 nodes cluser

2016-08-23 Thread Ken Gaillot 
On 08/23/2016 11:46 AM, Klaus Wenninger wrote:
> On 08/23/2016 06:26 PM, Radoslaw Garbacz wrote:
>> Hi,
>>
>> I would like to ask for settings (and hardware requirements) to have
>> corosync/pacemaker running on about 100 nodes cluster.
> Actually I had thought that 16 would be the limit for full
> pacemaker-cluster-nodes.
> For larger deployments pacemaker-remote should be the way to go. Were
> you speaking of a cluster with remote-nodes?
> 
> Regards,
> Klaus
>>
>> For now some nodes get totally frozen (high CPU, high network usage),
>> so that even login is not possible. By manipulating
>> corosync/pacemaker/kernel parameters I managed to run it on ~40 nodes
>> cluster, but I am not sure which parameters are critical, how to make
>> it more responsive and how to make the number of nodes even bigger.

16 is a practical limit without special hardware and tuning, so that's
often what companies that offer support for clusters will accept.

I know people have gone well higher than 16 with a lot of optimization,
but I think somewhere between 32 and 64 corosync can't keep up with the
messages. Your 40 nodes sounds about right. I'd be curious to hear what
you had to do (with hardware, OS tuning, and corosync tuning) to get
that far.

As Klaus mentioned, Pacemaker Remote is the preferred way to go beyond
that currently:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html

>> Thanks,
>>
>> -- 
>> Best Regards,
>>
>> Radoslaw Garbacz
>> XtremeData Incorporation

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync/pacemaker on ~100 nodes cluser

2016-08-23 Thread Klaus Wenninger 
On 08/23/2016 06:26 PM, Radoslaw Garbacz wrote:
> Hi,
>
> I would like to ask for settings (and hardware requirements) to have
> corosync/pacemaker running on about 100 nodes cluster.
Actually I had thought that 16 would be the limit for full
pacemaker-cluster-nodes.
For larger deployments pacemaker-remote should be the way to go. Were
you speaking of a cluster with remote-nodes?

Regards,
Klaus
>
> For now some nodes get totally frozen (high CPU, high network usage),
> so that even login is not possible. By manipulating
> corosync/pacemaker/kernel parameters I managed to run it on ~40 nodes
> cluster, but I am not sure which parameters are critical, how to make
> it more responsive and how to make the number of nodes even bigger.
>
> Thanks,
>
> -- 
> Best Regards,
>
> Radoslaw Garbacz
> XtremeData Incorporation
>
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org