Ferenc,
Jan Friesse <jfrie...@redhat.com> writes:
wf...@niif.hu writes:
In a 6-node cluster (vhbl03-08) the following happens 1-5 times a day
(in August; in May, it happened 0-2 times a day only, it's slowly
ramping up):
vhbl08 corosync[3687]: [TOTEM ] A processor failed, forming new configuration.
vhbl03 corosync[3890]: [TOTEM ] A processor failed, forming new configuration.
vhbl07 corosync[3805]: [MAIN ] Corosync main process was not scheduled for
4317.0054 ms (threshold is 2400.0000 ms). Consider token timeout increase.
^^^ This is main problem you have to solve. It usually means that
machine is too overloaded. [...]
Before I start tracing the scheduler, I'd like to ask something: what
wakes up the Corosync main process periodically? The token making a
full circle? (Please forgive my simplistic understanding of the TOTEM
protocol.) That would explain the recommendation in the log message,
but does not fit well with the overload assumption: totally idle nodes
could just as easily produce such warnings if there are no other regular
wakeup sources. (I'm looking at timer_function_scheduler_timeout but I
know too little of libqb to decide.)
Corosync main loop is based on epoll, so corosync is waked up ether by
receiving data (network socket or unix socket for services) or when
there are data to sent and socket is ready for non blocking write or
after timeout. This timeout is exactly what you call other wakeup resource.
Timeout is used for scheduling periodical tasks inside corosync.
One of periodical tasks is scheduler pause detector. It is basically
scheduled every (token_timeout / 3) msec and it computes diff between
current and last time. If diff is larger than (token_timeout * 0.8) it
displays warning.
As a start you can try what message say = Consider token timeout
increase. Currently you have 3 seconds, in theory 6 second should be
enough.
It was probably high time I realized that token timeout is scaled
automatically when one has a nodelist. When you say Corosync should
work OK with default settings up to 16 nodes, you assume this scaling is
in effect, don't you? On the other hand, I've got no nodelist in the
config, but token = 3000, which is less than the default 1000+4*650 with
six nodes, and this will get worse as the cluster grows.
This is described in corosync.conf man page (token_coefficient).
Final timeout is computed using totem.token as a base value. So if you
set totem.token to 3000 it means that final totem timeout value is not
3000 but (3000 + 4 * 650).
Regards,
Honza
Comments on the above ramblings welcome!
I'm grateful for all the valuable input poured into this thread by all
parties: it's proven really educative in quite unexpected ways beyond
what I was able to ask in the beginning.
_______________________________________________
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
