On 07/05/2011 10:08 AM, Vladislav Bogdanov wrote: > 05.07.2011 19:10, Steven Dake wrote: >> On 07/05/2011 07:26 AM, Vladislav Bogdanov wrote: >>> Hi all, >>> >>> Last days I see following messages in logs: >>> [TOTEM ] Process pause detected for XXX ms, flushing membership messages. >>> >>> After that ring is quickly re-established. >>> DLM/clvmd notifies this and switches to kern_stop waiting for fencing to >>> be done. Although what dlm_tool ls provides is really strange flags and >>> members differ between nodes. I have dumps of what has been happening in >>> dlm, and there are messages that fencing was done! >>> >>> On the other hand, pacemaker does not notify anything so fencing is not >>> done. This is rather strange, but for another list. >>> >>> Can anybody please explain what exactly that message means and what is >>> the correct reaction of upper services should be? >>> Can it be solely caused by network problems? >>> Can number of buffers in RX ring of ethernet card influence this (I did >>> some tuning there some time ago)? >>> >>> corosync 1.3.1, UDPU transport. >>> pacemaker-1.1-devel >>> dlm_controld.pcmk from 3.0.17 >>> clvmd 2.02.85 >>> clusterlib-3.1.1 >>> >> >> This indicates the kernel has paused scheduling (or corosync of corosync >> or corosync has blocked for the time value printed in the message. > > I suspected this, thanks for clarification. > >> Corosync is non-blocking. >> >> Are you running inside a VM? Increasing token is probably a necessity >> when running inside a VM on a heavily loaded host because kvm does not >> schedule as fairly as bare metal. >> >> Please provide feedback if this is bare metal or m. > > I see this both on one node in VM, and on bare metal hosts under high > load (30 vms are installing on each 12-core node, so CPU usage is quite > big). > > I removed eth RX ring buffer tuning from physical hosts (now it is > default 256 instead of max 4096). > Will see what will happen. > > This could be a problem of ethernet driver on bare metal nodes as well.
Which ethernet driver? > > With VM I'll try to increase its weight by cgroups. > > Steve, can you please also explain why I'm unable to move corosync to > another (non-default) CPU cgroup? Is this caused by a real-time > priority? I just wanted to increase its weight. > Not sure on cgroups question, but it should be running ahead of other processes assuming cgroups follow posix scheduler semantics. You could try with corosync -p (run without realtime priority) and see if cgroups can be manipulated that way. If you are running really heavy load, running a preemptible kernel config may be useful (if that is not already default). The kernel has changed so much since 5 years ago when I worked on it daily I have no idea how the scheduler actually works any longer. Regards -steve > Best, > Vladislav _______________________________________________ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
