Hi, Andrew I tested by the following versions. https://github.com/ClusterLabs/pacemaker/commit/3492fec7fe58a6fd94071632df27d3fd3fc3ffe3
load-threshold was checked at 60%, 40%, and 20%. However, the problem was not solved. It will not change but timeout will occur. Restriction of the number of jobs seems to be carried out correctly. However, since the synchronous message of CIB is sent ceaseless, it is processing there preferentially. Therefore, the internal IPC communication message is kept waiting. I think that I need to change the priority of message processing in order to solve this problem. Or when load is high, I think that processing which stops that DC sends a job is effective. The accumulated message may be processed while transmission of the job has stopped. However, it is expected that operation of the whole cluster becomes slow in that case. Does it happen with the problem which may occur when a priority is changed in what kind of case? And if known, I want you to tell me should be what the test. load-threshold 60% test report https://drive.google.com/file/d/0BwMFJItoO-fVOHB5S1ROOUJrams/edit?usp=sharing load-threshold 40% test report https://drive.google.com/file/d/0BwMFJItoO-fVemlqVUU2QkhEMW8/edit?usp=sharing load-threshold 20% test report https://drive.google.com/file/d/0BwMFJItoO-fVTWFTU2pqOF9pcms/edit?usp=sharing report tested by the commitment which changed the priority is also sent. https://github.com/yuusuke/pacemaker/commit/17a7cbe67c455f5f6d36a1e1bc255b4ab0039dd8 load-threshold 80% and CPG G_PRIORITY_DEFAULT test report https://drive.google.com/file/d/0BwMFJItoO-fVV1BoTjVQMk52WEU/edit?usp=sharing 2013/11/6 Andrew Beekhof <and...@beekhof.net>: > > On 5 Nov 2013, at 12:48 pm, yusuke iida <yusk.i...@gmail.com> wrote: > >> Hi, Andrew >> >> I tested by this commitment. >> https://github.com/beekhof/pacemaker/commit/145c782e432d8108ca865f994640cf5a62406363 >> >> However, the problem has not improved. >> It seems that it will be preferentially processed since the message of >> CPG is set as G_PRIORITY_MED. >> >> I suggest that you lower the priority of CPG instead. > > I worry about this change. > It may allow ipc clients to read out of date information (the pending cpg > messages almost certainly contain updates) and could result in updates being > lost (because they're not being made to the latest config+status). > > Could you try reducing the value of load-threshold? The default (80%) could > be too high. > >> How is this? >> https://github.com/yuusuke/pacemaker/commit/22a14318cc740b3043106609923f47039c3aa407 >> >> I did not find the method of lowering only the priority of the CPG >> message of a CIB process. >> >> Reports when the error came out were collected. >> I want you to note that it is delayed that an IPC message is processed >> as follows. >> >> Nov 01 21:53:52 [9246] vm01 crmd: (cib_native.c:397 ) trace: >> cib_native_perform_op_delegate: Async call, returning 32 >> (snip) >> Nov 01 21:55:57 [9241] vm01 cib: ( callbacks.c:688 ) info: >> cib_process_request: Forwarding cib_modify operation for section >> status to master (origin=local/crmd/32) >> >> Since size is large, I want you to download from the following. >> https://drive.google.com/file/d/0BwMFJItoO-fVWDg1Sjc2WXltUjQ/edit?usp=sharing >> >> Regards, >> Yusuke >> >> 2013/10/31 Andrew Beekhof <and...@beekhof.net>: >>> >>> On 29 Oct 2013, at 12:12 am, yusuke iida <yusk.i...@gmail.com> wrote: >>> >>>> Hi, Andrew >>>> >>>> I tested using following commit. >>>> https://github.com/beekhof/pacemaker/commit/b6fa1e650f64b1ba73fdb143f41323aa8cb3544e >>>> >>>> However, timeout of operation has still occurred. >>>> >>>> I analyzed the log. >>>> >>>> I am noting that it is late that the ipc message transmitted to cib >>>> from crmd of local is processed. >>>> Since the CIB synchronous message by which the CIB process came from >>>> the outside will have priority and will be processed, this happens? >>>> >>>> >>>> I made the following corrections so that the priority of the message >>>> which CIB processes might be changed. >>>> In this case, timeout does not occur. >>>> >>>> diff --git a/lib/cluster/cpg.c b/lib/cluster/cpg.c >>>> index 8522cbf..3a67998 100644 >>>> --- a/lib/cluster/cpg.c >>>> +++ b/lib/cluster/cpg.c >>>> @@ -212,7 +212,7 @@ pcmk_cpg_dispatch(gpointer user_data) >>>> int rc = 0; >>>> crm_cluster_t *cluster = (crm_cluster_t*) user_data; >>>> >>>> - rc = cpg_dispatch(cluster->cpg_handle, CS_DISPATCH_ALL); >>>> + rc = cpg_dispatch(cluster->cpg_handle, CS_DISPATCH_ONE); >>>> if (rc != CS_OK) { >>>> crm_err("Connection to the CPG API failed: %s (%d)", >>>> ais_error2text(rc), rc); >>>> cluster->cpg_handle = 0; >>>> diff --git a/lib/common/mainloop.c b/lib/common/mainloop.c >>>> index 18a67e6..d605288 100644 >>>> --- a/lib/common/mainloop.c >>>> +++ b/lib/common/mainloop.c >>>> @@ -482,7 +482,7 @@ gio_poll_dispatch_add(enum qb_loop_priority p, >>>> int32_t fd, int32_t evts, >>>> adaptor->p = p; >>>> adaptor->is_used = QB_TRUE; >>>> adaptor->source = >>>> - g_io_add_watch_full(channel, G_PRIORITY_DEFAULT, evts, >>>> gio_read_socket, adaptor, >>>> + g_io_add_watch_full(channel, G_PRIORITY_MEDIUM, evts, >>>> gio_read_socket, adaptor, >>>> gio_poll_destroy); >>>> >>>> /* Now that mainloop now holds a reference to channel, >>>> >>>> I do not know this fix is correct. >>>> Can't the comment to this correction be got? >>> >>> The CS_DISPATCH_ONE change looks ok: >>> https://github.com/beekhof/pacemaker/commit/6384053 >>> Did you try with just that? I'd like to avoid the mainloop priority change >>> if possible. >>> >>>> >>>> Regards, >>>> Yusuke >>>> >>>> 2013/10/20 Andrew Beekhof <and...@beekhof.net>: >>>>> >>>>> On 18/10/2013, at 10:12 PM, yusuke iida <yusk.i...@gmail.com> wrote: >>>>> >>>>>> Hi, Andrew >>>>>> >>>>>> Now, I am testing the configuration of one standby node and active node >>>>>> of 15. >>>>>> About 10 Dummy resources are started per node. >>>>>> >>>>>> If all the nodes are started with this composition, before all the >>>>>> resources start, it will take the time for about 20 minutes. >>>>>> >>>>>> And some resources have caused start timeout. >>>>>> probe is performed all at once by all the nodes at a start-up. >>>>>> The result is written in cib and synchronizes with all the nodes. >>>>>> This processing requires very high load. >>>>>> I think that timeout has occurred owing to it. >>>>> >>>>> More than likely, yes. >>>>> >>>>>> >>>>>> I am very interested in whether this problem is solvable, if you use >>>>>> throttle created now. >>>>> >>>>> I have been using it, I have found it more effective than batch-limit for >>>>> bounding CPU usage and avoiding timeouts. >>>>> I would be interested to hear your feedback if you have the time to do >>>>> some testing. >>>>> >>>>>> When is throttle due to be merged into the repository of ClusterLabs? >>>>> >>>>> It is queued up behind a compatibility patch that is needed for some >>>>> changes I made to the pacemaker-remote wire protocol. >>>>> >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> -- >>>>>> ---------------------------------------- >>>>>> METRO SYSTEMS CO., LTD >>>>>> >>>>>> Yusuke Iida >>>>>> Mail: yusk.i...@gmail.com >>>>>> ---------------------------------------- >>>>>> >>>>>> _______________________________________________ >>>>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>>> >>>>>> Project Home: http://www.clusterlabs.org >>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>> Bugs: http://bugs.clusterlabs.org >>>>> >>>>> >>>>> _______________________________________________ >>>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>> >>>>> Project Home: http://www.clusterlabs.org >>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>> Bugs: http://bugs.clusterlabs.org >>>>> >>>> >>>> >>>> >>>> -- >>>> ---------------------------------------- >>>> METRO SYSTEMS CO., LTD >>>> >>>> Yusuke Iida >>>> Mail: yusk.i...@gmail.com >>>> ---------------------------------------- >>>> >>>> _______________________________________________ >>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>> >>>> Project Home: http://www.clusterlabs.org >>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>> Bugs: http://bugs.clusterlabs.org >>> >>> >>> _______________________________________________ >>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>> >>> Project Home: http://www.clusterlabs.org >>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >> >> >> >> -- >> ---------------------------------------- >> METRO SYSTEMS CO., LTD >> >> Yusuke Iida >> Mail: yusk.i...@gmail.com >> ---------------------------------------- >> >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org > > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org -- ---------------------------------------- METRO SYSTEMS CO., LTD Yusuke Iida Mail: yusk.i...@gmail.com ---------------------------------------- _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org