Hello, Alexey. No, we don't include this ticket to 2.7. Should we?
ср, 19 дек. 2018 г. в 12:55, Alexey Goncharuk <alexey.goncha...@gmail.com>: > Folks, why did not we include IGNITE-10003 to ignite-2.7 release scope? > This causes an Ignite node to be stopped by default when checkpoint read > lock acquire times out. I expect a lot of Ignite 2.7 users will be affected > by this mistake. > > We should at least update the documentation and make users aware of a > workaround. > > чт, 25 окт. 2018 г. в 16:35, Alexey Goncharuk <alexey.goncha...@gmail.com > >: > > > Andrey, > > > > I still see that checkpoint read lock acquisition raises a > CRITICAL_ERROR, > > which by default will shut down local node. As far as I remember, we > > decided that by default thread timeout should not trigger node failure. > > Now, however, it does, because we ignore SYSTEM_WORKER_BLOCKED events in > > default configuration. > > > > Should we introduce another critical failure type > > CHECKPOINT_READ_LOCK_BLOCKED or use SYSTEM_WORKER_BLOCKED for checkpoint > > read lock acquire failure? > > > > --AG > > > > пт, 12 окт. 2018 г. в 8:29, Andrey Kuznetsov <stku...@gmail.com>: > > > >> Igniters, > >> > >> Now I spot blocking / long-running code arising from > >> {{GridDhtPartitionsExchangeFuture#init}} calls in partition-exchanger > >> thread, see [1]. Ideally, all blocking operations along all possible > code > >> paths should be guarded implicitly from critical failure detector to > avoid > >> the thread from being considered blocked. There is a pull request [2] > that > >> provides shallow solution. I didn't change code outside > >> {{GridDhtPartitionsExchangeFuture}}, otherwise it could be broken by any > >> upcoming change. Also, I didn't touch the code runnable by threads other > >> than partition-exchanger. So I have a number of guarded sections that > are > >> wider than they could be, and this potentially hides issues from failure > >> detector. Does this PR make sense? Or maybe it's better to exclude > >> partition-exchanger from critical threads registry at all? > >> > >> [1] https://issues.apache.org/jira/browse/IGNITE-9710 > >> [2] https://github.com/apache/ignite/pull/4962 > >> > >> > >> пт, 28 сент. 2018 г. в 18:56, Maxim Muzafarov <maxmu...@gmail.com>: > >> > >> > Andrey, Andrey > >> > > >> > > Thanks for being attentive! It's definitely a typo. Could you please > >> > create > >> > > an issue? > >> > > >> > I've created an issue [1] and prepared PR [2]. > >> > Please, review this change. > >> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9723 > >> > [2] https://github.com/apache/ignite/pull/4862 > >> > > >> > On Fri, 28 Sep 2018 at 16:58 Yakov Zhdanov <yzhda...@apache.org> > wrote: > >> > > >> > > Config option + mbean access. Does that make sense? > >> > > > >> > > Yakov > >> > > > >> > > On Fri, Sep 28, 2018, 17:17 Vladimir Ozerov <voze...@gridgain.com> > >> > wrote: > >> > > > >> > > > Then it should be config option. > >> > > > > >> > > > пт, 28 сент. 2018 г. в 13:15, Andrey Gura <ag...@apache.org>: > >> > > > > >> > > > > Guys, > >> > > > > > >> > > > > why we need both config option and system property? I believe > one > >> way > >> > > is > >> > > > > enough. > >> > > > > On Fri, Sep 28, 2018 at 12:38 PM Nikolay Izhikov < > >> > nizhi...@apache.org> > >> > > > > wrote: > >> > > > > > > >> > > > > > Ticket created - > >> https://issues.apache.org/jira/browse/IGNITE-9737 > >> > > > > > > >> > > > > > Fixed version is 2.7. > >> > > > > > > >> > > > > > В Пт, 28/09/2018 в 11:41 +0300, Alexey Goncharuk пишет: > >> > > > > > > Nikolay, I agree, a user should be able to disable both > thread > >> > > > liveness > >> > > > > > > check and checkpoint read lock timeout check from config > and a > >> > > system > >> > > > > > > property. > >> > > > > > > > >> > > > > > > пт, 28 сент. 2018 г. в 11:30, Nikolay Izhikov < > >> > nizhi...@apache.org > >> > > >: > >> > > > > > > > >> > > > > > > > Hello, Igniters. > >> > > > > > > > > >> > > > > > > > I found that this feature can't be disabled from config. > >> > > > > > > > The only way to disable it is from JMX bean. > >> > > > > > > > > >> > > > > > > > I think it very dangerous: If we have some corner case or > a > >> bug > >> > > in > >> > > > > this > >> > > > > > > > Watch Dog it can make Ignite unusable. > >> > > > > > > > I propose to implement possibility to disable this feature > >> > both - > >> > > > > from > >> > > > > > > > config and from JVM options. > >> > > > > > > > > >> > > > > > > > What do you think? > >> > > > > > > > > >> > > > > > > > В Чт, 27/09/2018 в 16:14 +0300, Andrey Kuznetsov пишет: > >> > > > > > > > > Maxim, > >> > > > > > > > > > >> > > > > > > > > Thanks for being attentive! It's definitely a typo. > Could > >> you > >> > > > > please > >> > > > > > > > > >> > > > > > > > create > >> > > > > > > > > an issue? > >> > > > > > > > > > >> > > > > > > > > чт, 27 сент. 2018 г. в 16:00, Maxim Muzafarov < > >> > > > maxmu...@gmail.com > >> > > > > >: > >> > > > > > > > > > >> > > > > > > > > > Folks, > >> > > > > > > > > > > >> > > > > > > > > > I've found in `GridCachePartitionExchangeManager:2684` > >> [1] > >> > > > > (master > >> > > > > > > > > >> > > > > > > > branch) > >> > > > > > > > > > exchange future wrapped > >> > > > > > > > > > with double `blockingSectionEnd` method. Is it > correct? > >> I > >> > > just > >> > > > > want to > >> > > > > > > > > > understand this change and > >> > > > > > > > > > how should I use this in the future. > >> > > > > > > > > > > >> > > > > > > > > > Should I file a new issue to fix this? I think here > >> > > > > > > > > >> > > > > > > > `blockingSectionBegin` > >> > > > > > > > > > method should be used. > >> > > > > > > > > > > >> > > > > > > > > > ------------- > >> > > > > > > > > > blockingSectionEnd(); > >> > > > > > > > > > > >> > > > > > > > > > try { > >> > > > > > > > > > resVer = exchFut.get(exchTimeout, > >> > TimeUnit.MILLISECONDS); > >> > > > > > > > > > } finally { > >> > > > > > > > > > blockingSectionEnd(); > >> > > > > > > > > > } > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > [1] > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePartitionExchangeManager.java#L2684 > >> > > > > > > > > > > >> > > > > > > > > > On Wed, 26 Sep 2018 at 22:47 Vyacheslav Daradur < > >> > > > > daradu...@gmail.com> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Andrey Gura, thank you for the answer! > >> > > > > > > > > > > > >> > > > > > > > > > > I agree that wrapping of 'init' method reduces the > >> profit > >> > > of > >> > > > > watchdog > >> > > > > > > > > > > service in case of PME worker, but in other cases, > we > >> > > should > >> > > > > wrap all > >> > > > > > > > > > > possible long sections on > >> GridDhtPartitionExchangeFuture. > >> > > For > >> > > > > example > >> > > > > > > > > > > 'onCacheChangeRequest' method or > >> > > > > > > > > > > 'cctx.affinity().onCacheChangeRequest' inside > because > >> it > >> > > may > >> > > > > take > >> > > > > > > > > > > significant time (reproducer attached). > >> > > > > > > > > > > > >> > > > > > > > > > > I only want to point out a possible issue which may > >> allow > >> > > to > >> > > > > end-user > >> > > > > > > > > > > halt the Ignite cluster accidentally. > >> > > > > > > > > > > > >> > > > > > > > > > > I'm sure that PME experts know how to fix this issue > >> > > > properly. > >> > > > > > > > > > > On Wed, Sep 26, 2018 at 10:28 PM Andrey Gura < > >> > > > ag...@apache.org > >> > > > > > > >> > > > > > > > > >> > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > Vyacheslav, > >> > > > > > > > > > > > > >> > > > > > > > > > > > Exchange worker is strongly tied with > >> > > > > > > > > > > > GridDhtPartitionExchangeFuture#init and it is ok. > >> > > Exchange > >> > > > > worker > >> > > > > > > > > >> > > > > > > > also > >> > > > > > > > > > > > shouldn't be blocked for long time but in reality > it > >> > > > > happens.It > >> > > > > > > > > >> > > > > > > > also > >> > > > > > > > > > > > means that your change doesn't make sense. > >> > > > > > > > > > > > > >> > > > > > > > > > > > What actually make sense it is identification of > >> places > >> > > > which > >> > > > > > > > > > > > intentionally blocking. May be some places/actions > >> > should > >> > > > be > >> > > > > > > > > >> > > > > > > > braced by > >> > > > > > > > > > > > blocking guards. > >> > > > > > > > > > > > > >> > > > > > > > > > > > If you have failing tests please make sure that > your > >> > > > > > > > > >> > > > > > > > failureHandler is > >> > > > > > > > > > > > NoOpFailureHandler or any other handler with > >> > > > > ignoreFailureTypes = > >> > > > > > > > > > > > [CRITICAL_WORKER_BLOCKED]. > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Wed, Sep 26, 2018 at 9:43 PM Vyacheslav > Daradur < > >> > > > > > > > > > > >> > > > > > > > > > daradu...@gmail.com> > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Hi Igniters! > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thank you for this important improvement! > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > I've looked through implementation and noticed > >> that > >> > > > > > > > > > > > > GridDhtPartitionsExchangeFuture#init has not > been > >> > > wrapped > >> > > > > in > >> > > > > > > > > >> > > > > > > > blocked > >> > > > > > > > > > > > > section. This means it easy to halt the node in > >> case > >> > of > >> > > > > > > > > >> > > > > > > > longrunning > >> > > > > > > > > > > > > actions during PME, for example when we create a > >> > cache > >> > > > with > >> > > > > > > > > > > > > StoreFactrory which connect to 3rd party DB. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > I'm not sure that it is the right behavior. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > I filled the issue [1] and prepared the PR [2] > >> with > >> > > > > reproducer > >> > > > > > > > > >> > > > > > > > and > >> > > > > > > > > > > > >> > > > > > > > > > > possible fix. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Andrey, could you please look at and confirm > that > >> it > >> > > > makes > >> > > > > sense? > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > [1] > >> > https://issues.apache.org/jira/browse/IGNITE-9710 > >> > > > > > > > > > > > > [2] https://github.com/apache/ignite/pull/4845 > >> > > > > > > > > > > > > On Mon, Sep 24, 2018 at 9:46 PM Andrey > Kuznetsov < > >> > > > > > > > > >> > > > > > > > stku...@gmail.com> > >> > > > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Denis, > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > I've created the ticket [1] with short > >> description > >> > of > >> > > > the > >> > > > > > > > > > > > >> > > > > > > > > > > functionality. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [1] > >> > > https://issues.apache.org/jira/browse/IGNITE-9679 > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > пн, 24 сент. 2018 г. в 17:46, Denis Magda < > >> > > > > dma...@apache.org>: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Andrey K. and G., > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thanks, do we have a documentation ticket > >> > created? > >> > > > > Prachi > >> > > > > > > > > > > >> > > > > > > > > > (copied) > >> > > > > > > > > > > can help > >> > > > > > > > > > > > > > > with the documentation. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > > Denis > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Mon, Sep 24, 2018 at 5:51 AM Andrey Gura > < > >> > > > > > > > > >> > > > > > > > ag...@apache.org> > >> > > > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Andrey, > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > finally your change is merged to master > >> branch. > >> > > > > > > > > >> > > > > > > > Congratulations > >> > > > > > > > > > > > >> > > > > > > > > > > and > >> > > > > > > > > > > > > > > > thank you very much! :) > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I think that the next step is feature that > >> will > >> > > > allow > >> > > > > > > > > >> > > > > > > > signal > >> > > > > > > > > > > > >> > > > > > > > > > > about > >> > > > > > > > > > > > > > > > blocked threads to the monitoring tools > via > >> > > MXBean. > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I hope you will continue development of > this > >> > > > feature > >> > > > > and > >> > > > > > > > > > > >> > > > > > > > > > provide > >> > > > > > > > > > > your > >> > > > > > > > > > > > > > > > vision in new JIRA issue. > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Tue, Sep 11, 2018 at 6:54 PM Andrey > >> > Kuznetsov > >> > > < > >> > > > > > > > > > > > >> > > > > > > > > > > stku...@gmail.com> > >> > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > David, Maxim! > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks a lot for you ideas. > >> Unfortunately, I > >> > > > can't > >> > > > > adopt > >> > > > > > > > > >> > > > > > > > all > >> > > > > > > > > > > > >> > > > > > > > > > > of them > >> > > > > > > > > > > > > > > > right > >> > > > > > > > > > > > > > > > > now: the scope is much broader than the > >> scope > >> > > of > >> > > > > the > >> > > > > > > > > >> > > > > > > > change I > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > implement. > >> > > > > > > > > > > > > > > > I > >> > > > > > > > > > > > > > > > > have had a talk to a group of Ignite > >> > commiters, > >> > > > > and we > >> > > > > > > > > >> > > > > > > > agreed > >> > > > > > > > > > > > >> > > > > > > > > > > to > >> > > > > > > > > > > > > > > complete > >> > > > > > > > > > > > > > > > > the change as follows. > >> > > > > > > > > > > > > > > > > - Blocking instructions in > system-critical > >> > > which > >> > > > > may > >> > > > > > > > > > > >> > > > > > > > > > resonably > >> > > > > > > > > > > last > >> > > > > > > > > > > > > > > long > >> > > > > > > > > > > > > > > > > should be explicitly excluded from the > >> > > > monitoring. > >> > > > > > > > > > > > > > > > > - Failure handlers should have a setting > >> to > >> > > > > suppress some > >> > > > > > > > > > > > >> > > > > > > > > > > failures on > >> > > > > > > > > > > > > > > > > per-failure-type basis. > >> > > > > > > > > > > > > > > > > According to this I have updated the > >> > > > > implementation: [1] > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > [1] > >> > https://github.com/apache/ignite/pull/4089 > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > пн, 10 сент. 2018 г. в 22:35, David > >> Harvey < > >> > > > > > > > > > > > >> > > > > > > > > > > syssoft...@gmail.com>: > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > When I've done this before,I've needed > >> to > >> > > find > >> > > > > the > >> > > > > > > > > >> > > > > > > > oldest > >> > > > > > > > > > > > >> > > > > > > > > > > thread, > >> > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > kill > >> > > > > > > > > > > > > > > > > > the node running that. From a > language > >> > > > > standpoint, > >> > > > > > > > > > > >> > > > > > > > > > Maxim's > >> > > > > > > > > > > "without > >> > > > > > > > > > > > > > > > > > progress" better than "heartbeat". > For > >> > > > > example, what > >> > > > > > > > > >> > > > > > > > I'm > >> > > > > > > > > > > > >> > > > > > > > > > > most > >> > > > > > > > > > > > > > > > interested > >> > > > > > > > > > > > > > > > > > in on a distributed system is which > >> thread > >> > > > > started the > >> > > > > > > > > >> > > > > > > > work > >> > > > > > > > > > > > >> > > > > > > > > > > it has > >> > > > > > > > > > > > > > > not > >> > > > > > > > > > > > > > > > > > completed the earliest, and when did > >> that > >> > > > thread > >> > > > > last > >> > > > > > > > > >> > > > > > > > make > >> > > > > > > > > > > > >> > > > > > > > > > > forward > >> > > > > > > > > > > > > > > > > > process. You don't want to kill a > >> node > >> > > > > because a > >> > > > > > > > > >> > > > > > > > thread > >> > > > > > > > > > > > >> > > > > > > > > > > is > >> > > > > > > > > > > > > > > waiting > >> > > > > > > > > > > > > > > > on a > >> > > > > > > > > > > > > > > > > > lock held by a thread that went > off-node > >> > and > >> > > > has > >> > > > > not > >> > > > > > > > > > > >> > > > > > > > > > gotten a > >> > > > > > > > > > > > > > > response. > >> > > > > > > > > > > > > > > > > > If you don't understand the dependency > >> > > > > relationships, > >> > > > > > > > > >> > > > > > > > you > >> > > > > > > > > > > > >> > > > > > > > > > > will make > >> > > > > > > > > > > > > > > > > > incorrect recovery decisions. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > On Mon, Sep 10, 2018 at 4:08 AM Maxim > >> > > > Muzafarov < > >> > > > > > > > > > > > >> > > > > > > > > > > maxmu...@gmail.com> > >> > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > I think we should find exact answers > >> to > >> > > these > >> > > > > > > > > >> > > > > > > > questions: > >> > > > > > > > > > > > > > > > > > > 1. What `critical` issue exactly > is? > >> > > > > > > > > > > > > > > > > > > 2. How can we find critical issues? > >> > > > > > > > > > > > > > > > > > > 3. How can we handle critical > issues? > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > First, > >> > > > > > > > > > > > > > > > > > > - Ignore uninterruptable actions > >> (e.g. > >> > > > > > > > > >> > > > > > > > worker\service > >> > > > > > > > > > > > >> > > > > > > > > > > shutdown) > >> > > > > > > > > > > > > > > > > > > - Long I/O operations (should be a > >> > > > > configurable > >> > > > > > > > > >> > > > > > > > timeout > >> > > > > > > > > > > > >> > > > > > > > > > > for each > >> > > > > > > > > > > > > > > > type of > >> > > > > > > > > > > > > > > > > > > usage) > >> > > > > > > > > > > > > > > > > > > - Infinite loops > >> > > > > > > > > > > > > > > > > > > - Stalled\deadlocked threads > (and\or > >> too > >> > > > many > >> > > > > parked > >> > > > > > > > > > > > >> > > > > > > > > > > threads, > >> > > > > > > > > > > > > > > > exclude > >> > > > > > > > > > > > > > > > > > I/O) > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Second, > >> > > > > > > > > > > > > > > > > > > - The working queue is without > >> progress > >> > > > (e.g. > >> > > > > disco, > >> > > > > > > > > > > > >> > > > > > > > > > > exchange > >> > > > > > > > > > > > > > > > queues) > >> > > > > > > > > > > > > > > > > > > - Work hasn't been completed since > >> the > >> > > last > >> > > > > > > > > >> > > > > > > > heartbeat > >> > > > > > > > > > > > >> > > > > > > > > > > (checking > >> > > > > > > > > > > > > > > > > > > milestones) > >> > > > > > > > > > > > > > > > > > > - Too many system resources used > by a > >> > > thread > >> > > > > for the > >> > > > > > > > > > > >> > > > > > > > > > long > >> > > > > > > > > > > period > >> > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > time > >> > > > > > > > > > > > > > > > > > > (allocated memory, CPU) > >> > > > > > > > > > > > > > > > > > > - Timing fields associated with > each > >> > > thread > >> > > > > status > >> > > > > > > > > > > > >> > > > > > > > > > > exceeded a > >> > > > > > > > > > > > > > > > maximum > >> > > > > > > > > > > > > > > > > > time > >> > > > > > > > > > > > > > > > > > > limit. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Third (not too many options here), > >> > > > > > > > > > > > > > > > > > > - `log everything` should be the > >> default > >> > > > > behaviour > >> > > > > > > > > >> > > > > > > > in > >> > > > > > > > > > > >> > > > > > > > > > all > >> > > > > > > > > > > these > >> > > > > > > > > > > > > > > > cases, > >> > > > > > > > > > > > > > > > > > > since it may be difficult to find > the > >> > cause > >> > > > > after the > >> > > > > > > > > > > > >> > > > > > > > > > > restart. > >> > > > > > > > > > > > > > > > > > > - Wait some interval of time and > kill > >> > the > >> > > > > hanging > >> > > > > > > > > >> > > > > > > > node > >> > > > > > > > > > > > >> > > > > > > > > > > (cluster > >> > > > > > > > > > > > > > > > should > >> > > > > > > > > > > > > > > > > > be > >> > > > > > > > > > > > > > > > > > > configured stable enough) > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Questions, > >> > > > > > > > > > > > > > > > > > > - Not sure, but can workers miss > >> their > >> > > > > heartbeat > >> > > > > > > > > > > > >> > > > > > > > > > > deadlines if CPU > >> > > > > > > > > > > > > > > > loads > >> > > > > > > > > > > > > > > > > > up > >> > > > > > > > > > > > > > > > > > > to 80%-90%? Bursts of momentary > >> overloads > >> > > can > >> > > > > be > >> > > > > > > > > > > > > > > > > > > expected behaviour as a normal > >> part > >> > of > >> > > > > system > >> > > > > > > > > > > > >> > > > > > > > > > > operations. > >> > > > > > > > > > > > > > > > > > > - Why do we decide that critical > >> thread > >> > > > should > >> > > > > > > > > >> > > > > > > > monitor > >> > > > > > > > > > > > >> > > > > > > > > > > each other? > >> > > > > > > > > > > > > > > > For > >> > > > > > > > > > > > > > > > > > > instance, if all the tasks were > >> blocked > >> > and > >> > > > > unable to > >> > > > > > > > > > > >> > > > > > > > > > run, > >> > > > > > > > > > > > > > > > > > > node reset would never occur. As > >> for > >> > > me, > >> > > > a > >> > > > > better > >> > > > > > > > > > > > >> > > > > > > > > > > solution is > >> > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > use > >> > > > > > > > > > > > > > > > > > a > >> > > > > > > > > > > > > > > > > > > separate monitor thread or pool > (maybe > >> > both > >> > > > > with > >> > > > > > > > > >> > > > > > > > software > >> > > > > > > > > > > > > > > > > > > and hardware checks) that not > only > >> > > checks > >> > > > > > > > > >> > > > > > > > heartbeats > >> > > > > > > > > > > > >> > > > > > > > > > > but > >> > > > > > > > > > > > > > > > monitors the > >> > > > > > > > > > > > > > > > > > > other system as well. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Mon, 10 Sep 2018 at 00:07 David > >> > Harvey < > >> > > > > > > > > > > > >> > > > > > > > > > > syssoft...@gmail.com> > >> > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > It would be safer to restart the > >> entire > >> > > > > cluster > >> > > > > > > > > >> > > > > > > > than to > >> > > > > > > > > > > > >> > > > > > > > > > > remove > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > last > >> > > > > > > > > > > > > > > > > > > > node for a cache that should be > >> > > redundant. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Sun, Sep 9, 2018, 4:00 PM > Andrey > >> > Gura > >> > > < > >> > > > > > > > > > > > >> > > > > > > > > > > ag...@apache.org> > >> > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Hi, > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > I agree with Yakov that we can > >> > provide > >> > > > some > >> > > > > > > > > >> > > > > > > > option > >> > > > > > > > > > > > >> > > > > > > > > > > that manage > >> > > > > > > > > > > > > > > > worker > >> > > > > > > > > > > > > > > > > > > > > liveness checker behavior in > case > >> of > >> > > > > observing > >> > > > > > > > > >> > > > > > > > that > >> > > > > > > > > > > > >> > > > > > > > > > > some worker > >> > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > blocked too long. > >> > > > > > > > > > > > > > > > > > > > > At least it will some > workaround > >> for > >> > > > > cases when > >> > > > > > > > > >> > > > > > > > node > >> > > > > > > > > > > > >> > > > > > > > > > > fails is > >> > > > > > > > > > > > > > > > too > >> > > > > > > > > > > > > > > > > > > > > annoying. > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Backups count threshold sounds > >> good > >> > > but I > >> > > > > don't > >> > > > > > > > > > > > >> > > > > > > > > > > understand how > >> > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > will > >> > > > > > > > > > > > > > > > > > > > > help in case of cluster hanging. > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > The simplest solution here is > >> alert > >> > in > >> > > > > cases of > >> > > > > > > > > > > > >> > > > > > > > > > > blocking of > >> > > > > > > > > > > > > > > some > >> > > > > > > > > > > > > > > > > > > > > critical worker (we can improve > >> > > > > WorkersRegistry > >> > > > > > > > > >> > > > > > > > for > >> > > > > > > > > > > > >> > > > > > > > > > > this > >> > > > > > > > > > > > > > > purpose > >> > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > expose list of blocked workers) > >> and > >> > > > > optionally > >> > > > > > > > > >> > > > > > > > call > >> > > > > > > > > > > > >> > > > > > > > > > > system > >> > > > > > > > > > > > > > > > configured > >> > > > > > > > > > > > > > > > > > > > > failure processor. BTW, failure > >> > > processor > >> > > > > can be > >> > > > > > > > > > > > >> > > > > > > > > > > extended in > >> > > > > > > > > > > > > > > > order to > >> > > > > > > > > > > > > > > > > > > > > perform any checks (e.g. backup > >> > count) > >> > > > and > >> > > > > decide > >> > > > > > > > > > > > >> > > > > > > > > > > whether it > >> > > > > > > > > > > > > > > > should > >> > > > > > > > > > > > > > > > > > > > > stop node or not. > >> > > > > > > > > > > > > > > > > > > > > On Sat, Sep 8, 2018 at 3:42 PM > >> Andrey > >> > > > > Kuznetsov < > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > stku...@gmail.com> > >> > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > David, Yakov, I understand > your > >> > > fears. > >> > > > > But > >> > > > > > > > > >> > > > > > > > liveness > >> > > > > > > > > > > > >> > > > > > > > > > > checks > >> > > > > > > > > > > > > > > deal > >> > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > _critical_ conditions, i.e. > when > >> > > such a > >> > > > > > > > > >> > > > > > > > condition > >> > > > > > > > > > > >> > > > > > > > > > is > >> > > > > > > > > > > met we > >> > > > > > > > > > > > > > > > > > conclude > >> > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > node as totally broken, and > >> there > >> > is > >> > > no > >> > > > > sense > >> > > > > > > > > >> > > > > > > > to > >> > > > > > > > > > > > >> > > > > > > > > > > keep it > >> > > > > > > > > > > > > > > alive > >> > > > > > > > > > > > > > > > > > > > regardless > >> > > > > > > > > > > > > > > > > > > > > > the data it contains. If we > >> want to > >> > > > give > >> > > > > it a > >> > > > > > > > > > > > >> > > > > > > > > > > chance, then > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > condition > >> > > > > > > > > > > > > > > > > > > > > > (long fsync etc.) should not > >> > > considered > >> > > > > as > >> > > > > > > > > >> > > > > > > > critical > >> > > > > > > > > > > > >> > > > > > > > > > > at all. > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > сб, 8 сент. 2018 г. в 15:18, > >> Yakov > >> > > > > Zhdanov < > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > yzhda...@apache.org>: > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Agree with David. We need to > >> have > >> > > an > >> > > > > > > > > >> > > > > > > > opporunity > >> > > > > > > > > > > > >> > > > > > > > > > > set backups > >> > > > > > > > > > > > > > > > count > >> > > > > > > > > > > > > > > > > > > > > threshold > >> > > > > > > > > > > > > > > > > > > > > > > (at runtime also!) that will > >> not > >> > > > allow > >> > > > > any > >> > > > > > > > > > > > >> > > > > > > > > > > automatic stop > >> > > > > > > > > > > > > > > if > >> > > > > > > > > > > > > > > > > > there > >> > > > > > > > > > > > > > > > > > > > > will be > >> > > > > > > > > > > > > > > > > > > > > > > a data loss. Andrey, what do > >> you > >> > > > think? > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > --Yakov > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > > > > > > > > > Best regards, > >> > > > > > > > > > > > > > > > > > > > > > Andrey Kuznetsov. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > > > > > > Maxim Muzafarov > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > > > > Best regards, > >> > > > > > > > > > > > > > > > > Andrey Kuznetsov. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > Best regards, > >> > > > > > > > > > > > > > Andrey Kuznetsov. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > -- > >> > > > > > > > > > > > > Best Regards, Vyacheslav D. > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > -- > >> > > > > > > > > > > Best Regards, Vyacheslav D. > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > -- > >> > > > > > > > > > -- > >> > > > > > > > > > Maxim Muzafarov > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > >> > > > >> > -- > >> > -- > >> > Maxim Muzafarov > >> > > >> > >> > >> -- > >> Best regards, > >> Andrey Kuznetsov. > >> > > >