[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13801409#comment-13801409 ] Omkar Vinit Joshi commented on YARN-1121: - [~bikassaha] we have a single dispatcher queue. so should we ignore other events when RM is going down and selectively only process rm state store writes related events? In any case we will have very short time before we actually get kill -9. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Omkar Vinit Joshi > Fix For: 2.2.1 > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13801504#comment-13801504 ] Bikas Saha commented on YARN-1121: -- The state store has its own dispatch queue. So basically in RMStateStore.stop() we need to block until the pending events get stored. At this point. Ideally before stopping the state store, the RM will shutdown all RPC services so that no new externally generated store events get added to the system. One of the main use cases for this jira is a user who has submitted a job and is currently polling for the app to be accepted. If the RM does not save this request then such users will get an unknown app error when the RM comes back up. If we are willing to live with this error then we may choose to not fix this jira. Because, like you say, the RM may not have time to actually flush pending events. So we cannot technically solve this completely. However, during graceful RM failover or admin controlled shutdown, we should have time to flush events and prevent such user facing errors. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Omkar Vinit Joshi > Fix For: 2.2.1 > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808451#comment-13808451 ] Jian He commented on YARN-1121: --- The unit test passed with the core code changes, and failed without those changes. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808457#comment-13808457
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610917/YARN-1121.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2310//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2310//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808530#comment-13808530
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610928/YARN-1121.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2312//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2312//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808704#comment-13808704 ] Zhijie Shen commented on YARN-1121: --- If we add the "drain" feature for AsyncDispatcher, DrainDispatcher seems to be unnecessary. OTOH, we can promote DrainDispatcher to the production code, and use it directly. I incline to the latter choice. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811094#comment-13811094
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611560/YARN-1121.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2346//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2346//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811552#comment-13811552
]
Hadoop QA commented on YARN-1121:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611645/YARN-1121.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2352//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2352//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813126#comment-13813126
]
Hadoop QA commented on YARN-1121:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611996/YARN-1121.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2361//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2361//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813301#comment-13813301 ] Zhijie Shen commented on YARN-1121: --- One typo: setDraningStop -> setDrainingStop > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813305#comment-13813305 ] Jian He commented on YARN-1121: --- bq. One typo: setDraningStop -> setDrainingStop Nice catch ! will fix it in the next patch. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813409#comment-13813409
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612047/YARN-1121.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2365//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2365//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2365//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813685#comment-13813685
]
Bikas Saha commented on YARN-1121:
--
There are 3 new booleans with 8 combinations possible between them. Which
combinations are legal? Which are impossible? Some comments will help
understand their interaction. Naming could be better. e.g. drainEventsOnStop
instead of drainingStopNeeded and drainOnStop instead of drainingStop.
{code}
+ private volatile boolean drained = true;
+ private volatile boolean drainingStopNeeded = false;
+ private volatile boolean drainingStop = false;
{code}
Typo {code} + LOG.info("Ignoring events as AsyncDispatcher is draning to
stop."); {code}
Isnt this almost a tight loop? Given that storing stuff will be over the
network and slow, why not have a wait notify between this thread and the
draining thread?
DrainEventHandler sounds misleading. It doesnt really drain. It ignores or
drops events. The other thing we can do is take a count of the number of
pending events to drain at service stop. Then make sure we drain only those
many, thus ignoring the new ones. This removes the need of drainingStop and
reduces the combinatorics of booleans.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813686#comment-13813686
]
Bikas Saha commented on YARN-1121:
--
Code for comment above - Isnt this almost a tight loop? Given that storing
stuff will be over the network and slow, why not have a wait notify between
this thread and the draining thread?
{code}
protected void serviceStop() throws Exception {
+if (drainingStopNeeded) {
+ drainingStop = true;
+ while(!drained) {
+Thread.yield();
+ }
{code}
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814138#comment-13814138 ] Jian He commented on YARN-1121: --- bq. drainingStop(renames to blockNewRequests): Typo, in the patch, it's actually named blockNewEvents. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814150#comment-13814150
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612223/YARN-1121.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2375//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814202#comment-13814202
]
Hadoop QA commented on YARN-1121:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612228/YARN-1121.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2376//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2376//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch,
> YARN-1121.6.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814410#comment-13814410 ] Bikas Saha commented on YARN-1121: -- bq. To do that, we may need to add things in dispatcher's runnable like "if(queueEmpty) notify", and this is likely to be invoked in every normal execution of the dispatch while loop if queue is empty, even it's not actually in stop phase, which may create more overhead, as this AsyncDispatcher is used everywhere. Can this be only enabled when serviceStop sets the drain events flag. In normal situations that flag will not be set. Replacing the eventHandler to DropEventHandler (instead of GenericEventHandler) may not be enough. Someone may have already gotten a GenericEventHandler object and may send events using that object. So new events will keep getting added to the queue from those cached GenericEventHandler object. So, I think keeping track of the number of events to drain and only draining those many events will be a more robust solution. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814512#comment-13814512
]
Hadoop QA commented on YARN-1121:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612294/YARN-1121.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2379//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2379//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.2.1
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch,
> YARN-1121.6.patch, YARN-1121.7.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817908#comment-13817908 ] Vinod Kumar Vavilapalli commented on YARN-1121: --- Looks good overall. Checking this in. One thing of note is that you are removing locking for service-life-cycle methods in RMStateStore. I verified that it seems fine - events coming in during serviceStop are ignored due to draining and other blocking calls are okay to happen. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.2.1 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817915#comment-13817915 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4707 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4707/]) YARN-1121. Changed ResourceManager's state-store to drain all events on shut-down. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1540232) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.3.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818099#comment-13818099 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #387 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/387/]) YARN-1121. Changed ResourceManager's state-store to drain all events on shut-down. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1540232) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.3.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818127#comment-13818127 ] Hudson commented on YARN-1121: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1604 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1604/]) YARN-1121. Changed ResourceManager's state-store to drain all events on shut-down. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1540232) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.3.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818135#comment-13818135 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1578 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1578/]) YARN-1121. Changed ResourceManager's state-store to drain all events on shut-down. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1540232) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.3.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844453#comment-13844453
]
Bikas Saha commented on YARN-1121:
--
If the thread gets interrupted or otherwise has an unexpected exit then it does
not look like drained will be set to true. And service stop will hang.
{code}
while (!stopped && !Thread.currentThread().isInterrupted()) {
+ drained = eventQueue.isEmpty();
{code}
Also, it would probably be better if we signaled an object when we exit the
above run() method and block on that signal instead of the following spin wait.
{code}
+ while(!drained) {
+Thread.yield();
+ }
{code}
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch,
> YARN-1121.6.patch, YARN-1121.7.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845060#comment-13845060
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618174/YARN-1121.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2643//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2643//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch,
> YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849327#comment-13849327
]
Bikas Saha commented on YARN-1121:
--
Is it enough to just check for drained==true?
{code}
+ synchronized (waitForDrained) {
+if (blockNewEvents && drained) {
+ waitForDrained.notify();
+}
+ }
{code}
Usual practice is to put the sync inside the while loop. Also, we should
probably do a wait(1000) and write a log so that its easy to debug when this
serviceStop gets stuck.
{code}
+ synchronized (waitForDrained) {
+while (!drained && eventHandlingThread.isAlive()) {
+ waitForDrained.wait();
+}
{code}
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch,
> YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849940#comment-13849940 ] Jian He commented on YARN-1121: --- bq. Is it enough to just check for drained==true? In the code, didn't I also check blockNewEvents, which is set true only when dispatcher is stopping? > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850088#comment-13850088 ] Bikas Saha commented on YARN-1121: -- I meant should it not be enough to just check for drained since the wait condition is only checking for drained. There could be errors if the notify and the wait conditions are different. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850151#comment-13850151 ] Jian He commented on YARN-1121: --- Oh, the reason I added the blockNewEvents check is that I don't want notify() to be called every time in the normal run if the queue is empty. That can possibly create more overhead. And only to be called if dispatcher is stopping. And In this scenario, I also think the sync should be outside the while loop to protect drained variable? > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850174#comment-13850174 ] Jian He commented on YARN-1121: --- bq. I also think the sync should be outside the while loop to protect drained variable? Disregard this comment, given drained is a volatile, I think it's fine to put sync inside the while loop. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852043#comment-13852043
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619381/YARN-1121.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2692//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2692//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch,
> YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch,
> YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854819#comment-13854819 ] Bikas Saha commented on YARN-1121: -- Sorry. The while loop for the wait() condition should be inside the sync block I think. My bad. +1 after that. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, > YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, > YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855101#comment-13855101 ] Jian He commented on YARN-1121: --- New patch changed the while loop inside the sync. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch, > YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch, > YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855105#comment-13855105
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620066/YARN-1121.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2714//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2714//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.10.patch,
> YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch,
> YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch,
> YARN-1121.8.patch, YARN-1121.9.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855150#comment-13855150 ] Bikas Saha commented on YARN-1121: -- +1. My only remaining concern is what if there is an exception in the event thread that prevents the execution of the notify line. I think we should be fine since then the isThreadAlive() check will prevent serviceStop() from getting stuck. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch, > YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch, > YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857726#comment-13857726 ] Jian He commented on YARN-1121: --- Now that we have the isThreadAlive check, I also believe it will prevent serviceStop from getting stuck. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch, > YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch, > YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857915#comment-13857915
]
Vinod Kumar Vavilapalli commented on YARN-1121:
---
This is unnecessary locking to go into the heart of our dispatcher loop. Let's
do it in the loop only if drainEventsOnStop is enabled?
Even otherwise, I don't see how the code in the dispatcher loop is useful other
than optimizing for the 1 sec wait in stop. I think the main change is checking
in stop if the dispatcher thread is alive. Just doing that should be enough?
{code}
while (!drained && eventHandlingThread.isAlive()) {
Thread.sleep(1000);
}
{code}
This should take care of all races and pay a small 1 second cost during stop
for some race conditions. Instead of the locking cost for every event. Some
JVMs do bias away the lock, but it can be avoided altogether..
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.10.patch,
> YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch,
> YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch,
> YARN-1121.8.patch, YARN-1121.9.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857956#comment-13857956 ] Bikas Saha commented on YARN-1121: -- >From the patch is wasnt clear to me that the lock was in the main loop. I >should have applied the patch to the code and seen the adjoining code. I agree >it does not make sense to put a lock in the inner loop for every event. It may >not be a 1 sec wait. These are all state store operations to an external >system with an unbounded time to complete and one thread will be spinning all >that time. Perhaps we could only to the notify when blockNewEvents is set to >true. That would be when we are actually waiting for the drain signal. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch, > YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch, > YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859141#comment-13859141 ] Jian He commented on YARN-1121: --- Agree, made the change to check the blockNewEvents flag before acquiring the lock and calling notify, and added the comment also. > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, > YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, > YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[
https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859157#comment-13859157
]
Hadoop QA commented on YARN-1121:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620880/YARN-1121.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2763//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2763//console
This message is automatically generated.
> RMStateStore should flush all pending store events before closing
> -
>
> Key: YARN-1121
> URL: https://issues.apache.org/jira/browse/YARN-1121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Jian He
> Fix For: 2.4.0
>
> Attachments: YARN-1121.1.patch, YARN-1121.10.patch,
> YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch,
> YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch,
> YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch
>
>
> on serviceStop it should wait for all internal pending events to drain before
> stopping.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859227#comment-13859227 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4941 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4941/]) YARN-1121. Addendum patch. Fixed AsyncDispatcher hang issue during stop due to a race condition caused by the previous patch. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554344) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, > YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, > YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859437#comment-13859437 ] Hudson commented on YARN-1121: -- FAILURE: Integrated in Hadoop-Yarn-trunk #438 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/438/]) YARN-1121. Addendum patch. Fixed AsyncDispatcher hang issue during stop due to a race condition caused by the previous patch. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554344) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, > YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, > YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859486#comment-13859486 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1630 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1630/]) YARN-1121. Addendum patch. Fixed AsyncDispatcher hang issue during stop due to a race condition caused by the previous patch. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554344) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, > YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, > YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859507#comment-13859507 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1655 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1655/]) YARN-1121. Addendum patch. Fixed AsyncDispatcher hang issue during stop due to a race condition caused by the previous patch. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1554344) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java > RMStateStore should flush all pending store events before closing > - > > Key: YARN-1121 > URL: https://issues.apache.org/jira/browse/YARN-1121 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1121.1.patch, YARN-1121.10.patch, > YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, > YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, > YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch > > > on serviceStop it should wait for all internal pending events to drain before > stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
