[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[
https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900109#comment-13900109
]
Hadoop QA commented on YARN-1551:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628673/YARN-1551.v06.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3094//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3094//console
This message is automatically generated.
> Allow user-specified reason for killApplication
> ---
>
> Key: YARN-1551
> URL: https://issues.apache.org/jira/browse/YARN-1551
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch,
> YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch,
> YARN-1551.v06.patch, YARN-1551.v06.patch
>
>
> This completes MAPREDUCE-5648
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041
[
https://issues.apache.org/jira/browse/YARN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900078#comment-13900078
]
Hadoop QA commented on YARN-1726:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628678/YARN-1726.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3095//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3095//console
This message is automatically generated.
> ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced
> in YARN-1041
>
>
> Key: YARN-1726
> URL: https://issues.apache.org/jira/browse/YARN-1726
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: YARN-1726.patch
>
>
> The YARN scheduler simulator failed when running Fair Scheduler, due to
> AbstractYarnScheduler introduced in YARN-1041. The ResourceSchedulerWrapper
> should inherit AbstractYarnScheduler, instead of implementing
> ResourceScheduler interface directly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900068#comment-13900068
]
Wei Yan commented on YARN-1296:
---
Re-look the code. Do we really need to add a lock to lock the
schedulerAllocateTimer? schedulerAllocateTimer is only updated in the
allocate() function.
> schedulerAllocateTimer is accessed without holding samplerLock in
> ResourceSchedulerWrapper
> --
>
> Key: YARN-1296
> URL: https://issues.apache.org/jira/browse/YARN-1296
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: yarn-1296-v1.patch
>
>
> Here is related code:
> {code}
> public Allocation allocate(ApplicationAttemptId attemptId,
> List resourceRequests,
> List containerIds,
> List strings, List strings2) {
> if (metricsON) {
> final Timer.Context context = schedulerAllocateTimer.time();
> {code}
> samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1724) Race condition in Fair Scheduler when continuous scheduling is turned on
[
https://issues.apache.org/jira/browse/YARN-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900057#comment-13900057
]
Hadoop QA commented on YARN-1724:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628663/YARN-1724.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3092//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3092//console
This message is automatically generated.
> Race condition in Fair Scheduler when continuous scheduling is turned on
> -
>
> Key: YARN-1724
> URL: https://issues.apache.org/jira/browse/YARN-1724
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-1724.patch
>
>
> If nodes resource allocations change during
> Collections.sort(nodeIdList, nodeAvailableResourceComparator);
> we'll hit:
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups
[
https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900056#comment-13900056
]
Hadoop QA commented on YARN-1297:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628664/YARN-1297-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3093//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3093//console
This message is automatically generated.
> Miscellaneous Fair Scheduler speedups
> -
>
> Key: YARN-1297
> URL: https://issues.apache.org/jira/browse/YARN-1297
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-1297-1.patch, YARN-1297-2.patch, YARN-1297.patch,
> YARN-1297.patch
>
>
> I ran the Fair Scheduler's core scheduling loop through a profiler to and
> identified a bunch of minimally invasive changes that can shave off a few
> milliseconds.
> The main one is demoting a couple INFO log messages to DEBUG, which brought
> my benchmark down from 16000 ms to 6000.
> A few others (which had way less of an impact) were
> * Most of the time in comparisons was being spent in Math.signum. I switched
> this to direct ifs and elses and it halved the percent of time spent in
> comparisons.
> * I removed some unnecessary instantiations of Resource objects
> * I made it so that queues' usage wasn't calculated from the applications up
> each time getResourceUsage was called.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041
[ https://issues.apache.org/jira/browse/YARN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1726: --- Description: The YARN scheduler simulator failed when running Fair Scheduler, due to AbstractYarnScheduler introduced in YARN-1041. The ResourceSchedulerWrapper should inherit AbstractYarnScheduler, instead of implementing ResourceScheduler interface directly. (was: The YARN scheduler simulator failed when running Fair Scheduler. This is because the new introduced AbstractYarnScheduler used for HA in YARN-1041. The ResourceSchedulerWrapper should inherit AbstractYarnScheduler, instead of implementing ResourceScheduler interface directly.) > ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced > in YARN-1041 > > > Key: YARN-1726 > URL: https://issues.apache.org/jira/browse/YARN-1726 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: YARN-1726.patch > > > The YARN scheduler simulator failed when running Fair Scheduler, due to > AbstractYarnScheduler introduced in YARN-1041. The ResourceSchedulerWrapper > should inherit AbstractYarnScheduler, instead of implementing > ResourceScheduler interface directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041
[ https://issues.apache.org/jira/browse/YARN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1726: -- Attachment: YARN-1726.patch > ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced > in YARN-1041 > > > Key: YARN-1726 > URL: https://issues.apache.org/jira/browse/YARN-1726 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: YARN-1726.patch > > > The YARN scheduler simulator failed when running Fair Scheduler. This is > because the new introduced AbstractYarnScheduler used for HA in YARN-1041. > The ResourceSchedulerWrapper should inherit AbstractYarnScheduler, instead of > implementing ResourceScheduler interface directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041
Wei Yan created YARN-1726: - Summary: ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041 Key: YARN-1726 URL: https://issues.apache.org/jira/browse/YARN-1726 Project: Hadoop YARN Issue Type: Bug Reporter: Wei Yan Assignee: Wei Yan Priority: Minor The YARN scheduler simulator failed when running Fair Scheduler. This is because the new introduced AbstractYarnScheduler used for HA in YARN-1041. The ResourceSchedulerWrapper should inherit AbstractYarnScheduler, instead of implementing ResourceScheduler interface directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1725) RM should provide an easier way for the app to reject a bad allocation
[ https://issues.apache.org/jira/browse/YARN-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900029#comment-13900029 ] Karthik Kambatla commented on YARN-1725: +1 to doing this. The last time there was discussion on something similar, there was a concern of too much state that in the RM. Doing it on the client side does get us past that. > RM should provide an easier way for the app to reject a bad allocation > -- > > Key: YARN-1725 > URL: https://issues.apache.org/jira/browse/YARN-1725 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha > > Currently, if the app gets a bad allocation then it can release the > container. However, the app now needs to request those resources again or > else the RM will not give it a new container in lieu of the one just > rejected. This makes the app writers life hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1515) Ability to dump the container threads and stop the containers in a single RPC
[ https://issues.apache.org/jira/browse/YARN-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900012#comment-13900012 ] Gera Shegalov commented on YARN-1515: - [~kkambatl] thanks for the suggestion! I am addressing it. > Ability to dump the container threads and stop the containers in a single RPC > - > > Key: YARN-1515 > URL: https://issues.apache.org/jira/browse/YARN-1515 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, nodemanager >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: YARN-1515.v01.patch, YARN-1515.v02.patch > > > This is needed to implement MAPREDUCE-5044 to enable thread diagnostics for > timed-out task attempts. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated YARN-1551: Attachment: YARN-1551.v06.patch the failed tests pass locally and there was no semantic change in v06. Re-uploading the patch assuming it's an intermittent issue. > Allow user-specified reason for killApplication > --- > > Key: YARN-1551 > URL: https://issues.apache.org/jira/browse/YARN-1551 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, > YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch, > YARN-1551.v06.patch, YARN-1551.v06.patch > > > This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1666) Make admin refreshNodes work across RM failover
[
https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1386#comment-1386
]
Hadoop QA commented on YARN-1666:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628631/YARN-1666.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3090//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3090//console
This message is automatically generated.
> Make admin refreshNodes work across RM failover
> ---
>
> Key: YARN-1666
> URL: https://issues.apache.org/jira/browse/YARN-1666
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch,
> YARN-1666.3.patch, YARN-1666.4.patch, YARN-1666.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1383#comment-1383
]
Hadoop QA commented on YARN-1525:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628660/YARN1525.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3091//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3091//console
This message is automatically generated.
> Web UI should redirect to active RM when HA is enabled.
> ---
>
> Key: YARN-1525
> URL: https://issues.apache.org/jira/browse/YARN-1525
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Cindy Li
> Attachments: YARN1525.patch, YARN1525.patch, YARN1525.patch.v1,
> YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch,
> YARN1525.v8.patch, YARN1525.v9.patch
>
>
> When failover happens, web UI should redirect to the current active rm.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1725) RM should provide an easier way for the app to reject a bad allocation
[ https://issues.apache.org/jira/browse/YARN-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1380#comment-1380 ] Sandy Ryza commented on YARN-1725: -- +1 to this. Can it be accomplished on the AMRMClient side? > RM should provide an easier way for the app to reject a bad allocation > -- > > Key: YARN-1725 > URL: https://issues.apache.org/jira/browse/YARN-1725 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha > > Currently, if the app gets a bad allocation then it can release the > container. However, the app now needs to request those resources again or > else the RM will not give it a new container in lieu of the one just > rejected. This makes the app writers life hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1725) RM should provide an easier way for the app to reject a bad allocation
Bikas Saha created YARN-1725: Summary: RM should provide an easier way for the app to reject a bad allocation Key: YARN-1725 URL: https://issues.apache.org/jira/browse/YARN-1725 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Currently, if the app gets a bad allocation then it can release the container. However, the app now needs to request those resources again or else the RM will not give it a new container in lieu of the one just rejected. This makes the app writers life hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899976#comment-13899976 ] Sandy Ryza commented on YARN-1297: -- Ran more rigorous benchmarks and the FairSharePolicy changes don't seem to have much impact. Uploading a patch with just the log message changes and the usage calculation changes. > Miscellaneous Fair Scheduler speedups > - > > Key: YARN-1297 > URL: https://issues.apache.org/jira/browse/YARN-1297 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1297-1.patch, YARN-1297-2.patch, YARN-1297.patch, > YARN-1297.patch > > > I ran the Fair Scheduler's core scheduling loop through a profiler to and > identified a bunch of minimally invasive changes that can shave off a few > milliseconds. > The main one is demoting a couple INFO log messages to DEBUG, which brought > my benchmark down from 16000 ms to 6000. > A few others (which had way less of an impact) were > * Most of the time in comparisons was being spent in Math.signum. I switched > this to direct ifs and elses and it halved the percent of time spent in > comparisons. > * I removed some unnecessary instantiations of Resource objects > * I made it so that queues' usage wasn't calculated from the applications up > each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1297) Miscellaneous Fair Scheduler speedups
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1297: - Attachment: YARN-1297-2.patch > Miscellaneous Fair Scheduler speedups > - > > Key: YARN-1297 > URL: https://issues.apache.org/jira/browse/YARN-1297 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1297-1.patch, YARN-1297-2.patch, YARN-1297.patch, > YARN-1297.patch > > > I ran the Fair Scheduler's core scheduling loop through a profiler to and > identified a bunch of minimally invasive changes that can shave off a few > milliseconds. > The main one is demoting a couple INFO log messages to DEBUG, which brought > my benchmark down from 16000 ms to 6000. > A few others (which had way less of an impact) were > * Most of the time in comparisons was being spent in Math.signum. I switched > this to direct ifs and elses and it halved the percent of time spent in > comparisons. > * I removed some unnecessary instantiations of Resource objects > * I made it so that queues' usage wasn't calculated from the applications up > each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1724) Race condition in Fair Scheduler when continuous scheduling is turned on
[ https://issues.apache.org/jira/browse/YARN-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1724: - Attachment: YARN-1724.patch > Race condition in Fair Scheduler when continuous scheduling is turned on > - > > Key: YARN-1724 > URL: https://issues.apache.org/jira/browse/YARN-1724 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1724.patch > > > If nodes resource allocations change during > Collections.sort(nodeIdList, nodeAvailableResourceComparator); > we'll hit: > java.lang.IllegalArgumentException: Comparison method violates its general > contract! -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[
https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899971#comment-13899971
]
Hadoop QA commented on YARN-1551:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628624/YARN-1551.v06.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestUberAM
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3087//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3087//console
This message is automatically generated.
> Allow user-specified reason for killApplication
> ---
>
> Key: YARN-1551
> URL: https://issues.apache.org/jira/browse/YARN-1551
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch,
> YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch,
> YARN-1551.v06.patch
>
>
> This completes MAPREDUCE-5648
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cindy Li updated YARN-1525: --- Attachment: YARN1525.patch Made changes w.r.t Karthik's comment. @Karthik, The test couldn't be moved to testRMHA because it needs to use MiniYarnCluster which has been setup well in testRMFailover, but not used at all in testRMHA. It would be nice if you can test in a secure cluster. Thanks. > Web UI should redirect to active RM when HA is enabled. > --- > > Key: YARN-1525 > URL: https://issues.apache.org/jira/browse/YARN-1525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Cindy Li > Attachments: YARN1525.patch, YARN1525.patch, YARN1525.patch.v1, > YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, > YARN1525.v8.patch, YARN1525.v9.patch > > > When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1692:
--
Attachment: yarn-1692-branch-2.3.patch
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.4.0
>
> Attachments: yarn-1692-branch-2.3.patch, yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1692:
--
Attachment: (was: yarn-1692-branch-2.3.patch)
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.4.0
>
> Attachments: yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1692:
--
Attachment: yarn-1692-branch-2.3.patch
This should apply for branch-2.3. It seems that the class in 2.3 is named
SchedulerApplication. Otherwise, the fix is identical.
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.4.0
>
> Attachments: yarn-1692-branch-2.3.patch, yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1724) Race condition in Fair Scheduler when continuous scheduling is turned on
Sandy Ryza created YARN-1724: Summary: Race condition in Fair Scheduler when continuous scheduling is turned on Key: YARN-1724 URL: https://issues.apache.org/jira/browse/YARN-1724 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza If nodes resource allocations change during Collections.sort(nodeIdList, nodeAvailableResourceComparator); we'll hit: java.lang.IllegalArgumentException: Comparison method violates its general contract! -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
Bikas Saha created YARN-1723: Summary: AMRMClientAsync missing blacklist addition and removal functionality Key: YARN-1723 URL: https://issues.apache.org/jira/browse/YARN-1723 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
[ https://issues.apache.org/jira/browse/YARN-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-1723: - Fix Version/s: 2.4.0 > AMRMClientAsync missing blacklist addition and removal functionality > > > Key: YARN-1723 > URL: https://issues.apache.org/jira/browse/YARN-1723 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Bikas Saha > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899933#comment-13899933
]
Sangjin Lee commented on YARN-1692:
---
Hmm, I thought that refactoring had already happened in branch-2.3, but I guess
I was mistaken. I'll post a separate patch for branch-2.3.
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.4.0
>
> Attachments: yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
[ https://issues.apache.org/jira/browse/YARN-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-1723: - Affects Version/s: 2.2.0 > AMRMClientAsync missing blacklist addition and removal functionality > > > Key: YARN-1723 > URL: https://issues.apache.org/jira/browse/YARN-1723 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Bikas Saha > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1722) AMRMProtocol should have a way of getting all the nodes in the cluster
Bikas Saha created YARN-1722: Summary: AMRMProtocol should have a way of getting all the nodes in the cluster Key: YARN-1722 URL: https://issues.apache.org/jira/browse/YARN-1722 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Bikas Saha Fix For: 2.4.0 There is no way for an AM to find out the names of all the nodes in the cluster via the AMRMProtocol. An AM can only at best ask for containers at * location. The only way to get that information is via the ClientRMProtocol but that is secured by Kerberos or RMDelegationToken while the AM has an AMRMToken. This is a pretty important piece of missing functionality. There are other jiras opened about getting cluster topology etc. but they havent been addressed due to a clear definition of cluster topology perhaps. Adding a means to at least get the node information would be a good first step. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899909#comment-13899909
]
Wei Yan commented on YARN-1296:
---
+1, the patch looks good to me. Looks also need to hold lock in the
handle(SchedulerEvent schedulerEvent) function.
> schedulerAllocateTimer is accessed without holding samplerLock in
> ResourceSchedulerWrapper
> --
>
> Key: YARN-1296
> URL: https://issues.apache.org/jira/browse/YARN-1296
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: yarn-1296-v1.patch
>
>
> Here is related code:
> {code}
> public Allocation allocate(ApplicationAttemptId attemptId,
> List resourceRequests,
> List containerIds,
> List strings, List strings2) {
> if (metricsON) {
> final Timer.Context context = schedulerAllocateTimer.time();
> {code}
> samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[
https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899905#comment-13899905
]
Hadoop QA commented on YARN-1721:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628614/YARN-1721.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3089//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3089//console
This message is automatically generated.
> When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
> -
>
> Key: YARN-1721
> URL: https://issues.apache.org/jira/browse/YARN-1721
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-1721.patch
>
>
> FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that
> allocate() can't be modifying it at the same time.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1570) Formatting the lines within 80 chars in YarnCommands.apt.vm
[
https://issues.apache.org/jira/browse/YARN-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899890#comment-13899890
]
Hadoop QA commented on YARN-1570:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628620/YARN-1570.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3088//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3088//console
This message is automatically generated.
> Formatting the lines within 80 chars in YarnCommands.apt.vm
> ---
>
> Key: YARN-1570
> URL: https://issues.apache.org/jira/browse/YARN-1570
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.2.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Attachments: YARN-1570.patch
>
>
> In YarnCommands.apt.vm, there are some lines longer than 80 characters.
> For example:
> {code}
> Yarn commands are invoked by the bin/yarn script. Running the yarn script
> without any arguments prints the description for all commands.
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899873#comment-13899873
]
Hudson commented on YARN-1578:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5160 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5160/])
YARN-1578. Fixed reading incomplete application attempt and container data in
FileSystemApplicationHistoryStore. Contributed by Shinichi Yamashita. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567816)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/FileSystemApplicationHistoryStore.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java
> Fix how to read history file in FileSystemApplicationHistoryStore
> -
>
> Key: YARN-1578
> URL: https://issues.apache.org/jira/browse/YARN-1578
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-321
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Fix For: 2.4.0
>
> Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
> YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
> screenshot.png, screenshot2.pdf
>
>
> I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
> After the job end and when I accessed Web UI of HistoryServer, it displayed
> "500". And HistoryServer daemon log was output as follows.
> {code}
> 2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI:
> /applicationhistory/appattempt/appattempt_1389146249925_0008_01
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> (snip...)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
> at
> org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
> (snip...)
> {code}
> I confirmed that there was container which was not finished from
> ApplicationHistory file.
> In ResourceManager daemon log, ResourceManager reserved this container, but
> did not allocate it.
> When FileSystemApplicationHistoryStore reads container information without
> finish data in history file, this problem occurs.
> In consideration of the case which there is not finish data, we should fix
> how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1345) Removing FINAL_SAVING from YarnApplicationAttemptState
[ https://issues.apache.org/jira/browse/YARN-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899872#comment-13899872 ] Hudson commented on YARN-1345: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5160 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5160/]) YARN-1345. Remove FINAL_SAVING state from YarnApplicationAttemptState. Contributed by Zhijie Shen (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567820) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationAttemptState.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java > Removing FINAL_SAVING from YarnApplicationAttemptState > -- > > Key: YARN-1345 > URL: https://issues.apache.org/jira/browse/YARN-1345 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-1345.1.patch, YARN-1345.2.patch > > > Whenever YARN-891 is done, we need to add the mapping of > RMAppAttemptState.FINAL_SAVING -> YarnApplicationAttemptState.FINAL_SAVING in > RMServerUtils#createApplicationAttemptState -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1345) Removing FINAL_SAVING from YarnApplicationAttemptState
[ https://issues.apache.org/jira/browse/YARN-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899865#comment-13899865 ] Jian He commented on YARN-1345: --- Committed to trunk and branch-2 , thanks Zhijie ! > Removing FINAL_SAVING from YarnApplicationAttemptState > -- > > Key: YARN-1345 > URL: https://issues.apache.org/jira/browse/YARN-1345 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-1345.1.patch, YARN-1345.2.patch > > > Whenever YARN-891 is done, we need to add the mapping of > RMAppAttemptState.FINAL_SAVING -> YarnApplicationAttemptState.FINAL_SAVING in > RMServerUtils#createApplicationAttemptState -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1666) Make admin refreshNodes work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899858#comment-13899858 ] Xuan Gong commented on YARN-1666: - add Private annotation to new APIs > Make admin refreshNodes work across RM failover > --- > > Key: YARN-1666 > URL: https://issues.apache.org/jira/browse/YARN-1666 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch, > YARN-1666.3.patch, YARN-1666.4.patch, YARN-1666.4.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1666) Make admin refreshNodes work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1666: Attachment: YARN-1666.4.patch > Make admin refreshNodes work across RM failover > --- > > Key: YARN-1666 > URL: https://issues.apache.org/jira/browse/YARN-1666 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch, > YARN-1666.3.patch, YARN-1666.4.patch, YARN-1666.4.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1685) [YARN-321] Logs link can be null so avoid NPE
[
https://issues.apache.org/jira/browse/YARN-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899837#comment-13899837
]
Zhijie Shen commented on YARN-1685:
---
Hi Mayank, some comments on the patch:
1. Please do not return "#" in the util method, which may not make sense to all
callers. Instead do null check in the related web page classes.
{code}
+ if(url == null){
+ return "#";
+ }
{code}
2. Please fix the test failures. And in TestRMContainerImpl, please assert both
the urls of container at running and completed states.
> [YARN-321] Logs link can be null so avoid NPE
> -
>
> Key: YARN-1685
> URL: https://issues.apache.org/jira/browse/YARN-1685
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Mayank Bansal
>Assignee: Mayank Bansal
> Fix For: YARN-321
>
> Attachments: YARN-1685-1.patch
>
>
> https://issues.apache.org/jira/browse/YARN-1413?focusedCommentId=13866416&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13866416
> https://issues.apache.org/jira/browse/YARN-1413?focusedCommentId=13866844&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13866844
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899833#comment-13899833 ] Gera Shegalov commented on YARN-1551: - s/v07/v06 in previous comment. > Allow user-specified reason for killApplication > --- > > Key: YARN-1551 > URL: https://issues.apache.org/jira/browse/YARN-1551 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, > YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch, > YARN-1551.v06.patch > > > This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated YARN-1551: Attachment: YARN-1551.v06.patch Thanks [~kkambatl] for the review. Getting rid of the redundant branching in v07.patch > Allow user-specified reason for killApplication > --- > > Key: YARN-1551 > URL: https://issues.apache.org/jira/browse/YARN-1551 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, > YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch, > YARN-1551.v06.patch > > > This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1570) Formatting the lines within 80 chars in YarnCommands.apt.vm
[
https://issues.apache.org/jira/browse/YARN-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-1570:
Attachment: YARN-1570.patch
Attaching a patch.
> Formatting the lines within 80 chars in YarnCommands.apt.vm
> ---
>
> Key: YARN-1570
> URL: https://issues.apache.org/jira/browse/YARN-1570
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.2.0
>Reporter: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Attachments: YARN-1570.patch
>
>
> In YarnCommands.apt.vm, there are some lines longer than 80 characters.
> For example:
> {code}
> Yarn commands are invoked by the bin/yarn script. Running the yarn script
> without any arguments prints the description for all commands.
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899819#comment-13899819
]
Zhijie Shen commented on YARN-1578:
---
Committed to trunk and branch-2. Thanks, [~sinchii]!
> Fix how to read history file in FileSystemApplicationHistoryStore
> -
>
> Key: YARN-1578
> URL: https://issues.apache.org/jira/browse/YARN-1578
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-321
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
> YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
> screenshot.png, screenshot2.pdf
>
>
> I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
> After the job end and when I accessed Web UI of HistoryServer, it displayed
> "500". And HistoryServer daemon log was output as follows.
> {code}
> 2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI:
> /applicationhistory/appattempt/appattempt_1389146249925_0008_01
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> (snip...)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
> at
> org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
> (snip...)
> {code}
> I confirmed that there was container which was not finished from
> ApplicationHistory file.
> In ResourceManager daemon log, ResourceManager reserved this container, but
> did not allocate it.
> When FileSystemApplicationHistoryStore reads container information without
> finish data in history file, this problem occurs.
> In consideration of the case which there is not finish data, we should fix
> how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899809#comment-13899809 ] Sandy Ryza commented on YARN-1721: -- My above comment does not appear to be issue, as we never take locks out on the queues. (This is OK because of the coarser-grained synchronization) > When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp > - > > Key: YARN-1721 > URL: https://issues.apache.org/jira/browse/YARN-1721 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1721.patch > > > FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that > allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1721: - Attachment: YARN-1721.patch > When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp > - > > Key: YARN-1721 > URL: https://issues.apache.org/jira/browse/YARN-1721 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1721.patch > > > FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that > allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899786#comment-13899786
]
Hudson commented on YARN-1692:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5159 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5159/])
Move YARN-1692 in CHANGES.txt (sandy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567793)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
YARN-1692. ConcurrentModificationException in fair scheduler AppSchedulable
(Sangjin Lee via Sandy Ryza) (sandy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567788)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.4.0
>
> Attachments: yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1666) Make admin refreshNodes work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1666: Attachment: YARN-1666.4.patch change ConfigurationProvider#getConfigurationInputStream to return InputStream. add RM_CONFIGURATION_FILES in YarnConfiguration which contains hadoop-policy.xml, yarn-site.xml, capacity-scheduler.xml, and core-site.xml. Also, HostsFileReader can take inputStream as paramter > Make admin refreshNodes work across RM failover > --- > > Key: YARN-1666 > URL: https://issues.apache.org/jira/browse/YARN-1666 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1666.1.patch, YARN-1666.2.patch, YARN-1666.2.patch, > YARN-1666.3.patch, YARN-1666.4.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1531) True up yarn command documentation
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899785#comment-13899785
]
Hudson commented on YARN-1531:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5159 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5159/])
YARN-1531. True up yarn command documentation (Akira Ajisaka via kasha) (kasha:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567775)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm
> True up yarn command documentation
> --
>
> Key: YARN-1531
> URL: https://issues.apache.org/jira/browse/YARN-1531
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: documentaion
> Fix For: 2.4.0
>
> Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
>
>
> There are some options which are not written to Yarn Command document.
> For example, "yarn rmadmin" command options are as follows:
> {code}
> Usage: yarn rmadmin
>-refreshQueues
>-refreshNodes
>-refreshSuperUserGroupsConfiguration
>-refreshUserToGroupsMappings
>-refreshAdminAcls
>-refreshServiceAcl
>-getGroups [username]
>-help [cmd]
>-transitionToActive
>-transitionToStandby
>-failover [--forcefence] [--forceactive]
>-getServiceState
>-checkHealth
> {code}
> But some of the new options such as "-getGroups", "-transitionToActive", and
> "-transitionToStandby" are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1578) Fix how to read history file in FileSystemApplicationHistoryStore
[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899783#comment-13899783
]
Zhijie Shen commented on YARN-1578:
---
+1 for the latest patch. Will commit it
> Fix how to read history file in FileSystemApplicationHistoryStore
> -
>
> Key: YARN-1578
> URL: https://issues.apache.org/jira/browse/YARN-1578
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-321
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: YARN-1578-2.patch, YARN-1578-3.patch, YARN-1578-4.patch,
> YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
> screenshot.png, screenshot2.pdf
>
>
> I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
> After the job end and when I accessed Web UI of HistoryServer, it displayed
> "500". And HistoryServer daemon log was output as follows.
> {code}
> 2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI:
> /applicationhistory/appattempt/appattempt_1389146249925_0008_01
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> (snip...)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
> at
> org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
> (snip...)
> {code}
> I confirmed that there was container which was not finished from
> ApplicationHistory file.
> In ResourceManager daemon log, ResourceManager reserved this container, but
> did not allocate it.
> When FileSystemApplicationHistoryStore reads container information without
> finish data in history file, this problem occurs.
> In consideration of the case which there is not finish data, we should fix
> how to read history file in FileSystemApplicationHistoryStore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899729#comment-13899729 ] Sandy Ryza commented on YARN-1721: -- To avoid deadlocks, we'll need to make sure that if we lock queues in move, we lock them before the apps they run in. > When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp > - > > Key: YARN-1721 > URL: https://issues.apache.org/jira/browse/YARN-1721 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that > allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1721) When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp
Sandy Ryza created YARN-1721: Summary: When moving app between queues in Fair Scheduler, grab lock on FSSchedulerApp Key: YARN-1721 URL: https://issues.apache.org/jira/browse/YARN-1721 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza FairScheduler.moveApplication should grab lock on FSSchedulerApp, so that allocate() can't be modifying it at the same time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899718#comment-13899718
]
Sandy Ryza commented on YARN-1692:
--
+1
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899697#comment-13899697 ] Karthik Kambatla commented on YARN-1551: My previous comment applies to the latest patch as well. Otherwise, +1. [~vinodkv] - do you have any other comments. Otherwise, can I go ahead and commit this later tomorrow? > Allow user-specified reason for killApplication > --- > > Key: YARN-1551 > URL: https://issues.apache.org/jira/browse/YARN-1551 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, > YARN-1551.v03.patch, YARN-1551.v04.patch, YARN-1551.v05.patch > > > This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1531) True up yarn command documentation
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1531:
---
Summary: True up yarn command documentation (was: Update yarn command
documentation)
> True up yarn command documentation
> --
>
> Key: YARN-1531
> URL: https://issues.apache.org/jira/browse/YARN-1531
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: documentaion
> Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
>
>
> There are some options which are not written to Yarn Command document.
> For example, "yarn rmadmin" command options are as follows:
> {code}
> Usage: yarn rmadmin
>-refreshQueues
>-refreshNodes
>-refreshSuperUserGroupsConfiguration
>-refreshUserToGroupsMappings
>-refreshAdminAcls
>-refreshServiceAcl
>-getGroups [username]
>-help [cmd]
>-transitionToActive
>-transitionToStandby
>-failover [--forcefence] [--forceactive]
>-getServiceState
>-checkHealth
> {code}
> But some of the new options such as "-getGroups", "-transitionToActive", and
> "-transitionToStandby" are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1531) Update yarn command documentation
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1531:
---
Summary: Update yarn command documentation (was: Update yarn command
document)
> Update yarn command documentation
> -
>
> Key: YARN-1531
> URL: https://issues.apache.org/jira/browse/YARN-1531
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: documentaion
> Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
>
>
> There are some options which are not written to Yarn Command document.
> For example, "yarn rmadmin" command options are as follows:
> {code}
> Usage: yarn rmadmin
>-refreshQueues
>-refreshNodes
>-refreshSuperUserGroupsConfiguration
>-refreshUserToGroupsMappings
>-refreshAdminAcls
>-refreshServiceAcl
>-getGroups [username]
>-help [cmd]
>-transitionToActive
>-transitionToStandby
>-failover [--forcefence] [--forceactive]
>-getServiceState
>-checkHealth
> {code}
> But some of the new options such as "-getGroups", "-transitionToActive", and
> "-transitionToStandby" are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1531) Update yarn command document
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899676#comment-13899676
]
Karthik Kambatla commented on YARN-1531:
+1. Committing this.
> Update yarn command document
>
>
> Key: YARN-1531
> URL: https://issues.apache.org/jira/browse/YARN-1531
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: documentaion
> Attachments: YARN-1531.2.patch, YARN-1531.3.patch, YARN-1531.patch
>
>
> There are some options which are not written to Yarn Command document.
> For example, "yarn rmadmin" command options are as follows:
> {code}
> Usage: yarn rmadmin
>-refreshQueues
>-refreshNodes
>-refreshSuperUserGroupsConfiguration
>-refreshUserToGroupsMappings
>-refreshAdminAcls
>-refreshServiceAcl
>-getGroups [username]
>-help [cmd]
>-transitionToActive
>-transitionToStandby
>-failover [--forcefence] [--forceactive]
>-getServiceState
>-checkHealth
> {code}
> But some of the new options such as "-getGroups", "-transitionToActive", and
> "-transitionToStandby" are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899668#comment-13899668
]
Karthik Kambatla commented on YARN-1525:
Thanks Cindy. Thanks for your patience through the reviews.
Functionally, the patch looks good. Some comments on the cosmetics:
* The following indentation needs to be fixed:
{code}
@Override
public boolean isStandbyMode() {
{code}
* Setting the redirect path in WebApp#isStandbyMode() is a little misleading.
Can we leave isStandbyMode() to only report whether the RM is in standby mode,
and may be getRedirectPath() can take care of constructing the path if
required.
* Rename RMWebApp#findRedirectPath to buildRedirectPath?
* Couple of lines longer than 80 chars
{code}
if (webApp.isStandbyMode() && !uri.equals("/" + webApp.name() +
"/cluster")) {
String redirectPath = webApp.getRedirectPath();
if (redirectPath != null && !redirectPath.isEmpty()) {
String redirectMsg =
"This is a standby resource manager, redirecting to the current
active one: "
+ redirectPath;
{code}
* TestRMFailover#testRMWebAppFailover
** Rename to testRMWebAppRedirect to capture the behavior better
** If it is not too much trouble, given the test doesn't do any failover, we
should probably move it to TestRMHA?
* Let us add Private-Unstable annotations to RMHAUtils so we can move the
methods or even remove this class if we don't see the code being reused.
I ll be glad to test the updated patch out on a secure cluster for verification.
> Web UI should redirect to active RM when HA is enabled.
> ---
>
> Key: YARN-1525
> URL: https://issues.apache.org/jira/browse/YARN-1525
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Cindy Li
> Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2,
> YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch,
> YARN1525.v9.patch
>
>
> When failover happens, web UI should redirect to the current active rm.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899650#comment-13899650 ] Zhijie Shen commented on YARN-1714: --- bq. 2. I'm not aware of any REST API does the same thing, can you give me some examples? You can have a look at RMWebServices#getApps. IMO, it's good to add all these filters in one round instead of one after the other, unless different filters really differs a lot in the way they're going to be presented. bq. 3. For now, users have to edit url, but we can also provide direct link where people can click to get in this page. Moreover, we can automatically take user name and call this new url directly in the future when turning on security. It's a good idea that in secured mode, the users login to the apps page that only shows their own app list. However, I still think some interactive method should be provided to users to filter apps, instead of editing URL directly. For example, I'm an Admin, and want to watch the applications from another user. Anyway, if your use case doesn't care about interaction on web UI, but the app list, the current REST API should already serve you well. Otherwise, we should make web UI more interactive. > Per user and per queue view in YARN RM > -- > > Key: YARN-1714 > URL: https://issues.apache.org/jira/browse/YARN-1714 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Siqi Li >Assignee: Siqi Li >Priority: Critical > Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, > YARN-1714.v3.patch > > > ResourceManager exposes either one or all jobs via WebUI. It would be good to > have filter for user so that see only their jobs. > Provide rest style url to access only user specified queue or user apps. > For instance, > http://hadoop-example.com:50030/cluster/user/toto > displays apps owned by toto > http://hadoop-example.com:50030/cluster/user/toto,glinda > displays apps owned by toto and glinda > http://hadoop-example.com:50030/cluster/queue/root.queue1 >displays apps in root.queue1 > http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 > displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899608#comment-13899608
]
Wei Yan commented on YARN-1296:
---
Sorry I missed this jira before.
I'll look into the patch. thanks.
> schedulerAllocateTimer is accessed without holding samplerLock in
> ResourceSchedulerWrapper
> --
>
> Key: YARN-1296
> URL: https://issues.apache.org/jira/browse/YARN-1296
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: yarn-1296-v1.patch
>
>
> Here is related code:
> {code}
> public Allocation allocate(ApplicationAttemptId attemptId,
> List resourceRequests,
> List containerIds,
> List strings, List strings2) {
> if (metricsON) {
> final Timer.Context context = schedulerAllocateTimer.time();
> {code}
> samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899603#comment-13899603 ] Cindy Li commented on YARN-1525: The test failure org.apache.hadoop.yarn.client.api.impl.TestNMClient is irrelevant. > Web UI should redirect to active RM when HA is enabled. > --- > > Key: YARN-1525 > URL: https://issues.apache.org/jira/browse/YARN-1525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Cindy Li > Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, > YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, > YARN1525.v9.patch > > > When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1158) ResourceManager UI can not show the logs of worker containers
[ https://issues.apache.org/jira/browse/YARN-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1158: -- Fix Version/s: (was: 2.3.0) Summary: ResourceManager UI can not show the logs of worker containers (was: ResourceManager UI has application stdout missing if application stdout is not in the same directory as AppMaster stdout) > ResourceManager UI can not show the logs of worker containers > - > > Key: YARN-1158 > URL: https://issues.apache.org/jira/browse/YARN-1158 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Tassapol Athiapinya > > Configure yarn-site.xml's yarn.nodemanager.local-dirs to multiple > directories. Turn on log aggregation. Run distributed shell application. If > an application writes AppMaster.stdout in one directory and stdout in another > directory. Goto ResourceManager web UI. Open up container logs. Only > AppMaster.stdout would appear. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899578#comment-13899578
]
Sandy Ryza commented on YARN-1296:
--
I'm not super familiar with this code, but [~ywskycn] might be?
> schedulerAllocateTimer is accessed without holding samplerLock in
> ResourceSchedulerWrapper
> --
>
> Key: YARN-1296
> URL: https://issues.apache.org/jira/browse/YARN-1296
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: yarn-1296-v1.patch
>
>
> Here is related code:
> {code}
> public Allocation allocate(ApplicationAttemptId attemptId,
> List resourceRequests,
> List containerIds,
> List strings, List strings2) {
> if (metricsON) {
> final Timer.Context context = schedulerAllocateTimer.time();
> {code}
> samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (YARN-1720) QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal
[ https://issues.apache.org/jira/browse/YARN-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved YARN-1720. Resolution: Duplicate > QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal > - > > Key: YARN-1720 > URL: https://issues.apache.org/jira/browse/YARN-1720 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Aditya Acharya >Assignee: Sandy Ryza > > The SecondaryGroupExistingQueue QueuePlacementRule's isTerminal() method > should always return false, not create, because it does not, in fact, create > a queue ever. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules
[ https://issues.apache.org/jira/browse/YARN-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899577#comment-13899577 ] Karthik Kambatla commented on YARN-1718: +1 > Fix a couple isTerminals in Fair Scheduler queue placement rules > - > > Key: YARN-1718 > URL: https://issues.apache.org/jira/browse/YARN-1718 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-1718.patch > > > SecondaryGroupExistingQueue and Default are incorrect -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899550#comment-13899550
]
Hadoop QA commented on YARN-1525:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628549/YARN1525.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3086//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3086//console
This message is automatically generated.
> Web UI should redirect to active RM when HA is enabled.
> ---
>
> Key: YARN-1525
> URL: https://issues.apache.org/jira/browse/YARN-1525
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Cindy Li
> Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2,
> YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch,
> YARN1525.v9.patch
>
>
> When failover happens, web UI should redirect to the current active rm.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899546#comment-13899546
]
Tsuyoshi OZAWA commented on YARN-1296:
--
[~sandyr], do you have any comment about this JIRA?
> schedulerAllocateTimer is accessed without holding samplerLock in
> ResourceSchedulerWrapper
> --
>
> Key: YARN-1296
> URL: https://issues.apache.org/jira/browse/YARN-1296
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: yarn-1296-v1.patch
>
>
> Here is related code:
> {code}
> public Allocation allocate(ApplicationAttemptId attemptId,
> List resourceRequests,
> List containerIds,
> List strings, List strings2) {
> if (metricsON) {
> final Timer.Context context = schedulerAllocateTimer.time();
> {code}
> samplerLock should be used to guard the access.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1158) ResourceManager UI has application stdout missing if application stdout is not in the same directory as AppMaster stdout
[ https://issues.apache.org/jira/browse/YARN-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899534#comment-13899534 ] Zhijie Shen commented on YARN-1158: --- Have some investigation in this problem. The root cause is that RM web UI only shows the log of AM container. For the logs of other containers, users need to go to the corresponding NM web UI to see it. The bad thing there's no links to redirect users. As AHS list the containers of the applications as well, and provide the link to the container logs, RM web UI is good to do the same thing to simply the way for users to look for container logs. > ResourceManager UI has application stdout missing if application stdout is > not in the same directory as AppMaster stdout > > > Key: YARN-1158 > URL: https://issues.apache.org/jira/browse/YARN-1158 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Tassapol Athiapinya > Fix For: 2.3.0 > > > Configure yarn-site.xml's yarn.nodemanager.local-dirs to multiple > directories. Turn on log aggregation. Run distributed shell application. If > an application writes AppMaster.stdout in one directory and stdout in another > directory. Goto ResourceManager web UI. Open up container logs. Only > AppMaster.stdout would appear. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899530#comment-13899530
]
Karthik Kambatla commented on YARN-1692:
Let me take a look later today.
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1717) Misc improvements to leveldb timeline store
[
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899512#comment-13899512
]
Hadoop QA commented on YARN-1717:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628544/YARN-1717.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3085//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3085//console
This message is automatically generated.
> Misc improvements to leveldb timeline store
> ---
>
> Key: YARN-1717
> URL: https://issues.apache.org/jira/browse/YARN-1717
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-1717.1.patch, YARN-1717.2.patch, YARN-1717.3.patch
>
>
> The leveldb timeline store implementation needs the following:
> * better documentation of its internal structures
> * braces for all control flow statements
> * simple locking to prevent issues related to concurrent writes
> * internal changes to enable deleting entities
> ** never overwrite existing primary filter entries
> ** add hidden reverse pointers to related entities
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1692) ConcurrentModificationException in fair scheduler AppSchedulable
[
https://issues.apache.org/jira/browse/YARN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899506#comment-13899506
]
Lohit Vijayarenu commented on YARN-1692:
+1 on the patch. Can anyone else review this as well.
> ConcurrentModificationException in fair scheduler AppSchedulable
>
>
> Key: YARN-1692
> URL: https://issues.apache.org/jira/browse/YARN-1692
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: yarn-1692.patch
>
>
> We saw a ConcurrentModificationException thrown in the fair scheduler:
> {noformat}
> 2014-02-07 01:40:01,978 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Exception in fair scheduler UpdateThread
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$ValueIterator.next(HashMap.java:954)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
> at java.lang.Thread.run(Thread.java:724)
> {noformat}
> The map that gets returned by FSSchedulerApp.getResourceRequests() are
> iterated on without proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cindy Li updated YARN-1525: --- Attachment: YARN1525.patch Karthik&Xuan, please take a look at the latest patch. I've removed the IDE generated format changes. > Web UI should redirect to active RM when HA is enabled. > --- > > Key: YARN-1525 > URL: https://issues.apache.org/jira/browse/YARN-1525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Cindy Li > Attachments: YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, > YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, > YARN1525.v9.patch > > > When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1717) Misc improvements to leveldb timeline store
[ https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1717: - Attachment: YARN-1717.3.patch Removed delete locks -- they're unnecessary in this implementation since deleting can only happen while ATS is offline. > Misc improvements to leveldb timeline store > --- > > Key: YARN-1717 > URL: https://issues.apache.org/jira/browse/YARN-1717 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1717.1.patch, YARN-1717.2.patch, YARN-1717.3.patch > > > The leveldb timeline store implementation needs the following: > * better documentation of its internal structures > * braces for all control flow statements > * simple locking to prevent issues related to concurrent writes > * internal changes to enable deleting entities > ** never overwrite existing primary filter entries > ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated YARN-1553:
-
Attachment: YARN-1553.008.patch
The v8 patch passes in the configuration in {{getRMDefaultPortNumber()}} so
that it can display the correct error message.
> Do not use HttpConfig.isSecure() in YARN
>
>
> Key: YARN-1553
> URL: https://issues.apache.org/jira/browse/YARN-1553
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
> YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
> YARN-1553.005.patch, YARN-1553.006.patch, YARN-1553.007.patch,
> YARN-1553.008.patch
>
>
> HDFS-5305 and related jira decide that each individual project will have
> their own configuration on http policy. {{HttpConfig.isSecure}} is a global
> static method which does not fit the design anymore. The same functionality
> should be moved into the YARN code base.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899458#comment-13899458 ] Siqi Li commented on YARN-1714: --- For question 1, yes we do have full text search over the app list. By doing this, we can filter the apps on the server side. So, user will not be bothered by other people's jobs. Also, once we need to add security, it's better to see the apps that you own. 2. I'm not aware of any REST API does the same thing, can you give me some examples? 3. For now, users have to edit url, but we can also provide direct link where people can click to get in this page. Moreover, we can automatically take user name and call this new url directly in the future when turning on security. 4. yes, we should reuse AppsBlock. > Per user and per queue view in YARN RM > -- > > Key: YARN-1714 > URL: https://issues.apache.org/jira/browse/YARN-1714 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Siqi Li >Assignee: Siqi Li >Priority: Critical > Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, > YARN-1714.v3.patch > > > ResourceManager exposes either one or all jobs via WebUI. It would be good to > have filter for user so that see only their jobs. > Provide rest style url to access only user specified queue or user apps. > For instance, > http://hadoop-example.com:50030/cluster/user/toto > displays apps owned by toto > http://hadoop-example.com:50030/cluster/user/toto,glinda > displays apps owned by toto and glinda > http://hadoop-example.com:50030/cluster/queue/root.queue1 >displays apps in root.queue1 > http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 > displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated YARN-1553:
-
Attachment: YARN-1553.007.patch
The v7 patch addresses Vinod's comments.
> Do not use HttpConfig.isSecure() in YARN
>
>
> Key: YARN-1553
> URL: https://issues.apache.org/jira/browse/YARN-1553
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
> YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
> YARN-1553.005.patch, YARN-1553.006.patch, YARN-1553.007.patch
>
>
> HDFS-5305 and related jira decide that each individual project will have
> their own configuration on http policy. {{HttpConfig.isSecure}} is a global
> static method which does not fit the design anymore. The same functionality
> should be moved into the YARN code base.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[
https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899446#comment-13899446
]
Hadoop QA commented on YARN-1417:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628531/YARN-1417.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3084//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3084//console
This message is automatically generated.
> RM may issue expired container tokens to AM while issuing new containers.
> -
>
> Key: YARN-1417
> URL: https://issues.apache.org/jira/browse/YARN-1417
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Jian He
>Priority: Blocker
> Attachments: YARN-1417.2.patch, YARN-1417.3.patch
>
>
> Today we create new container token when we create container in RM as a part
> of schedule cycle. However that container may get reserved or assigned. If
> the container gets reserved and remains like that (in reserved state) for
> more than container token expiry interval then RM will end up issuing
> container with expired token.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899448#comment-13899448
]
Haohui Mai commented on YARN-1553:
--
bq. In the few places that we used “//“, should we retain the old behavior and
explicitly use the URL with scheme?
This is a general way of handling links for both http and https. Previously
we've used your proposed scheme in the old HDFS UI, it was quite error-prone.
bq. getRMDefaultPortNumber() method should use the new method instead of using
RM_SERVICES_ADDRESS_CONF_KEYS_HTTP directly.
It is a static method so I don't quite see choosing between
{{RM_SERVICES_ADDRESS_CONF_KEYS_HTTP}} /
{{RM_SERVICES_ADDRESS_CONF_KEYS_HTTPS}} are possible. Do you have any idea?
> Do not use HttpConfig.isSecure() in YARN
>
>
> Key: YARN-1553
> URL: https://issues.apache.org/jira/browse/YARN-1553
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
> YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
> YARN-1553.005.patch, YARN-1553.006.patch
>
>
> HDFS-5305 and related jira decide that each individual project will have
> their own configuration on http policy. {{HttpConfig.isSecure}} is a global
> static method which does not fit the design anymore. The same functionality
> should be moved into the YARN code base.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[ https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1417: -- Attachment: YARN-1417.3.patch Patch that sets container token only when the container is pulled by the AM and adds the unit test. > RM may issue expired container tokens to AM while issuing new containers. > - > > Key: YARN-1417 > URL: https://issues.apache.org/jira/browse/YARN-1417 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Jian He >Priority: Blocker > Attachments: YARN-1417.2.patch, YARN-1417.3.patch > > > Today we create new container token when we create container in RM as a part > of schedule cycle. However that container may get reserved or assigned. If > the container gets reserved and remains like that (in reserved state) for > more than container token expiry interval then RM will end up issuing > container with expired token. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1625) mvn apache-rat:check outputs warning message in YARN-321 branch
[
https://issues.apache.org/jira/browse/YARN-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-1625:
--
Fix Version/s: (was: 3.0.0)
2.4.0
Setting the right fix-version.
> mvn apache-rat:check outputs warning message in YARN-321 branch
> ---
>
> Key: YARN-1625
> URL: https://issues.apache.org/jira/browse/YARN-1625
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-321
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Trivial
> Fix For: 2.4.0
>
> Attachments: YARN-1625.patch
>
>
> When I ran dev-support/test-patch.sh, following message output.
> {code}
> mvn apache-rat:check -DHadoopPatchProcess > /tmp/patchReleaseAuditOutput.txt
> 2>&1
> There appear to be 1 release audit warnings after applying the patch.
> {code}
> {code}
> !?
> /home/sinchii/git/YARN-321-test/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/applicationhistory/.keep
> Lines that start with ? in the release audit report indicate files that
> do not have an Apache license header.
> {code}
> To avoid release audit warning, it should fix pom.xml.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1613) Fix config name YARN_HISTORY_SERVICE_ENABLED
[ https://issues.apache.org/jira/browse/YARN-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1613: -- Fix Version/s: (was: 3.0.0) 2.4.0 Setting the right fix-version. > Fix config name YARN_HISTORY_SERVICE_ENABLED > > > Key: YARN-1613 > URL: https://issues.apache.org/jira/browse/YARN-1613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Akira AJISAKA > Labels: newbie > Fix For: 2.4.0 > > Attachments: YARN-1613.patch > > > YARN_HISTORY_SERVICE_ENABLED property name is "yarn.ahs..enabled", which is > wrong. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
[ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1530: - Attachment: application timeline design-20140210.pdf > [Umbrella] Store, manage and serve per-framework application-timeline data > -- > > Key: YARN-1530 > URL: https://issues.apache.org/jira/browse/YARN-1530 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli > Attachments: application timeline design-20140108.pdf, application > timeline design-20140116.pdf, application timeline design-20140130.pdf, > application timeline design-20140210.pdf > > > This is a sibling JIRA for YARN-321. > Today, each application/framework has to do store, and serve per-framework > data all by itself as YARN doesn't have a common solution. This JIRA attempts > to solve the storage, management and serving of per-framework data from > various applications, both running and finished. The aim is to change YARN to > collect and store data in a generic manner with plugin points for frameworks > to do their own thing w.r.t interpretation and serving. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899158#comment-13899158
]
Hudson commented on YARN-1719:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/])
YARN-1719. Fixed the root path related Jersey warnings produced in
ATSWebServices. Contributed by Billie Rinaldi. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java
> ATSWebServices produces jersey warnings
> ---
>
> Key: YARN-1719
> URL: https://issues.apache.org/jira/browse/YARN-1719
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Fix For: 2.4.0
>
> Attachments: YARN-1719.1.patch
>
>
> These don't appear to affect how the web services work, but the following
> warnings are logged:
> {noformat}
> WARNING: The following warnings have been detected with resource and/or
> provider
> classes:
> WARNING: A sub-resource method, public
> org.apache.hadoop.yarn.server.applicati
> onhistoryservice.webapp.ATSWebServices$AboutInfo
> org.apache.hadoop.yarn.server.a
> pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
> letRequest,javax.servlet.http.HttpServletResponse), with URI template, "/",
> is t
> reated as a resource method
> WARNING: A sub-resource method, public
> org.apache.hadoop.yarn.api.records.appt
> imeline.ATSPutErrors
> org.apache.hadoop.yarn.server.applicationhistoryservice.web
> app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
> let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
> ntities), with URI template, "/", is treated as a resource method
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899159#comment-13899159 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Protocol additions to allow moving apps between queues > -- > > Key: YARN-1496 > URL: https://issues.apache.org/jira/browse/YARN-1496 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, > YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899156#comment-13899156 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Fair Scheduler changes for moving apps between queues > - > > Key: YARN-1499 > URL: https://issues.apache.org/jira/browse/YARN-1499 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899161#comment-13899161 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Common scheduler changes for moving apps between queues > --- > > Key: YARN-1498 > URL: https://issues.apache.org/jira/browse/YARN-1498 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, > YARN-1498.patch, YARN-1498.patch > > > This JIRA is to track changes that aren't in particular schedulers but that > help them support moving apps between queues. In particular, it makes sure > that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line
[ https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899160#comment-13899160 ] Hudson commented on YARN-1497: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Expose moving apps between queues on the command line > - > > Key: YARN-1497 > URL: https://issues.apache.org/jira/browse/YARN-1497 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899157#comment-13899157 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > RM changes for moving apps between queues > - > > Key: YARN-1504 > URL: https://issues.apache.org/jira/browse/YARN-1504 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1504-1.patch, YARN-1504.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899153#comment-13899153 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1696 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1696/]) Fix YARN-1697 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567553) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567356) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java > NodeManager reports negative running containers > --- > > Key: YARN-1697 > URL: https://issues.apache.org/jira/browse/YARN-1697 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.4.0 > > Attachments: YARN-1697.patch > > > We're seeing the NodeManager metrics report a negative number of running > containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1641) ZK store should attempt a write periodically to ensure it is still Active
[ https://issues.apache.org/jira/browse/YARN-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899135#comment-13899135 ] Hudson commented on YARN-1641: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5154 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5154/]) YARN-1641. ZK store should attempt a write periodically to ensure it is still Active. (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567628) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java > ZK store should attempt a write periodically to ensure it is still Active > - > > Key: YARN-1641 > URL: https://issues.apache.org/jira/browse/YARN-1641 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.4.0 > > Attachments: yarn-1641-1.patch, yarn-1641-2.patch > > > Fencing in ZK store kicks in when the RM tries to write something to the > store. If the RM doesn't write anything to the store, it doesn't get fenced > and can continue to assume being the Active. > By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can > ensure it gets fenced. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899113#comment-13899113 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Common scheduler changes for moving apps between queues > --- > > Key: YARN-1498 > URL: https://issues.apache.org/jira/browse/YARN-1498 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, > YARN-1498.patch, YARN-1498.patch > > > This JIRA is to track changes that aren't in particular schedulers but that > help them support moving apps between queues. In particular, it makes sure > that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line
[ https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899112#comment-13899112 ] Hudson commented on YARN-1497: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Expose moving apps between queues on the command line > - > > Key: YARN-1497 > URL: https://issues.apache.org/jira/browse/YARN-1497 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899111#comment-13899111 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Protocol additions to allow moving apps between queues > -- > > Key: YARN-1496 > URL: https://issues.apache.org/jira/browse/YARN-1496 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, > YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899108#comment-13899108 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > RM changes for moving apps between queues > - > > Key: YARN-1504 > URL: https://issues.apache.org/jira/browse/YARN-1504 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1504-1.patch, YARN-1504.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899109#comment-13899109
]
Hudson commented on YARN-1719:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/])
YARN-1719. Fixed the root path related Jersey warnings produced in
ATSWebServices. Contributed by Billie Rinaldi. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java
> ATSWebServices produces jersey warnings
> ---
>
> Key: YARN-1719
> URL: https://issues.apache.org/jira/browse/YARN-1719
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Fix For: 2.4.0
>
> Attachments: YARN-1719.1.patch
>
>
> These don't appear to affect how the web services work, but the following
> warnings are logged:
> {noformat}
> WARNING: The following warnings have been detected with resource and/or
> provider
> classes:
> WARNING: A sub-resource method, public
> org.apache.hadoop.yarn.server.applicati
> onhistoryservice.webapp.ATSWebServices$AboutInfo
> org.apache.hadoop.yarn.server.a
> pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
> letRequest,javax.servlet.http.HttpServletResponse), with URI template, "/",
> is t
> reated as a resource method
> WARNING: A sub-resource method, public
> org.apache.hadoop.yarn.api.records.appt
> imeline.ATSPutErrors
> org.apache.hadoop.yarn.server.applicationhistoryservice.web
> app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
> let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
> ntities), with URI template, "/", is treated as a resource method
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899104#comment-13899104 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Fix YARN-1697 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567553) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567356) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java > NodeManager reports negative running containers > --- > > Key: YARN-1697 > URL: https://issues.apache.org/jira/browse/YARN-1697 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.4.0 > > Attachments: YARN-1697.patch > > > We're seeing the NodeManager metrics report a negative number of running > containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899107#comment-13899107 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1671 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1671/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Fair Scheduler changes for moving apps between queues > - > > Key: YARN-1499 > URL: https://issues.apache.org/jira/browse/YARN-1499 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899008#comment-13899008 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > RM changes for moving apps between queues > - > > Key: YARN-1504 > URL: https://issues.apache.org/jira/browse/YARN-1504 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1504-1.patch, YARN-1504.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899009#comment-13899009
]
Hudson commented on YARN-1719:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/479/])
YARN-1719. Fixed the root path related Jersey warnings produced in
ATSWebServices. Contributed by Billie Rinaldi. (zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java
> ATSWebServices produces jersey warnings
> ---
>
> Key: YARN-1719
> URL: https://issues.apache.org/jira/browse/YARN-1719
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Fix For: 2.4.0
>
> Attachments: YARN-1719.1.patch
>
>
> These don't appear to affect how the web services work, but the following
> warnings are logged:
> {noformat}
> WARNING: The following warnings have been detected with resource and/or
> provider
> classes:
> WARNING: A sub-resource method, public
> org.apache.hadoop.yarn.server.applicati
> onhistoryservice.webapp.ATSWebServices$AboutInfo
> org.apache.hadoop.yarn.server.a
> pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
> letRequest,javax.servlet.http.HttpServletResponse), with URI template, "/",
> is t
> reated as a resource method
> WARNING: A sub-resource method, public
> org.apache.hadoop.yarn.api.records.appt
> imeline.ATSPutErrors
> org.apache.hadoop.yarn.server.applicationhistoryservice.web
> app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
> let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
> ntities), with URI template, "/", is treated as a resource method
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899007#comment-13899007 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Fair Scheduler changes for moving apps between queues > - > > Key: YARN-1499 > URL: https://issues.apache.org/jira/browse/YARN-1499 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899012#comment-13899012 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Common scheduler changes for moving apps between queues > --- > > Key: YARN-1498 > URL: https://issues.apache.org/jira/browse/YARN-1498 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, > YARN-1498.patch, YARN-1498.patch > > > This JIRA is to track changes that aren't in particular schedulers but that > help them support moving apps between queues. In particular, it makes sure > that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899010#comment-13899010 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #479 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/479/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Protocol additions to allow moving apps between queues > -- > > Key: YARN-1496 > URL: https://issues.apache.org/jira/browse/YARN-1496 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 3.0.0 > > Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, > YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
