[jira] [Commented] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873123#comment-13873123 ] Tsuyoshi OZAWA commented on YARN-1474: -- Main updates are as follows: 1. Made AbstractYarnScheduler AbstractService. 2. reinitialize() API just executes reloading configuration or reset initialization, because this API is called from AdminService#refreshQueues(). 3. setRMContext() API is for initializing RMContext in schedulers. This is because we can only pass Configuration to serviceInit() API. 4. Scheduers' serviceInit(Configuration conf) execute initialization of configurations or threads. 5. Scheduler's serviceStart() start threads. 6. Scheduler's serviceStop() don't anything currently. Just calling super.serviceStop(). [~kkambatl] and [~vinodkv], do you have feedbacks? > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.4.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.2.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1474) Make schedulers services
[
https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873117#comment-13873117
]
Hadoop QA commented on YARN-1474:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623327/YARN-1474.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2894//console
This message is automatically generated.
> Make schedulers services
>
>
> Key: YARN-1474
> URL: https://issues.apache.org/jira/browse/YARN-1474
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: 2.4.0
>Reporter: Sandy Ryza
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-1474.1.patch, YARN-1474.2.patch
>
>
> Schedulers currently have a reinitialize but no start and stop. Fitting them
> into the YARN service model would make things more coherent.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1474: - Attachment: YARN-1474.2.patch Updated a patch based on Karthik's approach. Sorry for delaying. I needed to redesign reinitialize() API and design what we should do in serviceInit/serviceStop/serviceStart APIs to take this ticket. > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.4.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.2.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1609) Add Service Container type to NodeManager in YARN
[ https://issues.apache.org/jira/browse/YARN-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-1609: - Attachment: Add Service Container type to NodeManager in YARN-V1.pdf Attached a draft design doc, credit to [~lvm] and [~zjffdu] for this idea and design. Hope to get your thoughts about this, thanks! > Add Service Container type to NodeManager in YARN > - > > Key: YARN-1609 > URL: https://issues.apache.org/jira/browse/YARN-1609 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Add Service Container type to NodeManager in YARN-V1.pdf > > > From our work to support running OpenMPI on YARN (MAPREDUCE-2911), we found > that it’s important to have framework specific daemon process manage the > tasks on each node directly. The daemon process, most likely similar in other > frameworks as well, provides critical services to tasks running on that > node(for example “wireup”, spawn user process in large numbers at once etc). > In YARN, it’s hard, if not possible, to have the those processes to be > managed by YARN. > We propose to extend the container model on NodeManager side to support > “Service Container” to run/manage such framework daemon/services process. We > believe this is very useful to other application framework developers as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1609) Add Service Container type to NodeManager in YARN
Wangda Tan created YARN-1609: Summary: Add Service Container type to NodeManager in YARN Key: YARN-1609 URL: https://issues.apache.org/jira/browse/YARN-1609 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.2.0 Reporter: Wangda Tan Assignee: Wangda Tan >From our work to support running OpenMPI on YARN (MAPREDUCE-2911), we found >that it’s important to have framework specific daemon process manage the tasks >on each node directly. The daemon process, most likely similar in other >frameworks as well, provides critical services to tasks running on that >node(for example “wireup”, spawn user process in large numbers at once etc). >In YARN, it’s hard, if not possible, to have the those processes to be managed >by YARN. We propose to extend the container model on NodeManager side to support “Service Container” to run/manage such framework daemon/services process. We believe this is very useful to other application framework developers as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[
https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872958#comment-13872958
]
Hadoop QA commented on YARN-1608:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623287/yarn-1608-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2893//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2893//console
This message is automatically generated.
> LinuxContainerExecutor has a few DEBUG messages at INFO level
> -
>
> Key: YARN-1608
> URL: https://issues.apache.org/jira/browse/YARN-1608
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Trivial
> Labels: log
> Attachments: yarn-1608-1.patch
>
>
> LCE has a few INFO level log messages meant to be at debug level. In fact,
> they are logged both at INFO and DEBUG.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[ https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872938#comment-13872938 ] Alejandro Abdelnur commented on YARN-1608: -- +1 > LinuxContainerExecutor has a few DEBUG messages at INFO level > - > > Key: YARN-1608 > URL: https://issues.apache.org/jira/browse/YARN-1608 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Trivial > Labels: log > Attachments: yarn-1608-1.patch > > > LCE has a few INFO level log messages meant to be at debug level. In fact, > they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[ https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872937#comment-13872937 ] Alejandro Abdelnur commented on YARN-1608: -- +1 > LinuxContainerExecutor has a few DEBUG messages at INFO level > - > > Key: YARN-1608 > URL: https://issues.apache.org/jira/browse/YARN-1608 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Trivial > Labels: log > Attachments: yarn-1608-1.patch > > > LCE has a few INFO level log messages meant to be at debug level. In fact, > they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[ https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1608: --- Priority: Trivial (was: Minor) > LinuxContainerExecutor has a few DEBUG messages at INFO level > - > > Key: YARN-1608 > URL: https://issues.apache.org/jira/browse/YARN-1608 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Trivial > Labels: log > Attachments: yarn-1608-1.patch > > > LCE has a few INFO level log messages meant to be at debug level. In fact, > they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
[ https://issues.apache.org/jira/browse/YARN-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1608: --- Attachment: yarn-1608-1.patch Trivial patch. > LinuxContainerExecutor has a few DEBUG messages at INFO level > - > > Key: YARN-1608 > URL: https://issues.apache.org/jira/browse/YARN-1608 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Minor > Labels: log > Attachments: yarn-1608-1.patch > > > LCE has a few INFO level log messages meant to be at debug level. In fact, > they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1608) LinuxContainerExecutor has a few DEBUG messages at INFO level
Karthik Kambatla created YARN-1608: -- Summary: LinuxContainerExecutor has a few DEBUG messages at INFO level Key: YARN-1608 URL: https://issues.apache.org/jira/browse/YARN-1608 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor LCE has a few INFO level log messages meant to be at debug level. In fact, they are logged both at INFO and DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1607) TestRM expects the capacity scheduler
Sandy Ryza created YARN-1607: Summary: TestRM expects the capacity scheduler Key: YARN-1607 URL: https://issues.apache.org/jira/browse/YARN-1607 Project: Hadoop YARN Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza We should either explicitly set the Capacity Scheduler or make it scheduler-agnostic -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872786#comment-13872786 ] Hudson commented on YARN-1351: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5006 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5006/]) YARN-1351. Invalid string format in Fair Scheduler log warn message (Konstantin Weitz via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558597) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java > Invalid string format in Fair Scheduler log warn message > > > Key: YARN-1351 > URL: https://issues.apache.org/jira/browse/YARN-1351 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Konstantin Weitz >Assignee: Konstantin Weitz > Fix For: 2.4.0 > > Attachments: fixprnt.patch > > > While trying to print a warning, two values of the wrong type (Resource > instead of int) are passed into a String.format method call, leading to a > runtime exception, in the file: > _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. > The warning was intended to be printed whenever the resources don't fit into > each other, either because the number of virtual cores or the memory is too > small. I changed the %d's into %s, this way the warning will contain both the > cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1573) ZK store should pick a private password for root-node-acls
[
https://issues.apache.org/jira/browse/YARN-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872769#comment-13872769
]
Hadoop QA commented on YARN-1573:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623242/yarn-1573-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2891//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2891//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2891//console
This message is automatically generated.
> ZK store should pick a private password for root-node-acls
> --
>
> Key: YARN-1573
> URL: https://issues.apache.org/jira/browse/YARN-1573
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-1573-1.patch
>
>
> Currently, when HA is enabled, ZK store uses cluster-timestamp as the
> password for root node ACLs to give the Active RM exclusive access to the
> store. A more private value like a random number might be better.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1351: - Assignee: Konstantin Weitz > Invalid string format in Fair Scheduler log warn message > > > Key: YARN-1351 > URL: https://issues.apache.org/jira/browse/YARN-1351 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Konstantin Weitz >Assignee: Konstantin Weitz > Fix For: 2.4.0 > > Attachments: fixprnt.patch > > > While trying to print a warning, two values of the wrong type (Resource > instead of int) are passed into a String.format method call, leading to a > runtime exception, in the file: > _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. > The warning was intended to be printed whenever the resources don't fit into > each other, either because the number of virtual cores or the memory is too > small. I changed the %d's into %s, this way the warning will contain both the > cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872767#comment-13872767 ] Sandy Ryza commented on YARN-1351: -- (Sorry for the delay) > Invalid string format in Fair Scheduler log warn message > > > Key: YARN-1351 > URL: https://issues.apache.org/jira/browse/YARN-1351 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Konstantin Weitz >Assignee: Konstantin Weitz > Fix For: 2.4.0 > > Attachments: fixprnt.patch > > > While trying to print a warning, two values of the wrong type (Resource > instead of int) are passed into a String.format method call, leading to a > runtime exception, in the file: > _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. > The warning was intended to be printed whenever the resources don't fit into > each other, either because the number of virtual cores or the memory is too > small. I changed the %d's into %s, this way the warning will contain both the > cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1606) yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java
[
https://issues.apache.org/jira/browse/YARN-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872745#comment-13872745
]
Hadoop QA commented on YARN-1606:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623244/yarn-1606-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2892//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2892//console
This message is automatically generated.
> yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in
> yarn-default.xml and YarnConfiguration.java
> ---
>
> Key: YARN-1606
> URL: https://issues.apache.org/jira/browse/YARN-1606
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Labels: conf, newbie
> Attachments: yarn-1606-1.patch
>
>
> Looks like YARN-1029 changed at one location, but not the other.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1606) yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java
[ https://issues.apache.org/jira/browse/YARN-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872725#comment-13872725 ] Sandy Ryza commented on YARN-1606: -- +1 > yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in > yarn-default.xml and YarnConfiguration.java > --- > > Key: YARN-1606 > URL: https://issues.apache.org/jira/browse/YARN-1606 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Minor > Labels: conf, newbie > Attachments: yarn-1606-1.patch > > > Looks like YARN-1029 changed at one location, but not the other. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1595) Test failures on YARN-321 branch
[
https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872722#comment-13872722
]
Hadoop QA commented on YARN-1595:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623245/YARN-1595-20140115.1.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2890//console
This message is automatically generated.
> Test failures on YARN-321 branch
>
>
> Key: YARN-1595
> URL: https://issues.apache.org/jira/browse/YARN-1595
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Attachments: YARN-1595-20140115.1.txt, YARN-1595-20140115.txt
>
>
> mvn test doesn't pass on YARN-321 branch anymore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1587) [YARN-321] Merge Patch for YARN-321
[
https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872707#comment-13872707
]
Hadoop QA commented on YARN-1587:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623214/YARN-1587-20140115.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapred.TestYARNRunner
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryClientService
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2887//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2887//console
This message is automatically generated.
> [YARN-321] Merge Patch for YARN-321
> ---
>
> Key: YARN-1587
> URL: https://issues.apache.org/jira/browse/YARN-1587
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Mayank Bansal
>Assignee: Vinod Kumar Vavilapalli
> Fix For: YARN-321
>
> Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt,
> YARN-1587-20140115.txt, YARN-321-merge-1.patch
>
>
> Merge Patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1587) [YARN-321] Merge Patch for YARN-321
[
https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872705#comment-13872705
]
Hadoop QA commented on YARN-1587:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623214/YARN-1587-20140115.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-assemblies
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapred.TestYARNRunner
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryClientService
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
The following test timeouts occurred in
hadoop-assemblies
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2888//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2888//console
This message is automatically generated.
> [YARN-321] Merge Patch for YARN-321
> ---
>
> Key: YARN-1587
> URL: https://issues.apache.org/jira/browse/YARN-1587
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Mayank Bansal
>Assignee: Vinod Kumar Vavilapalli
> Fix For: YARN-321
>
> Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt,
> YARN-1587-20140115.txt, YARN-321-merge-1.patch
>
>
> Merge Patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872709#comment-13872709 ] Karthik Kambatla commented on YARN-1410: bq. But, I am not sure why we need to use these annotation in RM restart. Adding to what Bikas has already said. Consider RM restarting while an app submission is in-flight. The possibilities are exactly the same as in the case of failover. bq. Also, the AtMostOnce and Idempotent annotation are only used when RetryDecision is FAILOVER_AND_RETRY. So, this is another reason why we do not have them in RM restart case (For the RM restart, the valid RetryDecision is RETRY). Precisely my point. IMO, RM restart should not be directly using RetryUptoFixedTime or whatever it is currently using. Eventually, when RetryPolicy#shouldRetry is called, the annotations should be taken into account. > Handle client failover during 2 step client API's like app submission > - > > Key: YARN-1410 > URL: https://issues.apache.org/jira/browse/YARN-1410 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, > YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > App submission involves > 1) creating appId > 2) using that appId to submit an ApplicationSubmissionContext to the user. > The client may have obtained an appId from an RM, the RM may have failed > over, and the client may submit the app to the new RM. > Since the new RM has a different notion of cluster timestamp (used to create > app id) the new RM may reject the app submission resulting in unexpected > failure on the client side. > The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[ https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872698#comment-13872698 ] Konstantin Weitz commented on YARN-1351: Because of the simplicity of the fix, writing a test case would not be worthwhile. Because of the locality and size of the change, I used manual inspection to verify that the patch fixes the problem and does not introduce new problems. Sandy Ryza, do agree with this reasoning, or do you have any suggestions on how to improve the patch? > Invalid string format in Fair Scheduler log warn message > > > Key: YARN-1351 > URL: https://issues.apache.org/jira/browse/YARN-1351 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Konstantin Weitz > Attachments: fixprnt.patch > > > While trying to print a warning, two values of the wrong type (Resource > instead of int) are passed into a String.format method call, leading to a > runtime exception, in the file: > _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_. > The warning was intended to be printed whenever the resources don't fit into > each other, either because the number of virtual cores or the memory is too > small. I changed the %d's into %s, this way the warning will contain both the > cores and the memory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1351) Invalid string format in Fair Scheduler log warn message
[
https://issues.apache.org/jira/browse/YARN-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Weitz updated YARN-1351:
---
Description:
While trying to print a warning, two values of the wrong type (Resource instead
of int) are passed into a String.format method call, leading to a runtime
exception, in the file:
_trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
The warning was intended to be printed whenever the resources don't fit into
each other, either because the number of virtual cores or the memory is too
small. I changed the %d's into %s, this way the warning will contain both the
cores and the memory.
was:
While trying to print a warning, two values of the wrong type (Resource instead
of int) are passed into a String.format method call, leading to a runtime
exception, in the file:
_trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
The warning was intended to be printed whenever the resources don't fit into
each other, either because the number of virtual cores or the memory is too
small. I changed the %d's into %s, this way the warning will contain both the
cores and the memory.
Following the patch that fixes the issue:
Index:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
===
---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
(revision 1535589)
+++
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
(working copy)
@@ -471,7 +471,7 @@
if (maxQueueResources.containsKey(queueName) &&
minQueueResources.containsKey(queueName)
&& !Resources.fitsIn(minQueueResources.get(queueName),
maxQueueResources.get(queueName))) {
- LOG.warn(String.format("Queue %s has max resources %d less than min
resources %d",
+ LOG.warn(String.format("Queue %s has max resources %s less than min
resources %s",
queueName, maxQueueResources.get(queueName),
minQueueResources.get(queueName)));
}
}
> Invalid string format in Fair Scheduler log warn message
>
>
> Key: YARN-1351
> URL: https://issues.apache.org/jira/browse/YARN-1351
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Konstantin Weitz
> Attachments: fixprnt.patch
>
>
> While trying to print a warning, two values of the wrong type (Resource
> instead of int) are passed into a String.format method call, leading to a
> runtime exception, in the file:
> _trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
> The warning was intended to be printed whenever the resources don't fit into
> each other, either because the number of virtual cores or the memory is too
> small. I changed the %d's into %s, this way the warning will contain both the
> cores and the memory.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1595) Test failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1595: -- Attachment: YARN-1595-20140115.1.txt Tx for the review Zhijie! Here's an updated patch with the yarn-default.xml changes. > Test failures on YARN-321 branch > > > Key: YARN-1595 > URL: https://issues.apache.org/jira/browse/YARN-1595 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: YARN-1595-20140115.1.txt, YARN-1595-20140115.txt > > > mvn test doesn't pass on YARN-321 branch anymore. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1606) yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java
[ https://issues.apache.org/jira/browse/YARN-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1606: --- Attachment: yarn-1606-1.patch Trivial patch. > yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in > yarn-default.xml and YarnConfiguration.java > --- > > Key: YARN-1606 > URL: https://issues.apache.org/jira/browse/YARN-1606 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Minor > Labels: conf, newbie > Attachments: yarn-1606-1.patch > > > Looks like YARN-1029 changed at one location, but not the other. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1573) ZK store should pick a private password for root-node-acls
[ https://issues.apache.org/jira/browse/YARN-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1573: --- Attachment: yarn-1573-1.patch Trivial patch that uses a random number instead of cluster-timestamp. Even though it is unlikely, it is okay for the random numbers generated to be same as the usernames are different. We only care about a malicious user not being able to guess the password. Further, users can set their own ACLs for the root node for even better security. > ZK store should pick a private password for root-node-acls > -- > > Key: YARN-1573 > URL: https://issues.apache.org/jira/browse/YARN-1573 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-1573-1.patch > > > Currently, when HA is enabled, ZK store uses cluster-timestamp as the > password for root node ACLs to give the Active RM exclusive access to the > store. A more private value like a random number might be better. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1595) Test failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872635#comment-13872635 ] Zhijie Shen commented on YARN-1595: --- It seems that yarn-default.xml needs to be updated accordingly > Test failures on YARN-321 branch > > > Key: YARN-1595 > URL: https://issues.apache.org/jira/browse/YARN-1595 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: YARN-1595-20140115.txt > > > mvn test doesn't pass on YARN-321 branch anymore. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1606) yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java
Karthik Kambatla created YARN-1606: -- Summary: yarn.resourcemanager.zk-timeout-ms has inconsistent defaults in yarn-default.xml and YarnConfiguration.java Key: YARN-1606 URL: https://issues.apache.org/jira/browse/YARN-1606 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Looks like YARN-1029 changed at one location, but not the other. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1600) RM does not startup when security is enabled without spnego configured
[ https://issues.apache.org/jira/browse/YARN-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872586#comment-13872586 ] Haohui Mai commented on YARN-1600: -- I think at least in the short term that the earlier approach of YARN-1463 could work. We'll need to revisit the issue of supporting SPNEGO and third-party HTTP authentication in the longer term. > RM does not startup when security is enabled without spnego configured > -- > > Key: YARN-1600 > URL: https://issues.apache.org/jira/browse/YARN-1600 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Priority: Blocker > > We have a custom auth filter in front of our various UI pages that handles > user authentication. However currently the RM assumes that if security is > enabled then the user must have configured spnego as well for the RM web > pages which is not true in our case. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1605) Fix formatting issues with new module in YARN-321 branch
Vinod Kumar Vavilapalli created YARN-1605: - Summary: Fix formatting issues with new module in YARN-321 branch Key: YARN-1605 URL: https://issues.apache.org/jira/browse/YARN-1605 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli There are a bunch of formatting issues. I'm restricting myself for a sweep of all the files in the new module. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1595) Test failures on YARN-321 branch
[
https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872569#comment-13872569
]
Hadoop QA commented on YARN-1595:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623213/YARN-1595-20140115.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2889//console
This message is automatically generated.
> Test failures on YARN-321 branch
>
>
> Key: YARN-1595
> URL: https://issues.apache.org/jira/browse/YARN-1595
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Attachments: YARN-1595-20140115.txt
>
>
> mvn test doesn't pass on YARN-321 branch anymore.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1587) [YARN-321] Merge Patch for YARN-321
[ https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1587: -- Summary: [YARN-321] Merge Patch for YARN-321 (was: [YARN-321] MERGE Patch for YARN-321) > [YARN-321] Merge Patch for YARN-321 > --- > > Key: YARN-1587 > URL: https://issues.apache.org/jira/browse/YARN-1587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Vinod Kumar Vavilapalli > Fix For: YARN-321 > > Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt, > YARN-1587-20140115.txt, YARN-321-merge-1.patch > > > Merge Patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1587) [YARN-321] MERGE Patch for YARN-321
[ https://issues.apache.org/jira/browse/YARN-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1587: -- Attachment: YARN-1587-20140115.txt Updated patch = all YARN-321 changes + YARN-1595. > [YARN-321] MERGE Patch for YARN-321 > --- > > Key: YARN-1587 > URL: https://issues.apache.org/jira/browse/YARN-1587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Vinod Kumar Vavilapalli > Fix For: YARN-321 > > Attachments: YARN-1587-20140113.txt, YARN-1587-20140114.txt, > YARN-1587-20140115.txt, YARN-321-merge-1.patch > > > Merge Patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1595) Test failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1595: -- Attachment: YARN-1595-20140115.txt Here's a patch that addresses all test failures. The core change is to make application-history-service optional at both the RM as well as the client. > Test failures on YARN-321 branch > > > Key: YARN-1595 > URL: https://issues.apache.org/jira/browse/YARN-1595 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: YARN-1595-20140115.txt > > > mvn test doesn't pass on YARN-321 branch anymore. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1584) Support explicit failover when automatic failover is enabled
[ https://issues.apache.org/jira/browse/YARN-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872514#comment-13872514 ] Bikas Saha commented on YARN-1584: -- And what use case does this solve other than make RM1 become standby? Downside is a lot of churn in client that are currently using YARN as they switch from RM1 to RM2? If RM1 is going to go back and join the election anyways then why are we doing this? Secondly, I dont see how this works when there are 3 RMs in the leader election. I dont think we want to restrict our model to only support 2 RMs. Lastly, isnt this something that can be done entirely on the client side. Client would first call transitionToStandby() on all RM's other than the one that the admin want to make active. It will determine if the desired RM has become active (or timeout) and then make the remaining RMs' transitionToActive() and rejoin the election. > Support explicit failover when automatic failover is enabled > > > Key: YARN-1584 > URL: https://issues.apache.org/jira/browse/YARN-1584 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-1029 adds automatic failover support. However, users can't explicitly > ask for a failover from one RM to the other without stopping the other RM. > Stopping the RM until the other RM takes over and then restarting the first > RM is more involving and exposes the RM-ensemble to SPOF for a longer > duration. > It would be nice to allow explicit failover through yarn rmadmin -failover > command. > PS: HDFS supports -failover option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872461#comment-13872461 ] Zhijie Shen commented on YARN-1603: --- [~djp], would you mind merging it to YARN-321 branch as well? > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872467#comment-13872467 ] Zhijie Shen commented on YARN-1603: --- Oh, never mind. I'm guessing there're other commits need to be merged as well. We can take care of it. > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1584) Support explicit failover when automatic failover is enabled
[ https://issues.apache.org/jira/browse/YARN-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872384#comment-13872384 ] Karthik Kambatla commented on YARN-1584: bq. What exactly are the chain of events that we think should happen? The proposal is to replicate what ZKFC does to support -failover. When a user issues yarn rmadmin -failover rm* rm2, it means the admin wants to failover to RM2. The chain of events following that are as follows: # RM2 checks if it is already active. Returns if it is. # RM2 identifies who the current Active is. Assume RM1. # RM2 asks RM1 to quitElection, and rejoin after 'n' milliseconds. 'n' could be equal to RM_ZK_TIMEOUT_MS. # RM2 waits to become Active. # If it becomes Active, reaches out to RM1 asking it to joinElection() # If doesn't become Active in 'n' milliseconds, -failover fails. The ease comes from the above 6 steps (at least 3 - 3, 4, 5) being handled by a single admin command "-failover". On top of that, manually doing those steps takes longer than the RMs doing that for us with millisecond waits. Practically (tested it out), it is as fast as issuing the -transitionToActive or -transitionToStandby commands. > Support explicit failover when automatic failover is enabled > > > Key: YARN-1584 > URL: https://issues.apache.org/jira/browse/YARN-1584 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-1029 adds automatic failover support. However, users can't explicitly > ask for a failover from one RM to the other without stopping the other RM. > Stopping the RM until the other RM takes over and then restarting the first > RM is more involving and exposes the RM-ensemble to SPOF for a longer > duration. > It would be nice to allow explicit failover through yarn rmadmin -failover > command. > PS: HDFS supports -failover option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
[ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872300#comment-13872300 ] Robert Joseph Evans commented on YARN-1530: --- I agree that we need to think about load and plan for something that can handle at least 20x the current load but preferably 100x. However, I am not that sure that the load will be a huge problem at least for current MR clusters. We have seen very large jobs as well, but 700 MB history file job does not finish instantly. I took a look at a 3500 node cluster we have that is under fairly heavy load, and looking at the done directory for yesterday, I saw what amounted to about 1.7MB/sec of data on average. Gigabit Ethernet should be able to handle 15 to 20 times this (assuming that we read as much data as we write, and that the storage may require some replication). I am fine with the proposed solution by [~lohit] so long as the history service always provides a restful interface and the AM can decide if it wants to use it, or go through a different higher load channel. Otherwise non-java based AMs would not necessarily be able to write to the history service. I am also a bit nervous about using the history service for recovery or as a backend for the current MR APIs if we have a pub/sub system as a link between the applications and the history service. I don't think it is a show stopper, it just opens the door for a number of corner cases that will have to be dealt with, like an MR AM crashes badly and the client goes to the history service to get the counters/etc, when does the history service know that all of the events for the MR AM have been processed so it can return those counters, or perhaps other data? I am not totally sure what data may be a show stopper for this, but the lag means all applications have to be sure that they don't use the history service for split brain problems or things like that. > [Umbrella] Store, manage and serve per-framework application-timeline data > -- > > Key: YARN-1530 > URL: https://issues.apache.org/jira/browse/YARN-1530 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli > Attachments: application timeline design-20140108.pdf > > > This is a sibling JIRA for YARN-321. > Today, each application/framework has to do store, and serve per-framework > data all by itself as YARN doesn't have a common solution. This JIRA attempts > to solve the storage, management and serving of per-framework data from > various applications, both running and finished. The aim is to change YARN to > collect and store data in a generic manner with plugin points for frameworks > to do their own thing w.r.t interpretation and serving. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872287#comment-13872287 ] Hudson commented on YARN-1601: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5001 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5001/]) YARN-1601. 3rd party JARs are missing from hadoop-dist output. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558465) * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/pom.xml > 3rd party JARs are missing from hadoop-dist output > -- > > Key: YARN-1601 > URL: https://issues.apache.org/jira/browse/YARN-1601 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.4.0 > > Attachments: YARN-1601.patch > > > With the build changes of YARN-888 we are leaving out all 3rd party JArs used > directly by YARN under /share/hadoop/yarn/lib/. > We did not notice this when running minicluster because they all happen to be > in the classpath from hadoop-common and hadoop-yarn. > As 3d party JARs are not 'public' interfaces we cannot rely on them being > provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd > party dependency that yarn uses this would break yarn if yarn does not pull > that dependency explicitly). > Also, this will break bigtop hadoop build when they move to use branch-2 as > they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872270#comment-13872270 ] Alejandro Abdelnur commented on YARN-1601: -- Thanks Steve. Thanks Sean, regarding the stax-api JAR, I've opened another JIRA to fix that, HADOOP-10235. > 3rd party JARs are missing from hadoop-dist output > -- > > Key: YARN-1601 > URL: https://issues.apache.org/jira/browse/YARN-1601 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-1601.patch > > > With the build changes of YARN-888 we are leaving out all 3rd party JArs used > directly by YARN under /share/hadoop/yarn/lib/. > We did not notice this when running minicluster because they all happen to be > in the classpath from hadoop-common and hadoop-yarn. > As 3d party JARs are not 'public' interfaces we cannot rely on them being > provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd > party dependency that yarn uses this would break yarn if yarn does not pull > that dependency explicitly). > Also, this will break bigtop hadoop build when they move to use branch-2 as > they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872235#comment-13872235 ] Sean Mackrory commented on YARN-1601: - +1 (non-committer) - I can confirm the problem from a Bigtop perspective. If you were to try and build the "hadoop-client" package it would break because there wouldn't be any other Hadoop package that would provide one of the dependencies Maven determined was needed for a YARN client (specifically, stax-api 1.0-2, which has a different group ID than stax-api 1.0.1). > 3rd party JARs are missing from hadoop-dist output > -- > > Key: YARN-1601 > URL: https://issues.apache.org/jira/browse/YARN-1601 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-1601.patch > > > With the build changes of YARN-888 we are leaving out all 3rd party JArs used > directly by YARN under /share/hadoop/yarn/lib/. > We did not notice this when running minicluster because they all happen to be > in the classpath from hadoop-common and hadoop-yarn. > As 3d party JARs are not 'public' interfaces we cannot rely on them being > provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd > party dependency that yarn uses this would break yarn if yarn does not pull > that dependency explicitly). > Also, this will break bigtop hadoop build when they move to use branch-2 as > they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872043#comment-13872043 ] Hudson commented on YARN-1598: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1645 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1645/]) YARN-1598. HA-related rmadmin commands don't work on a secure cluster (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558251) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java > HA-related rmadmin commands don't work on a secure cluster > -- > > Key: YARN-1598 > URL: https://issues.apache.org/jira/browse/YARN-1598 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Critical > Fix For: 2.4.0 > > Attachments: yarn-1598-1.patch > > > The HA-related commands like -getServiceState -checkHealth etc. don't work in > a secure cluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872042#comment-13872042 ] Hudson commented on YARN-1603: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1645 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1645/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558340) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java.orig * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java.orig svn merge -c -1558337 . for reverting YARN-1603, which only partly committed. (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558339) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872050#comment-13872050 ] Hudson commented on YARN-1603: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1670/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558340) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java.orig * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java.orig svn merge -c -1558337 . for reverting YARN-1603, which only partly committed. (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558339) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872051#comment-13872051 ] Hudson commented on YARN-1598: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1670/]) YARN-1598. HA-related rmadmin commands don't work on a secure cluster (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558251) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java > HA-related rmadmin commands don't work on a secure cluster > -- > > Key: YARN-1598 > URL: https://issues.apache.org/jira/browse/YARN-1598 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Critical > Fix For: 2.4.0 > > Attachments: yarn-1598-1.patch > > > The HA-related commands like -getServiceState -checkHealth etc. don't work in > a secure cluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1567) In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload
[ https://issues.apache.org/jira/browse/YARN-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872044#comment-13872044 ] Hudson commented on YARN-1567: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1645 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1645/]) YARN-1567. In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558228) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java > In Fair Scheduler, allow empty queues to change between leaf and parent on > allocation file reload > - > > Key: YARN-1567 > URL: https://issues.apache.org/jira/browse/YARN-1567 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.4.0 > > Attachments: YARN-1567-1.patch, YARN-1567-2.patch, YARN-1567-3.patch, > YARN-1567.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1567) In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload
[ https://issues.apache.org/jira/browse/YARN-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872052#comment-13872052 ] Hudson commented on YARN-1567: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1670/]) YARN-1567. In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558228) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java > In Fair Scheduler, allow empty queues to change between leaf and parent on > allocation file reload > - > > Key: YARN-1567 > URL: https://issues.apache.org/jira/browse/YARN-1567 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.4.0 > > Attachments: YARN-1567-1.patch, YARN-1567-2.patch, YARN-1567-3.patch, > YARN-1567.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871940#comment-13871940 ] Hudson commented on YARN-1603: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5000 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5000/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558340) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java.orig * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java.orig svn merge -c -1558337 . for reverting YARN-1603, which only partly committed. (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558339) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1567) In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload
[ https://issues.apache.org/jira/browse/YARN-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871933#comment-13871933 ] Hudson commented on YARN-1567: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #453 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/453/]) YARN-1567. In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558228) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java > In Fair Scheduler, allow empty queues to change between leaf and parent on > allocation file reload > - > > Key: YARN-1567 > URL: https://issues.apache.org/jira/browse/YARN-1567 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.4.0 > > Attachments: YARN-1567-1.patch, YARN-1567-2.patch, YARN-1567-3.patch, > YARN-1567.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871931#comment-13871931 ] Hudson commented on YARN-1603: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #453 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/453/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871932#comment-13871932 ] Hudson commented on YARN-1598: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #453 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/453/]) YARN-1598. HA-related rmadmin commands don't work on a secure cluster (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558251) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java > HA-related rmadmin commands don't work on a secure cluster > -- > > Key: YARN-1598 > URL: https://issues.apache.org/jira/browse/YARN-1598 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Critical > Fix For: 2.4.0 > > Attachments: yarn-1598-1.patch > > > The HA-related commands like -getServiceState -checkHealth etc. don't work in > a secure cluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871923#comment-13871923 ] Steve Loughran commented on YARN-1601: -- confirmed that problem exists & patch does as claimed, +1 > 3rd party JARs are missing from hadoop-dist output > -- > > Key: YARN-1601 > URL: https://issues.apache.org/jira/browse/YARN-1601 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-1601.patch > > > With the build changes of YARN-888 we are leaving out all 3rd party JArs used > directly by YARN under /share/hadoop/yarn/lib/. > We did not notice this when running minicluster because they all happen to be > in the classpath from hadoop-common and hadoop-yarn. > As 3d party JARs are not 'public' interfaces we cannot rely on them being > provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd > party dependency that yarn uses this would break yarn if yarn does not pull > that dependency explicitly). > Also, this will break bigtop hadoop build when they move to use branch-2 as > they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[
https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871921#comment-13871921
]
Steve Loughran commented on YARN-1601:
--
after the patch
{code}
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn/
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//.DS_Store
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-api-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-client-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-tests-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/activation-1.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/aopalliance-1.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/asm-3.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-cli-1.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-codec-1.4.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-compress-1.4.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-httpclient-3.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-io-2.4.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-lang-2.6.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/commons-logging-1.1.3.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/guava-11.0.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/guice-3.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/guice-servlet-3.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-core-asl-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-jaxrs-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-mapper-asl-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jackson-xc-1.8.8.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/javax.inject-1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jaxb-api-2.2.2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jaxb-impl-2.2.3-1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-client-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-core-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-guice-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-json-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jersey-server-1.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jettison-1.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jetty-6.1.26.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jetty-util-6.1.26.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jline-0.9.94.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/jsr305-1.3.9.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/log4j-1.2.17.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/protobuf-java-2.5.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/servlet-api-2.5.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/stax-api-1.0-2.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/stax-api-1.0.1.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/xz-1.0.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//lib/zookeeper-3.4.5.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-api-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/ta
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[
https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871920#comment-13871920
]
Steve Loughran commented on YARN-1601:
--
before the patch
{code}
$ find hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn/
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn/
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-api-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-client-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-common-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-tests-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-api-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-distributedshell-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-applications-unmanaged-am-launcher-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-client-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-client-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-common-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-common-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-common-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-common-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-nodemanager-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-resourcemanager-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-tests-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//sources/hadoop-yarn-server-web-proxy-2.4.0-SNAPSHOT-test-sources.jar
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//test
hadoop-dist/target/hadoop-2.4.0-SNAPSHOT/share/hadoop/yarn//test/hadoop-yarn-server-tests-2.4.0-SNAPSHOT-tests.jar
{code}
> 3rd party JARs are missing from hadoop-dist output
> --
>
> Key: YARN-1601
> URL: https://issues.apache.org/jira/browse/YARN-1601
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: YARN-1601.patch
>
>
> With the build changes of YARN-888 we are leaving out all 3rd party JArs used
> directly by YARN under /share/hadoop/yarn/lib/.
> We did not notice this when running minicluster because they all happen to be
> in the classpath from hadoop-common and hadoop-yarn.
> As 3d party JARs are not 'public' interfaces we cannot rely on them being
> provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd
> party dependency that yarn uses this would break yarn if yarn does not pull
> that dependency explicitl
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871906#comment-13871906 ] Hudson commented on YARN-1603: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4999 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4999/]) YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) (junping_du: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558337) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871859#comment-13871859 ] Junping Du commented on YARN-1603: -- +1. I will commit it shortly. > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1604) Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig
[ https://issues.apache.org/jira/browse/YARN-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-1604: - Attachment: YARN-1604.patch A simple fix to remove both files. > Remove unnecessary files TestFifoScheduler.java.orig and > FairScheduler.java.orig > > > Key: YARN-1604 > URL: https://issues.apache.org/jira/browse/YARN-1604 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Junping Du >Priority: Minor > Attachments: YARN-1604.patch > > > In previous commit of YARN-1041, 2 files TestFifoScheduler.java.orig and > FairScheduler.java.orig are get committed which is not necessary. Remove both > of them in this JIRA. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1604) Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig
[ https://issues.apache.org/jira/browse/YARN-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-1604. -- Resolution: Duplicate > Remove unnecessary files TestFifoScheduler.java.orig and > FairScheduler.java.orig > > > Key: YARN-1604 > URL: https://issues.apache.org/jira/browse/YARN-1604 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Junping Du >Priority: Minor > Attachments: YARN-1604.patch > > > In previous commit of YARN-1041, 2 files TestFifoScheduler.java.orig and > FairScheduler.java.orig are get committed which is not necessary. Remove both > of them in this JIRA. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1604) Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig
Junping Du created YARN-1604: Summary: Remove unnecessary files TestFifoScheduler.java.orig and FairScheduler.java.orig Key: YARN-1604 URL: https://issues.apache.org/jira/browse/YARN-1604 Project: Hadoop YARN Issue Type: Improvement Reporter: Junping Du Priority: Minor In previous commit of YARN-1041, 2 files TestFifoScheduler.java.orig and FairScheduler.java.orig are get committed which is not necessary. Remove both of them in this JIRA. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1603) Remove two *.orig files which were unexpectedly committed
[ https://issues.apache.org/jira/browse/YARN-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871845#comment-13871845 ] Steve Loughran commented on YARN-1603: -- +1 > Remove two *.orig files which were unexpectedly committed > - > > Key: YARN-1603 > URL: https://issues.apache.org/jira/browse/YARN-1603 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Minor > Attachments: YARN-1603.1.patch > > > FairScheduler.java.orig and TestFifoScheduler.java.orig -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1601) 3rd party JARs are missing from hadoop-dist output
[ https://issues.apache.org/jira/browse/YARN-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871837#comment-13871837 ] Steve Loughran commented on YARN-1601: -- Makes sense. Let me do a branch-2 build before and after this patch & see what comes out > 3rd party JARs are missing from hadoop-dist output > -- > > Key: YARN-1601 > URL: https://issues.apache.org/jira/browse/YARN-1601 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: YARN-1601.patch > > > With the build changes of YARN-888 we are leaving out all 3rd party JArs used > directly by YARN under /share/hadoop/yarn/lib/. > We did not notice this when running minicluster because they all happen to be > in the classpath from hadoop-common and hadoop-yarn. > As 3d party JARs are not 'public' interfaces we cannot rely on them being > provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd > party dependency that yarn uses this would break yarn if yarn does not pull > that dependency explicitly). > Also, this will break bigtop hadoop build when they move to use branch-2 as > they expect to find jars in /share/hadoop/yarn/lib/ -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1104) NMs to support rolling logs of stdout & stderr
[ https://issues.apache.org/jira/browse/YARN-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871834#comment-13871834 ] Steve Loughran commented on YARN-1104: -- in Hoya we're setting up log4j for rolling logs, both in the AM and in deployed HBase containers. We'll have to see how well this works with long-lived, busy clusters. YARN- highlights how the web UI can't handle big logs > NMs to support rolling logs of stdout & stderr > -- > > Key: YARN-1104 > URL: https://issues.apache.org/jira/browse/YARN-1104 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.1.0-beta >Reporter: Steve Loughran > > Currently NMs stream the stdout and stderr streams of a container to a file. > For longer lived processes those files need to be rotated so that the log > doesn't overflow -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1457) YARN single node install issues on mvn clean install assembly:assembly on mapreduce project
[ https://issues.apache.org/jira/browse/YARN-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated YARN-1457: -- Component/s: (was: documentation) site > YARN single node install issues on mvn clean install assembly:assembly on > mapreduce project > --- > > Key: YARN-1457 > URL: https://issues.apache.org/jira/browse/YARN-1457 > Project: Hadoop YARN > Issue Type: Bug > Components: site >Affects Versions: 2.0.5-alpha >Reporter: Rekha Joshi >Priority: Minor > Labels: mvn > Attachments: yarn-mvn-mapreduce.txt > > > YARN single node install - > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html > On Mac OSX 10.7.3, Java 1.6, Protobuf 2.5.0 and hadoop-2.0.5-alpha.tar, mvn > clean install -DskipTests succeds after a YARN fix on pom.xml(using 2.5.0 > protobuf) > But on hadoop-mapreduce-project mvn install fails for tests with below errors > $ mvn clean install assembly:assembly -Pnative > errors as in atatched yarn-mvn-mapreduce,txt > On $mvn clean install assembly:assembly -DskipTests > Reactor Summary: > [INFO] > [INFO] hadoop-mapreduce-client ... SUCCESS [2.410s] > [INFO] hadoop-mapreduce-client-core .. SUCCESS [13.781s] > [INFO] hadoop-mapreduce-client-common SUCCESS [8.486s] > [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [0.774s] > [INFO] hadoop-mapreduce-client-app ... SUCCESS [4.409s] > [INFO] hadoop-mapreduce-client-hs SUCCESS [1.618s] > [INFO] hadoop-mapreduce-client-jobclient . SUCCESS [4.470s] > [INFO] hadoop-mapreduce-client-hs-plugins SUCCESS [0.561s] > [INFO] Apache Hadoop MapReduce Examples .. SUCCESS [1.620s] > [INFO] hadoop-mapreduce .. FAILURE [10.107s] > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 49.606s > [INFO] Finished at: Thu Nov 28 16:20:52 GMT+05:30 2013 > [INFO] Final Memory: 34M/118M > [INFO] > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-assembly-plugin:2.3:assembly (default-cli) on > project hadoop-mapreduce: Error reading assemblies: No assembly descriptors > found. -> [Help 1] > $mvn package -Pdist -DskipTests=true -Dtar > works > The documentation needs to be updated for possible issues and resolutions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
