[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[
https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694829#comment-14694829
]
zhoulinlin commented on YARN-3689:
--
Thanks! I thought "Higher Interger indicates Lower priority", so I am wrong.
The current code is fine.
> FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file,
> the "s1" and "s2" should change position when compare priority
> -
>
> Key: YARN-3689
> URL: https://issues.apache.org/jira/browse/YARN-3689
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, scheduler
>Affects Versions: 2.5.0
>Reporter: zhoulinlin
>Assignee: Ajith S
>
> In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should
> change position when compare priority.
> I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the
> same queue.
> The result is below:
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> before sort --
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0001 appPririty:4
> appStartTime:1432094170038
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0002 appPririty:2
> appStartTime:1432094173131
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> after sort %
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0001 appPririty:4
> appStartTime:1432094170038
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0002 appPririty:2
> appStartTime:1432094173131
> But when change the "s1" and "s2" position like below:
> public int compare(Schedulable s1, Schedulable s2) {
> int res = s2.getPriority().compareTo(s1.getPriority());
> .}
> The result:
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> before sort --
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0009 appPririty:4
> appStartTime:1432092992503
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0010 appPririty:2
> appStartTime:1432092996437
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> after sort %
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0010 appPririty:2
> appStartTime:1432092996437
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0009 appPririty:4
> appStartTime:1432092992503
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694840#comment-14694840
]
Hadoop QA commented on YARN-4025:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 36s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 24s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 44s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 50s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 26s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 45m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750226/YARN-4025-YARN-2928.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / bcd755e |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8839/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8839/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8839/console |
This message was automatically generated.
> Deal with byte representations of Longs in writer code
> --
>
> Key: YARN-4025
> URL: https://issues.apache.org/jira/browse/YARN-4025
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sangjin Lee
> Attachments: YARN-4025-YARN-2928.001.patch,
> YARN-4025-YARN-2928.002.patch
>
>
> Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
> code. There seem to be some places in the code where there are conversions
> between Long to byte[] to String for easier argument passing between function
> calls. Then these values end up being converted back to byte[] while storing
> in hbase.
> It would be better to pass around byte[] or the Longs themselves as
> applicable.
> This may result in some api changes (store function) as well in adding a few
> more function calls like getColumnQualifier which accepts a pre-encoded byte
> array. It will be in addition to the existing api which accepts a String and
> the ColumnHelper to return a byte[] column name instead of a String one.
> Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[
https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoulinlin resolved YARN-3689.
--
Resolution: Not A Problem
> FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file,
> the "s1" and "s2" should change position when compare priority
> -
>
> Key: YARN-3689
> URL: https://issues.apache.org/jira/browse/YARN-3689
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, scheduler
>Affects Versions: 2.5.0
>Reporter: zhoulinlin
>Assignee: Ajith S
>
> In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should
> change position when compare priority.
> I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the
> same queue.
> The result is below:
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> before sort --
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0001 appPririty:4
> appStartTime:1432094170038
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0002 appPririty:2
> appStartTime:1432094173131
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> after sort %
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0001 appPririty:4
> appStartTime:1432094170038
> 2015-05-20 11:57:41,449 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432094103221_0002 appPririty:2
> appStartTime:1432094173131
> But when change the "s1" and "s2" position like below:
> public int compare(Schedulable s1, Schedulable s2) {
> int res = s2.getPriority().compareTo(s1.getPriority());
> .}
> The result:
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> before sort --
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0009 appPririty:4
> appStartTime:1432092992503
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0010 appPririty:2
> appStartTime:1432092996437
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> after sort %
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0010 appPririty:2
> appStartTime:1432092996437
> 2015-05-20 11:36:37,119 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> appName:application_1432090734333_0009 appPririty:4
> appStartTime:1432092992503
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694988#comment-14694988
]
Hadoop QA commented on YARN-3964:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 21m 31s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 8 new or modified test files. |
| {color:green}+1{color} | javac | 10m 14s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 52s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 44s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 12m 24s | The patch appears to introduce 8
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 7m 30s | Tests failed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 2m 18s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 0m 30s | Tests passed in
hadoop-yarn-server-common. |
| {color:green}+1{color} | yarn tests | 6m 57s | Tests passed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 55m 20s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 136m 32s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-common |
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.yarn.util.TestRackResolver |
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750244/YARN-3964.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 40f8151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8840/console |
This message was automatically generated.
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow us
[jira] [Commented] (YARN-4024) YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat
[ https://issues.apache.org/jira/browse/YARN-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695011#comment-14695011 ] Hong Zhiguo commented on YARN-4024: --- There's DNS cache in InetAddress. What's the benefit to have another layer of cache in memory? Maybe easier to control? > YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat > -- > > Key: YARN-4024 > URL: https://issues.apache.org/jira/browse/YARN-4024 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Hong Zhiguo > > Currently, YARN RM NodesListManager will resolve IP address every time when > node doing heartbeat. When DNS server becomes slow, NM heartbeat will be > blocked and cannot make progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695038#comment-14695038 ] Rohith Sharma K S commented on YARN-4014: - Tried the syntax app-id,but options does not take app-id as valid input. May this is the reason other commands has camel case. > Support user cli interface in for Application Priority > -- > > Key: YARN-4014 > URL: https://issues.apache.org/jira/browse/YARN-4014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4014-V1.patch > > > Track the changes for user-RM client protocol i.e ApplicationClientProtocol > changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4014: Attachment: 0001-YARN-4014.patch > Support user cli interface in for Application Priority > -- > > Key: YARN-4014 > URL: https://issues.apache.org/jira/browse/YARN-4014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch > > > Track the changes for user-RM client protocol i.e ApplicationClientProtocol > changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695085#comment-14695085 ] Rohith Sharma K S commented on YARN-4014: - Updated the working patch with test cases, kindly review it. > Support user cli interface in for Application Priority > -- > > Key: YARN-4014 > URL: https://issues.apache.org/jira/browse/YARN-4014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch > > > Track the changes for user-RM client protocol i.e ApplicationClientProtocol > changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695084#comment-14695084 ] Rohith Sharma K S commented on YARN-4014: - thanks Sunil G for the review.. bq. In ApplicationCLI, public static final String SET_PRIORITY = "setPriority"; Done, changed to updatePriority bq. In future --appId can be used with other parameters also, correct? Yes, Done bq. updateApplicationPriority can throw NumberFormatException Since exception is directly thrown back to client cli, I think this should be fine. bq. ClientRMService.java has few commented code. Yes , Since YARN-3887 was not committed, I was used that patch to compile but while uploading patch I commented for HadoopQA compilation. Now I have uncommented those lines. > Support user cli interface in for Application Priority > -- > > Key: YARN-4014 > URL: https://issues.apache.org/jira/browse/YARN-4014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch > > > Track the changes for user-RM client protocol i.e ApplicationClientProtocol > changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695120#comment-14695120 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #286 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/286/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4031) Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
[ https://issues.apache.org/jira/browse/YARN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695117#comment-14695117 ] Hudson commented on YARN-4031: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #286 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/286/]) YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) (rkanter: rev dc2340c60e33f903f8fd34958ec746c989016191) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServer.java * hadoop-yarn-project/CHANGES.txt > Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer > - > > Key: YARN-4031 > URL: https://issues.apache.org/jira/browse/YARN-4031 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver, webapp >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-4031-v2.patch, YARN-4031.patch > > > We should add the JvmPauseMonitor to ApplicationHistoryServer and > WebAppProxyServer like what we do in YARN-4019 for ResourceManager and > NodeManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695127#comment-14695127 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1016 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1016/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4031) Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
[ https://issues.apache.org/jira/browse/YARN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695124#comment-14695124 ] Hudson commented on YARN-4031: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1016 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1016/]) YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) (rkanter: rev dc2340c60e33f903f8fd34958ec746c989016191) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServer.java * hadoop-yarn-project/CHANGES.txt > Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer > - > > Key: YARN-4031 > URL: https://issues.apache.org/jira/browse/YARN-4031 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver, webapp >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-4031-v2.patch, YARN-4031.patch > > > We should add the JvmPauseMonitor to ApplicationHistoryServer and > WebAppProxyServer like what we do in YARN-4019 for ResourceManager and > NodeManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695191#comment-14695191 ] Allen Wittenauer commented on YARN-4014: Try it without the dash. "--appid". The reason many commands use camel case is because Hadoop was written in Java and most Hadoop devs have clearly never used the command line on a regular basis. > Support user cli interface in for Application Priority > -- > > Key: YARN-4014 > URL: https://issues.apache.org/jira/browse/YARN-4014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch > > > Track the changes for user-RM client protocol i.e ApplicationClientProtocol > changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695191#comment-14695191 ] Allen Wittenauer edited comment on YARN-4014 at 8/13/15 1:08 PM: - Try it without the dash. "-appid". The reason many commands use camel case is because Hadoop was written in Java and most Hadoop devs have clearly never used the command line on a regular basis. was (Author: aw): Try it without the dash. "--appid". The reason many commands use camel case is because Hadoop was written in Java and most Hadoop devs have clearly never used the command line on a regular basis. > Support user cli interface in for Application Priority > -- > > Key: YARN-4014 > URL: https://issues.apache.org/jira/browse/YARN-4014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch > > > Track the changes for user-RM client protocol i.e ApplicationClientProtocol > changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4041) Slow delegation token renewal can severely prolong RM recovery
[ https://issues.apache.org/jira/browse/YARN-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695316#comment-14695316 ] Jason Lowe commented on YARN-4041: -- bq. IIRR, synchronous recovery was to fail-fast if recovery doesn't work. With the proposed change, what happens when the recovery fails? Arguably the same thing that happens when the RM goes to renew tokens on a live application and fails without a restart. IIRC this is not fatal to either the RM nor the application when this occurs today. In general I think we should make restarting as orthogonal as possible to token renewals, and ideally RM restart should not cause an out-of-band token renewal storm. > Slow delegation token renewal can severely prolong RM recovery > -- > > Key: YARN-4041 > URL: https://issues.apache.org/jira/browse/YARN-4041 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Sunil G > > When the RM does a work-preserving restart it synchronously tries to renew > delegation tokens for every active application. If a token server happens to > be down or is running slow and a lot of the active apps were using tokens > from that server then it can have a huge impact on the time it takes the RM > to process the restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4050) NM event dispatcher may blocked by LogAggregationService if NameNode is slow
sandflee created YARN-4050: -- Summary: NM event dispatcher may blocked by LogAggregationService if NameNode is slow Key: YARN-4050 URL: https://issues.apache.org/jira/browse/YARN-4050 Project: Hadoop YARN Issue Type: Bug Reporter: sandflee env: nm restart and log aggregation is enabled. NN is almost dead, when we restart NM, NM event dispatcher is blocked until NN returns to normal.It seems. NM recovered app and send APPLICATION_START event to log aggregation service, it will check log dir permission in HDFS(BLOCKED) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4044) Running applications information changes such as movequeue is not published to TimeLine server
[ https://issues.apache.org/jira/browse/YARN-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-4044: -- Attachment: 0001-YARN-4044.patch Uploading an initial version of patch. cc/[~xgong], [~rohithsharma] > Running applications information changes such as movequeue is not published > to TimeLine server > -- > > Key: YARN-4044 > URL: https://issues.apache.org/jira/browse/YARN-4044 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-4044.patch > > > SystemMetricsPublisher need to expose an appUpdated api to update any change > for a running application. > Events can be > - change of queue for a running application. > - change of application priority for a running application. > This ticket intends to handle both RM and timeline side changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4051) ContainerKillEvent is lost when container is In New State and is recovering
sandflee created YARN-4051: -- Summary: ContainerKillEvent is lost when container is In New State and is recovering Key: YARN-4051 URL: https://issues.apache.org/jira/browse/YARN-4051 Project: Hadoop YARN Issue Type: Bug Reporter: sandflee Assignee: sandflee As in YARN-4050, NM event dispatcher is blocked, and container is in New state, when we finish application, the container still alive even after NM event dispatcher is unblocked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4051) ContainerKillEvent is lost when container is In New State and is recovering
[ https://issues.apache.org/jira/browse/YARN-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandflee updated YARN-4051: --- Attachment: YARN-4051.01.patch > ContainerKillEvent is lost when container is In New State and is recovering > > > Key: YARN-4051 > URL: https://issues.apache.org/jira/browse/YARN-4051 > Project: Hadoop YARN > Issue Type: Bug >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-4051.01.patch > > > As in YARN-4050, NM event dispatcher is blocked, and container is in New > state, when we finish application, the container still alive even after NM > event dispatcher is unblocked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-4040) container complete msg should passed to AM,even if the container is released.
[
https://issues.apache.org/jira/browse/YARN-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandflee resolved YARN-4040.
Resolution: Not A Problem
If AM release a container, the complete msg(released by AM) is stored by
RMAppAttempt, and could be pulled by AM.
> container complete msg should passed to AM,even if the container is released.
> -
>
> Key: YARN-4040
> URL: https://issues.apache.org/jira/browse/YARN-4040
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: sandflee
>Assignee: sandflee
>
> If AM release a container, and container completes not killed by rm, there is
> a race condition the container complete msg are lost, because In
> RMNodeImpl.handleContainerStatus:
> if (containersToClean.contains(containerId)) {
> LOG.info("Container " + containerId + " already scheduled for "
> + "cleanup, no further processing");
> continue;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695391#comment-14695391 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2213 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2213/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4031) Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
[ https://issues.apache.org/jira/browse/YARN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695388#comment-14695388 ] Hudson commented on YARN-4031: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2213 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2213/]) YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) (rkanter: rev dc2340c60e33f903f8fd34958ec746c989016191) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/CHANGES.txt > Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer > - > > Key: YARN-4031 > URL: https://issues.apache.org/jira/browse/YARN-4031 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver, webapp >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-4031-v2.patch, YARN-4031.patch > > > We should add the JvmPauseMonitor to ApplicationHistoryServer and > WebAppProxyServer like what we do in YARN-4019 for ResourceManager and > NodeManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4031) Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
[ https://issues.apache.org/jira/browse/YARN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695455#comment-14695455 ] Hudson commented on YARN-4031: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #275 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/275/]) YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) (rkanter: rev dc2340c60e33f903f8fd34958ec746c989016191) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java > Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer > - > > Key: YARN-4031 > URL: https://issues.apache.org/jira/browse/YARN-4031 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver, webapp >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-4031-v2.patch, YARN-4031.patch > > > We should add the JvmPauseMonitor to ApplicationHistoryServer and > WebAppProxyServer like what we do in YARN-4019 for ResourceManager and > NodeManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695458#comment-14695458 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #275 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/275/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/CHANGES.txt > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695483#comment-14695483 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #283 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/283/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4031) Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
[ https://issues.apache.org/jira/browse/YARN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695480#comment-14695480 ] Hudson commented on YARN-4031: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #283 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/283/]) YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) (rkanter: rev dc2340c60e33f903f8fd34958ec746c989016191) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServer.java > Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer > - > > Key: YARN-4031 > URL: https://issues.apache.org/jira/browse/YARN-4031 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver, webapp >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-4031-v2.patch, YARN-4031.patch > > > We should add the JvmPauseMonitor to ApplicationHistoryServer and > WebAppProxyServer like what we do in YARN-4019 for ResourceManager and > NodeManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695522#comment-14695522 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2232 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2232/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4031) Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
[ https://issues.apache.org/jira/browse/YARN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695519#comment-14695519 ] Hudson commented on YARN-4031: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2232 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2232/]) YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) (rkanter: rev dc2340c60e33f903f8fd34958ec746c989016191) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java > Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer > - > > Key: YARN-4031 > URL: https://issues.apache.org/jira/browse/YARN-4031 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver, webapp >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-4031-v2.patch, YARN-4031.patch > > > We should add the JvmPauseMonitor to ApplicationHistoryServer and > WebAppProxyServer like what we do in YARN-4019 for ResourceManager and > NodeManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4051) ContainerKillEvent is lost when container is In New State and is recovering
[
https://issues.apache.org/jira/browse/YARN-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695564#comment-14695564
]
Hadoop QA commented on YARN-4051:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 11s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 48s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 38s | The applied patch generated 5
new checkstyle issues (total was 96, now 101). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 18s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 6m 7s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 43m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750297/YARN-4051.01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 53bef9c |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8842/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8842/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8842/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8842/console |
This message was automatically generated.
> ContainerKillEvent is lost when container is In New State and is recovering
>
>
> Key: YARN-4051
> URL: https://issues.apache.org/jira/browse/YARN-4051
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: sandflee
>Assignee: sandflee
> Attachments: YARN-4051.01.patch
>
>
> As in YARN-4050, NM event dispatcher is blocked, and container is in New
> state, when we finish application, the container still alive even after NM
> event dispatcher is unblocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4052) Set SO_KEEPALIVE on NM servers
Nathan Roberts created YARN-4052: Summary: Set SO_KEEPALIVE on NM servers Key: YARN-4052 URL: https://issues.apache.org/jira/browse/YARN-4052 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.7.1 Reporter: Nathan Roberts Shuffle handler does not set SO_KEEPALIVE so we've seen cases where FDs/sockets get stuck in ESTABLISHED state indefinitely because the server did not see the client leave (network cut or otherwise). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2076) Minor error in TestLeafQueue files
[ https://issues.apache.org/jira/browse/YARN-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He resolved YARN-2076. --- Resolution: Cannot Reproduce > Minor error in TestLeafQueue files > -- > > Key: YARN-2076 > URL: https://issues.apache.org/jira/browse/YARN-2076 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Chen He >Assignee: Chen He >Priority: Minor > Labels: test > Attachments: YARN-2076.patch > > > "numNodes" should be 2 instead of 3 in testReservationExchange() since only > two nodes are defined. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3924) Submitting an application to standby ResourceManager should respond better than Connection Refused
[ https://issues.apache.org/jira/browse/YARN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695639#comment-14695639 ] Robert Kanter commented on YARN-3924: - I think the original mention of JobTracker here is because Oozie names the field in the workflow "job-tracker", regardless of whether you put a JobTracker or ResourceManager address there. So, yes, if using Hadoop 2, the JobTracker is not involved at all. > Submitting an application to standby ResourceManager should respond better > than Connection Refused > -- > > Key: YARN-3924 > URL: https://issues.apache.org/jira/browse/YARN-3924 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Dustin Cote >Assignee: Ajith S >Priority: Minor > > When submitting an application directly to a standby resource manager, the > resource manager responds with 'Connection Refused' rather than indicating > that it is a standby resource manager. Because the resource manager is aware > of its own state, I feel like we can have the 8032 port open for standby > resource managers and reject the request with something like 'Cannot process > application submission from this standby resource manager'. > This would be especially helpful for debugging oozie problems when users put > in the wrong address for the 'jobtracker' (i.e. they don't put the logical RM > address but rather point to a specific resource manager). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4044) Running applications information changes such as movequeue is not published to TimeLine server
[
https://issues.apache.org/jira/browse/YARN-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695646#comment-14695646
]
Hadoop QA commented on YARN-4044:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 30s | Pre-patch trunk has 7 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 11s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 33s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests | 53m 30s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 95m 20s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750296/0001-YARN-4044.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 53bef9c |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8843/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8843/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8843/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8843/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8843/console |
This message was automatically generated.
> Running applications information changes such as movequeue is not published
> to TimeLine server
> --
>
> Key: YARN-4044
> URL: https://issues.apache.org/jira/browse/YARN-4044
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, timelineserver
>Affects Versions: 2.7.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Critical
> Attachments: 0001-YARN-4044.patch
>
>
> SystemMetricsPublisher need to expose an appUpdated api to update any change
> for a running application.
> Events can be
> - change of queue for a running application.
> - change of application priority for a running application.
> This ticket intends to handle both RM and timeline side changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3816) [Aggregation] App-level Aggregation for YARN system metrics
[ https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695653#comment-14695653 ] Varun Saxena commented on YARN-3816: When we are doing sum operation, what if the value after is sum outside the range of data type ? Do we assume it will be within limits ? Especially aggregation values over a longer time period may well go beyond limits. > [Aggregation] App-level Aggregation for YARN system metrics > --- > > Key: YARN-3816 > URL: https://issues.apache.org/jira/browse/YARN-3816 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Junping Du > Attachments: Application Level Aggregation of Timeline Data.pdf, > YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch > > > We need application level aggregation of Timeline data: > - To present end user aggregated states for each application, include: > resource (CPU, Memory) consumption across all containers, number of > containers launched/completed/failed, etc. We need this for apps while they > are running as well as when they are done. > - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be > aggregated to show details of states in framework level. > - Other level (Flow/User/Queue) aggregation can be more efficient to be based > on Application-level aggregations rather than raw entity-level data as much > less raws need to scan (with filter out non-aggregated entities, like: > events, configurations, etc.). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3816) [Aggregation] App-level Aggregation for YARN system metrics
[ https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695658#comment-14695658 ] Varun Saxena commented on YARN-3816: When we are doing sum operation, what if the value after sum goes beyond the upper range of data type ? Do we assume it will be within limits ? Especially aggregation values over a longer time period may well go beyond limits > [Aggregation] App-level Aggregation for YARN system metrics > --- > > Key: YARN-3816 > URL: https://issues.apache.org/jira/browse/YARN-3816 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Junping Du > Attachments: Application Level Aggregation of Timeline Data.pdf, > YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch > > > We need application level aggregation of Timeline data: > - To present end user aggregated states for each application, include: > resource (CPU, Memory) consumption across all containers, number of > containers launched/completed/failed, etc. We need this for apps while they > are running as well as when they are done. > - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be > aggregated to show details of states in framework level. > - Other level (Flow/User/Queue) aggregation can be more efficient to be based > on Application-level aggregations rather than raw entity-level data as much > less raws need to scan (with filter out non-aggregated entities, like: > events, configurations, etc.). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4037) Hadoop - failed redirect for container
[ https://issues.apache.org/jira/browse/YARN-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695687#comment-14695687 ] Gagan commented on YARN-4037: - Hi Mohammed, I am not sure what difference came in hadoop configs but I configured everything from scratch using the following link https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html#YARN_on_Single_Node The error "Failed while trying to construct the redirect url to the log server. Log Server url may not be configured java.lang.Exception: Unknown container" disappeared and I could see standard err, out files, which contain the following Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. I tried setting various values of set HADOOP_HEAPSIZE=512 but in vain, a value over 512 did not start the daemons. I have 8 gb ram. Could you help me on the same ? > Hadoop - failed redirect for container > -- > > Key: YARN-4037 > URL: https://issues.apache.org/jira/browse/YARN-4037 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.1 > Environment: Windows 7, Apache Hadoop 2.7.1 >Reporter: Gagan > Attachments: mapred-site.xml, yarn-site.xml > > > I believe this issue has been addressed earlier in > https://issues.apache.org/jira/browse/YARN-1473 though I am not sure because > the description of the JIRA does not talk about the following message > Failed while trying to construct the redirect url to the log server. Log > Server url may not be configured > java.lang.Exception: Unknown container. Container either has not started or > has already completed or doesn't belong to this node at all. > Could some one look at the same and provide detail on the root cause and > resolution ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4029) Update LogAggregationStatus to store on finish
[ https://issues.apache.org/jira/browse/YARN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-4029: --- Description: Currently the log aggregation status is not getting updated to Store. When RM is restarted will show NOT_START. Steps to reproduce 1.Submit mapreduce application 2.Wait for completion 3.Once application is completed switch RM *Log Aggregation Status* are changing *Log Aggregation Status* from SUCCESS to NOT_START was: Steps to reproduce 1.Submit mapreduce application 2.Wait for completion 3.Once application is completed switch RM *Log Aggregation Status* and *Diagnostics* are changing *Log Aggregation Status* from SUCCESS to NOT_START *Diagnostics* from EMPTY to null Summary: Update LogAggregationStatus to store on finish (was: LogAggregationStatus and Diagnostics value wrong after RM recovery) > Update LogAggregationStatus to store on finish > -- > > Key: YARN-4029 > URL: https://issues.apache.org/jira/browse/YARN-4029 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: Image.jpg > > > Currently the log aggregation status is not getting updated to Store. When RM > is restarted will show NOT_START. > Steps to reproduce > > 1.Submit mapreduce application > 2.Wait for completion > 3.Once application is completed switch RM > *Log Aggregation Status* are changing > *Log Aggregation Status* from SUCCESS to NOT_START -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4028) AppBlock page key update and diagnostics value null on recovery
[
https://issues.apache.org/jira/browse/YARN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4028:
---
Priority: Minor (was: Trivial)
Description:
All keys ends with *:* adding the same in *Log Aggregation Status* for
consistency
Also Diagnostics value shown as null on recovery
was:
All keys ends with *:* adding the same in *Log Aggregation Status* for
consistency
Issue Type: Bug (was: Improvement)
Summary: AppBlock page key update and diagnostics value null on
recovery (was: AppsBlock page keep key consistent)
In {{RMAppImpl}} during recovery diagnostics is set as below
{{RMAppImpl#recover(RMState state)}}
{code}
this.diagnostics.append(appState.getDiagnostics());
{code}
So null gets appended to diagnostics;
AppBlock checks for null but value is string 'null'
> AppBlock page key update and diagnostics value null on recovery
> ---
>
> Key: YARN-4028
> URL: https://issues.apache.org/jira/browse/YARN-4028
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4028.patch, Image.jpg
>
>
> All keys ends with *:* adding the same in *Log Aggregation Status* for
> consistency
> Also Diagnostics value shown as null on recovery
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4028) AppBlock page key update and diagnostics value null on recovery
[ https://issues.apache.org/jira/browse/YARN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-4028: --- Attachment: 0002-YARN-4028.patch Attaching patch for review after handling both the issues > AppBlock page key update and diagnostics value null on recovery > --- > > Key: YARN-4028 > URL: https://issues.apache.org/jira/browse/YARN-4028 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: 0001-YARN-4028.patch, 0002-YARN-4028.patch, Image.jpg > > > All keys ends with *:* adding the same in *Log Aggregation Status* for > consistency > Also Diagnostics value shown as null on recovery -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-221) NM should provide a way for AM to tell it not to aggregate logs.
[
https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated YARN-221:
-
Attachment: YARN-221-9.patch
I had offline discussion with Xuan about the API. To support this as public
interface just like {{AuxiliaryService}} so that YARN framework developers can
develop some customized policy, it might be better to have its own
{{ContainerLogContext}}.
The latest patch has the following updates.
* Use ContainerLogContext.
* Move ContainerLogAggregationPolicy to yarn.server.api package.
* Fix the documentation in AppLogAggregatorImpl.
> NM should provide a way for AM to tell it not to aggregate logs.
>
>
> Key: YARN-221
> URL: https://issues.apache.org/jira/browse/YARN-221
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation, nodemanager
>Reporter: Robert Joseph Evans
>Assignee: Ming Ma
> Attachments: YARN-221-6.patch, YARN-221-7.patch, YARN-221-8.patch,
> YARN-221-9.patch, YARN-221-trunk-v1.patch, YARN-221-trunk-v2.patch,
> YARN-221-trunk-v3.patch, YARN-221-trunk-v4.patch, YARN-221-trunk-v5.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the
> logs should not be aggregated, that they should be aggregated with a high
> priority, or that they should be aggregated but with a lower priority. The
> AM should be able to do this in the ContainerLaunch context to provide a
> default value, but should also be able to update the value when the container
> is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid
> connection to the NN at all.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[
https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695839#comment-14695839
]
Hadoop QA commented on YARN-4014:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 47s | The applied patch generated 4
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 31s | The applied patch generated 6
new checkstyle issues (total was 31, now 37). |
| {color:red}-1{color} | whitespace | 0m 13s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 19s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 119m 56s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | yarn tests | 0m 29s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 7m 5s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 2m 2s | Tests failed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 53m 23s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 232m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.util.TestRackResolver |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| Timed out tests |
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750273/0001-YARN-4014.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 53bef9c |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8841/console |
This message was automatically generated.
> Support user cli interface in for Application Priority
> --
>
> Key: YARN-4014
> URL: https://issues.apache.org/jira/browse/YARN-4014
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client, resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch
>
>
> Track the changes for user-RM client protocol i.e ApplicationClientProtocol
> changes and discussions in this jira.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4028) AppBlock page key update and diagnostics value null on recovery
[
https://issues.apache.org/jira/browse/YARN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695984#comment-14695984
]
Hadoop QA commented on YARN-4028:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 22s | Pre-patch trunk has 7 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests | 53m 39s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 95m 29s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750360/0002-YARN-4028.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b73181f |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8844/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8844/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8844/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8844/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8844/console |
This message was automatically generated.
> AppBlock page key update and diagnostics value null on recovery
> ---
>
> Key: YARN-4028
> URL: https://issues.apache.org/jira/browse/YARN-4028
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4028.patch, 0002-YARN-4028.patch, Image.jpg
>
>
> All keys ends with *:* adding the same in *Log Aggregation Status* for
> consistency
> Also Diagnostics value shown as null on recovery
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4047) ClientRMService getApplications has high scheduler lock contention
[ https://issues.apache.org/jira/browse/YARN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696054#comment-14696054 ] Jian He commented on YARN-4047: --- lgtm +1 > ClientRMService getApplications has high scheduler lock contention > -- > > Key: YARN-4047 > URL: https://issues.apache.org/jira/browse/YARN-4047 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe > Labels: 2.6.1-candidate > Attachments: YARN-4047.001.patch > > > The getApplications call can be particuarly expensive because the code can > call checkAccess on every application being tracked by the RM. checkAccess > will often call scheduler.checkAccess which will grab the big scheduler lock. > This can cause a lot of contention with the scheduler thread which is busy > trying to process node heartbeats, app allocation requests, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4005) Completed container whose app is finished is not removed from NMStateStore
[ https://issues.apache.org/jira/browse/YARN-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696078#comment-14696078 ] Hudson commented on YARN-4005: -- FAILURE: Integrated in Hadoop-trunk-Commit #8295 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8295/]) YARN-4005. Completed container whose app is finished is possibly not removed from NMStateStore. Contributed by Jun Gong (jianhe: rev 38aed1a94ed7b6da62e2445b5610bc02b1cddeeb) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Completed container whose app is finished is not removed from NMStateStore > -- > > Key: YARN-4005 > URL: https://issues.apache.org/jira/browse/YARN-4005 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Fix For: 2.8.0 > > Attachments: YARN-4005.01.patch > > > If a container is completed and its corresponding app is finished, NM only > removes it from its context and does not add it to > 'recentlyStoppedContainers' when calling 'getContainerStatuses'. Then NM will > not remove it from NMStateStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3862) Decide which contents to retrieve and send back in response in TimelineReader
[ https://issues.apache.org/jira/browse/YARN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696131#comment-14696131 ] Li Lu commented on YARN-3862: - Hi [~varun_saxena], thanks for working on this! After looking at the WIP patch, I have some general questions and think some discussions will definitely be helpful. My first confusion is about the relationship between this JIRA and YARN-3863. Both JIRAs appear to be related to timeline filter designs. However, if we'd like to support filters in timeline v2, I think a natural workflow may be: # Filter object model and support them in FS storage implementation for tests. # Supporting timeline filters in our HBase reader # Connecting the filter implementation to our existing web services interface Each steps may not necessarily be a separate JIRA. If I understand correctly we're including some changes in step 1 and 2 in this JIRA, and you're planning to add WS support later. So I think this will nicely address our current request on timeline filters. What is the plan for YARN-3863 then? Or I'm missing some major piece? Secondly it's about the role of our planned filters. I notice you're putting filters in a package within timeline reader. My feeling is that the concept of timeline filter may become a part of our object model, so that client users can easily communicate? A follow up question from this is, are we treating our timeline filters as pure-data objects (models) or something that include both filter data and the way we actually filter (models+algorithm)? I slightly prefer the former because this may decouple the filter information (in timeline server land) and the storage implementations. I agree our filter design should at least slightly favor HBase filters, but converting a timeline filter to an HBase filter should be restricted in the land of the actual storage. Finally a quick question: is it easy, or possible, for us to implement a "paging filter"? We have a lot of web UI use cases that may need such a filter. > Decide which contents to retrieve and send back in response in TimelineReader > - > > Key: YARN-3862 > URL: https://issues.apache.org/jira/browse/YARN-3862 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-3862-YARN-2928.wip.01.patch > > > Currently, we will retrieve all the contents of the field if that field is > specified in the query API. In case of configs and metrics, this can become a > lot of data even though the user doesn't need it. So we need to provide a way > to query only a set of configs or metrics. > As a comma spearated list of configs/metrics to be returned will be quite > cumbersome to specify, we have to support either of the following options : > # Prefix match > # Regex > # Group the configs/metrics and query that group. > We also need a facility to specify a metric time window to return metrics in > a that window. This may be useful in plotting graphs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-221) NM should provide a way for AM to tell it not to aggregate logs.
[
https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696137#comment-14696137
]
Hadoop QA commented on YARN-221:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 44s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 20s | The applied patch generated 1
new checkstyle issues (total was 212, now 212). |
| {color:green}+1{color} | whitespace | 2m 27s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 24s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 7m 44s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 19s | Tests failed in
hadoop-common. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 56s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 7m 35s | Tests passed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 53m 19s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 141m 10s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
| | hadoop.yarn.util.TestRackResolver |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750361/YARN-221-9.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b73181f |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8845/console |
This message was automatically generated.
> NM should provide a way for AM to tell it not to aggregate logs.
>
>
> Key: YARN-221
> URL: https://issues.apache.org/jira/browse/YARN-221
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation, nodemanager
>Reporter: Robert Joseph Evans
>Assignee: Ming Ma
> Attachments: YARN-221-6.patch, YARN-221-7.patch, YARN-221-8.patch,
> YARN-221-9.patch, YARN-221-trunk-v1.patch, YARN-221-trunk-v2.patch,
> YARN-221-trunk-v3.patch, YARN-221-trunk-v4.patch, YARN-221-trunk-v5.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the
> logs should not be aggregated, that they should be aggregated with a high
> priority, or that they should be aggregated but with a lower priority. The
> AM should be able to do this in the ContainerLaunch context to provide a
> default value, but should also be able to update the value when the container
> is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid
> connection to the NN at all.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3863) Enhance filters in TimelineReader
[ https://issues.apache.org/jira/browse/YARN-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696163#comment-14696163 ] Joep Rottinghuis commented on YARN-3863: [~varun_saxena] in YARN-3862 you refer to this jira saying that YARN-3862 depends on this one. But YARN-3862 already will implement some filters, so what work are you envisioning happening here rather than there? > Enhance filters in TimelineReader > - > > Key: YARN-3863 > URL: https://issues.apache.org/jira/browse/YARN-3863 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > > Currently filters in timeline reader will return an entity only if all the > filter conditions hold true i.e. only AND operation is supported. We can > support OR operation for the filters as well. Additionally as primary backend > implementation is HBase, we can design our filters in a manner, where they > closely resemble HBase Filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3999) RM hangs on draining events
[ https://issues.apache.org/jira/browse/YARN-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-3999: -- Summary: RM hangs on draining events (was: RM hangs on draing events) > RM hangs on draining events > --- > > Key: YARN-3999 > URL: https://issues.apache.org/jira/browse/YARN-3999 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.7.2 > > Attachments: YARN-3999-branch-2.7.patch, YARN-3999.1.patch, > YARN-3999.2.patch, YARN-3999.2.patch, YARN-3999.3.patch, YARN-3999.4.patch, > YARN-3999.5.patch, YARN-3999.patch, YARN-3999.patch > > > If external systems like ATS, or ZK becomes very slow, draining all the > events take a lot of time. If this time becomes larger than 10 mins, all > applications will expire. Fixes include: > 1. add a timeout and stop the dispatcher even if not all events are drained. > 2. Move ATS service out from RM active service so that RM doesn't need to > wait for ATS to flush the events when transitioning to standby. > 3. Stop client-facing services (ClientRMService etc.) first so that clients > get fast notification that RM is stopping/transitioning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4047) ClientRMService getApplications has high scheduler lock contention
[ https://issues.apache.org/jira/browse/YARN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696245#comment-14696245 ] Hudson commented on YARN-4047: -- FAILURE: Integrated in Hadoop-trunk-Commit #8296 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8296/]) YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe (jianhe: rev 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java > ClientRMService getApplications has high scheduler lock contention > -- > > Key: YARN-4047 > URL: https://issues.apache.org/jira/browse/YARN-4047 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Jason Lowe > Labels: 2.6.1-candidate > Fix For: 2.7.2 > > Attachments: YARN-4047.001.patch > > > The getApplications call can be particuarly expensive because the code can > call checkAccess on every application being tracked by the RM. checkAccess > will often call scheduler.checkAccess which will grab the big scheduler lock. > This can cause a lot of contention with the scheduler thread which is busy > trying to process node heartbeats, app allocation requests, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3987) am container complete msg ack to NM once RM receive it
[ https://issues.apache.org/jira/browse/YARN-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696244#comment-14696244 ] Hudson commented on YARN-3987: -- FAILURE: Integrated in Hadoop-trunk-Commit #8296 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8296/]) YARN-3987. Send AM container completed msg to NM once AM finishes. Contributed by sandflee (jianhe: rev 0a030546e24c55662a603bb63c9029ad0ccf43fc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * hadoop-yarn-project/CHANGES.txt > am container complete msg ack to NM once RM receive it > -- > > Key: YARN-3987 > URL: https://issues.apache.org/jira/browse/YARN-3987 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: sandflee >Assignee: sandflee > Fix For: 2.8.0 > > Attachments: YARN-3987.001.patch, YARN-3987.002.patch > > > In our cluster we set max-am-attempts to a very very large num, and > unfortunately our am crash after launched, leaving too many completed > container(AM container) in NM. completed container is removed from NM and > NMStateStore only if container complete is passed to AM, but if AM couldn't > be launched, the completed AM container couldn't be cleaned, and may eat up > NM heap memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696268#comment-14696268 ] Li Lu commented on YARN-4025: - Hi [~sjlee0], thanks for the work! The patch overall LGTM. One small question: The way to deserializing event key, ts, and infokey appears to be a little bit isolated from the serialization part in the writer. Maybe it's helpful to put them in a centralized place? It might be a little bit confusing for the hardcoded 0, 1, and 2 indices, so maybe it's also helpful to mark down the structure of the entityid:ts:infokey structure somewhere close? > Deal with byte representations of Longs in writer code > -- > > Key: YARN-4025 > URL: https://issues.apache.org/jira/browse/YARN-4025 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sangjin Lee > Attachments: YARN-4025-YARN-2928.001.patch, > YARN-4025-YARN-2928.002.patch > > > Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl > code. There seem to be some places in the code where there are conversions > between Long to byte[] to String for easier argument passing between function > calls. Then these values end up being converted back to byte[] while storing > in hbase. > It would be better to pass around byte[] or the Longs themselves as > applicable. > This may result in some api changes (store function) as well in adding a few > more function calls like getColumnQualifier which accepts a pre-encoded byte > array. It will be in addition to the existing api which accepts a String and > the ColumnHelper to return a byte[] column name instead of a String one. > Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696271#comment-14696271 ] Li Lu commented on YARN-4025: - Oh and, BTW, why are we changing Separator.VALUES? > Deal with byte representations of Longs in writer code > -- > > Key: YARN-4025 > URL: https://issues.apache.org/jira/browse/YARN-4025 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sangjin Lee > Attachments: YARN-4025-YARN-2928.001.patch, > YARN-4025-YARN-2928.002.patch > > > Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl > code. There seem to be some places in the code where there are conversions > between Long to byte[] to String for easier argument passing between function > calls. Then these values end up being converted back to byte[] while storing > in hbase. > It would be better to pass around byte[] or the Longs themselves as > applicable. > This may result in some api changes (store function) as well in adding a few > more function calls like getColumnQualifier which accepts a pre-encoded byte > array. It will be in addition to the existing api which accepts a String and > the ColumnHelper to return a byte[] column name instead of a String one. > Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4005) Completed container whose app is finished is not removed from NMStateStore
[ https://issues.apache.org/jira/browse/YARN-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696285#comment-14696285 ] Jun Gong commented on YARN-4005: Thanks [~jianhe] for the review and commit! > Completed container whose app is finished is not removed from NMStateStore > -- > > Key: YARN-4005 > URL: https://issues.apache.org/jira/browse/YARN-4005 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Fix For: 2.8.0 > > Attachments: YARN-4005.01.patch > > > If a container is completed and its corresponding app is finished, NM only > removes it from its context and does not add it to > 'recentlyStoppedContainers' when calling 'getContainerStatuses'. Then NM will > not remove it from NMStateStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4028) AppBlock page key update and diagnostics value null on recovery
[
https://issues.apache.org/jira/browse/YARN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696301#comment-14696301
]
Bibin A Chundatt commented on YARN-4028:
Test case failures are not related to this patch.
No testadded since its only Only UI change
{code}
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testNormalContainerAllocationWhenDNSUnavailable
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable
{code}
> AppBlock page key update and diagnostics value null on recovery
> ---
>
> Key: YARN-4028
> URL: https://issues.apache.org/jira/browse/YARN-4028
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4028.patch, 0002-YARN-4028.patch, Image.jpg
>
>
> All keys ends with *:* adding the same in *Log Aggregation Status* for
> consistency
> Also Diagnostics value shown as null on recovery
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-4029) Update LogAggregationStatus to store on finish
[ https://issues.apache.org/jira/browse/YARN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-4029: - Assignee: Sunil G (was: Bibin A Chundatt) > Update LogAggregationStatus to store on finish > -- > > Key: YARN-4029 > URL: https://issues.apache.org/jira/browse/YARN-4029 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Sunil G > Attachments: Image.jpg > > > Currently the log aggregation status is not getting updated to Store. When RM > is restarted will show NOT_START. > Steps to reproduce > > 1.Submit mapreduce application > 2.Wait for completion > 3.Once application is completed switch RM > *Log Aggregation Status* are changing > *Log Aggregation Status* from SUCCESS to NOT_START -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4029) Update LogAggregationStatus to store on finish
[
https://issues.apache.org/jira/browse/YARN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696358#comment-14696358
]
Sunil G commented on YARN-4029:
---
We have {{yarn.log-aggregation-enable}} configuration at yarn level. So I feel
if its disabled, we can also update Not_Enabled instead of Failed if its fine.
> Update LogAggregationStatus to store on finish
> --
>
> Key: YARN-4029
> URL: https://issues.apache.org/jira/browse/YARN-4029
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Sunil G
> Attachments: Image.jpg
>
>
> Currently the log aggregation status is not getting updated to Store. When RM
> is restarted will show NOT_START.
> Steps to reproduce
>
> 1.Submit mapreduce application
> 2.Wait for completion
> 3.Once application is completed switch RM
> *Log Aggregation Status* are changing
> *Log Aggregation Status* from SUCCESS to NOT_START
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3987) am container complete msg ack to NM once RM receive it
[ https://issues.apache.org/jira/browse/YARN-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696433#comment-14696433 ] sandflee commented on YARN-3987: Thanks [~jianhe]! > am container complete msg ack to NM once RM receive it > -- > > Key: YARN-3987 > URL: https://issues.apache.org/jira/browse/YARN-3987 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: sandflee >Assignee: sandflee > Fix For: 2.8.0 > > Attachments: YARN-3987.001.patch, YARN-3987.002.patch > > > In our cluster we set max-am-attempts to a very very large num, and > unfortunately our am crash after launched, leaving too many completed > container(AM container) in NM. completed container is removed from NM and > NMStateStore only if container complete is passed to AM, but if AM couldn't > be launched, the completed AM container couldn't be cleaned, and may eat up > NM heap memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4050) NM event dispatcher may blocked by LogAggregationService if NameNode is slow
[ https://issues.apache.org/jira/browse/YARN-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandflee reassigned YARN-4050: -- Assignee: sandflee > NM event dispatcher may blocked by LogAggregationService if NameNode is slow > > > Key: YARN-4050 > URL: https://issues.apache.org/jira/browse/YARN-4050 > Project: Hadoop YARN > Issue Type: Bug >Reporter: sandflee >Assignee: sandflee > > env: nm restart and log aggregation is enabled. > NN is almost dead, when we restart NM, NM event dispatcher is blocked until > NN returns to normal.It seems. NM recovered app and send APPLICATION_START > event to log aggregation service, it will check log dir permission in > HDFS(BLOCKED) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[
https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696594#comment-14696594
]
Sunil G commented on YARN-4014:
---
Hi [~rohithsharma]
Thank you. Overall the patch looks good. Some minor nits
- In ApplicationClientProtocolPBServiceImpl, you may try like below
{code}
catch (YarnException | IOException e) {
throw new ServiceException(e);
}
{code}
- In ClientRMService
{code}
+if (EnumSet.of(RMAppState.NEW, RMAppState.NEW_SAVING, RMAppState.FAILED,
+RMAppState.FINAL_SAVING, RMAppState.FINISHING, RMAppState.FINISHED,
+RMAppState.KILLED, RMAppState.KILLING, RMAppState.FAILED).contains(
+application.getState()))
{code}
Could we have a lookup method for this rather checking it directly. May be
other apis can use this.
- In testUpdateApplicationPriorityRequest, Could we pass an invalid AppID and
check that error condition also.
- While printing message
{code}
+pw.println(" -appId ApplicationId can be used
with any other");
+pw.println(" sub commands in future.
Currently it is");
+pw.println(" used along only with
-set-priority");
{code}
-set-priority can be changed to -updatePriority
> Support user cli interface in for Application Priority
> --
>
> Key: YARN-4014
> URL: https://issues.apache.org/jira/browse/YARN-4014
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client, resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4014-V1.patch, 0001-YARN-4014.patch
>
>
> Track the changes for user-RM client protocol i.e ApplicationClientProtocol
> changes and discussions in this jira.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
