[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743032#comment-14743032 ] Hudson commented on YARN-4126: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2331 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2331/]) YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 2.8.0 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3901: - Attachment: YARN-3901-YARN-2928.7.patch Attaching patch v7 that addresses Sangjin's review suggestions as well as those discussed with Joep offline. I have also fixed the findbugs warnings. > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch, > YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch, > YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743087#comment-14743087
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 12s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 50s | The applied patch generated
14 new checkstyle issues (total was 234, now 241). |
| {color:red}-1{color} | whitespace | 0m 6s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 54m 31s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 94m 30s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e1b1d7e |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743097#comment-14743097 ] Hudson commented on YARN-4126: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #383 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/383/]) YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 2.8.0 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743155#comment-14743155 ] Hudson commented on YARN-4126: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1121 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1121/]) YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 2.8.0 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4034) Render cluster Max Priority in scheduler metrics in RM web UI
[ https://issues.apache.org/jira/browse/YARN-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743167#comment-14743167 ] Jian He commented on YARN-4034: --- looks good overall, "Maximum Cluster Application Priority", how about "Maximum Cluster Level Application Priority" > Render cluster Max Priority in scheduler metrics in RM web UI > - > > Key: YARN-4034 > URL: https://issues.apache.org/jira/browse/YARN-4034 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, webapp >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Minor > Attachments: 0001-YARN-4034.patch, YARN-4034.PNG > > > Currently Scheduler Metric renders the common scheduler metrics in RM web UI. > It would be helpful for the user to know what is the configured cluster max > priority from web UI. > So, in RM web UI front page, Scheduler Metrics can render configured max > cluster priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[
https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743191#comment-14743191
]
Hadoop QA commented on YARN-1651:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 3m 13s | YARN-1197 compilation may be
broken. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 23 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 54s | The applied patch generated 65
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 17s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 35m 36s | The patch has 177 line(s)
that end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 7m 3s | The patch appears to introduce 8
new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 9m 19s | Tests failed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | tools/hadoop tests | 0m 52s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 6m 56s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 0s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-server-common. |
| {color:green}+1{color} | yarn tests | 55m 37s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 143m 38s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-common |
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests | hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755671/YARN-1651-10.YARN-1197.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | YARN-1197 / 78ad04d |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/diffJavadocWarnings.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9117/console |
This message was automatically generated.
> CapacityScheduler side changes to support increase/decrease container
> resource.
> ---
>
> Key: YARN-1651
> URL: https://issues.apache.org/jira/browse/YARN-1651
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager, scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-1651-1.YARN-1197.patch,
> YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch,
> YARN-1651-3.YARN-1
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743209#comment-14743209
]
Jian He commented on YARN-3635:
---
lgtm
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743259#comment-14743259 ] Hudson commented on YARN-4126: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #368 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/368/]) YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 2.8.0 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4155) TestLogAggregationService.testLogAggregationServiceWithInterval failing
Steve Loughran created YARN-4155:
Summary:
TestLogAggregationService.testLogAggregationServiceWithInterval failing
Key: YARN-4155
URL: https://issues.apache.org/jira/browse/YARN-4155
Project: Hadoop YARN
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Environment: Jenkins
Reporter: Steve Loughran
Priority: Critical
Test failing on Jenkins:
{{TestLogAggregationService.testLogAggregationServiceWithInterval}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4155) TestLogAggregationService.testLogAggregationServiceWithInterval failing
[
https://issues.apache.org/jira/browse/YARN-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743296#comment-14743296
]
Steve Loughran commented on YARN-4155:
--
{code}
java.lang.AssertionError: expected:<-1> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1993)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithInterval(TestLogAggregationService.java:1887)
{code}
Note that in the assertion in the source, the expected and actual arguments are
in the wrong order. This assert expected "1", but got "-1".
The patch should include reversing these arguments, and adding some explanatory
message to the assert
> TestLogAggregationService.testLogAggregationServiceWithInterval failing
> ---
>
> Key: YARN-4155
> URL: https://issues.apache.org/jira/browse/YARN-4155
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Priority: Critical
>
> Test failing on Jenkins:
> {{TestLogAggregationService.testLogAggregationServiceWithInterval}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743299#comment-14743299 ] Hudson commented on YARN-4126: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2308 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2308/]) YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 2.8.0 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2578) NM does not failover timely if RM node network connection fails
[
https://issues.apache.org/jira/browse/YARN-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved YARN-2578.
Resolution: Duplicate
I'm closing this as duplicate of HADOOP-11252. Please reopen if you disagree.
> NM does not failover timely if RM node network connection fails
> ---
>
> Key: YARN-2578
> URL: https://issues.apache.org/jira/browse/YARN-2578
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.5.1
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-2578.002.patch, YARN-2578.patch
>
>
> The NM does not fail over correctly when the network cable of the RM is
> unplugged or the failure is simulated by a "service network stop" or a
> firewall that drops all traffic on the node. The RM fails over to the standby
> node when the failure is detected as expected. The NM should than re-register
> with the new active RM. This re-register takes a long time (15 minutes or
> more). Until then the cluster has no nodes for processing and applications
> are stuck.
> Reproduction test case which can be used in any environment:
> - create a cluster with 3 nodes
> node 1: ZK, NN, JN, ZKFC, DN, RM, NM
> node 2: ZK, NN, JN, ZKFC, DN, RM, NM
> node 3: ZK, JN, DN, NM
> - start all services make sure they are in good health
> - kill the network connection of the RM that is active using one of the
> network kills from above
> - observe the NN and RM failover
> - the DN's fail over to the new active NN
> - the NM does not recover for a long time
> - the logs show a long delay and traces show no change at all
> The stack traces of the NM all show the same set of threads. The main thread
> which should be used in the re-register is the "Node Status Updater" This
> thread is stuck in:
> {code}
> "Node Status Updater" prio=10 tid=0x7f5a6cc99800 nid=0x18d0 in
> Object.wait() [0x7f5a51fc1000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xed62f488> (a org.apache.hadoop.ipc.Client$Call)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1395)
> - locked <0xed62f488> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy26.nodeHeartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:80)
> {code}
> The client connection which goes through the proxy can be traced back to the
> ResourceTrackerPBClientImpl. The generated proxy does not time out and we
> should be using a version which takes the RPC timeout (from the
> configuration) as a parameter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4148) When killing app, RM releases app's resource before they are released by NM
[ https://issues.apache.org/jira/browse/YARN-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743358#comment-14743358 ] Jun Gong commented on YARN-4148: I have some thoughts. Proposal A: NM records its total resource and available resource. When launching a container, NM checks available resource and waits until there is enough resource for container. But there might be a time gap from AM's perspective, AM thinks it has launched container, however container might be waiting for its resource. Proposal B: RM does not release app's resource until containers actually finish and NM releases the resource. It seems a little complex. I prefer proposal A. Any suggestion or feedback is greatly appreciated. > When killing app, RM releases app's resource before they are released by NM > --- > > Key: YARN-4148 > URL: https://issues.apache.org/jira/browse/YARN-4148 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Jun Gong >Assignee: Jun Gong > > When killing a app, RM scheduler releases app's resource as soon as possible, > then it might allocate these resource for new requests. But NM have not > released them at that time. > The problem was found when we supported GPU as a resource(YARN-4122). Test > environment: a NM had 6 GPUs, app A used all 6 GPUs, app B was requesting 3 > GPUs. Killed app A, then RM released A's 6 GPUs, and allocated 3 GPUs to B. > But when B tried to start container on NM, NM found it didn't have 3 GPUs to > allocate because it had not released A's GPUs. > I think the problem also exists for CPU/Memory. It might cause OOM when > memory is overused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4155) TestLogAggregationService.testLogAggregationServiceWithInterval failing
[
https://issues.apache.org/jira/browse/YARN-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4155:
---
Attachment: 0001-YARN-4155.patch
Attaching patch for the same.
# Windows was not able to run have changed for that
# Assert part have changed as requested.
In locally when i ran its working fine
> TestLogAggregationService.testLogAggregationServiceWithInterval failing
> ---
>
> Key: YARN-4155
> URL: https://issues.apache.org/jira/browse/YARN-4155
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Priority: Critical
> Attachments: 0001-YARN-4155.patch
>
>
> Test failing on Jenkins:
> {{TestLogAggregationService.testLogAggregationServiceWithInterval}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4140) RM container allocation delayed incase of app submitted to Nodelabel partition
[
https://issues.apache.org/jira/browse/YARN-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4140:
---
Attachment: 0003-YARN-4140.patch
Attaching updated patch with testcases
> RM container allocation delayed incase of app submitted to Nodelabel partition
> --
>
> Key: YARN-4140
> URL: https://issues.apache.org/jira/browse/YARN-4140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4140.patch, 0002-YARN-4140.patch,
> 0003-YARN-4140.patch
>
>
> Trying to run application on Nodelabel partition I found that the
> application execution time is delayed by 5 – 10 min for 500 containers .
> Total 3 machines 2 machines were in same partition and app submitted to same.
> After enabling debug was able to find the below
> # From AM the container ask is for OFF-SWITCH
> # RM allocating all containers to NODE_LOCAL as shown in logs below.
> # So since I was having about 500 containers time taken was about – 6 minutes
> to allocate 1st map after AM allocation.
> # Tested with about 1K maps using PI job took 17 minutes to allocate next
> container after AM allocation
> Once 500 container allocation on NODE_LOCAL is done the next container
> allocation is done on OFF_SWITCH
> {code}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> /default-rack, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location: *, Relax
> Locality: true, Node Label Expression: 3}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-143, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-117, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
>
> {code}
> 2015-09-09 14:35:45,467 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:45,831 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,469 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,832 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
> {code}
> dsperf@host-127:/opt/bibin/dsperf/HAINSTALL/install/hadoop/resourcemanager/logs1>
> cat hadoop-dsperf-resourcemanager-host-127.log | grep "NODE_LOCAL" | grep
> "root.b.b1" | wc -l
> 500
> {code}
>
> (Consumes about 6 minutes)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4140) RM container allocation delayed incase of app submitted to Nodelabel partition
[
https://issues.apache.org/jira/browse/YARN-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743483#comment-14743483
]
Hadoop QA commented on YARN-4140:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 56s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 18s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 50s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 54m 58s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 95m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755709/0003-YARN-4140.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9120/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9120/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9120/console |
This message was automatically generated.
> RM container allocation delayed incase of app submitted to Nodelabel partition
> --
>
> Key: YARN-4140
> URL: https://issues.apache.org/jira/browse/YARN-4140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4140.patch, 0002-YARN-4140.patch,
> 0003-YARN-4140.patch
>
>
> Trying to run application on Nodelabel partition I found that the
> application execution time is delayed by 5 – 10 min for 500 containers .
> Total 3 machines 2 machines were in same partition and app submitted to same.
> After enabling debug was able to find the below
> # From AM the container ask is for OFF-SWITCH
> # RM allocating all containers to NODE_LOCAL as shown in logs below.
> # So since I was having about 500 containers time taken was about – 6 minutes
> to allocate 1st map after AM allocation.
> # Tested with about 1K maps using PI job took 17 minutes to allocate next
> container after AM allocation
> Once 500 container allocation on NODE_LOCAL is done the next container
> allocation is done on OFF_SWITCH
> {code}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> /default-rack, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location: *, Relax
> Locality: true, Node Label Expression: 3}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-143, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-117, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server
[jira] [Commented] (YARN-3212) RMNode State Transition Update with DECOMMISSIONING state
[ https://issues.apache.org/jira/browse/YARN-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743495#comment-14743495 ] Junping Du commented on YARN-3212: -- Thanks [~leftnoteasy] for review and comments! bq. 1. Why shutdown a "decommissioning" NM if it is doing heartbeat. Should we allow it continue heartbeat, since RM needs to know about container finished / killed information. We don't shutdown a "decommissioning" NM. On the contrary, we differentiates nodes in decommissioning from others which get false in nodesListManager.isValidNode() check so it can still get running instead of decommissioned. bq. 2. Do we have timeout of graceful decomission? Which will update a node to "DECOMMISSIONED" after the timeout. There are some discussions in umbrella JIRA (https://issues.apache.org/jira/browse/YARN-914?focusedCommentId=14314653&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14314653), so we decide to track timeout in CLI instead of RM. The CLI patch (YARN-3225) also shows that. bq. 3. If I understand correct, decommissioning is another running state, except: We cannot allocate any new containers to it. Exactly. Another different is available resource should get updated with each running container get finished. bq. If answer to question #2 is no, I suggest to rename RMNodeEventType.DECOMISSION_WITH_TIMEOUT to GRACEFUL_DECOMISSION, since it doesn't have a "real" timeout. Already replied above that we support timeout in CLI. DECOMISSION_WITH_TIMEOUT sounds more clear comparing with old DECOMMISSION event. Thoughts? bq. Why this is need? .addTransition(NodeState.DECOMMISSIONING, NodeState.DECOMMISSIONING, RMNodeEventType.DECOMMISSION_WITH_TIMEOUT, new DecommissioningNodeTransition(NodeState.DECOMMISSIONING)) If not adding this transition, an InvalidStateTransitionException will get thrown in our state machine which sounds not right for a normal operation. bq. Should we simply ignore the DECOMMISSION_WITH_TIMEOUT event? No. RM should aware this event so later do some precisely update on available resource, etc. (YARN-3223). bq. Is there specific considerations that transfer UNHEALTHY to DECOMISSIONED when DECOMMISSION_WITH_TIMEOUT received? Is it better to transfer it to DECOMISSIONING since it has some containers running on it? I don't have a strong preference in this case. However, my previous consideration is UNHEALTHY event comes from machine monitor which indicate the node is not quite suitable for containers keep running while DECOMMISSION_WITH_TIMEOUT comes from user who is prefer to decommission a batch of nodes without affecting app/container running if there are currently running *normally*. So I think make it get decommissioned sounds a simpler way before we have more operation experience with this new feature. I have similiar view on discussion above on UNHEALTHY event to a decommissioning event (https://issues.apache.org/jira/browse/YARN-3212?focusedCommentId=14693360&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14693360). May be we can retrospect on this later? bq. One suggestion of how to handle node update to scheduler: I think you can add a field "isDecomissioning" to NodeUpdateSchedulerEvent, and scheduler can do all updates except allocate container. Thanks for good suggestion here. YARN-3223 will handle the balance of NM's total resource and used resource (so available resource is always 0). So this could be an option that we can use this way (new scheduler event) to keep NM resource balanced. There are also other options too so we can move the discussion to that JIRA I think. > RMNode State Transition Update with DECOMMISSIONING state > - > > Key: YARN-3212 > URL: https://issues.apache.org/jira/browse/YARN-3212 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Junping Du >Assignee: Junping Du > Attachments: RMNodeImpl - new.png, YARN-3212-v1.patch, > YARN-3212-v2.patch, YARN-3212-v3.patch, YARN-3212-v4.1.patch, > YARN-3212-v4.patch, YARN-3212-v5.1.patch, YARN-3212-v5.patch > > > As proposed in YARN-914, a new state of “DECOMMISSIONING” will be added and > can transition from “running” state triggered by a new event - > “decommissioning”. > This new state can be transit to state of “decommissioned” when > Resource_Update if no running apps on this NM or NM reconnect after restart. > Or it received DECOMMISSIONED event (after timeout from CLI). > In addition, it can back to “running” if user decides to cancel previous > decommission by calling recommission on the same node. The reaction to other > events is similar to RUNNING state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4072) ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager to support JvmPauseMonitor as a service
[ https://issues.apache.org/jira/browse/YARN-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-4072: -- Attachment: 0002-YARN-4072.patch Updating test case. Removed the check which counts the no: of child services. This check is not needed and not adding much value. > ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager > to support JvmPauseMonitor as a service > > > Key: YARN-4072 > URL: https://issues.apache.org/jira/browse/YARN-4072 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: 0001-YARN-4072.patch, 0002-YARN-4072.patch, > HADOOP-12407-001.patch > > > As JvmPauseMonitor is made as an AbstractService, subsequent method changes > are needed in all places which uses the monitor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4072) ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager to support JvmPauseMonitor as a service
[ https://issues.apache.org/jira/browse/YARN-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-4072: -- Attachment: HADOOP-12321-005-aggregated.patch Attaching aggregated patch to check Jenkins. > ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager > to support JvmPauseMonitor as a service > > > Key: YARN-4072 > URL: https://issues.apache.org/jira/browse/YARN-4072 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: 0001-YARN-4072.patch, 0002-YARN-4072.patch, > HADOOP-12321-005-aggregated.patch, HADOOP-12407-001.patch > > > As JvmPauseMonitor is made as an AbstractService, subsequent method changes > are needed in all places which uses the monitor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2691) User level API support for priority label
[ https://issues.apache.org/jira/browse/YARN-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743582#comment-14743582 ] Sunil G commented on YARN-2691: --- Hi [~rohithsharma] As we are not planning for Label support for priority, I feel we can close this ticket. pls share ur opinion. > User level API support for priority label > - > > Key: YARN-2691 > URL: https://issues.apache.org/jira/browse/YARN-2691 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Sunil G >Assignee: Rohith Sharma K S > Labels: BB2015-05-TBR > Attachments: YARN-2691.patch, YARN-2691.patch > > > Support for handling Application-Priority label coming from client to > ApplicationSubmissionContext. > Common api support for user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-313) Add Admin API for supporting node resource configuration in command line
[ https://issues.apache.org/jira/browse/YARN-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743588#comment-14743588 ] Jian He commented on YARN-313: -- lgtm > Add Admin API for supporting node resource configuration in command line > > > Key: YARN-313 > URL: https://issues.apache.org/jira/browse/YARN-313 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Junping Du >Assignee: Inigo Goiri >Priority: Critical > Attachments: YARN-313-sample.patch, YARN-313-v1.patch, > YARN-313-v10.patch, YARN-313-v11.patch, YARN-313-v2.patch, YARN-313-v3.patch, > YARN-313-v4.patch, YARN-313-v5.patch, YARN-313-v6.patch, YARN-313-v7.patch, > YARN-313-v8.patch, YARN-313-v9.patch > > > We should provide some admin interface, e.g. "yarn rmadmin -refreshResources" > to support changes of node's resource specified in a config file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-313) Add Admin API for supporting node resource configuration in command line
[ https://issues.apache.org/jira/browse/YARN-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743591#comment-14743591 ] Jian He commented on YARN-313: -- lgtm, +1 > Add Admin API for supporting node resource configuration in command line > > > Key: YARN-313 > URL: https://issues.apache.org/jira/browse/YARN-313 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Junping Du >Assignee: Inigo Goiri >Priority: Critical > Attachments: YARN-313-sample.patch, YARN-313-v1.patch, > YARN-313-v10.patch, YARN-313-v11.patch, YARN-313-v2.patch, YARN-313-v3.patch, > YARN-313-v4.patch, YARN-313-v5.patch, YARN-313-v6.patch, YARN-313-v7.patch, > YARN-313-v8.patch, YARN-313-v9.patch > > > We should provide some admin interface, e.g. "yarn rmadmin -refreshResources" > to support changes of node's resource specified in a config file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-313) Add Admin API for supporting node resource configuration in command line
[ https://issues.apache.org/jira/browse/YARN-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743592#comment-14743592 ] Junping Du commented on YARN-313: - +1 too. Will commit it shortly based on Jenkins' result. > Add Admin API for supporting node resource configuration in command line > > > Key: YARN-313 > URL: https://issues.apache.org/jira/browse/YARN-313 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client >Reporter: Junping Du >Assignee: Inigo Goiri >Priority: Critical > Attachments: YARN-313-sample.patch, YARN-313-v1.patch, > YARN-313-v10.patch, YARN-313-v11.patch, YARN-313-v2.patch, YARN-313-v3.patch, > YARN-313-v4.patch, YARN-313-v5.patch, YARN-313-v6.patch, YARN-313-v7.patch, > YARN-313-v8.patch, YARN-313-v9.patch > > > We should provide some admin interface, e.g. "yarn rmadmin -refreshResources" > to support changes of node's resource specified in a config file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[
https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743594#comment-14743594
]
Sunil G commented on YARN-4141:
---
HI [~rohithsharma]
As we are not updating priority if app is in final states, I feel action is not
success from user point of view. Because if user tries to verify the priority
from server side, it will be still old value. Also in
{{RMAuditLogger.logSuccess}}, we cannot supply any useful messages such as
"priority change is skipped since app is in final states". Hence I thought we
can print logFailure for now. How do you feel? cc/[~jianhe]
> Runtime Application Priority change should not throw exception for
> applications at finishing states
> ---
>
> Key: YARN-4141
> URL: https://issues.apache.org/jira/browse/YARN-4141
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch
>
>
> As suggested by [~jlowe] in
> [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035]
> , its good that if YARN can suppress exceptions during change application
> priority calls for applications at its finishing stages.
> Currently it will be difficult for clients to handle this. This will be
> similar to kill application behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[ https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743602#comment-14743602 ] Jason Lowe commented on YARN-4141: -- When the user tries to verify the priority they will get an app report showing the job is in a terminal state which will explain why the priority isn't updated. This is akin to killing an application that already completed. We don't retroactively mark it as killed, and we also don't report an error to the user when they tried to kill the terminated application. > Runtime Application Priority change should not throw exception for > applications at finishing states > --- > > Key: YARN-4141 > URL: https://issues.apache.org/jira/browse/YARN-4141 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch > > > As suggested by [~jlowe] in > [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035] > , its good that if YARN can suppress exceptions during change application > priority calls for applications at its finishing stages. > Currently it will be difficult for clients to handle this. This will be > similar to kill application behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2691) User level API support for priority label
[
https://issues.apache.org/jira/browse/YARN-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743601#comment-14743601
]
Hadoop QA commented on YARN-2691:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12683391/YARN-2691.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9125/console |
This message was automatically generated.
> User level API support for priority label
> -
>
> Key: YARN-2691
> URL: https://issues.apache.org/jira/browse/YARN-2691
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Sunil G
>Assignee: Rohith Sharma K S
> Labels: BB2015-05-TBR
> Attachments: YARN-2691.patch, YARN-2691.patch
>
>
> Support for handling Application-Priority label coming from client to
> ApplicationSubmissionContext.
> Common api support for user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS
[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743621#comment-14743621 ] Jason Lowe commented on YARN-3942: -- Thanks for trying out the patch, [~gss2002]! Sorry to hear it's been crashing in your setup. If you could provide a bit more details on the types of crashes (e.g.: stacktraces) that would be very helpful. If they are OOM type of crashes then it could either be the Hive session reuse problem that Hitesh mentioned or possibly there are too many jobs being cached simultaneously for the heap size being used by the ATS. > Timeline store to read events from HDFS > --- > > Key: YARN-3942 > URL: https://issues.apache.org/jira/browse/YARN-3942 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-3942.001.patch > > > This adds a new timeline store plugin that is intended as a stop-gap measure > to mitigate some of the issues we've seen with ATS v1 while waiting for ATS > v2. The intent of this plugin is to provide a workable solution for running > the Tez UI against the timeline server on a large-scale clusters running many > thousands of jobs per day. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743622#comment-14743622
]
Jian He commented on YARN-3635:
---
+1, committing soon if no further comments.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[ https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-4141: -- Attachment: 0003-YARN-4141.patch Thank you [~jlowe] for sharing the comments. Yes. I understood the case as in comparison with kill operation. The app will be finished when user checks reports again. so its fine there. Attaching patch by addressing this point. > Runtime Application Priority change should not throw exception for > applications at finishing states > --- > > Key: YARN-4141 > URL: https://issues.apache.org/jira/browse/YARN-4141 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch, > 0003-YARN-4141.patch > > > As suggested by [~jlowe] in > [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035] > , its good that if YARN can suppress exceptions during change application > priority calls for applications at its finishing stages. > Currently it will be difficult for clients to handle this. This will be > similar to kill application behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4142) add a way for an attempt to report an attempt failure
[ https://issues.apache.org/jira/browse/YARN-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743628#comment-14743628 ] Jason Lowe commented on YARN-4142: -- One issue with sending diagnostics on the heartbeat is what to do when an AM ends up sending them _every_ heartbeat. If they all concatenate then this could be a significant memory burden on the RM. If subsequent diagnostics eclipse earlier ones then it's less of an issue, but then there will be situations where that may be undesirable. Maybe we only store the last N Kbytes of diagnostic information from an AM attempt to keep it from overwhelming the RM. > add a way for an attempt to report an attempt failure > - > > Key: YARN-4142 > URL: https://issues.apache.org/jira/browse/YARN-4142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > Currently AMs can report a failure with exit code and diagnostics text —but > only when exiting to a failed state. If the AM terminates for any other > reason there's no information held in the RM, just the logs somewhere —and we > know they don't always last. > When an application explicitly terminates an attempt, it would be nice if it > could optionally report something to the RM before it exited. The most > recent set of these could then be included in Application Reports, so > allowing client apps to count attempt failures and get exit details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743646#comment-14743646 ] Wangda Tan commented on YARN-1651: -- javadocs warnings are not related to this patch, mr test failure tracked by: https://issues.apache.org/jira/browse/MAPREDUCE-6475. Findbugs warnings are not related to this patch: https://issues.apache.org/jira/browse/YARN-1644?focusedCommentId=14739800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14739800. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, > YARN-1651-3.YARN-1197.patch, YARN-1651-4.YARN-1197.patch, > YARN-1651-5.YARN-1197.patch, YARN-1651-6.YARN-1197.patch, > YARN-1651-7.YARN-1197.patch, YARN-1651-8.YARN-1197.patch, > YARN-1651-9.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-313) Add Admin API for supporting node resource configuration in command line
[
https://issues.apache.org/jira/browse/YARN-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743656#comment-14743656
]
Hadoop QA commented on YARN-313:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 8m 12s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 11s | The applied patch generated 1
new checkstyle issues (total was 230, now 230). |
| {color:green}+1{color} | whitespace | 0m 7s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 33s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 6m 57s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 0s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 39m 19s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 97m 46s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing |
| | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog |
| | hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| | hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels |
| | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| |
hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
|
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| | hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetrics |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate |
| | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
h
[jira] [Commented] (YARN-4142) add a way for an attempt to report an attempt failure
[ https://issues.apache.org/jira/browse/YARN-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743668#comment-14743668 ] Sunil G commented on YARN-4142: --- Yes [~jlowe]. RM need to have some safeguard mechanism such as no: of diagnostic messages to store per attempt OR last some KBytes of diagnostics etc (configuration based may be?). This will be helpful in saving the memory. I will try making these changes and will share a prototype if no issues. Is it ok to go ahead? > add a way for an attempt to report an attempt failure > - > > Key: YARN-4142 > URL: https://issues.apache.org/jira/browse/YARN-4142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > Currently AMs can report a failure with exit code and diagnostics text —but > only when exiting to a failed state. If the AM terminates for any other > reason there's no information held in the RM, just the logs somewhere —and we > know they don't always last. > When an application explicitly terminates an attempt, it would be nice if it > could optionally report something to the RM before it exited. The most > recent set of these could then be included in Application Reports, so > allowing client apps to count attempt failures and get exit details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4142) add a way for an attempt to report an attempt failure
[ https://issues.apache.org/jira/browse/YARN-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743719#comment-14743719 ] Jason Lowe commented on YARN-4142: -- Sure, as long as there's a limiting mechanism it should be OK to prototype for adding diagnostics to heartbeats. We may also want the ability to unregister cleanly yet still launch another attempt which is another way to accomplish the original JIRA request, but we can do that separately if needed. As Steve mentioned above, having the ability to update diagnostics while the app is running could be useful for scenarios outside of attempt failure. > add a way for an attempt to report an attempt failure > - > > Key: YARN-4142 > URL: https://issues.apache.org/jira/browse/YARN-4142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > Currently AMs can report a failure with exit code and diagnostics text —but > only when exiting to a failed state. If the AM terminates for any other > reason there's no information held in the RM, just the logs somewhere —and we > know they don't always last. > When an application explicitly terminates an attempt, it would be nice if it > could optionally report something to the RM before it exited. The most > recent set of these could then be included in Application Reports, so > allowing client apps to count attempt failures and get exit details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4072) ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager to support JvmPauseMonitor as a service
[
https://issues.apache.org/jira/browse/YARN-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743746#comment-14743746
]
Hadoop QA commented on YARN-4072:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 28m 15s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 9m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 5m 55s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 2m 8s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 10m 41s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 0s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 5m 49s | Tests passed in
hadoop-mapreduce-client-hs. |
| {color:green}+1{color} | yarn tests | 3m 8s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| {color:red}-1{color} | yarn tests | 6m 30s | Tests failed in
hadoop-yarn-server-nodemanager. |
| {color:red}-1{color} | yarn tests | 0m 19s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:red}-1{color} | yarn tests | 0m 11s | Tests failed in
hadoop-yarn-server-web-proxy. |
| {color:red}-1{color} | hdfs tests | 0m 22s | Tests failed in hadoop-hdfs. |
| {color:red}-1{color} | hdfs tests | 0m 12s | Tests failed in
hadoop-hdfs-nfs. |
| | | 109m 24s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.nodemanager.TestNodeStatusUpdater |
| |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| Timed out tests |
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown |
| Failed build | hadoop-yarn-server-resourcemanager |
| | hadoop-yarn-server-web-proxy |
| | hadoop-hdfs |
| | hadoop-hdfs-nfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755742/HADOOP-12321-005-aggregated.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| hadoop-yarn-server-applicationhistoryservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-web-proxy test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-nfs test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/artifact/patchprocess/testrun_hadoop-hdfs-nfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9124/console |
This message was automatically generated.
> ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager
> to support JvmPauseMonitor as a service
>
>
> Key: YARN-4072
> URL: https://issues.apache.org/jira/browse/YARN-4072
> Project: Hadoop YARN
> Issue Type: Improvement
>
[jira] [Updated] (YARN-4149) yarn logs -am should provide an option to fetch all the log files
[
https://issues.apache.org/jira/browse/YARN-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-4149:
Attachment: YARN-4149.002.patch
Thanks for the review Xuan!
bq. Can we do the same for fetching regular containers logs? In current patch,
looks like that we only do it for am containers
Fixed.
bq. If the application is finished, looks like that we could not get any logs
if we specify --logfiles ALL. Because we can not get information from web
service call. Just set requestedLogFiles as NULL if the application is finished
which should fix this.
Fixed.
bq.Are the test case failures related ?
One of them was. I've fixed it.
> yarn logs -am should provide an option to fetch all the log files
> -
>
> Key: YARN-4149
> URL: https://issues.apache.org/jira/browse/YARN-4149
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, nodemanager
>Affects Versions: 2.7.1
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4149.001.patch, YARN-4149.002.patch
>
>
> From [~gopalv] -
> {quote}
> Trying to collect a hanging Tez AM logs, by killing the container and running
> yarn logs -applicationId application_1437098194051_0178 -am ALL
> The output contains only one log file, which does not contain any of the
> actual execution logs, only the initialization logs.
> From YARN-3347, I note that
> // if we do not specify the value for CONTAINER_LOG_FILES option,
> // we will only output syslog
> This means that the person calling the yarn logs command has to list it out
> like this, to collect logs
> yarn logs -applicationId application_1437098194051_0178 -am ALL -logFiles \
> syslog_dag_1437098194051_0178_2_post,\
> dag_1437098194051_0178_2-tez-dag.pb.txt,\
> syslog_dag_1437098194051_0178_2,\
> syslog_dag_1437098194051_0178_1_post,\
> syslog_dag_1437098194051_0178_1,\
> syslog,\
> stdout,\
> stderr,\
> dag_1437098194051_0178_2.dot,\
> dag_1437098194051_0178_1.dot,\
> dag_1437098194051_0178_1-tez-dag.pb.txt
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4009) CORS support for ResourceManager REST API
[ https://issues.apache.org/jira/browse/YARN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743796#comment-14743796 ] Varun Vasudev commented on YARN-4009: - [~jeagles] - did you get a chance to review the patch? Thanks! > CORS support for ResourceManager REST API > - > > Key: YARN-4009 > URL: https://issues.apache.org/jira/browse/YARN-4009 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Prakash Ramachandran >Assignee: Varun Vasudev > Attachments: YARN-4009.001.patch > > > Currently the REST API's do not have CORS support. This means any UI (running > in browser) cannot consume the REST API's. For ex Tez UI would like to use > the REST API for getting application, application attempt information exposed > by the API's. > It would be very useful if CORS is enabled for the REST API's. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4142) add a way for an attempt to report an attempt failure
[ https://issues.apache.org/jira/browse/YARN-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743802#comment-14743802 ] Sunil G commented on YARN-4142: --- Yes [~jlowe]. A limiting mechanism will be added for diagnostics. I will share a prototype. "unregister cleanly from an attempt, and launch a new attempt" is also a useful one, its good if we can track the same in separate ticket. will raise a ticket if its fine to go in that direction also. > add a way for an attempt to report an attempt failure > - > > Key: YARN-4142 > URL: https://issues.apache.org/jira/browse/YARN-4142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > Currently AMs can report a failure with exit code and diagnostics text —but > only when exiting to a failed state. If the AM terminates for any other > reason there's no information held in the RM, just the logs somewhere —and we > know they don't always last. > When an application explicitly terminates an attempt, it would be nice if it > could optionally report something to the RM before it exited. The most > recent set of these could then be included in Application Reports, so > allowing client apps to count attempt failures and get exit details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4142) add a way for an attempt to report an attempt failure
[ https://issues.apache.org/jira/browse/YARN-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-4142: - Assignee: Sunil G > add a way for an attempt to report an attempt failure > - > > Key: YARN-4142 > URL: https://issues.apache.org/jira/browse/YARN-4142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Sunil G > > Currently AMs can report a failure with exit code and diagnostics text —but > only when exiting to a failed state. If the AM terminates for any other > reason there's no information held in the RM, just the logs somewhere —and we > know they don't always last. > When an application explicitly terminates an attempt, it would be nice if it > could optionally report something to the RM before it exited. The most > recent set of these could then be included in Application Reports, so > allowing client apps to count attempt failures and get exit details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[
https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743824#comment-14743824
]
Hadoop QA commented on YARN-4141:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 55s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 58s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 30s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 53m 42s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 57s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755752/0003-YARN-4141.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9126/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9126/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9126/console |
This message was automatically generated.
> Runtime Application Priority change should not throw exception for
> applications at finishing states
> ---
>
> Key: YARN-4141
> URL: https://issues.apache.org/jira/browse/YARN-4141
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch,
> 0003-YARN-4141.patch
>
>
> As suggested by [~jlowe] in
> [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035]
> , its good that if YARN can suppress exceptions during change application
> priority calls for applications at its finishing stages.
> Currently it will be difficult for clients to handle this. This will be
> similar to kill application behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1510) Make NMClient support change container resources
[ https://issues.apache.org/jira/browse/YARN-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MENG DING updated YARN-1510: Attachment: YARN-1510-YARN-1197.1.patch Attaching the first version for review. I got a bunch of unrelated Javadoc and checkstyle warnings during my own testing. Will see if I get the same warnings after submitting the patch. > Make NMClient support change container resources > > > Key: YARN-1510 > URL: https://issues.apache.org/jira/browse/YARN-1510 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1510-YARN-1197.1.patch > > > As described in YARN-1197, YARN-1449, we need add API in NMClient to support > 1) sending request of increase/decrease container resource limits > 2) get succeeded/failed changed containers response from NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[
https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743869#comment-14743869
]
Sangjin Lee commented on YARN-3901:
---
Thanks for updating the patch [~vrushalic]. It seems that the patch does not
apply cleanly because {{TimelineSchemaCreator.java}} has been updated for
YARN-4102. Could you please update the patch so it applies cleanly? Thanks.
https://builds.apache.org/job/PreCommit-YARN-Build/9119/console
> Populate flow run data in the flow_run & flow activity tables
> -
>
> Key: YARN-3901
> URL: https://issues.apache.org/jira/browse/YARN-3901
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Attachments: YARN-3901-YARN-2928.1.patch,
> YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch,
> YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch,
> YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch
>
>
> As per the schema proposed in YARN-3815 in
> https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf
> filing jira to track creation and population of data in the flow run table.
> Some points that are being considered:
> - Stores per flow run information aggregated across applications, flow version
> RM’s collector writes to on app creation and app completion
> - Per App collector writes to it for metric updates at a slower frequency
> than the metric updates to application table
> primary key: cluster ! user ! flow ! flow run id
> - Only the latest version of flow-level aggregated metrics will be kept, even
> if the entity and application level keep a timeseries.
> - The running_apps column will be incremented on app creation, and
> decremented on app completion.
> - For min_start_time the RM writer will simply write a value with the tag for
> the applicationId. A coprocessor will return the min value of all written
> values. -
> - Upon flush and compactions, the min value between all the cells of this
> column will be written to the cell without any tag (empty tag) and all the
> other cells will be discarded.
> - Ditto for the max_end_time, but then the max will be kept.
> - Tags are represented as #type:value. The type can be not set (0), or can
> indicate running (1) or complete (2). In those cases (for metrics) only
> complete app metrics are collapsed on compaction.
> - The m! values are aggregated (summed) upon read. Only when applications are
> completed (indicated by tag type 2) can the values be collapsed.
> - The application ids that have completed and been aggregated into the flow
> numbers are retained in a separate column for historical tracking: we don’t
> want to re-aggregate for those upon replay
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4141) Runtime Application Priority change should not throw exception for applications at finishing states
[ https://issues.apache.org/jira/browse/YARN-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743923#comment-14743923 ] Sunil G commented on YARN-4141: --- These failed tests are passing locally. > Runtime Application Priority change should not throw exception for > applications at finishing states > --- > > Key: YARN-4141 > URL: https://issues.apache.org/jira/browse/YARN-4141 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-4141.patch, 0002-YARN-4141.patch, > 0003-YARN-4141.patch > > > As suggested by [~jlowe] in > [MAPREDUCE-5870-comment|https://issues.apache.org/jira/browse/MAPREDUCE-5870?focusedCommentId=14737035&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14737035] > , its good that if YARN can suppress exceptions during change application > priority calls for applications at its finishing stages. > Currently it will be difficult for clients to handle this. This will be > similar to kill application behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4149) yarn logs -am should provide an option to fetch all the log files
[
https://issues.apache.org/jira/browse/YARN-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743973#comment-14743973
]
Hadoop QA commented on YARN-4149:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 1s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 9m 6s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 13m 18s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 51s | The applied patch generated 8
new checkstyle issues (total was 40, now 46). |
| {color:red}-1{color} | checkstyle | 1m 12s | The applied patch generated 6
new checkstyle issues (total was 19, now 25). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 46s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 49s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 48s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 7m 9s | Tests failed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 7m 52s | Tests failed in
hadoop-yarn-server-nodemanager. |
| | | 65m 0s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-client |
| Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| Timed out tests | org.apache.hadoop.yarn.server.nodemanager.TestEventFlow |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755764/YARN-4149.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9128/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
https://builds.apache.org/job/PreCommit-YARN-Build/9128/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9128/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9128/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9128/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9128/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9128/console |
This message was automatically generated.
> yarn logs -am should provide an option to fetch all the log files
> -
>
> Key: YARN-4149
> URL: https://issues.apache.org/jira/browse/YARN-4149
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, nodemanager
>Affects Versions: 2.7.1
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4149.001.patch, YARN-4149.002.patch
>
>
> From [~gopalv] -
> {quote}
> Trying to collect a hanging Tez AM logs, by killing the container and running
> yarn logs -applicationId application_1437098194051_0178 -am ALL
> The output contains only one log file, which does not contain any of the
> actual execution logs, only the initialization logs.
> From YARN-3347, I note that
> // if we do not specify the value for CONTAINER_LOG_FILES option,
> // we will only output syslog
> This means that the person calling the yarn logs command has to list it out
> like this, to collect logs
> yarn logs -applicationId application_1437098194051_0178 -am ALL -logFiles \
> syslog_dag_1437098194051_0178_2_post,\
> dag_1437098194051_0178_2-tez-dag.pb.txt,\
> syslog_dag_1437098194051_0178_2,\
> syslog_dag_1437098194051_0178_1_post,\
> syslog_dag_1437098194051_0178_1,\
> syslog,\
> stdout,\
> stderr,\
> dag_1437098194051_0178_2.dot,\
> dag_1437098194051_0178_1.dot,\
> dag_1437098194051_0178_1-tez-dag.pb.txt
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744006#comment-14744006 ] Anubhav Dhoot commented on YARN-3784: - Hi [~sunilg] this does not include support for FairScheduler. Are we planning to add that here or have a separate jira to track that work? Thx > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4151) findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MENG DING updated YARN-4151:
Attachment: YARN-4151.1.patch
Attaching the patch that addresses the findbugs warnings. Some of the warnings
are introduced by YARN-4055 and YARN-3948
I am not really sure if we need to fix the following though. We may still want
to keep the branch as is, as different scheduler pages may have different
column definitions for the cluster page.
{code}
@@ -51,8 +51,7 @@ private static String getAppsTableColumnDefs(
sb.append("[\n")
.append("{'sType':'string', 'aTargets': [0]")
.append(", 'mRender': parseHadoopID }")
- .append("\n, {'sType':'numeric', 'aTargets': " +
- (isFairSchedulerPage ? "[6, 7]": "[6, 7]"))
+ .append("\n, {'sType':'numeric', 'aTargets': [6, 7]")
.append(", 'mRender': renderHadoopDate }")
.append("\n, {'sType':'numeric', bSearchable:false, 'aTargets':");
if (isFairSchedulerPage) {
{code}
> findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2513) Host framework UIs in YARN for use with the ATS
[ https://issues.apache.org/jira/browse/YARN-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-2513: -- Attachment: YARN-2513.v5.patch Fixing (hopefully) the checkstyle warnings and adding a unit test to show functionality. > Host framework UIs in YARN for use with the ATS > --- > > Key: YARN-2513 > URL: https://issues.apache.org/jira/browse/YARN-2513 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Labels: 2.6.1-candidate > Attachments: YARN-2513-v1.patch, YARN-2513-v2.patch, > YARN-2513.v3.patch, YARN-2513.v4.patch, YARN-2513.v5.patch > > > Allow for pluggable UIs as described by TEZ-8. Yarn can provide the > infrastructure to host java script and possible java UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4151) findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744232#comment-14744232
]
Hadoop QA commented on YARN-4151:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 13s | Pre-patch trunk has 7 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 33s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings, and fixes 7 pre-existing warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-server-common. |
| | | 38m 35s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755799/YARN-4151.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9130/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9130/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9130/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9130/console |
This message was automatically generated.
> findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744270#comment-14744270 ] Li Lu commented on YARN-3901: - OK, thanks for the info! > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch, > YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch, > YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1510) Make NMClient support change container resources
[
https://issues.apache.org/jira/browse/YARN-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744282#comment-14744282
]
Hadoop QA commented on YARN-1510:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 3m 25s | YARN-1197 compilation may be
broken. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 39s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 10m 49s | The applied patch generated 65
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 14s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 37s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 10s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 7m 6s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 53m 54s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 98m 43s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-client |
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755771/YARN-1510-YARN-1197.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-1197 / 78ad04d |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/artifact/patchprocess/diffJavadocWarnings.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9129/console |
This message was automatically generated.
> Make NMClient support change container resources
>
>
> Key: YARN-1510
> URL: https://issues.apache.org/jira/browse/YARN-1510
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1510-YARN-1197.1.patch
>
>
> As described in YARN-1197, YARN-1449, we need add API in NMClient to support
> 1) sending request of increase/decrease container resource limits
> 2) get succeeded/failed changed containers response from NM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[
https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744320#comment-14744320
]
Joep Rottinghuis commented on YARN-3901:
[~sjlee0]
bq. I remember Joep Rottinghuis mentioning that an unset timestamp is
equivalent to Cell.getTimestamp() returning Long.MAX_VALUE. Joep Rottinghuis?
Yes, if you see the org.apache.hadoop.hbase.client.Put#addColumn method without
a timestamp, it uses the timestamp from the Put. There are two constructors for
a Put, one with the timestamp, one without. The one without uses
HConstants.LATEST_TIMESTAMP which is defined as:
{code}
/**
* Timestamp to use when we want to refer to the latest cell.
* This is the timestamp sent by clients when no timestamp is specified on
* commit.
*/
public static final long LATEST_TIMESTAMP = Long.MAX_VALUE;
{code}
That is then used (indirectly through LATEST_TIMESTAMP_BYTES) in the KeyValue
class in the #isLatestTimestamp method, which in turn is used in the
KeyValue#updateLatestStamp that sets it to "now" on the server side.
I'm not 100% sure (we need to test this) but I'm assuming that the
transformation of this isLatestTimestamp happens after coprocessors or never at
all (the cells might just be written with the latest timestamp, and one might
not have the ability to ask what the row looked like at any particular time at
all). I thought this might be overwritten on the server side, but can't find
that code now.
> Populate flow run data in the flow_run & flow activity tables
> -
>
> Key: YARN-3901
> URL: https://issues.apache.org/jira/browse/YARN-3901
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
> Attachments: YARN-3901-YARN-2928.1.patch,
> YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch,
> YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch,
> YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch
>
>
> As per the schema proposed in YARN-3815 in
> https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf
> filing jira to track creation and population of data in the flow run table.
> Some points that are being considered:
> - Stores per flow run information aggregated across applications, flow version
> RM’s collector writes to on app creation and app completion
> - Per App collector writes to it for metric updates at a slower frequency
> than the metric updates to application table
> primary key: cluster ! user ! flow ! flow run id
> - Only the latest version of flow-level aggregated metrics will be kept, even
> if the entity and application level keep a timeseries.
> - The running_apps column will be incremented on app creation, and
> decremented on app completion.
> - For min_start_time the RM writer will simply write a value with the tag for
> the applicationId. A coprocessor will return the min value of all written
> values. -
> - Upon flush and compactions, the min value between all the cells of this
> column will be written to the cell without any tag (empty tag) and all the
> other cells will be discarded.
> - Ditto for the max_end_time, but then the max will be kept.
> - Tags are represented as #type:value. The type can be not set (0), or can
> indicate running (1) or complete (2). In those cases (for metrics) only
> complete app metrics are collapsed on compaction.
> - The m! values are aggregated (summed) upon read. Only when applications are
> completed (indicated by tag type 2) can the values be collapsed.
> - The application ids that have completed and been aggregated into the flow
> numbers are retained in a separate column for historical tracking: we don’t
> want to re-aggregate for those upon replay
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4151) findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744334#comment-14744334
]
MENG DING commented on YARN-4151:
-
This patch fixes findbugs warnings, so no test cases are added.
> findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2513) Host framework UIs in YARN for use with the ATS
[
https://issues.apache.org/jira/browse/YARN-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744374#comment-14744374
]
Hadoop QA commented on YARN-2513:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 21m 45s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 3s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 2s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 38s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 5s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 3m 11s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| | | 57m 17s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755807/YARN-2513.v5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 6955771 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9131/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9131/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9131/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-applicationhistoryservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9131/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9131/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9131/console |
This message was automatically generated.
> Host framework UIs in YARN for use with the ATS
> ---
>
> Key: YARN-2513
> URL: https://issues.apache.org/jira/browse/YARN-2513
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Labels: 2.6.1-candidate
> Attachments: YARN-2513-v1.patch, YARN-2513-v2.patch,
> YARN-2513.v3.patch, YARN-2513.v4.patch, YARN-2513.v5.patch
>
>
> Allow for pluggable UIs as described by TEZ-8. Yarn can provide the
> infrastructure to host java script and possible java UIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1510) Make NMClient support change container resources
[ https://issues.apache.org/jira/browse/YARN-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MENG DING updated YARN-1510: Attachment: YARN-1510-YARN-1197.2.patch * The javadoc warnings are not related to this patch. * The findbugs warnings are fixed * The summary shows test failure, but the report doesn't show any failed tests. Attach new patch. > Make NMClient support change container resources > > > Key: YARN-1510 > URL: https://issues.apache.org/jira/browse/YARN-1510 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1510-YARN-1197.1.patch, YARN-1510-YARN-1197.2.patch > > > As described in YARN-1197, YARN-1449, we need add API in NMClient to support > 1) sending request of increase/decrease container resource limits > 2) get succeeded/failed changed containers response from NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2513) Host framework UIs in YARN for use with the ATS
[
https://issues.apache.org/jira/browse/YARN-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744388#comment-14744388
]
Jonathan Eagles commented on YARN-2513:
---
Checkstyle issue is not related to patch.
{quote}
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:1:
File length is 2,153 lines (max allowed is 2,000).
{quote}
[~hitesh], [~xgong], [~zjshen], can you have another look at this patch when
you get a chance?
> Host framework UIs in YARN for use with the ATS
> ---
>
> Key: YARN-2513
> URL: https://issues.apache.org/jira/browse/YARN-2513
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Labels: 2.6.1-candidate
> Attachments: YARN-2513-v1.patch, YARN-2513-v2.patch,
> YARN-2513.v3.patch, YARN-2513.v4.patch, YARN-2513.v5.patch
>
>
> Allow for pluggable UIs as described by TEZ-8. Yarn can provide the
> infrastructure to host java script and possible java UIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4151) findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1475#comment-1475
]
Wangda Tan commented on YARN-4151:
--
LGTM, committing...
> findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-4151:
-
Summary: Fix findbugs errors in hadoop-yarn-server-common module (was:
findbugs errors in hadoop-yarn-server-common module)
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4156) testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler
Anubhav Dhoot created YARN-4156: --- Summary: testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler Key: YARN-4156 URL: https://issues.apache.org/jira/browse/YARN-4156 Project: Hadoop YARN Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot The assumes the scheduler is CapacityScheduler without configuring it as such. This causes it to fail if the default is something else such as the FairScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4156) testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated YARN-4156: Attachment: YARN-4156.001.patch Uploading patch that configures the scheduler to be CapcacityScheduler > testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler > -- > > Key: YARN-4156 > URL: https://issues.apache.org/jira/browse/YARN-4156 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-4156.001.patch > > > The assumes the scheduler is CapacityScheduler without configuring it as > such. This causes it to fail if the default is something else such as the > FairScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3901: - Attachment: YARN-3901-YARN-2928.8.patch Attaching v8 patch. Thanks [~sjlee0] for the code review and helping restructure the loop in FlowScanner#nextInternal > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch, > YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch, > YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch, > YARN-3901-YARN-2928.8.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4140) RM container allocation delayed incase of app submitted to Nodelabel partition
[
https://issues.apache.org/jira/browse/YARN-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744487#comment-14744487
]
Wangda Tan commented on YARN-4140:
--
Thanks [~bibinchundatt],
Few comments:
- It's no need to maintain an {{anyPrioritymap}}, getResourceRequest(Priority,
resourceName) should enough.
Current logic seems not correct to me, it should cover following cases:
- Newly-added-non-ANY request should have same labelExpression of ANY request.
(done in your patch already)
- If ANY request newly added OR labelExpression changed of ANY request, you
should update ALL resource requests' labelExpression.
And if you agree about above logic, tests should cover above cases as well.
> RM container allocation delayed incase of app submitted to Nodelabel partition
> --
>
> Key: YARN-4140
> URL: https://issues.apache.org/jira/browse/YARN-4140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4140.patch, 0002-YARN-4140.patch,
> 0003-YARN-4140.patch
>
>
> Trying to run application on Nodelabel partition I found that the
> application execution time is delayed by 5 – 10 min for 500 containers .
> Total 3 machines 2 machines were in same partition and app submitted to same.
> After enabling debug was able to find the below
> # From AM the container ask is for OFF-SWITCH
> # RM allocating all containers to NODE_LOCAL as shown in logs below.
> # So since I was having about 500 containers time taken was about – 6 minutes
> to allocate 1st map after AM allocation.
> # Tested with about 1K maps using PI job took 17 minutes to allocate next
> container after AM allocation
> Once 500 container allocation on NODE_LOCAL is done the next container
> allocation is done on OFF_SWITCH
> {code}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> /default-rack, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location: *, Relax
> Locality: true, Node Label Expression: 3}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-143, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-117, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
>
> {code}
> 2015-09-09 14:35:45,467 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:45,831 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,469 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,832 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
> {code}
> dsperf@host-127:/opt/bibin/dsperf/HAINSTALL/insta
[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3901: - Attachment: YARN-3901-YARN-2928.8.patch > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch, > YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch, > YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch, > YARN-3901-YARN-2928.8.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3901: - Attachment: (was: YARN-3901-YARN-2928.8.patch) > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch, > YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch, > YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch, > YARN-3901-YARN-2928.8.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3717) Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API
[
https://issues.apache.org/jira/browse/YARN-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744495#comment-14744495
]
Wangda Tan commented on YARN-3717:
--
Thanks [~Naganarasimha], patch generally looks good, one nit:
- Could you put and to static final constant
variable of {{NodeLabel}}?
> Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API
> -
>
> Key: YARN-3717
> URL: https://issues.apache.org/jira/browse/YARN-3717
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: 3717_cluster_test_snapshots.zip, RMLogsForHungJob.log,
> YARN-3717.20150822-1.patch, YARN-3717.20150824-1.patch,
> YARN-3717.20150825-1.patch, YARN-3717.20150826-1.patch,
> YARN-3717.20150911-1.patch, YARN-3717.20150912-1.patch
>
>
> 1> Add the default-node-Label expression for each queue in scheduler page.
> 2> In Application/Appattempt page show the app configured node label
> expression for AM and Job
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744541#comment-14744541
]
Hudson commented on YARN-4151:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8450 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8450/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4149) yarn logs -am should provide an option to fetch all the log files
[
https://issues.apache.org/jira/browse/YARN-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744566#comment-14744566
]
Xuan Gong commented on YARN-4149:
-
Thanks, [~vvasudev] The patch looks good overall. One nit:
* Could you check whether the logFiles array contains "ALL" instead of just
checking whether "ALL" is the first item in logFiles array?
* Could you comment whether the testcase failures/findbug are related or not ?
> yarn logs -am should provide an option to fetch all the log files
> -
>
> Key: YARN-4149
> URL: https://issues.apache.org/jira/browse/YARN-4149
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, nodemanager
>Affects Versions: 2.7.1
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4149.001.patch, YARN-4149.002.patch
>
>
> From [~gopalv] -
> {quote}
> Trying to collect a hanging Tez AM logs, by killing the container and running
> yarn logs -applicationId application_1437098194051_0178 -am ALL
> The output contains only one log file, which does not contain any of the
> actual execution logs, only the initialization logs.
> From YARN-3347, I note that
> // if we do not specify the value for CONTAINER_LOG_FILES option,
> // we will only output syslog
> This means that the person calling the yarn logs command has to list it out
> like this, to collect logs
> yarn logs -applicationId application_1437098194051_0178 -am ALL -logFiles \
> syslog_dag_1437098194051_0178_2_post,\
> dag_1437098194051_0178_2-tez-dag.pb.txt,\
> syslog_dag_1437098194051_0178_2,\
> syslog_dag_1437098194051_0178_1_post,\
> syslog_dag_1437098194051_0178_1,\
> syslog,\
> stdout,\
> stderr,\
> dag_1437098194051_0178_2.dot,\
> dag_1437098194051_0178_1.dot,\
> dag_1437098194051_0178_1-tez-dag.pb.txt
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4074) [timeline reader] implement support for querying for flows and flow runs
[
https://issues.apache.org/jira/browse/YARN-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-4074:
--
Attachment: YARN-4074-YARN-2928.POC.005.patch
The POC v.5 patch posted.
It mostly rebases with the v.8 patch for YARN-3901.
It should apply cleanly on top of the v.8 patch for YARN-3901. Again, your
comments are greatly appreciated.
> [timeline reader] implement support for querying for flows and flow runs
>
>
> Key: YARN-4074
> URL: https://issues.apache.org/jira/browse/YARN-4074
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: YARN-4074-YARN-2928.POC.001.patch,
> YARN-4074-YARN-2928.POC.002.patch, YARN-4074-YARN-2928.POC.003.patch,
> YARN-4074-YARN-2928.POC.004.patch, YARN-4074-YARN-2928.POC.005.patch
>
>
> Implement support for querying for flows and flow runs.
> We should be able to query for the most recent N flows, etc.
> This includes changes to the {{TimelineReader}} API if necessary, as well as
> implementation of the API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated YARN-1897: -- Attachment: YARN-1897-5.patch Based on the offline discussion with [~djp] [~ste...@apache.org] [~xgong] w.r.t. to YARN-4131 and signal container functionality, we decided to speed up the open source effort for signal container functionality. Instead of defining a comprehensive API to all scenarios, we prefer to addressing immediate scenarios first. Here is the draft patch that provides the core signal container functionality so that users can use YARN CLI to capture thread dump of a container or kill a container. Things that will be covered by other jiras are: * WebUI support. * Thread dump support on windows. * Batch support so that AM or applications can send several ordered requests at once. * Allow applications to specify custom command and have NM run it against the container. > Define SignalContainerRequest and SignalContainerResponse > - > > Key: YARN-1897 > URL: https://issues.apache.org/jira/browse/YARN-1897 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897-4.patch, > YARN-1897-5.patch, YARN-1897.1.patch > > > We need to define SignalContainerRequest and SignalContainerResponse first as > they are needed by other sub tasks. SignalContainerRequest should use > OS-independent commands and provide a way to application to specify "reason" > for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-1897) CLI and core support for signal container functionality
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated YARN-1897: -- Summary: CLI and core support for signal container functionality (was: Define SignalContainerRequest and SignalContainerResponse) > CLI and core support for signal container functionality > --- > > Key: YARN-1897 > URL: https://issues.apache.org/jira/browse/YARN-1897 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897-4.patch, > YARN-1897-5.patch, YARN-1897.1.patch > > > We need to define SignalContainerRequest and SignalContainerResponse first as > they are needed by other sub tasks. SignalContainerRequest should use > OS-independent commands and provide a way to application to specify "reason" > for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744678#comment-14744678
]
Hudson commented on YARN-4151:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #391 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/391/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744685#comment-14744685
]
Hudson commented on YARN-4151:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #385 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/385/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4156) testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744691#comment-14744691
]
Hadoop QA commented on YARN-4156:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 7m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 52s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 54m 40s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 74m 22s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755859/YARN-4156.001.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 7b5cf53 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9133/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9133/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9133/console |
This message was automatically generated.
> testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler
> --
>
> Key: YARN-4156
> URL: https://issues.apache.org/jira/browse/YARN-4156
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-4156.001.patch
>
>
> The assumes the scheduler is CapacityScheduler without configuring it as
> such. This causes it to fail if the default is something else such as the
> FairScheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744705#comment-14744705
]
Hudson commented on YARN-4151:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1124 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1124/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-4154) Tez Build with hadoop 2.6.1 fails due to MiniYarnCluster change
[
https://issues.apache.org/jira/browse/YARN-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned YARN-4154:
---
Assignee: Akira AJISAKA
> Tez Build with hadoop 2.6.1 fails due to MiniYarnCluster change
> ---
>
> Key: YARN-4154
> URL: https://issues.apache.org/jira/browse/YARN-4154
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jeff Zhang
>Assignee: Akira AJISAKA
>Priority: Blocker
>
> {code}
> [ERROR]
> /mnt/nfs0/jzhang/tez-autobuild/tez/tez-plugins/tez-yarn-timeline-history/src/test/java/org/apache/tez/tests/MiniTezClusterWithTimeline.java:[92,5]
> no suitable constructor found for
> MiniYARNCluster(java.lang.String,int,int,int,int,boolean)
> constructor
> org.apache.hadoop.yarn.server.MiniYARNCluster.MiniYARNCluster(java.lang.String,int,int,int,int)
> is not applicable
> (actual and formal argument lists differ in length)
> constructor
> org.apache.hadoop.yarn.server.MiniYARNCluster.MiniYARNCluster(java.lang.String,int,int,int)
> is not applicable
> (actual and formal argument lists differ in length)
> {code}
> MR might have the same issue.
> \cc [~vinodkv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-1646) Support change container resource in RM
[ https://issues.apache.org/jira/browse/YARN-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He resolved YARN-1646. --- Resolution: Duplicate close as a dup of YARN-1651 > Support change container resource in RM > --- > > Key: YARN-1646 > URL: https://issues.apache.org/jira/browse/YARN-1646 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744715#comment-14744715
]
Hudson commented on YARN-4151:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2333 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2333/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1197) Support changing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744724#comment-14744724 ] Jian He commented on YARN-1197: --- Thanks [~mding] and [~leftnoteasy] for all the hard work ! Now that the majority of the patches are in, we plan to merge YARN-1197 branch into trunk in next few days. > Support changing resources of an allocated container > > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan > Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, > YARN-1197_Design.2015.06.24.pdf, YARN-1197_Design.2015.07.07.pdf, > YARN-1197_Design.2015.08.21.pdf, YARN-1197_Design.pdf > > > The current YARN resource management logic assumes resource allocated to a > container is fixed during the lifetime of it. When users want to change a > resource > of an allocated container the only way is releasing it and allocating a new > container with expected size. > Allowing run-time changing resources of an allocated container will give us > better control of resource usage in application side -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4154) Tez Build with hadoop 2.6.1 fails due to MiniYarnCluster change
[
https://issues.apache.org/jira/browse/YARN-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744727#comment-14744727
]
Akira AJISAKA commented on YARN-4154:
-
In branch-2, YARN-2890 (d21ef79) was reverted and another patch (55b794e) was
committed later. We should cherry-pick 55b794e instead of d21ef79 to
branch-2.6.1.
> Tez Build with hadoop 2.6.1 fails due to MiniYarnCluster change
> ---
>
> Key: YARN-4154
> URL: https://issues.apache.org/jira/browse/YARN-4154
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jeff Zhang
>Assignee: Akira AJISAKA
>Priority: Blocker
>
> {code}
> [ERROR]
> /mnt/nfs0/jzhang/tez-autobuild/tez/tez-plugins/tez-yarn-timeline-history/src/test/java/org/apache/tez/tests/MiniTezClusterWithTimeline.java:[92,5]
> no suitable constructor found for
> MiniYARNCluster(java.lang.String,int,int,int,int,boolean)
> constructor
> org.apache.hadoop.yarn.server.MiniYARNCluster.MiniYARNCluster(java.lang.String,int,int,int,int)
> is not applicable
> (actual and formal argument lists differ in length)
> constructor
> org.apache.hadoop.yarn.server.MiniYARNCluster.MiniYARNCluster(java.lang.String,int,int,int)
> is not applicable
> (actual and formal argument lists differ in length)
> {code}
> MR might have the same issue.
> \cc [~vinodkv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4140) RM container allocation delayed incase of app submitted to Nodelabel partition
[
https://issues.apache.org/jira/browse/YARN-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744735#comment-14744735
]
Bibin A Chundatt commented on YARN-4140:
Hi [~leftnoteasy]
Thnks for the comments
Have a doubt regarding the below point
{quote}
If ANY request newly added OR labelExpression changed of ANY request, you
should update ALL resource requests' labelExpression.
{quote}
This case will happen only when at run time we are changing label for an
application rt??
Is it possible to change label of an application at runtime??
> RM container allocation delayed incase of app submitted to Nodelabel partition
> --
>
> Key: YARN-4140
> URL: https://issues.apache.org/jira/browse/YARN-4140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4140.patch, 0002-YARN-4140.patch,
> 0003-YARN-4140.patch
>
>
> Trying to run application on Nodelabel partition I found that the
> application execution time is delayed by 5 – 10 min for 500 containers .
> Total 3 machines 2 machines were in same partition and app submitted to same.
> After enabling debug was able to find the below
> # From AM the container ask is for OFF-SWITCH
> # RM allocating all containers to NODE_LOCAL as shown in logs below.
> # So since I was having about 500 containers time taken was about – 6 minutes
> to allocate 1st map after AM allocation.
> # Tested with about 1K maps using PI job took 17 minutes to allocate next
> container after AM allocation
> Once 500 container allocation on NODE_LOCAL is done the next container
> allocation is done on OFF_SWITCH
> {code}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> /default-rack, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location: *, Relax
> Locality: true, Node Label Expression: 3}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-143, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-117, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
>
> {code}
> 2015-09-09 14:35:45,467 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:45,831 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,469 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,832 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
> {code}
> dsperf@host-127:/opt/bibin/dsperf/HAINSTALL/install/hadoop/resourcemanager/logs1>
> cat hadoop-dsperf-resourcemanager-host-127.log | grep "NODE_LOCAL" | grep
> "root.b.b1" | wc -l
> 500
> {code}
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744759#comment-14744759
]
Hudson commented on YARN-4151:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2310 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2310/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4151) Fix findbugs errors in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744770#comment-14744770
]
Hudson commented on YARN-4151:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #370 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/370/])
YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding
via wangda) (wangda: rev e2a02702178db60150cc0c2253d48b8532a474c2)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java
* hadoop-yarn-project/CHANGES.txt
> Fix findbugs errors in hadoop-yarn-server-common module
> ---
>
> Key: YARN-4151
> URL: https://issues.apache.org/jira/browse/YARN-4151
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0
>
> Attachments: YARN-4151.1.patch, findbugs.xml
>
>
> 7 findbugs warnings are found in hadoop-yarn-server-common module which needs
> to be fixed:
> {code}
> [INFO]
> [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @
> hadoop-yarn-server-common ---
> [INFO] Fork Value is true
> [java] Warnings generated: 7
> [INFO] Done FindBugs Analysis
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744789#comment-14744789 ] Sangjin Lee commented on YARN-3901: --- Again the jenkins result wasn't posted. Here it is: -1 overall | Vote | Subsystem | Runtime | Comment | -1 | pre-patch | 16m 1s| Findbugs (version ) appears to be | | || broken on YARN-2928. | +1 |@author | 0m 0s | The patch does not contain any | | || @author tags. | +1 | tests included | 0m 0s | The patch appears to include 2 new | | || or modified test files. | +1 | javac | 8m 4s | There were no new javac warning | | || messages. | -1 |javadoc | 7m 42s| The applied patch generated 166 | | || additional warning messages. | +1 | release audit | 0m 21s| The applied patch does not increase | | || the total number of release audit | | || warnings. | +1 | checkstyle | 0m 19s| There were no new checkstyle | | || issues. | -1 | whitespace | 0m 47s| The patch has 1 line(s) that end in | | || whitespace. Use git apply | | || --whitespace=fix. | +1 |install | 1m 33s| mvn install still works. | +1 |eclipse:eclipse | 0m 40s| The patch built with | | || eclipse:eclipse. | +1 | findbugs | 0m 52s| The patch does not introduce any | | || new Findbugs (version 3.0.0) | | || warnings. | +1 | yarn tests | 1m 56s| Tests passed in | | || hadoop-yarn-server-timelineservice. | | | 38m 22s | || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12755862/YARN-3901-YARN-2928.8.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | YARN-2928 / b1960e0 | | javadoc | /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/diffJavadocWarnings.txt | | whitespace | /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/whitespace.txt | | hadoop-yarn-server-timelineservice test log | /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9134/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.2.patch, YARN-3901-YARN-2928.3.patch, > YARN-3901-YARN-2928.4.patch, YARN-3901-YARN-2928.5.patch, > YARN-3901-YARN-2928.6.patch, YARN-3901-YARN-2928.7.patch, > YARN-3901-YARN-2928.8.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744806#comment-14744806 ] Wangda Tan commented on YARN-1651: -- Thanks [~jianhe] and [~mding] for very thorough review! > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: YARN-1197 > > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, > YARN-1651-3.YARN-1197.patch, YARN-1651-4.YARN-1197.patch, > YARN-1651-5.YARN-1197.patch, YARN-1651-6.YARN-1197.patch, > YARN-1651-7.YARN-1197.patch, YARN-1651-8.YARN-1197.patch, > YARN-1651-9.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4157) Merge YARN-1197 back to trunk
Wangda Tan created YARN-4157: Summary: Merge YARN-1197 back to trunk Key: YARN-4157 URL: https://issues.apache.org/jira/browse/YARN-4157 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1197) Support changing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744814#comment-14744814 ] Wangda Tan commented on YARN-1197: -- +1, created YARN-4157 to run Jenkins build for diff before merge. > Support changing resources of an allocated container > > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan > Attachments: YARN-1197 old-design-docs-patches-for-reference.zip, > YARN-1197_Design.2015.06.24.pdf, YARN-1197_Design.2015.07.07.pdf, > YARN-1197_Design.2015.08.21.pdf, YARN-1197_Design.pdf > > > The current YARN resource management logic assumes resource allocated to a > container is fixed during the lifetime of it. When users want to change a > resource > of an allocated container the only way is releasing it and allocating a new > container with expected size. > Allowing run-time changing resources of an allocated container will give us > better control of resource usage in application side -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4157) Merge YARN-1197 back to trunk
[ https://issues.apache.org/jira/browse/YARN-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4157: - Attachment: YARN-1197.diff.1.patch Attached diff between branch YARN-1197 and trunk. (Ver.1) > Merge YARN-1197 back to trunk > - > > Key: YARN-4157 > URL: https://issues.apache.org/jira/browse/YARN-4157 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1197.diff.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4000) RM crashes with NPE if leaf queue becomes parent queue during restart
[ https://issues.apache.org/jira/browse/YARN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744884#comment-14744884 ] Jian He commented on YARN-4000: --- [~varun_saxena], thanks for working on the patch, some comments: - We may not need a new RMAppKillEvent; we can add a new diagnostics field into the existing RMAppEvent, which will be useful for other types of events too. - the appDiagnosticsBeforeKilling in RMAppImpl is also not needed, we can reuse the diagnostics object. - CapacityScheduler#addApplication is now getting a bit complex, could you separate a new method called recoverApplication and put the recovery logic in there ? > RM crashes with NPE if leaf queue becomes parent queue during restart > - > > Key: YARN-4000 > URL: https://issues.apache.org/jira/browse/YARN-4000 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-4000.01.patch, YARN-4000.02.patch > > > This is a similar situation to YARN-2308. If an application is active in > queue A and then the RM restarts with a changed capacity scheduler > configuration where queue A becomes a parent queue to other subqueues then > the RM will crash with a NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3717) Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API
[ https://issues.apache.org/jira/browse/YARN-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3717: Attachment: YARN-3717.20150915-1.patch Attaching a patch with fixing [~wangda]'s review comments. Also have made use of this contstant variable in other web pages > Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API > - > > Key: YARN-3717 > URL: https://issues.apache.org/jira/browse/YARN-3717 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: 3717_cluster_test_snapshots.zip, RMLogsForHungJob.log, > YARN-3717.20150822-1.patch, YARN-3717.20150824-1.patch, > YARN-3717.20150825-1.patch, YARN-3717.20150826-1.patch, > YARN-3717.20150911-1.patch, YARN-3717.20150912-1.patch, > YARN-3717.20150915-1.patch > > > 1> Add the default-node-Label expression for each queue in scheduler page. > 2> In Application/Appattempt page show the app configured node label > expression for AM and Job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4157) Merge YARN-1197 back to trunk
[ https://issues.apache.org/jira/browse/YARN-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744904#comment-14744904 ] Karthik Kambatla commented on YARN-4157: I am fine with using a separate JIRA for this, but could we have just done this on YARN-1197? I am assuming details of the contents of the merge etc. will be included in the merge-vote thread. > Merge YARN-1197 back to trunk > - > > Key: YARN-4157 > URL: https://issues.apache.org/jira/browse/YARN-4157 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1197.diff.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4149) yarn logs -am should provide an option to fetch all the log files
[
https://issues.apache.org/jira/browse/YARN-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-4149:
Attachment: YARN-4149.003.patch
bq. Could you check whether the logFiles array contains "ALL" instead of just
checking whether "ALL" is the first item in logFiles array?
Fixed.
I've also addressed the findbugs warnings.
> yarn logs -am should provide an option to fetch all the log files
> -
>
> Key: YARN-4149
> URL: https://issues.apache.org/jira/browse/YARN-4149
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, nodemanager
>Affects Versions: 2.7.1
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4149.001.patch, YARN-4149.002.patch,
> YARN-4149.003.patch
>
>
> From [~gopalv] -
> {quote}
> Trying to collect a hanging Tez AM logs, by killing the container and running
> yarn logs -applicationId application_1437098194051_0178 -am ALL
> The output contains only one log file, which does not contain any of the
> actual execution logs, only the initialization logs.
> From YARN-3347, I note that
> // if we do not specify the value for CONTAINER_LOG_FILES option,
> // we will only output syslog
> This means that the person calling the yarn logs command has to list it out
> like this, to collect logs
> yarn logs -applicationId application_1437098194051_0178 -am ALL -logFiles \
> syslog_dag_1437098194051_0178_2_post,\
> dag_1437098194051_0178_2-tez-dag.pb.txt,\
> syslog_dag_1437098194051_0178_2,\
> syslog_dag_1437098194051_0178_1_post,\
> syslog_dag_1437098194051_0178_1,\
> syslog,\
> stdout,\
> stderr,\
> dag_1437098194051_0178_2.dot,\
> dag_1437098194051_0178_1.dot,\
> dag_1437098194051_0178_1-tez-dag.pb.txt
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
