[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756464#comment-15756464 ] Joep Rottinghuis commented on YARN-4061: [~gtCarrera9] I've filed a separate jira HBASE-17327 to be able to deal with lazy connection creation. Once we know how to tackle it from an HBase perspective, I can adjust the SpoolingBuffredMutator approach to use it. > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Joep Rottinghuis > Labels: YARN-5355 > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6010) Fix findbugs, site warnings in yarn-services-api module
[
https://issues.apache.org/jira/browse/YARN-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756457#comment-15756457
]
Hadoop QA commented on YARN-6010:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
42s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-services-api in yarn-native-services
failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
in yarn-native-services has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6010 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843702/YARN-6010-yarn-native-services.01.patch
|
| Optional Tests | asflicense findbugs xml compile javac javadoc
mvninstall mvnsite unit |
| uname | Linux 2ecab78a2271 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 4ce02c3 |
| Default Java | 1.8.0_111 |
| mvnsite |
https://builds.apache.org/job/PreCommit-YARN-Build/14357/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-api.txt
|
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/14357/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-api-warnings.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/14357/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
ht
[jira] [Updated] (YARN-6010) Fix findbugs, site warnings in yarn-services-api module
[ https://issues.apache.org/jira/browse/YARN-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6010: -- Attachment: YARN-6010-yarn-native-services.01.patch > Fix findbugs, site warnings in yarn-services-api module > --- > > Key: YARN-6010 > URL: https://issues.apache.org/jira/browse/YARN-6010 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-6010-yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6010) Fix findbugs, site warnings in yarn-services-api module
[ https://issues.apache.org/jira/browse/YARN-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reassigned YARN-6010: - Assignee: Jian He > Fix findbugs, site warnings in yarn-services-api module > --- > > Key: YARN-6010 > URL: https://issues.apache.org/jira/browse/YARN-6010 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6010-yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6010) Fix findbugs, site warnings in yarn-services-api module
Jian He created YARN-6010: - Summary: Fix findbugs, site warnings in yarn-services-api module Key: YARN-6010 URL: https://issues.apache.org/jira/browse/YARN-6010 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6009) RM fails to start during an upgrade - Failed to load/recover state (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
[
https://issues.apache.org/jira/browse/YARN-6009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756428#comment-15756428
]
Rohith Sharma K S commented on YARN-6009:
-
[~gsaha]
This is mainly because cluster was running YARN-4205 patch initially. And
trying to upgrade with other timeout patches cluster.
But in other timeout patch I think YARN-5611, the design of timeout value that
will be stored in RMStateStore and validation during application submission are
got changed.
This is causing the recovery failure for upgrade from YARN-4205 patch. So, any
upgrade from YARN-4205 patch to latest do not work properly.
With YARN-4205+YARN-5611 cluster, this issue does not appear.
To move forward in your cluster to make cluster up, This application need to be
deleted from state store using CLI {{./yarn resourcemanager
-remove-application-from-state-store $appId}} so that RM service will be up.
> RM fails to start during an upgrade - Failed to load/recover state
> (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
> --
>
> Key: YARN-6009
> URL: https://issues.apache.org/jira/browse/YARN-6009
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Gour Saha
>Assignee: Rohith Sharma K S
>Priority: Critical
>
> ResourceManager fails to start during an upgrade with the following
> exceptions -
> Exception 1:
> {color:red}
> {code}
> 2016-12-09 14:57:23,508 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(328)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>,
> maximumAllocation=<>, asynchronousScheduling=false,
> asyncScheduleInterval=5ms
> 2016-12-09 14:57:23,509 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(863)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:129)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:859)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when
> transitioning to Active mode
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:318)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
> ... 4 more
> Caused by: org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
> value=0 for type=LIFETIME
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:991)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1032)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1028)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1028)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
> ... 5 more
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Invalid
> application timeout, value=0 for type=LIFETIME
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:463)
>
[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings
[
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756425#comment-15756425
]
Hadoop QA commented on YARN-5967:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
in yarn-native-services has 262 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-slider in yarn-native-services failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services
failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider:
The patch generated 64 new + 1228 unchanged - 94 fixed = 1292 total (was 1322)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
generated 0 new + 6 unchanged - 256 fixed = 6 total (was 262) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-slider in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
27s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-slider in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {col
[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5967: -- Attachment: YARN-5967-yarn-native-services.09.patch > Fix slider core module findbugs warnings > - > > Key: YARN-5967 > URL: https://issues.apache.org/jira/browse/YARN-5967 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5967-yarn-native-services.01.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.03.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.05.patch, > YARN-5967-yarn-native-services.06.patch, > YARN-5967-yarn-native-services.07.patch, > YARN-5967-yarn-native-services.08.patch, > YARN-5967-yarn-native-services.08.patch, > YARN-5967-yarn-native-services.09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6001) Improve moveApplicationQueues command line
[
https://issues.apache.org/jira/browse/YARN-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756375#comment-15756375
]
Yufei Gu commented on YARN-6001:
Make sense to me. Thanks to point out.
It would be swell to be explicit about the direction, like "changetoqueue",
“migratetoqueue”. It is obvious for the people who actually write the command
to move app that the parameter queue should be the target queue instead of
source queue, but may confuse some admins or support guys who maintain system
with pre-baked scripts.
> Improve moveApplicationQueues command line
> --
>
> Key: YARN-6001
> URL: https://issues.apache.org/jira/browse/YARN-6001
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-6001.0001.patch
>
>
> As an example, below command is used to move an application across queue.
> {noformat}./yarn application -movetoqueue application_1479894790219_0002
> -queue default{noformat}
> Inline with other application's attribute modification features such as
> updateTimeout or priority, movetoqueue cli command lacks unification and more
> complex or error prone.
> Suggesting below cli command which can be used.
> {noformat}
> ./yarn application -appId application_1479894790219_0002 -move default
> {noformat}
> This is inline with other commands such as
> {noformat}
> ./yarn application -appId application_1479894790219_0002 -updatePriority 8
> ./yarn application -appId application_1479894790219_0002 -updateLifetime 10
> {noformat}
> Old *movetoqueue* command could be still kept, but we can mark it as
> deprecated and mention in help message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5968) Fix slider core module javadocs
[
https://issues.apache.org/jira/browse/YARN-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756222#comment-15756222
]
Hadoop QA commented on YARN-5968:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
47s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
39s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 5m
1s{color} | {color:red} hadoop-yarn in yarn-native-services failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
19s{color} | {color:red} hadoop-yarn in yarn-native-services failed. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 4m
24s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
14s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 120 new +
6463 unchanged - 2055 fixed = 6583 total (was 8518) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 4s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.timeline.webapp.TestTimelineWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5968 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843694/YARN-5968-yarn-native-services.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 097dc95f581d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 4ce02c3 |
| Default Java | 1.8.0_111 |
| mvnsite |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14355/console |
[jira] [Updated] (YARN-5968) Fix slider core module javadocs
[ https://issues.apache.org/jira/browse/YARN-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5968: -- Assignee: Billie Rinaldi > Fix slider core module javadocs > --- > > Key: YARN-5968 > URL: https://issues.apache.org/jira/browse/YARN-5968 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Billie Rinaldi > Attachments: YARN-5968-yarn-native-services.01.patch, > YARN-5968-yarn-native-services.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756123#comment-15756123 ] Billie Rinaldi commented on YARN-5967: -- +1, the new patch address all of my suggestions. Thanks, [~jianhe]! > Fix slider core module findbugs warnings > - > > Key: YARN-5967 > URL: https://issues.apache.org/jira/browse/YARN-5967 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5967-yarn-native-services.01.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.03.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.05.patch, > YARN-5967-yarn-native-services.06.patch, > YARN-5967-yarn-native-services.07.patch, > YARN-5967-yarn-native-services.08.patch, > YARN-5967-yarn-native-services.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5968) Fix slider core module javadocs
[ https://issues.apache.org/jira/browse/YARN-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-5968: - Attachment: YARN-5968-yarn-native-services.02.patch > Fix slider core module javadocs > --- > > Key: YARN-5968 > URL: https://issues.apache.org/jira/browse/YARN-5968 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-5968-yarn-native-services.01.patch, > YARN-5968-yarn-native-services.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings
[
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756100#comment-15756100
]
Hadoop QA commented on YARN-5967:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
4s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-5967 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843691/YARN-5967-yarn-native-services.08.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14354/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix slider core module findbugs warnings
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch,
> YARN-5967-yarn-native-services.02.patch,
> YARN-5967-yarn-native-services.02.patch,
> YARN-5967-yarn-native-services.03.patch,
> YARN-5967-yarn-native-services.04.patch,
> YARN-5967-yarn-native-services.04.patch,
> YARN-5967-yarn-native-services.05.patch,
> YARN-5967-yarn-native-services.06.patch,
> YARN-5967-yarn-native-services.07.patch,
> YARN-5967-yarn-native-services.08.patch,
> YARN-5967-yarn-native-services.08.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756098#comment-15756098 ] Jian He commented on YARN-5967: --- btw. this patch is the latest https://issues.apache.org/jira/secure/attachment/12843691/YARN-5967-yarn-native-services.08.patch > Fix slider core module findbugs warnings > - > > Key: YARN-5967 > URL: https://issues.apache.org/jira/browse/YARN-5967 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5967-yarn-native-services.01.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.03.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.05.patch, > YARN-5967-yarn-native-services.06.patch, > YARN-5967-yarn-native-services.07.patch, > YARN-5967-yarn-native-services.08.patch, > YARN-5967-yarn-native-services.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4990) Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion
[
https://issues.apache.org/jira/browse/YARN-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756095#comment-15756095
]
Hadoop QA commented on YARN-4990:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
29s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-4990 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843686/YARN-4990.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0392e8cc1bde 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fcbe152 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14352/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14352/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Re-direction of a particular log file within in a container in NM UI does not
> redirect properly to Log Server ( history ) on container completion
> -
>
> Key: YARN-4990
> URL: https://issues.apache.org/jira/b
[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5967: -- Attachment: YARN-5967-yarn-native-services.08.patch > Fix slider core module findbugs warnings > - > > Key: YARN-5967 > URL: https://issues.apache.org/jira/browse/YARN-5967 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5967-yarn-native-services.01.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.03.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.05.patch, > YARN-5967-yarn-native-services.06.patch, > YARN-5967-yarn-native-services.07.patch, > YARN-5967-yarn-native-services.08.patch, > YARN-5967-yarn-native-services.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5967: -- Attachment: YARN-5967-yarn-native-services.08.patch > Fix slider core module findbugs warnings > - > > Key: YARN-5967 > URL: https://issues.apache.org/jira/browse/YARN-5967 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5967-yarn-native-services.01.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.03.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.05.patch, > YARN-5967-yarn-native-services.06.patch, > YARN-5967-yarn-native-services.07.patch, > YARN-5967-yarn-native-services.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings
[
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756052#comment-15756052
]
Hadoop QA commented on YARN-5967:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-5967 does not apply to yarn-native-services. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843687/YARN-5967-yarn-native-services.07.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14353/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix slider core module findbugs warnings
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch,
> YARN-5967-yarn-native-services.02.patch,
> YARN-5967-yarn-native-services.02.patch,
> YARN-5967-yarn-native-services.03.patch,
> YARN-5967-yarn-native-services.04.patch,
> YARN-5967-yarn-native-services.04.patch,
> YARN-5967-yarn-native-services.05.patch,
> YARN-5967-yarn-native-services.06.patch,
> YARN-5967-yarn-native-services.07.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5967: -- Attachment: YARN-5967-yarn-native-services.07.patch Thanks Billie for the thorough review ! Fixed all of them > Fix slider core module findbugs warnings > - > > Key: YARN-5967 > URL: https://issues.apache.org/jira/browse/YARN-5967 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5967-yarn-native-services.01.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.02.patch, > YARN-5967-yarn-native-services.03.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.04.patch, > YARN-5967-yarn-native-services.05.patch, > YARN-5967-yarn-native-services.06.patch, > YARN-5967-yarn-native-services.07.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4990) Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion
[
https://issues.apache.org/jira/browse/YARN-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756029#comment-15756029
]
Xuan Gong commented on YARN-4990:
-
It is not easy to add a test case for it. But did a manual test.
{code}
http://xuanmacbook-pro.home:8042/node/containerlogs/container_1481936892133_0005_01_01/xuan
{code}
would redirect to
{code}
http://localhost:8188/applicationhistory/logs/xuanmacbook-pro.home:53565/container_1481936892133_0005_01_01/container_1481936892133_0005_01_01/xuan
{code}
{code}
http://xuanmacbook-pro.home:8042/node/containerlogs/container_1481936892133_0005_01_01/xuan/syslog/?start=10
{code}
would redirect to
{code}
http://localhost:8188/applicationhistory/logs/xuanmacbook-pro.home:53565/container_1481936892133_0005_01_01/container_1481936892133_0005_01_01/xuan/syslog?start=10
{code}
Also verified that we do have the correct offset
> Re-direction of a particular log file within in a container in NM UI does not
> redirect properly to Log Server ( history ) on container completion
> -
>
> Key: YARN-4990
> URL: https://issues.apache.org/jira/browse/YARN-4990
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Xuan Gong
> Attachments: YARN-4990.1.patch
>
>
> The NM does the redirection to the history server correctly. However if the
> user is viewing or has a link to a particular specific file, the redirect
> ends up going to the top level page for the container and not redirecting to
> the specific file. Additionally, the start param to show logs from the offset
> 0 also goes missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4990) Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion
[ https://issues.apache.org/jira/browse/YARN-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756015#comment-15756015 ] Xuan Gong commented on YARN-4990: - The issue is that we did not parse/append containerLogType(logFile name)when we generate the redirect log link for the container logs. Attached a trivial patch to fix it > Re-direction of a particular log file within in a container in NM UI does not > redirect properly to Log Server ( history ) on container completion > - > > Key: YARN-4990 > URL: https://issues.apache.org/jira/browse/YARN-4990 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Attachments: YARN-4990.1.patch > > > The NM does the redirection to the history server correctly. However if the > user is viewing or has a link to a particular specific file, the redirect > ends up going to the top level page for the container and not redirecting to > the specific file. Additionally, the start param to show logs from the offset > 0 also goes missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5952) Create REST API for changing YARN scheduler configurations
[ https://issues.apache.org/jira/browse/YARN-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756013#comment-15756013 ] Jonathan Hung commented on YARN-5952: - Hi [~leftnoteasy] and [~xgong], attached 002 patch containing REST object definitions, as well as some implementation for converting this into objects as defined in YARN-6005. As discussed offline, can you take a look? Thanks! > Create REST API for changing YARN scheduler configurations > -- > > Key: YARN-5952 > URL: https://issues.apache.org/jira/browse/YARN-5952 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5952.001.patch, YARN-5952.002.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4990) Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion
[ https://issues.apache.org/jira/browse/YARN-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4990: Attachment: YARN-4990.1.patch > Re-direction of a particular log file within in a container in NM UI does not > redirect properly to Log Server ( history ) on container completion > - > > Key: YARN-4990 > URL: https://issues.apache.org/jira/browse/YARN-4990 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Attachments: YARN-4990.1.patch > > > The NM does the redirection to the history server correctly. However if the > user is viewing or has a link to a particular specific file, the redirect > ends up going to the top level page for the container and not redirecting to > the specific file. Additionally, the start param to show logs from the offset > 0 also goes missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5952) Create REST API for changing YARN scheduler configurations
[ https://issues.apache.org/jira/browse/YARN-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5952: Attachment: YARN-5952.002.patch > Create REST API for changing YARN scheduler configurations > -- > > Key: YARN-5952 > URL: https://issues.apache.org/jira/browse/YARN-5952 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5952.001.patch, YARN-5952.002.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-5524:
Issue Type: Sub-task (was: Bug)
Parent: YARN-4904
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5655) TestContainerManagerSecurity#testNMTokens is asserting
[
https://issues.apache.org/jira/browse/YARN-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5655:
--
Attachment: mvn_test.out
My local branch is synched with trunk. Attaching output of:
$ mvn test -Dtest=TestContainerManagerSecurity
> TestContainerManagerSecurity#testNMTokens is asserting
> --
>
> Key: YARN-5655
> URL: https://issues.apache.org/jira/browse/YARN-5655
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jason Lowe
>Assignee: Robert Kanter
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: YARN-5655.001.patch, mvn_test.out
>
>
> TestContainerManagerSecurity has been failing recently in 2.8:
> {noformat}
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 80.928 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 44.478 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:394)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:337)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 34.964 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:333)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5756) Add state-machine implementation for queues
[
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755898#comment-15755898
]
Hadoop QA commented on YARN-5756:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 610 unchanged - 3 fixed = 613 total (was 613) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 42m
38s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5756 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843662/YARN-5756.7.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 7c317c1fe731 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f121645 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14351/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14351/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoo
[jira] [Commented] (YARN-5993) Allow native services quicklinks to be exported for each component
[
https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755889#comment-15755889
]
Gour Saha commented on YARN-5993:
-
[~billie.rinaldi] Please fix the following findbug warning and the 6 new
checkstyle issues as per the jenkins report.
1 findbug warning -
At DockerProviderService.java:line 368
{code}
org.apache.slider.providers.docker.DockerProviderService.publishExportGroups(String,
String, String, String, List) makes inefficient use of keySet iterator instead
of entrySet iterator
{code}
h6.
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/core/registry/docstore/ExportEntry.java
equals and hashcode methods:
{code}
return containerId.equals(that.containerId);
return containerId.hashCode();
{code}
Please add null check for containerId in both the methods.
h6.
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/core/registry/docstore/PublishedExports.java
{code}
public Map> sortedEntries() {
TreeMap> sortedEntries = new TreeMap<>();
{code}
Change sortedEntries type from TreeMap to Map
h6.
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/providers/ProviderUtils.java
Remove Iterator import.
{code}
public synchronized void publishExportGroup(
Map> exportGroup,
StateAccessForProviders amState, String groupName) {
{code}
Why is this method synchronized now?
Method publishExportGroup:
{code}
TreeSet values = new TreeSet<>();
{code}
We can change values type from TreeSet to Set.
h6.
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/providers/docker/DockerProviderService.java
Method notifyContainerCompleted:
{code}
for (Set entrySet : exportMap.values()) {
{code}
Can you rename entrySet to entries or exportEntries or something more
appropriate to avoid confusion to the reader who might think why it is
exportMap.values() and not exportMap.entrySet()?
Method publishExportGroups:
1.
{code}
replaceTokens.put(String.format(IP_KEY_FORMAT, roleNameKey), ips.get(0));
.
.
replaceTokens.put(String.format(IP_KEY_FORMAT, roleGroupKey),
ips.get(0));
{code}
Since we are discarding all but the first IP, I am thinking that we should
probably introduce a IPS_KEY_FORMAT of $\{%s_IPS\} and replace it with a comma
separated list of IPs when there are more than one IP for a single container.
This gives application owners a placeholder to fetch all the IPs as well (if
they want to).
2.
Question:
Just to confirm, if roleNameKey is null then there is no roleGroupKey, right?
3.
{code}
if (value.contains(VARIABLE_INDICATOR)) {
// not all variables have been substituted, so do not export
continue;
}
{code}
I think we should not block here, since it will help application owners to know
which variables were not substituted, rather than thinking that exports don’t
work at all. What do you think?
Method getExportSet:
{code}
protected Set getExportSet(String key) {
{code}
Can we rename this method to getExportEntries(String key) ?
h6.
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/server/appmaster/web/view/IndexBlock.java
Method enumeratePublishedExports:
{code}
if (entry.getValue().size() > 0) {
{code}
Check entry.getValue() for null
> Allow native services quicklinks to be exported for each component
> --
>
> Key: YARN-5993
> URL: https://issues.apache.org/jira/browse/YARN-5993
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5993-yarn-native-services.001.patch
>
>
> The quicklinks export capability changed in switching from the agent provider
> to the docker provider, and currently the docker provider only allows one
> component to export links. We should improve this capability to be more in
> line with the previous agent capability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5952) Create REST API for changing YARN scheduler configurations
[ https://issues.apache.org/jira/browse/YARN-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5952: Summary: Create REST API for changing YARN scheduler configurations (was: Create REST API for changing YARN configurations) > Create REST API for changing YARN scheduler configurations > -- > > Key: YARN-5952 > URL: https://issues.apache.org/jira/browse/YARN-5952 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5952.001.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6005) Protocol for scheduler configuration changes between client and RM
[ https://issues.apache.org/jira/browse/YARN-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-6005: Summary: Protocol for scheduler configuration changes between client and RM (was: Protocol for configuration changes between client and RM) > Protocol for scheduler configuration changes between client and RM > -- > > Key: YARN-6005 > URL: https://issues.apache.org/jira/browse/YARN-6005 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-6005.001.patch > > > For configuration change protocol and associated builder objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755881#comment-15755881
]
Xuan Gong commented on YARN-5524:
-
any update [~vrushalic] ?
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5655) TestContainerManagerSecurity#testNMTokens is asserting
[
https://issues.apache.org/jira/browse/YARN-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755734#comment-15755734
]
Robert Kanter commented on YARN-5655:
-
[~asuresh], can you point me to an instance of it failing? I just ran the test
100 times locally and it never failed.
> TestContainerManagerSecurity#testNMTokens is asserting
> --
>
> Key: YARN-5655
> URL: https://issues.apache.org/jira/browse/YARN-5655
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jason Lowe
>Assignee: Robert Kanter
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: YARN-5655.001.patch
>
>
> TestContainerManagerSecurity has been failing recently in 2.8:
> {noformat}
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 80.928 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 44.478 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:394)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:337)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 34.964 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:333)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5756) Add state-machine implementation for queues
[ https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755732#comment-15755732 ] Xuan Gong commented on YARN-5756: - [~wangda] Thanks for the view. Attached a patch for all the comments > Add state-machine implementation for queues > --- > > Key: YARN-5756 > URL: https://issues.apache.org/jira/browse/YARN-5756 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch, > YARN-5756.4.patch, YARN-5756.5.patch, YARN-5756.6.patch, YARN-5756.6.patch, > YARN-5756.7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5756) Add state-machine implementation for queues
[ https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5756: Attachment: YARN-5756.7.patch > Add state-machine implementation for queues > --- > > Key: YARN-5756 > URL: https://issues.apache.org/jira/browse/YARN-5756 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch, > YARN-5756.4.patch, YARN-5756.5.patch, YARN-5756.6.patch, YARN-5756.6.patch, > YARN-5756.7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6009) RM fails to start during an upgrade - Failed to load/recover state (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
Gour Saha created YARN-6009:
---
Summary: RM fails to start during an upgrade - Failed to
load/recover state (YarnException: Invalid application timeout, value=0 for
type=LIFETIME)
Key: YARN-6009
URL: https://issues.apache.org/jira/browse/YARN-6009
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Gour Saha
Priority: Critical
ResourceManager fails to start during an upgrade with the following exceptions
-
Exception 1:
{color:red}
{code}
2016-12-09 14:57:23,508 INFO capacity.CapacityScheduler
(CapacityScheduler.java:initScheduler(328)) - Initialized CapacityScheduler
with calculator=class
org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
minimumAllocation=<>, maximumAllocation=<>, asynchronousScheduling=false, asyncScheduleInterval=5ms
2016-12-09 14:57:23,509 WARN ha.ActiveStandbyElector
(ActiveStandbyElector.java:becomeActive(863)) - Exception handling the winning
of election
org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
at
org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:129)
at
org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:859)
at
org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when
transitioning to Active mode
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:318)
at
org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
... 4 more
Caused by: org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
value=0 for type=LIFETIME
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:991)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1032)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1028)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1028)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
... 5 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Invalid application
timeout, value=0 for type=LIFETIME
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:463)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1184)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:594)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
... 13 more
{code}
{color}
Exception 2:
{color:red}
{code}
2016-12-09 14:57:26,162 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(790)) -
application_1477927786494_0008 State change from NEW to FINISHED
2016-12-09 14:57:26,162 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(599)) - Failed to load/recover state
org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
value=0 for type=LIFETIME
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:463)
at
o
[jira] [Updated] (YARN-6009) RM fails to start during an upgrade - Failed to load/recover state (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
[
https://issues.apache.org/jira/browse/YARN-6009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gour Saha updated YARN-6009:
Assignee: Rohith Sharma K S
> RM fails to start during an upgrade - Failed to load/recover state
> (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
> --
>
> Key: YARN-6009
> URL: https://issues.apache.org/jira/browse/YARN-6009
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Gour Saha
>Assignee: Rohith Sharma K S
>Priority: Critical
>
> ResourceManager fails to start during an upgrade with the following
> exceptions -
> Exception 1:
> {color:red}
> {code}
> 2016-12-09 14:57:23,508 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(328)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>,
> maximumAllocation=<>, asynchronousScheduling=false,
> asyncScheduleInterval=5ms
> 2016-12-09 14:57:23,509 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(863)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:129)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:859)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when
> transitioning to Active mode
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:318)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
> ... 4 more
> Caused by: org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
> value=0 for type=LIFETIME
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:991)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1032)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1028)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1028)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
> ... 5 more
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Invalid
> application timeout, value=0 for type=LIFETIME
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:463)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1184)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:594)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> ... 13 more
> {code}
> {color}
> Exception 2:
> {color:red}
> {code}
> 2016-12-09 14:57:26,162 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(790)) -
> application_1477927786494_0008 State change from NEW to FINISHED
> 2016-12-09 14:57:26,162 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:serviceStart(599)) - Failed to load/recover state
> org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
> val
[jira] [Commented] (YARN-6009) RM fails to start during an upgrade - Failed to load/recover state (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
[
https://issues.apache.org/jira/browse/YARN-6009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755546#comment-15755546
]
Gour Saha commented on YARN-6009:
-
/cc [~rohithsharma]
> RM fails to start during an upgrade - Failed to load/recover state
> (YarnException: Invalid application timeout, value=0 for type=LIFETIME)
> --
>
> Key: YARN-6009
> URL: https://issues.apache.org/jira/browse/YARN-6009
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Gour Saha
>Assignee: Rohith Sharma K S
>Priority: Critical
>
> ResourceManager fails to start during an upgrade with the following
> exceptions -
> Exception 1:
> {color:red}
> {code}
> 2016-12-09 14:57:23,508 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(328)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>,
> maximumAllocation=<>, asynchronousScheduling=false,
> asyncScheduleInterval=5ms
> 2016-12-09 14:57:23,509 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(863)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:129)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:859)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when
> transitioning to Active mode
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:318)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
> ... 4 more
> Caused by: org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.yarn.exceptions.YarnException: Invalid application timeout,
> value=0 for type=LIFETIME
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:991)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1032)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1028)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1028)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
> ... 5 more
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Invalid
> application timeout, value=0 for type=LIFETIME
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateApplicationTimeouts(RMServerUtils.java:305)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:365)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:330)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:463)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1184)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:594)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> ... 13 more
> {code}
> {color}
> Exception 2:
> {color:red}
> {code}
> 2016-12-09 14:57:26,162 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(790)) -
> application_1477927786494_0008 State change from NEW to FINISHED
> 2016-12-09 14:57:26,162 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:serviceStart(599)) - Failed to load/recover state
> org.apache.hadoop.yarn.exceptions.Yarn
[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings
[
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755543#comment-15755543
]
Billie Rinaldi commented on YARN-5967:
--
[~jianhe], this is looking great so far. Here are some specific comments. I
tried to point out places where it would be easy to fix an error rather than
ignoring the warning, but I also don't think any of these are a real problem.
* There is a non-generated class in the proto package. Maybe we should move
that to a different package?
* For SliderKeys COMPONENT_KEYS_TO_SKIP, we could make it a List and use
Collections.unmodifiableList(Arrays.asList(…)) instead of ignoring
MS_MUTABLE_ARRAY.
* StringBuffer in AbstractActionArgs could be StringBuilder.
* We should specify Charset.forName("UTF-8”) or StandardCharsets.UTF_8 instead
of ignoring DM_DEFAULT_ENCODING, where possible. FileReader can be switched to
an InputStreamReader. PrintStream has a constructor which takes a File and a
charset name as a String. Not sure about PrintWriter.
* I am not sure we should ignore OS_OPEN_STREAM, but I did not look at the
individual cases of this error.
* In AppDefinitionPersister, let’s move File defPath = new File(value) after
the if (SliderUtils.isUnset(value)) check.
* Since OutstandingRequestTracker newerThan is Serializable now, does it need a
serialVersionUID?
* For OutstandingRequest, would it be better to have a trivial equals method
that calls super.equals instead of ignoring EQ_DOESNT_OVERRIDE_EQUALS?
* In RoleInstance, we could make output private instead of ignoring
UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD.
* In Probe, could we make bootstrapStarted and bootstrapFinished private
instead of ignoring URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD? I see there is a
comment saying to leave them alone, but maybe making them private isn’t what
the comment meant.
* In PublisherResource, I think we could make getAMClassPath return a List
rather than ignoring DMI_COLLECTION_OF_URLS.
* The ContainerStatsBlock error still needs to be handled, correct?
> Fix slider core module findbugs warnings
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch,
> YARN-5967-yarn-native-services.02.patch,
> YARN-5967-yarn-native-services.02.patch,
> YARN-5967-yarn-native-services.03.patch,
> YARN-5967-yarn-native-services.04.patch,
> YARN-5967-yarn-native-services.04.patch,
> YARN-5967-yarn-native-services.05.patch,
> YARN-5967-yarn-native-services.06.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5996) Native services AM kills app on AMRMClientAsync onError call
[ https://issues.apache.org/jira/browse/YARN-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-5996: Fix Version/s: yarn-native-services > Native services AM kills app on AMRMClientAsync onError call > > > Key: YARN-5996 > URL: https://issues.apache.org/jira/browse/YARN-5996 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-5996-yarn-native-services.001.patch, > YARN-5996-yarn-native-services.002.patch > > > The AMRMClientAsync onError callback occurred due to an InterruptedException > in this case. The AM may need to kill itself once the client reaches this > state, but it should not kill the entire application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch
[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-5877:
---
Attachment: 5877 nm logs of test.log
Attaching nm logs of test.
> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 5877 nm logs of test.log, Dockerfile,
> YARN-5877.0001.patch, YARN-5877.0002.patch, YARN-5877.0003.patch,
> YARN-5877.0004.patch, YARN-5877.0005.patch, YARN-5877.0006.patch,
> bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
>
> Environment variables that containers may override rather
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
>
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755361#comment-15755361
]
Miklos Szegedi commented on YARN-5936:
--
Thank you, for the reply [~zhengchenyu]!
{quote}
First，The linux kernel source code of cpu bandwidth will add too many timer,
and add more function to be called.
Secondly, limit utilization ratio will lead to bad performance.
{quote}
I did an experiment with he following cpu heavy app:
{code}
//a.c
int main() {
int i;
int j = 0;
for (i = 0; i < 10; ++i) {
j++;
}
return j & 1;
}
{code}
I ran it in parallel in a single cgroup, multiple cgroups and multipre cgroups
with CPU throttling enabled on a single CPU.
{code}
for j in `seq 1 10`; do export i=$j;sh -c 'time ./a.out&'; done
for j in `seq 1 10`; do export i=$j;sh -c 'echo $$ >/cgroup/cpu/$i/tasks;echo
-1 >/cgroup/cpu/$i/cpu.cfs_quota_us;time ./a.out&'; done
for j in `seq 1 10`; do export i=$j;sh -c 'echo $$ >/cgroup/cpu/$i/tasks;echo
1 >/cgroup/cpu/$i/cpu.cfs_quota_us;time ./a.out&'; done
{code}
The runtime in the first case (no cgroups) was 24.7154, the second (no group
throttle) was 24.6907 seconds on average, the runtime in the latter case was
24.7469 respectively.
The difference less than 0.25% in these cases. I ran it a few more times and I
received very similar numbers.
This means to me that what you are seeing is the utilization drop, if the
container group limits the CPU usage and not an inefficiency in the Linux
kernel.
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
>
[jira] [Commented] (YARN-5641) Localizer leaves behind tarballs after container is complete
[
https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755347#comment-15755347
]
Hadoop QA commented on YARN-5641:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 39s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
| |
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor |
| |
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5641 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843617/YARN-5641.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1a6d3b6de260 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f121645 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14349/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14349/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14349/console |
| Powered by | Apache Yetus 0.
[jira] [Commented] (YARN-5957) NMWebAppFilter currently drops end of URL during container log redirection
[
https://issues.apache.org/jira/browse/YARN-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755334#comment-15755334
]
Hadoop QA commented on YARN-5957:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
57s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843097/YARN-5957.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6de5d6499647 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f121645 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14348/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14348/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NMWebAppFilter currently drops end of URL during container log redirection
> --
>
> Key: YARN-5957
> URL: https://issues.apache.org/jira/browse/YARN-5957
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-5957.001.patch, YARN-5957.002.patch,
> YARN-5957.003.patch
>
>
> When the NM redirec
[jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755304#comment-15755304 ] Wangda Tan commented on YARN-5889: -- bq. For preemption calculation, one of the main problem could have been about the free resources I think so, but I think the *minimum* preemption threshold is we should not preempt res from user if used of the user < queue-capacity * MULP. The higher threshold is, the safer of the preemption. bq. I think in a busier and short-living app's cluster, we may recalculate more. I agree with [~eepayne], we should be able to make preemption / allocation UL calculation independent (or at least it's better to not allocation UL has dependency on preemption UL). [~sunilg] could you add some details here? bq. On this note, could I update a patch with approach mentioned above. Please go ahead when you get chance :). We should generally agree the approach, there are some debatable details, but starting prototype can help us understand the scope. Thoughts? > Improve user-limit calculation in capacity scheduler > > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, > YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6005) Protocol for configuration changes between client and RM
[ https://issues.apache.org/jira/browse/YARN-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755264#comment-15755264 ] Jonathan Hung commented on YARN-6005: - Hi [~xgong] and [~leftnoteasy], uploaded a patch containing the protocol changes. Can you take a look? If this looks OK we will update this ticket the associated PBImpl objects (and tests). > Protocol for configuration changes between client and RM > > > Key: YARN-6005 > URL: https://issues.apache.org/jira/browse/YARN-6005 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-6005.001.patch > > > For configuration change protocol and associated builder objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6005) Protocol for configuration changes between client and RM
[ https://issues.apache.org/jira/browse/YARN-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-6005: Attachment: YARN-6005.001.patch > Protocol for configuration changes between client and RM > > > Key: YARN-6005 > URL: https://issues.apache.org/jira/browse/YARN-6005 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-6005.001.patch > > > For configuration change protocol and associated builder objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5864) Capacity Scheduler preemption for fragmented cluster
[
https://issues.apache.org/jira/browse/YARN-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755233#comment-15755233
]
Wangda Tan commented on YARN-5864:
--
Thanks [~curino] for the quick response!
All great points, I will cover them in the doc, and will cover at least this
feature related "tunables".
> Capacity Scheduler preemption for fragmented cluster
> -
>
> Key: YARN-5864
> URL: https://issues.apache.org/jira/browse/YARN-5864
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-5864.poc-0.patch
>
>
> YARN-4390 added preemption for reserved container. However, we found one case
> that large container cannot be allocated even if all queues are under their
> limit.
> For example, we have:
> {code}
> Two queues, a and b, capacity 50:50
> Two nodes: n1 and n2, each of them have 50 resource
> Now queue-a uses 10 on n1 and 10 on n2
> queue-b asks for one single container with resource=45.
> {code}
> The container could be reserved on any of the host, but no preemption will
> happen because all queues are under their limits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755181#comment-15755181 ] Eric Payne commented on YARN-5889: -- bq. I think in a busier and short-living app's cluster, we may recalculate more. This is probably true for the active-user limit. For the preemption limit, my thinking is that it only needs to be calculated during a preemption cycle. Thoughts? > Improve user-limit calculation in capacity scheduler > > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, > YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3955) Support for priority ACLs in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755127#comment-15755127
]
Sunil G commented on YARN-3955:
---
HI [~leftnoteasy]
Thanks for the comments. I will work on tests/findbugs/ and comments.
In general I have one doubt below point.
bq.Instead of call methods of appPriorityAclManager (I prefer to remove it, see
#1). Can we call scheduler#checkAndGetApplicationPriority to achieve the same
goal?
I think with {{QueueACLsManager}}, we took ACLs out of CS and managed
externally. Yes, it was to plugin external ACLs policies, however I thought it
was good. I think that was the reason I went with a PriorityACLManager model.
Do you see adding PrioirtyACLs back to scheduler is better? If so, as suggested
we can leverage from scheduler apis. Thoughts?
> Support for priority ACLs in CapacityScheduler
> --
>
> Key: YARN-3955
> URL: https://issues.apache.org/jira/browse/YARN-3955
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: ApplicationPriority-ACL.pdf,
> ApplicationPriority-ACLs-v2.pdf, YARN-3955.0001.patch, YARN-3955.0002.patch,
> YARN-3955.0003.patch, YARN-3955.v0.patch, YARN-3955.v1.patch,
> YARN-3955.wip1.patch
>
>
> Support will be added for User-level access permission to use different
> application-priorities. This is to avoid situations where all users try
> running max priority in the cluster and thus degrading the value of
> priorities.
> Access Control Lists can be set per priority level within each queue. Below
> is an example configuration that can be added in capacity scheduler
> configuration
> file for each Queue level.
> yarn.scheduler.capacity.root...acl=user1,user2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755117#comment-15755117 ] Joep Rottinghuis commented on YARN-4061: For the next version of the patch I'll modify the code so that even if HBase is down we can create a SpoolingBufferedMutatorImpl. I'll check a little whether we need to also push down the connection creation into the code. Part of the issue there is that we now will be creating multiple connections rather than having one shared one. Keep in mind that we have a BufferedMutator per table. It would be unfortunate to create 5 connections. wrt. "at least once" here is my reply from the Google doc: bq. Yes, we guarantee at least once, if one considers spooling the mutation a delivery. The code to replay those mutations is still missing. I'm not sure if that should belong here, or if that should be a separate thing. bq. From the Yarn timeline perspective we set the timestamp on each put explicitly exactly for the reason of delayed mutations. We want to know when they happened, and not when they were submitted. For our usecase, that makes the puts idempotent. Other use-cases may not need this requirement, but they do need to deal with duplicate puts. > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Joep Rottinghuis > Labels: YARN-5355 > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755070#comment-15755070
]
Sunil G commented on YARN-5889:
---
Generally I am also agreeing with the direction at which we are going towards.
Few points from end:
- For preemption calculation, one of the main problem could have been about the
*free resources* in the queue even when some users are over-utilizing its
resource quota (these users could become active/non-active). Because preemption
module will be handling {free_resources + to_be_preempted_resources} and need
to think more like scheduler.
- Above point will play a big factor to decide when preemption need to kick in.
It could be when free/used become very smaller OR it could also be when there
is a lot of violation from few users which holds resource more than MULP but
became non-active users.
As far as I understood, we will still have pre-computed user-limit model. But
this cache will be computed based on any event change on resource changes for
non-active users. I think in a busier and short-living app's cluster, we may
recalculate more. But I think preemption module will have a better accuracy.
On this note, could I update a patch with approach mentioned above. I think
free resource also need to be part to trigger preemption. But for user-limit
calculation, I will be making changes in {{ActiveUserManager}} to track of
non-active-users as well with a state to reflect changes in resource.
> Improve user-limit calculation in capacity scheduler
>
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch,
> YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755015#comment-15755015 ] Konstantinos Karanasos commented on YARN-5646: -- Thanks [~asuresh]! > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map
[
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755009#comment-15755009
]
Miklos Szegedi commented on YARN-5714:
--
{quote}
are you sure about the double % ? i've tested it on windows 7, 8, 8.1 and 10
and double-ing the % does not escape it.
set A=toto
set B=%A%
set C=%%A%%
echo %A% %B% %C%
produce toto %toto% %toto%, so i always got A's value, so both B and C depend
on A.
{quote}
Yes. However, it only works this way, if you are running within a batch file
like Yarn does. Are you using these commands typing to the console?
{quote}
I still prefer to keep it the way it is because it is the last line of the
while that does this test. Personnally when i read code that double check
things I tend to think the code is still under dev and not yet production-ready
and then i start to tripple check everything i read. So for me, such double
check would lead in code maintenance/review time increase. if this is really a
blocking issue and does not permit the patch to be accepted, then i'll change
it.
{quote}
It is okay, it is not a high priority issue.
{quote}
I've reworked the code to match you algorithm so the comment (and its tipo) no
longer exist
{quote}
Thank you!
{quote}
I've tested it with deps in reverse order and having each env entry depending
on each of its successors and with 100 entries. On my computer it returns in 0
or 16ms (no lesser time precision). My algorithm start to be slower in a
perceptible way with the same test case when we have more than 200 entries. it
hits the second for computation when i have more than ~700 entries. So I've
implemented the new algorithm (more or less, i use recursion to handle the
stack stuff) but i need to rework the tests too because both algorithm do not
exactly order the same way, even if both of them do the job. I'm currently on
it (just to let you know).
{quote}
Thank you!
> ContainerExecutor does not order environment map
>
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch,
> YARN-5714.003.patch, YARN-5714.004.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped
> based on the order internally used by the map implementation (hash based). It
> does not take into consideration that some env varibales may refer each
> other, and so that some env variables must be declared before those
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a
> wrong value and so native libraries weren't loaded. jobs were running but not
> at their best efficiency. This is just a use case falling into that bug, but
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate
> to 5 days for packaging the patch in the right fashion for JIRA + try my best
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will
> only implement the unix version on a 1st release. I'm not used to windows env
> variables syntax so it will take me more time/research for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754986#comment-15754986 ] Hudson commented on YARN-5646: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11007 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11007/]) YARN-5646. Add documentation and update config parameter names for (arun suresh: rev 2273a74c1f3895163046cca09ff5e983df301d22) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/OpportunisticContainers.md * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMROpportunisticMaps.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestDistributedScheduling.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2904) Use linux cgroups to enhance container tear down
[
https://issues.apache.org/jira/browse/YARN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754975#comment-15754975
]
Nathan Roberts commented on YARN-2904:
--
Simple streaming job that does the following illustrates tasks escaping.
(/usr/bin/timeout does a setpgrp() which puts it in its own session).
{noformat}
#!/bin/bash
/usr/bin/timeout 1d /bin/sleep 1000
{noformat}
Mesos has apparently addressed this a couple of different ways including 1)
freeze_container->kill_all_processes_in_container->unfreeze_container; or 2)
use a private PID NS within the container and then kill PID1 within the
container.
> Use linux cgroups to enhance container tear down
>
>
> Key: YARN-2904
> URL: https://issues.apache.org/jira/browse/YARN-2904
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Nathan Roberts
>
> If we are launching yarn containers within cgroups, linux provides some
> guarantees that can help completely tear down a container. Specifically,
> linux guarantees that tasks can't escape a cgroup. We can use this fact to
> tear down a yarn container without leaking tasks.
> Today, a SIGTERM is sent to the session (normally lead by bash). When the
> session leader exits, the LCE sees this and assumes all resources have been
> given back to the system. This is not guaranteed. Example: YARN-2809
> implements a workaround that is only necessary because tasks are still
> lingering within the cgroup when the nodemanager attempts to delete it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-5641: -- Attachment: YARN-5641.005.patch [~jlowe], since HADOOP-13709 was committed, we can add in the localizer-specific piece. Here's a patch that attempts to shutdown the thread pool threads, kills all subprocesses, and then waits for the thread pool threads to terminate. If they do not terminate in 10 seconds, then we will call System.exit() to force the shutdown hook to be invoked, which will call destroyAllProcesses() again to make sure that all of the spawned subprocesses have been killed. > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5646: -- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-beta1) > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754894#comment-15754894
]
Arun Suresh commented on YARN-5646:
---
The checkstyle warnings can be ignored to retain the style of existing
variables in {{MRJobConfig}}
The findbugs warnings are not related and appear because I think mvn could not
find the findBugs.xml file.
The test failures are also unrelated and seem to work fine locally (except for
TestContainerManagerSecurity for which, i think we need to re-open YARN-5655)
Committing this to trunk shortly.
> Add documentation and update config parameter names for scheduling of
> OPPORTUNISTIC containers
> --
>
> Key: YARN-5646
> URL: https://issues.apache.org/jira/browse/YARN-5646
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5646.001.patch, YARN-5646.002.patch,
> YARN-5646.003.patch, YARN-5646.004.patch
>
>
> This is for adding documentation regarding the scheduling of OPPORTUNISTIC
> containers.
> It includes both the centralized (YARN-5220) and the distributed (YARN-2877)
> scheduling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6001) Improve moveApplicationQueues command line
[
https://issues.apache.org/jira/browse/YARN-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754877#comment-15754877
]
Sunil G commented on YARN-6001:
---
Thanks [~yufeigu]
Generally I understood your point. Since we already have a "movetoqueue"
implemented in a different way, i think we need to deprecate the same. But we
could not remove the internal code as its formally released cli. We could
remove from help, and create a new cli for now.
I think "changeQueue" may be more meaningful in that line since we cant use
"movetoqueue"
For eg:
{noformat}
./yarn application -appId application_1479894790219_0002 -changeQueue default
{noformat}
Thoughts?
> Improve moveApplicationQueues command line
> --
>
> Key: YARN-6001
> URL: https://issues.apache.org/jira/browse/YARN-6001
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-6001.0001.patch
>
>
> As an example, below command is used to move an application across queue.
> {noformat}./yarn application -movetoqueue application_1479894790219_0002
> -queue default{noformat}
> Inline with other application's attribute modification features such as
> updateTimeout or priority, movetoqueue cli command lacks unification and more
> complex or error prone.
> Suggesting below cli command which can be used.
> {noformat}
> ./yarn application -appId application_1479894790219_0002 -move default
> {noformat}
> This is inline with other commands such as
> {noformat}
> ./yarn application -appId application_1479894790219_0002 -updatePriority 8
> ./yarn application -appId application_1479894790219_0002 -updateLifetime 10
> {noformat}
> Old *movetoqueue* command could be still kept, but we can mark it as
> deprecated and mention in help message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5655) TestContainerManagerSecurity#testNMTokens is asserting
[
https://issues.apache.org/jira/browse/YARN-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754878#comment-15754878
]
Arun Suresh commented on YARN-5655:
---
[~rkanter], [~templedf], Looks like this still fails at times on trunk... can
you take a look ?
> TestContainerManagerSecurity#testNMTokens is asserting
> --
>
> Key: YARN-5655
> URL: https://issues.apache.org/jira/browse/YARN-5655
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jason Lowe
>Assignee: Robert Kanter
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: YARN-5655.001.patch
>
>
> TestContainerManagerSecurity has been failing recently in 2.8:
> {noformat}
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 80.928 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> testContainerManager[0](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 44.478 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:394)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:337)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Time elapsed: 34.964 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:333)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5646) Update config parameter names and add documentation for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5646: -- Summary: Update config parameter names and add documentation for scheduling of OPPORTUNISTIC containers (was: Documentation for scheduling of OPPORTUNISTIC containers) > Update config parameter names and add documentation for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5646: -- Summary: Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers (was: Update config parameter names and add documentation for scheduling of OPPORTUNISTIC containers) > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754635#comment-15754635
]
Sunil G commented on YARN-5956:
---
Yes. This test failure is not related.
Generally patch looks fine for me. I will wait for [~rohithsharma] and
[~templedf] if they have some more comments. Thank you.
> Refactor ClientRMService
>
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch,
> YARN-5956.03.patch, YARN-5956.04.patch, YARN-5956.05.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch
[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754630#comment-15754630
]
Sunil G commented on YARN-5877:
---
Thanks [~bibinchundatt]
I tested this patch locally with LCE as well as with Regular Executor.
[~bibinchundatt] also has tested in docker too.
Patch generally looks fine for me +1. MAPREDUCE-6704 also seems working.
[~templedf], any comments? I could commit the patch later today if there are no
objections.
> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch,
> YARN-5877.0003.patch, YARN-5877.0004.patch, YARN-5877.0005.patch,
> YARN-5877.0006.patch, bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
>
> Environment variables that containers may override rather
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
>
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5650) Render Application Timeout value in web UI(new and old)
[ https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754620#comment-15754620 ] Sunil G commented on YARN-5650: --- +1 from end. I will commit later today if there are no objections. > Render Application Timeout value in web UI(new and old) > --- > > Key: YARN-5650 > URL: https://issues.apache.org/jira/browse/YARN-5650 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Rohith Sharma K S >Assignee: Akhil PB > Attachments: YARN-5650.001.patch, YARN-5650.002.patch, > YARN-5650.003.patch, YARN-5650.004.patch, YARN-5650.005.patch, > screenshot-1.png, screenshot-2.png > > > This is to track timeout rendering in old and new yarn web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5956) Refactor ClientRMService
[
https://issues.apache.org/jira/browse/YARN-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754495#comment-15754495
]
Kai Sasaki commented on YARN-5956:
--
[~sunilg] Test failure of {{TestRMRestart}} cannot be reproduced on local. Is
it intermittent failure?
> Refactor ClientRMService
>
>
> Key: YARN-5956
> URL: https://issues.apache.org/jira/browse/YARN-5956
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5956.01.patch, YARN-5956.02.patch,
> YARN-5956.03.patch, YARN-5956.04.patch, YARN-5956.05.patch
>
>
> Some refactoring can be done in {{ClientRMService}}.
> - Remove redundant variable declaration
> - Fill in missing javadocs
> - Proper variable access modifier
> - Fix some typos in method name and exception messages
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5864) Capacity Scheduler preemption for fragmented cluster
[
https://issues.apache.org/jira/browse/YARN-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754101#comment-15754101
]
Carlo Curino commented on YARN-5864:
[~wangda] I like the direction of specifying more clearly what happens. I think
working on a design doc that spells this out would be very valuable, I am happy
to review and brainstorm with you if you think it is useful. (But FYI: I am on
parental leave, and traveling abroad till mid-Jan.)
In writing the document, in particular I think you should address the semantics
from all points of view, e.g., which guarantees do I get as a user of any of
the queues (not just the one we are preempting in favor of)? It is clear that
if I am running over-capacity I can be preempted, but what happens if I am
(safely?) within my capacity? (This is related to the "abuses" I was describing
before, e.g., one in which I ask for massive containers on the nodes I want,
and then resize them down, after you have killed anyone in my way).
Looking further ahead: Ideally, this document you are starting to capture the
semantics of this feature can be expanded to slowly cover all "tunables" of the
scheduler, and explore the many complex interactions among features and the
semantics we can derive from that (I bet we might be able to get rid of some
redundancies). This could become part of the documentation of YARN. Even nicer
would be to codify this with SLS driven tests (so that any future feature will
not mess up with the semantics you are capturing, without us noticing).
> Capacity Scheduler preemption for fragmented cluster
> -
>
> Key: YARN-5864
> URL: https://issues.apache.org/jira/browse/YARN-5864
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-5864.poc-0.patch
>
>
> YARN-4390 added preemption for reserved container. However, we found one case
> that large container cannot be allocated even if all queues are under their
> limit.
> For example, we have:
> {code}
> Two queues, a and b, capacity 50:50
> Two nodes: n1 and n2, each of them have 50 resource
> Now queue-a uses 10 on n1 and 10 on n2
> queue-b asks for one single container with resource=45.
> {code}
> The container could be reserved on any of the host, but no preemption will
> happen because all queues are under their limits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map
[
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753981#comment-15753981
]
Remi Catherinot commented on YARN-5714:
---
Hi, thanks for the review too :)
# are you sure about the double % ? i've tested it on windows 7, 8, 8.1 and 10
and double-ing the % does not escape it.
{noformat}
set A=toto
set B=%A%
set C=%%A%%
echo %A% %B% %C%
{noformat}
produce _toto %toto% %toto%_, so i always got _A_'s value, so both _B_ and _C_
depend on _A_.
# I still prefer to keep it the way it is because it is the last line of the
while that does this test. Personnally when i read code that double check
things I tend to think the code is still under dev and not yet production-ready
and then i start to tripple check everything i read. So for me, such double
check would lead in code maintenance/review time increase. if this is really a
blocking issue and does not permit the patch to be accepted, then i'll change
it.
# I've reworked the code to match you algorithm so the comment (and its tipo)
no longer exist
# I've tested it with deps in reverse order and having each env entry depending
on each of its successors and with 100 entries. On my computer it returns in 0
or 16ms (no lesser time precision). My algorithm start to be slower in a
perceptible way with the same test case when we have more than 200 entries. it
hits the second for computation when i have more than ~700 entries. So I've
implemented the new algorithm (more or less, i use recursion to handle the
stack stuff) but i need to rework the tests too because both algorithm do not
exactly order the same way, even if both of them do the job. I'm currently on
it (just to let you know).
> ContainerExecutor does not order environment map
>
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch,
> YARN-5714.003.patch, YARN-5714.004.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped
> based on the order internally used by the map implementation (hash based). It
> does not take into consideration that some env varibales may refer each
> other, and so that some env variables must be declared before those
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a
> wrong value and so native libraries weren't loaded. jobs were running but not
> at their best efficiency. This is just a use case falling into that bug, but
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate
> to 5 days for packaging the patch in the right fashion for JIRA + try my best
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will
> only implement the unix version on a 1st release. I'm not used to windows env
> variables syntax so it will take me more time/research for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance
[
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753958#comment-15753958
]
Sunil G commented on YARN-5995:
---
[~piaoyu zhang]
Thanks for the patch.
Generally I think that its better to discuss what all metrics are we going to
publish from StateStore and the type of the metric etc. I think it will help to
streamline the most needed metrics to be exposed as of priority.
Currently you have exposed {{Duration}} for all type of apis from StateStore
interface. I think this is too much of an information and we need to see
whether we need all such metrics and what we can infer out of same. I think
{{Duration}} overall for all api's may be more meaningful.So we can see how
fast the underlying storage is finishing each api writes and gives a metric
like {{no: of write ops per secs}} etc. Because I am not may not be much
worried about each api's storage duration.
I think we also need data size distribution. We may need to see how much data
is stored over time and its latency. As mentioned in YARN-3936, I also think
that *write latency* and *data size distribution* are major to start with.
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1
>Reporter: zhangyubiao
> Labels: patch
> Attachments: YARN-5995.0001.patch, YARN-5995.0002.patch,
> YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition
> performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753853#comment-15753853
]
Hadoop QA commented on YARN-5646:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 49s{color} | {color:orange} root: The patch generated 2 new + 954 unchanged
- 2 fixed = 956 total (was 956) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red}
patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6
[jira] [Commented] (YARN-3409) Add constraint node labels
[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753839#comment-15753839
]
Sunil G commented on YARN-3409:
---
Short summary from the offline discussion with [~naganarasimha...@apache.org]
[~leftnoteasy] [~kkaranasos] [~grey] [~varun_saxena]. I am taking the liberty
of summarizing. Folks, please pitch in and add more points if some points are
missed. Thank you.
*Basic requirements*
{{Attribute}} will be an entity which can be associated with a node. Few notes
about attribute.
- It will be a subclass of Node Labels
- Attribute could be considered as (key, value, type)
- Default type will be *string* for now.
- We could also have predefined types and it could be {{string}}. Later we
could add like number etc. This is to be discussed more.
- Boundary between resource and attribute should be whether its consumable or
not.
- Could use empty string to express “boolean” attributes
Create separate JIRA to define ConstraintExpression (add field to
ResourceRequest and change scheduler). Within this, we can have different
constraint types, e.g., AttributeConstraintExpression,
AffinityCostraintExpression. etc,
Few items to be discussed further more:
- Predefined attributed to created before use or not?
- Could we use existing label API / REST API or create new APIs? (An initial
consensus for this is to use existing API. We could discuss more.)
- Could have separate (dynamic) node labels for the (anti-)affinity constraints
> Add constraint node labels
> --
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
> YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to
> determinate how resources of a special set of nodes could be shared by a
> group of entities (like teams, departments, etc.). Partitions of a cluster
> has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has
> priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40%
> minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of
> node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >=
> 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6003) yarn-ui build failure caused by debug 2.4.0
[
https://issues.apache.org/jira/browse/YARN-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753808#comment-15753808
]
Sreenath Somarajapuram commented on YARN-6003:
--
It because npm uses semantic versioning (semver) rules and the dependencies are
not locked. Semver relies on package developers to not making mistakes, and
here we stumbled upon a package with a bug.
Best option to fix this issue would to move to yarn package manager instead of
npm. It also comes with added advantages like faster build time & offline build
support.
A patch for the same is under investigation.
> yarn-ui build failure caused by debug 2.4.0
> ---
>
> Key: YARN-6003
> URL: https://issues.apache.org/jira/browse/YARN-6003
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build, yarn-ui-v2
>Reporter: Akira Ajisaka
>Priority: Minor
>
> The recent build failure:
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/255/artifact/out/patch-compile-root.txt
> {noformat}
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/debug/debug.js:126
> debug.color = selectColor(namespae);
> ^
> ReferenceError: namespae is not defined
> at createDebug
> (/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/debug/debug.js:126:29)
> at Object.
> (/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/ember-cli/lib/models/project.js:16:43)
> at Module._compile (module.js:456:26)
> at Object.Module._extensions..js (module.js:474:10)
> at Module.load (module.js:356:32)
> at Function.Module._load (module.js:312:12)
> at Module.require (module.js:364:17)
> at require (module.js:380:17)
> at Object.
> (/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/ember-cli/lib/cli/index.js:4:21)
> at Module._compile (module.js:456:26)
> {noformat}
> build@2.4.0 is broken. https://github.com/visionmedia/debug/issues/347
> Maybe we need to set the version to 2.4.1 explicitly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
