[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422307#comment-15422307 ] Zhankun Tang commented on YARN-4266: [~sidharta-s], Any progress/update on this? We would like to help pushing this if you will. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5455) LinuxContainerExecutor needs Javadocs
[ https://issues.apache.org/jira/browse/YARN-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422304#comment-15422304 ] Varun Vasudev commented on YARN-5455: - +1 for the latest patch. I'll commit this tomorrow if no one objects. > LinuxContainerExecutor needs Javadocs > - > > Key: YARN-5455 > URL: https://issues.apache.org/jira/browse/YARN-5455 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-5455.001.patch, YARN-5455.002.patch, > YARN-5455.003.patch, YARN-5455.004.patch > > > 'Nuff said. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5042) Mount /sys/fs/cgroup into Docker containers as read only mount
[
https://issues.apache.org/jira/browse/YARN-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422291#comment-15422291
]
Varun Vasudev commented on YARN-5042:
-
Thanks for the updated patch [~luhuichun]! Just one change -
{code} +.addMountLocation("/sys/fs/cgroup", "/sys/fs/cgroup:ro", true);
{code}
Can you please set createSource to false? We don't want to create
/sys/fs/cgroup on the host machine if it doesn't exist.
> Mount /sys/fs/cgroup into Docker containers as read only mount
> --
>
> Key: YARN-5042
> URL: https://issues.apache.org/jira/browse/YARN-5042
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Varun Vasudev
>Assignee: luhuichun
> Attachments: YARN-5042.001.patch, YARN-5042.002.patch,
> YARN-5042.003.patch, YARN-5042.004.patch
>
>
> Containers running systemd need access to /sys/fs/cgroup. We should mount it
> into the container as a read only mount.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422289#comment-15422289 ] Varun Vasudev commented on YARN-5430: - +1 for the latest patch. I'll commit it tomorrow if no one objects. > Get container's ip and host from NM > --- > > Key: YARN-5430 > URL: https://issues.apache.org/jira/browse/YARN-5430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch, > YARN-5430.4.patch, YARN-5430.5.patch, YARN-5430.6.patch, YARN-5430.7.patch, > YARN-5430.8.patch > > > In YARN-4757, we introduced a DNS mechanism for containers. That's based on > the assumption that we can get the container's ip and host information and > store it in the registry-service. This jira aims to get the container's ip > and host from the NM, primarily docker container -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5152) [YARN-3368] Avoid reloading web page after selecting different queues in scheduler page
[
https://issues.apache.org/jira/browse/YARN-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422234#comment-15422234
]
Hadoop QA commented on YARN-5152:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 6s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 39s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:d13f52f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823820/YARN-5152-YARN-3368.0002.patch
|
| JIRA Issue | YARN-5152 |
| Optional Tests | asflicense |
| uname | Linux c2f3fc5546ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3368 / 6d0fd40 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12783/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Avoid reloading web page after selecting different queues in
> scheduler page
> ---
>
> Key: YARN-5152
> URL: https://issues.apache.org/jira/browse/YARN-5152
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Padma Priya Nagaraj
> Attachments: YARN-5152-YARN-3368-0002.patch,
> YARN-5152-YARN-3368.0001.patch, YARN-5152-YARN-3368.0002.patch,
> queue_onclick.png, queues_mouseover.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5152) [YARN-3368] Avoid reloading web page after selecting different queues in scheduler page
[ https://issues.apache.org/jira/browse/YARN-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1542#comment-1542 ] Padma Priya Nagaraj commented on YARN-5152: --- Hi [~sunilg], I have uploaded a new patch with the ASF policy details added in yarn-queue-apps-tests.js and yarn-queue-test.js > [YARN-3368] Avoid reloading web page after selecting different queues in > scheduler page > --- > > Key: YARN-5152 > URL: https://issues.apache.org/jira/browse/YARN-5152 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Padma Priya Nagaraj > Attachments: YARN-5152-YARN-3368-0002.patch, > YARN-5152-YARN-3368.0001.patch, YARN-5152-YARN-3368.0002.patch, > queue_onclick.png, queues_mouseover.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5152) [YARN-3368] Avoid reloading web page after selecting different queues in scheduler page
[ https://issues.apache.org/jira/browse/YARN-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Padma Priya Nagaraj updated YARN-5152: -- Attachment: YARN-5152-YARN-3368.0002.patch > [YARN-3368] Avoid reloading web page after selecting different queues in > scheduler page > --- > > Key: YARN-5152 > URL: https://issues.apache.org/jira/browse/YARN-5152 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Padma Priya Nagaraj > Attachments: YARN-5152-YARN-3368-0002.patch, > YARN-5152-YARN-3368.0001.patch, YARN-5152-YARN-3368.0002.patch, > queue_onclick.png, queues_mouseover.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422099#comment-15422099
]
Rohith Sharma K S commented on YARN-5521:
-
Right, this need to be reviewed and committed. Let me also look at yarn-5375.
Thanks.
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Fix For: 2.9.0
>
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[
https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422081#comment-15422081
]
Hadoop QA commented on YARN-4090:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 40s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 5s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805578/YARN-4090.002.patch |
| JIRA Issue | YARN-4090 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 55cb850923d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9daa997 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12782/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12782/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Make Collections.sort() more efficient in FSParentQueue.java
>
>
> Key: YARN-4090
> URL: https://issues.apache.org/jira/browse/YARN-4090
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Xianyin Xin
>
[jira] [Commented] (YARN-4586) YARN ATS: ATS leveldb fails to come up
[
https://issues.apache.org/jira/browse/YARN-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422071#comment-15422071
]
SeanWong commented on YARN-4586:
Hi, I encounter the same problem:
Error: .../yarn/timeline/leveldb-timeline-store.ldb/LOCK: No such file or
directory
So if someone can give me some suggestions? Thanks
> YARN ATS: ATS leveldb fails to come up
> --
>
> Key: YARN-4586
> URL: https://issues.apache.org/jira/browse/YARN-4586
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Environment: hadoop2.6.0+tez0.7.0
>Reporter: Lvpenglin
>
> My environment is hadoop2.6.0+tez0.7.0,when I start yarn timelineserver
> occurred this problem,I can not solve it ,can you help me or give some
> suggestion? Thank you very much.
> {code}
> STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on
> 2014-11-13T21:10Z
> STARTUP_MSG: java = 1.7.0_79
> /
> 16/01/13 00:01:29 INFO applicationhistoryservice.ApplicationHistoryServer:
> registered UNIX signal handlers for [TERM, HUP, INT]
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/td/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/td/tez-jars/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 16/01/13 00:01:31 WARN applicationhistoryservice.ApplicationHistoryServer:
> The filesystem based application history store is deprecated.
> 16/01/13 00:01:31 INFO impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 16/01/13 00:01:31 INFO impl.MetricsSystemImpl: Scheduled snapshot period at
> 10 second(s).
> 16/01/13 00:01:31 INFO impl.MetricsSystemImpl: ApplicationHistoryServer
> metrics system started
> 16/01/13 00:01:31 INFO timeline.LeveldbTimelineStore: Using leveldb path
> file:/home/td/temp/yarn/timeline/leveldb-timeline-store.ldb
> 16/01/13 00:01:31 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore failed in state
> INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO
> error:
> /home/td/file:/home/td/temp/yarn/timeline/leveldb-timeline-store.ldb/LOCK: No
> such file or directory
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /home/td/file:/home/td/temp/yarn/timeline/leveldb-timeline-store.ldb/LOCK: No
> such file or directory
> at
> org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:219)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:99)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:157)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:167)
> 16/01/13 00:01:31 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
> failed in state INITED; cause:
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /home/td/file:/home/td/temp/yarn/timeline/leveldb-timeline-store.ldb/LOCK: No
> such file or directory
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /home/td/file:/home/td/temp/yarn/timeline/leveldb-timeline-store.ldb/LOCK: No
> such file or directory
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.serve
[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
[
https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422033#comment-15422033
]
He Tianyi commented on YARN-5479:
-
Yes. And It's a vast improvement.
I simulated a scenario similar to production (with 10s of queues, hundres of
running apps) and benchmark showing 2x more faster for heartbeat processing.
> FairScheduler: Scheduling performance improvement
> -
>
> Key: YARN-5479
> URL: https://issues.apache.org/jira/browse/YARN-5479
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for
> scheduling. As number of nodes grows, more events need to be processed in
> time in FairScheduler. Also, increased number of applications & queues slows
> down processing of each single event.
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds.
> This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of
> some nodes cannot get processed before next heartbeat. This brings
> inefficiency handling burst requests (i.e. newly submitted MapReduce
> application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single
> cluster is scaled to several K of nodes (even with {{assignmultiple}}
> enabled).
> This issue proposes to perform several optimization towards performance in
> FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped
> (or should this be called 'delayed sorting'?). we can either start another
> dedicated thread to do the sorting & updating, or actually perform sorting
> after current result have been used several times (say sort once in every 100
> calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at
> least 2 objects ({{ResourceImpl}} and its proto builder) is created each time
> (using 'immutable' apis). the overhead can be eliminated with a
> light-weighted implementation of Resource, which do not instantiate a builder
> until necessary, because most instances are used as intermediate result in
> scheduler instead of being exchanged via IPC. Also, {{createResource}} is
> using reflection, which can be replaced by a plain {{new}} (for scheduler
> usage only). furthermore, perhaps we could 'intern' resource to avoid
> allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to
> {{update}}, making root queue metrics eventual consistent (which may
> satisfies most of the needs). or introduce counters to {{getResourceUsage}}
> and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5475) Test failed for TestAggregatedLogFormat on trunk
[
https://issues.apache.org/jira/browse/YARN-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422013#comment-15422013
]
Hadoop QA commented on YARN-5475:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823797/YARN-5475.02.patch |
| JIRA Issue | YARN-5475 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bc0d2cb59aff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9daa997 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12781/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12781/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Test failed for TestAggregatedLogFormat on trunk
>
>
> Key: YARN-5475
> URL: https://issues.apache.org/jira/browse/YARN-5475
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junping Du
>Assignee: Jun Gong
> Attachments: YARN-5475.01.patch, YARN-5475.02.patch
>
>
> From some jenkins run:
> https://builds.apache.org/job/PreCommit-YARN-Build/12651/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
> The error message fo
[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
[
https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422002#comment-15422002
]
Xianyin Xin commented on YARN-5479:
---
[~He Tianyi], hope YARN-4090 can provide some information, in which the locked
resourceusage was snapshoted and such the performance was improved greatly.
> FairScheduler: Scheduling performance improvement
> -
>
> Key: YARN-5479
> URL: https://issues.apache.org/jira/browse/YARN-5479
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for
> scheduling. As number of nodes grows, more events need to be processed in
> time in FairScheduler. Also, increased number of applications & queues slows
> down processing of each single event.
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds.
> This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of
> some nodes cannot get processed before next heartbeat. This brings
> inefficiency handling burst requests (i.e. newly submitted MapReduce
> application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single
> cluster is scaled to several K of nodes (even with {{assignmultiple}}
> enabled).
> This issue proposes to perform several optimization towards performance in
> FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped
> (or should this be called 'delayed sorting'?). we can either start another
> dedicated thread to do the sorting & updating, or actually perform sorting
> after current result have been used several times (say sort once in every 100
> calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at
> least 2 objects ({{ResourceImpl}} and its proto builder) is created each time
> (using 'immutable' apis). the overhead can be eliminated with a
> light-weighted implementation of Resource, which do not instantiate a builder
> until necessary, because most instances are used as intermediate result in
> scheduler instead of being exchanged via IPC. Also, {{createResource}} is
> using reflection, which can be replaced by a plain {{new}} (for scheduler
> usage only). furthermore, perhaps we could 'intern' resource to avoid
> allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to
> {{update}}, making root queue metrics eventual consistent (which may
> satisfies most of the needs). or introduce counters to {{getResourceUsage}}
> and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5475) Test failed for TestAggregatedLogFormat on trunk
[
https://issues.apache.org/jira/browse/YARN-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-5475:
---
Attachment: YARN-5475.02.patch
Fix checkstyle error.
> Test failed for TestAggregatedLogFormat on trunk
>
>
> Key: YARN-5475
> URL: https://issues.apache.org/jira/browse/YARN-5475
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junping Du
> Attachments: YARN-5475.01.patch, YARN-5475.02.patch
>
>
> From some jenkins run:
> https://builds.apache.org/job/PreCommit-YARN-Build/12651/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
> The error message for the test is:
> {noformat}
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.114 sec <<<
> FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat
> testReadAcontainerLogs1(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat)
> Time elapsed: 0.012 sec <<< ERROR!
> java.io.IOException: Unable to create directory :
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/TestAggregatedLogFormat/testReadAcontainerLogs1/srcFiles/application_1_0001/container_1_0001_01_01/subDir
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.getOutputStreamWriter(TestAggregatedLogFormat.java:403)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.writeSrcFile(TestAggregatedLogFormat.java:382)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testReadAcontainerLog(TestAggregatedLogFormat.java:211)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testReadAcontainerLogs1(TestAggregatedLogFormat.java:185)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5475) Test failed for TestAggregatedLogFormat on trunk
[
https://issues.apache.org/jira/browse/YARN-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong reassigned YARN-5475:
--
Assignee: Jun Gong
> Test failed for TestAggregatedLogFormat on trunk
>
>
> Key: YARN-5475
> URL: https://issues.apache.org/jira/browse/YARN-5475
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junping Du
>Assignee: Jun Gong
> Attachments: YARN-5475.01.patch, YARN-5475.02.patch
>
>
> From some jenkins run:
> https://builds.apache.org/job/PreCommit-YARN-Build/12651/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
> The error message for the test is:
> {noformat}
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.114 sec <<<
> FAILURE! - in org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat
> testReadAcontainerLogs1(org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat)
> Time elapsed: 0.012 sec <<< ERROR!
> java.io.IOException: Unable to create directory :
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/TestAggregatedLogFormat/testReadAcontainerLogs1/srcFiles/application_1_0001/container_1_0001_01_01/subDir
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.getOutputStreamWriter(TestAggregatedLogFormat.java:403)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.writeSrcFile(TestAggregatedLogFormat.java:382)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testReadAcontainerLog(TestAggregatedLogFormat.java:211)
> at
> org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat.testReadAcontainerLogs1(TestAggregatedLogFormat.java:185)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5467) InputValidator for the FederationStateStore internal APIs
[
https://issues.apache.org/jira/browse/YARN-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421978#comment-15421978
]
Subru Krishnan commented on YARN-5467:
--
Thanks [~giovanni.fumarola] for working on this. Good job especially with the
test coverage.
I have a few minor comments:
* Move the _InputValidator_ classes to
*org.apache.hadoop.yarn.server.federation.store.utils* package.
* Align the _InputValidator_ class names with the Store interface names.
* Consider logging the exception before throwing. This way we need to only
log in single place for any input validation errors instead of in each store
impl.
* In {{FederationApplicationInputValidator::checkApplicationId}}, we should
also check if the value if parseable into an _ApplicationId_.
* There are few typos in the exception message, for e.g:
*checkSubClusterPolicyConfiguration*, *checkQueue*, etc.
* Remove the *Correct* suffix from tests as they check both valid & invalid
scenarios.
* Nit: move the null tests before the invalid tests.
* Can you integrate the _InputValidators_with {{MemoryFederationStateStore}}
as now they are not being used anywhere.
> InputValidator for the FederationStateStore internal APIs
> -
>
> Key: YARN-5467
> URL: https://issues.apache.org/jira/browse/YARN-5467
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-5467-YARN-2915.v0.patch,
> YARN-5467-YARN-2915.v1.patch, YARN-5467-YARN-2915.v2.patch
>
>
> We need to check for mandatory fields, well formed-ness (for address fields)
> etc of input params to FederationStateStore. This is common across all Store
> implementations and can be used as a _fail-fast_ mechanism on the client side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3672) Create Facade for Federation State and Policy Store
[
https://issues.apache.org/jira/browse/YARN-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421922#comment-15421922
]
Hadoop QA commented on YARN-3672:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 43s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 29s
{color} | {color:red} root: The patch generated 16 new + 214 unchanged - 0
fixed = 230 total (was 214) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common
generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 19s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.logaggregation.TestAggregatedLogFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issue
[jira] [Commented] (YARN-5047) Refactor nodeUpdate across schedulers
[
https://issues.apache.org/jira/browse/YARN-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421919#comment-15421919
]
Hadoop QA commented on YARN-5047:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 664 unchanged - 13 fixed = 666 total (was 677)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 39s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823777/YARN-5047.010.patch |
| JIRA Issue | YARN-5047 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fa5c31d3e5dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 864f878 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12778/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12778/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12778/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12778/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop
[jira] [Updated] (YARN-3649) Allow configurable prefix for hbase table names (like prod, exp, test etc)
[ https://issues.apache.org/jira/browse/YARN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3649: - Attachment: YARN-3649-YARN-5355.002.patch Uploading patch YARN-3649-YARN-5355.002.patch that I think fixes the checkstyle and javadoc issues as well as the documentation point from Varun. I also fixed several other checkstyle issues. > Allow configurable prefix for hbase table names (like prod, exp, test etc) > -- > > Key: YARN-3649 > URL: https://issues.apache.org/jira/browse/YARN-3649 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > Attachments: YARN-3649-YARN-2928.01.patch, > YARN-3649-YARN-5355.002.patch, YARN-3649-YARN-5355.01.patch > > > As per [~jrottinghuis]'s suggestion in YARN-3411, it will be a good idea to > have a configurable prefix for hbase table names. > This way we can easily run a staging, a test, a production and whatever setup > in the same HBase instance / without having to override every single table in > the config. > One could simply overwrite the default prefix and you're off and running. > For prefix, potential candidates are "tst" "prod" "exp" etc. Once can then > still override one tablename if needed, but managing one whole setup will be > easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421912#comment-15421912
]
Hadoop QA commented on YARN-5524:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 1 new + 98 unchanged - 1 fixed = 99 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 52s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823774/YARN-5524.002.patch |
| JIRA Issue | YARN-5524 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d437d2735b9a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 864f878 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12780/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12780/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12780/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12780/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12780/console |
| Powered by | Apache
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421868#comment-15421868
]
Hadoop QA commented on YARN-5524:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 1 new + 98 unchanged - 1 fixed = 99 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 4s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823774/YARN-5524.002.patch |
| JIRA Issue | YARN-5524 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 46a821413a26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 864f878 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12777/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12777/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12777/console |
| Powered by | Apache Y
[jira] [Updated] (YARN-3672) Create Facade for Federation State and Policy Store
[
https://issues.apache.org/jira/browse/YARN-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-3672:
-
Attachment: YARN-3672-YARN-2915-v4.patch
Thanks [~jianhe] for your feedback.
bq. How will the RM failover play together with YARN-3673 ? Let’s say
subCluster1(RM1, RM2), subCluster2(RM3, RM4). Looks like the implementation
will ignore cluster intra-failover and do cluster inter-failover only ?
The implementation will handle only cluster intra-failover as the RM failover
proxy in YARN-3673 will be seeded based on _subClusterId_. The information on
the StateStore will get updated as part of RM active services initialization
(YARN-3671). In your example, the RM failover proxy will be a _connection_ to
subCluster1 which will initially point to say RM1 which is the current primary.
Suppose RM1 fails over to RM2, RM2 will now heartbeat the StateStore against
subCluster1 and we will auto-update the proxy to connect to RM2 (by querying
getSubClusterInfo(subCluster1) on the Facade).
The cluster inter-failover is determined by the policies (YARN-5323) as that
defines how a queue spans multiple sub-clusters and the {{Router/AMRMProxy}}
will create a RM failover proxy per subCluster in the policy.
Makes sense?
bq. question for such API. It asks for a specific subCluster info, do we still
need the filterInactiveSubClusters flag ? Even if it’s required, the behavior
for the if/else is inconsistent, the if case is honoring the flag, while the
else doesn’t.
Good catch. I have removed filterInactiveSubClusters from getSubClusterInfo.
bq. I think we should not reuse these two configs for retry, the default value
of both is zero.
Valid point. I was trying to reuse existing configs as we have to add a few for
Federation on top of the too many existing ones. I looked at the {{RMProxy}}
and have replaced them with better fitting ones.
I have updated the patch (v4) accordingly.
> Create Facade for Federation State and Policy Store
> ---
>
> Key: YARN-3672
> URL: https://issues.apache.org/jira/browse/YARN-3672
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3672-YARN-2915-v1.patch,
> YARN-3672-YARN-2915-v2.patch, YARN-3672-YARN-2915-v3.patch,
> YARN-3672-YARN-2915-v4.patch
>
>
> This JIRA proposes creating a facade for Federation State and Policy Store to
> simply access and have a common place for cache management etc that can be
> used by both Router & AMRMProxy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5047) Refactor nodeUpdate across schedulers
[ https://issues.apache.org/jira/browse/YARN-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5047: - Attachment: YARN-5047.010.patch - Removed unused imports in CapacityScheduler > Refactor nodeUpdate across schedulers > - > > Key: YARN-5047 > URL: https://issues.apache.org/jira/browse/YARN-5047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, scheduler >Affects Versions: 3.0.0-alpha1 >Reporter: Ray Chiang >Assignee: Ray Chiang > Attachments: YARN-5047.001.patch, YARN-5047.002.patch, > YARN-5047.003.patch, YARN-5047.004.patch, YARN-5047.005.patch, > YARN-5047.006.patch, YARN-5047.007.patch, YARN-5047.008.patch, > YARN-5047.009.patch, YARN-5047.010.patch > > > FairScheduler#nodeUpdate() and CapacityScheduler#nodeUpdate() have a lot of > commonality in their code. See about refactoring the common parts into > AbstractYARNScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3649) Allow configurable prefix for hbase table names (like prod, exp, test etc)
[ https://issues.apache.org/jira/browse/YARN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421838#comment-15421838 ] Vrushali C commented on YARN-3649: -- Thanks [~varun_saxena]! I will upload a patch shortly that addresses the documentation changes and the checkstyle ones. bq. I guess there can be differing opinions on this but should we have prod as default instead of dev ? Yes, so my thinking is that if a new person wants to try this out in their setup and then tries out something to run with default options and if it is set to default to "prod.", then they will start writing to prod tables unintentionally. But when the param on the cluster configs on client nodes is explicitly set to "prod.", then that is fine and that would mean any user writing to timeline service will write to production tables, which is what we would want on those client nodes. > Allow configurable prefix for hbase table names (like prod, exp, test etc) > -- > > Key: YARN-3649 > URL: https://issues.apache.org/jira/browse/YARN-3649 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > Attachments: YARN-3649-YARN-2928.01.patch, > YARN-3649-YARN-5355.01.patch > > > As per [~jrottinghuis]'s suggestion in YARN-3411, it will be a good idea to > have a configurable prefix for hbase table names. > This way we can easily run a staging, a test, a production and whatever setup > in the same HBase instance / without having to override every single table in > the config. > One could simply overwrite the default prefix and you're off and running. > For prefix, potential candidates are "tst" "prod" "exp" etc. Once can then > still override one tablename if needed, but managing one whole setup will be > easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-5524:
-
Attachment: YARN-5524.002.patch
Uploading patch v2 to address checkstyle warnings (although first patch
followed the existing coding style).
Not sure if the checkstyle warning about function length is to be fixed in this
patch. The LogsCLI#run function without this patch is already longer than 150
lines.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421800#comment-15421800 ] Jason Lowe commented on YARN-3388: -- [~nroberts] the patch no longer applies to trunk. Could you please rebase? Looks good overall, but it would be nice to have some whitespace around the calculateUserUsageRatio, recalculateQueueUsageRatio, and getUsageRatio methods for readability. > Allocation in LeafQueue could get stuck because DRF calculator isn't well > supported when computing user-limit > - > > Key: YARN-3388 > URL: https://issues.apache.org/jira/browse/YARN-3388 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Attachments: YARN-3388-v0.patch, YARN-3388-v1.patch, > YARN-3388-v2.patch, YARN-3388-v3.patch > > > When there are multiple active users in a queue, it should be possible for > those users to make use of capacity up-to max_capacity (or close). The > resources should be fairly distributed among the active users in the queue. > This works pretty well when there is a single resource being scheduled. > However, when there are multiple resources the situation gets more complex > and the current algorithm tends to get stuck at Capacity. > Example illustrated in subsequent comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421786#comment-15421786
]
Hadoop QA commented on YARN-5524:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 5 new + 98 unchanged - 1 fixed = 103 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 26s
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823766/YARN-5524.001.patch |
| JIRA Issue | YARN-5524 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c56ffd212dd5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 03dea65 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12776/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12776/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12776/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Com
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421781#comment-15421781
]
sandflee commented on YARN-5521:
Thanks [~varun_saxena], [~sunilg], [~bibinchundatt]
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Fix For: 2.9.0
>
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5525) Make log aggregation service class configurable
[ https://issues.apache.org/jira/browse/YARN-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5525: - Assignee: Botong Huang > Make log aggregation service class configurable > --- > > Key: YARN-5525 > URL: https://issues.apache.org/jira/browse/YARN-5525 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Reporter: Giovanni Matteo Fumarola >Assignee: Botong Huang >Priority: Minor > > Make the log aggregation class configurable and extensible, so that > alternative log aggregation behaviors like app specific log aggregation > directory, log aggregation format can be implemented and plugged in. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5525) Make log aggregation service class configurable
Giovanni Matteo Fumarola created YARN-5525: -- Summary: Make log aggregation service class configurable Key: YARN-5525 URL: https://issues.apache.org/jira/browse/YARN-5525 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Reporter: Giovanni Matteo Fumarola Priority: Minor Make the log aggregation class configurable and extensible, so that alternative log aggregation behaviors like app specific log aggregation directory, log aggregation format can be implemented and plugged in. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-5524:
-
Attachment: YARN-5524.001.patch
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-5524:
-
Attachment: (was: YARN-5524.001.patch)
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5265) Make HBase configuration for the timeline service configurable
[
https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421722#comment-15421722
]
Hadoop QA commented on YARN-5265:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 33s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
9s {color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 1s
{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org
[jira] [Updated] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-5524:
-
Attachment: YARN-5524.001.patch
Uploading v1.
[~prasanth_j] does this seem to be along the lines you were thinking? This will
check for extra arguments and return an exit of -1 (similar to other incorrect
argument situations). Added some test cases.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421714#comment-15421714
]
Hadoop QA commented on YARN-5519:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 2 new + 0 unchanged - 2 fixed = 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 16s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823760/YARN-5519-YARN-2915.v4.patch
|
| JIRA Issue | YARN-5519 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 86689485be9f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / b689f55 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12775/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12775/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12775/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add SubClusterId in AddApplicationHomeSubClu
[jira] [Comment Edited] (YARN-5520) [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues
[
https://issues.apache.org/jira/browse/YARN-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421682#comment-15421682
]
Min Shen edited comment on YARN-5520 at 8/15/16 9:15 PM:
-
[~venkateshrin],
Thanks for providing feedbacks on this ticket.
To answer your questions, I'd like to use the following example to make the
explanation more clear:
Assume we have 4 queues configured, i.e. root.orgA, root.orgB, root.default,
and root.public
root.orgA's capacity and max capacity are configured as 45% and 45%,
respectively. Same for root.orgB.
root.default is configured as 0%, 0% while root.public is configured as 5% and
30%.
Preemption is also enabled. Thus, root.orgA and root.orgB each has 45% of
guaranteed resources, while root.public has access to certain elastic resources
w/o too much guarantee.
Also, assume we have 3 users, userA which belongs to orgA, userB which belongs
to orgB, and userC which also belongs to orgB.
Admins want to route users to their corresponding organization queue, so they
have configured the following in capacity-scheduler.xml:
{noformat}
u:userA:orgA, u:userB:orgB, u:userC:orgB
{noformat}
In addition, YarnConfiguration.DEFAULT_QUEUE_NAME is set to "default".
In my proposed change, when
{{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to false,
user's application will always get submitted to whichever queue the user
requests, or root.default if user does not specify a queue.
When {{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true,
we have the following possible scenarios:
# userA/userB/userC submits jobs which do not specify a queue. My proposed
change will override the application's queue with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.default, root.orgA, or root.orgB. The
application's queue will still be overridden with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.public. The application will be
submitted to root.public. This could happen in the following case: userB
consumed all available resources in root.orgB but userA is not using resources
in root.orgA. In the mean time, userC wants to launch his job. If we enforce
queue overriding for all queues, then userC has to wait for userB's job to
release resources. However, if we disable queue overriding for root.public,
userC can use root.public to get resources much more quickly.
In this way, the admin can override queues for applications submitted to a
certain subset of queues in the cluster, while still allowing users to use the
"adhoc" queues. It also distinguishes well between the cases when queue
overriding is enabled vs. when it's not, since users have to explicitly specify
to use the "adhoc" queues in order to disable queue overriding. As a result,
the users should understand disabling queue overriding comes with the cost of
less resource guarantees (because of preemption).
We can introduce an additional parameter
{{yarn.scheduler.capacity.queue-mappings.disabled.queues}} to control the list
of queues where queue overriding is disabled when
{{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true.
was (Author: mshen):
[~venkateshrin],
Thanks for providing feedbacks on this ticket.
To answer your questions, I'd like to use the following example to make the
explanation more clear:
Assume we have 4 queues configured, i.e. root.orgA, root.orgB, root.default,
and root.public
root.orgA's capacity and max capacity are configured as 45% and 45%,
respectively. Same for root.orgB.
root.default is configured as 0%, 0% while root.public is configured as 5% and
30%.
Preemption is also enabled. Thus, root.orgA and root.orgB each has 45% of
guaranteed resources, while root.public has access to certain elastic resources
w/o too much guarantee.
Also, assume we have 3 users, userA which belongs to orgA, userB which belongs
to orgB, and userC which also belongs to orgB.
Admins want to route users to their corresponding organization queue, so they
have configured the following in capacity-scheduler.xml:
{noformat}
u:userA:orgA, u:userB:orgB, u:userC:orgB
{noformat}
In addition, YarnConfiguration.DEFAULT_QUEUE_NAME is set to "default".
In my proposed change, when
{{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to false,
user's application will always get submitted to whichever queue the user
requests, or root.default if user does not specify a queue.
When {{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true,
we have the following possible scenarios:
# userA/userB/userC submits jobs which do not specify a queue. My proposed
change will override the application's queue with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.default, root.orgA,
[jira] [Comment Edited] (YARN-5520) [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues
[
https://issues.apache.org/jira/browse/YARN-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421682#comment-15421682
]
Min Shen edited comment on YARN-5520 at 8/15/16 9:14 PM:
-
[~venkateshrin],
Thanks for providing feedbacks on this ticket.
To answer your questions, I'd like to use the following example to make the
explanation more clear:
Assume we have 4 queues configured, i.e. root.orgA, root.orgB, root.default,
and root.public
root.orgA's capacity and max capacity are configured as 45% and 45%,
respectively. Same for root.orgB.
root.default is configured as 0%, 0% while root.public is configured as 5% and
30%.
Preemption is also enabled. Thus, root.orgA and root.orgB each has 45% of
guaranteed resources, while root.public has access to certain elastic resources
w/o too much guarantee.
Also, assume we have 3 users, userA which belongs to orgA, userB which belongs
to orgB, and userC which also belongs to orgB.
Admins want to route users to their corresponding organization queue, so they
have configured the following in capacity-scheduler.xml:
{noformat}
u:userA:orgA, u:userB:orgB, u:userC:orgB
{noformat}
In addition, YarnConfiguration.DEFAULT_QUEUE_NAME is set to "default".
In my proposed change, when
{{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to false,
user's application will always get submitted to whichever queue the user
requests, or root.default if user does not specify a queue.
When {{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true,
we have the following possible scenarios:
# userA/userB/userC submits jobs which do not specify a queue. My proposed
change will override the application's queue with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.default, root.orgA, or root.orgB. The
application's queue will still be overridden with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.public. The application will be
submitted to root.public. This could happen in the following case: userB
consumed all available resources in root.orgB but userA is not using resources
in root.orgA. In the mean time, userC wants to launch his job. If we enforce
queue overriding for all queues, then userC has to wait for userB's job to
release resources. However, if we disable queue overriding for root.public,
userC can use root.public to get resources much more quickly.
In this way, the admin can override queues for applications submitted to a
certain subset of queues in the cluster, while still allowing users to use the
"adhoc" queues. It also distinguishes well between the cases when queue
overriding is enabled vs. when it's not, since users have to explicitly specify
to use the "adhoc" queues in order to disable queue overriding. As a result,
the users should understand disabling queue overriding comes with the cost of
less resource guarantees (because of preemption).
We can introduce an additional parameter
{{yarn.scheduler.capacity.queue-mappings.disabled.queues}} to control the list
of queues where queue overriding is disabled when
{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true.
was (Author: mshen):
[~venkateshrin],
Thanks for providing feedbacks on this ticket.
To answer your questions, I'd like to use the following example to make the
explanation more clear:
Assume we have 4 queues configured, i.e. root.orgA, root.orgB, root.default,
and root.public
root.orgA's capacity and max capacity are configured as 45% and 45%,
respectively. Same for root.orgB.
root.default is configured as 0%, 0% while root.public is configured as 5% and
30%.
Preemption is also enabled. Thus, root.orgA and root.orgB each has 45% of
guaranteed resources, while root.public has access to certain elastic resources
w/o too much guarantee.
Also, assume we have 3 users, userA which belongs to orgA, userB which belongs
to orgB, and userC which also belongs to orgB.
Admins want to route users to their corresponding organization queue, so they
have configured the following in capacity-scheduler.xml:
{noformat}
u:userA:orgA, u:userB:orgB, u:userC:orgB
{noformat}
In addition, YarnConfiguration.DEFAULT_QUEUE_NAME is set to "default".
In my proposed change, when
{{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to false,
user's application will always get submitted to whichever queue the user
requests, or root.default if user does not specify a queue.
When {{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true,
we have the following possible scenarios:
# userA/userB/userC submits jobs which do not specify a queue. My proposed
change will override the application's queue with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.default, root.orgA, o
[jira] [Updated] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ellen Hui updated YARN-5519:
Attachment: YARN-5519-YARN-2915.v4.patch
* Add [~subru]'s javadoc suggestions
* Fix typos in {{FederationStateStoreBaseTest}}
* Remove the line in {{MemoryFederationStore}} that modifies a
{{SubClusterInfo}}'s lastStartTime timestamp during registration
> Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
>
>
> Key: YARN-5519
> URL: https://issues.apache.org/jira/browse/YARN-5519
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Ellen Hui
> Attachments: YARN-5519-YARN-2915.v1.patch,
> YARN-5519-YARN-2915.v2.patch, YARN-5519-YARN-2915.v3.patch,
> YARN-5519-YARN-2915.v4.patch
>
>
> This JIRA tracks the addition of SubClusterId into
> AddApplicationHomeSubClusterResponse.
> in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659],
> to handle better fail-over scenario the response needs SubclusterId as field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5520) [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues
[
https://issues.apache.org/jira/browse/YARN-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421682#comment-15421682
]
Min Shen commented on YARN-5520:
[~venkateshrin],
Thanks for providing feedbacks on this ticket.
To answer your questions, I'd like to use the following example to make the
explanation more clear:
Assume we have 4 queues configured, i.e. root.orgA, root.orgB, root.default,
and root.public
root.orgA's capacity and max capacity are configured as 45% and 45%,
respectively. Same for root.orgB.
root.default is configured as 0%, 0% while root.public is configured as 5% and
30%.
Preemption is also enabled. Thus, root.orgA and root.orgB each has 45% of
guaranteed resources, while root.public has access to certain elastic resources
w/o too much guarantee.
Also, assume we have 3 users, userA which belongs to orgA, userB which belongs
to orgB, and userC which also belongs to orgB.
Admins want to route users to their corresponding organization queue, so they
have configured the following in capacity-scheduler.xml:
{noformat}
u:userA:orgA, u:userB:orgB, u:userC:orgB
{noformat}
In addition, YarnConfiguration.DEFAULT_QUEUE_NAME is set to "default".
In my proposed change, when
{{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to false,
user's application will always get submitted to whichever queue the user
requests, or root.default if user does not specify a queue.
When {{yarn.scheduler.capacity.queue-mappings-override.enable}} is set to true,
we have the following possible scenarios:
# userA/userB/userC submits jobs which do not specify a queue. My proposed
change will override the application's queue with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.default, root.orgA, or root.orgB. The
application's queue will still be overridden with the one specified in the
queue mappings configuration.
# userA/userB/userC submits jobs to root.public. The application will be
submitted to root.public. This could happen in the following case: userB
consumed all available resources in root.orgB but userA is not using resources
in root.orgA. In the mean time, userC wants to launch his job. If we enforce
queue overriding for all queues, then userC has to wait for userB's job to
release resources. However, if we disable queue overriding for root.public,
userC can use root.public to get resources much more quickly.
In this way, the admin can override queues for applications submitted to a
certain subset of queues in the cluster, while still allowing users to use the
"adhoc" queues. It also distinguishes well between the cases when queue
overriding is enabled vs. when it's not, since users have to explicitly specify
to use the "adhoc" queues in order to disable queue overriding. As a result,
the users should understand disabling queue overriding comes with the cost of
less resource guarantees (because of preemption).
> [Capacity Scheduler] Change the logic for when to trigger user/group mappings
> to queues
> ---
>
> Key: YARN-5520
> URL: https://issues.apache.org/jira/browse/YARN-5520
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.6.0, 2.7.0, 2.6.1
>Reporter: Min Shen
>
> In YARN-2411, the feature in Capacity Scheduler to support user/group based
> mappings to queues was introduced.
> In the original implementation, the configuration key
> {{yarn.scheduler.capacity.queue-mappings-override.enable}} was added to
> control when to enable overriding user requested queues.
> However, even if this configuration is set to false, queue overriding could
> still happen if the user didn't request for any specific queue or choose to
> simply submit his job to "default" queue, according to the following if
> condition which triggers queue overriding:
> {code}
> if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
> || overrideWithQueueMappings)
> {code}
> This logic does not seem very reasonable, as there's no way to fully disable
> queue overriding when mappings are configured inside capacity-scheduler.xml.
> In addition, in our environment, we have setup a few organization dedicated
> queues as well as some "adhoc" queues. The organization dedicated queues have
> better resource guarantees and we want to be able to route users to the
> corresponding organization queues. On the other hand, the "adhoc" queues have
> less resource guarantees but everyone can use it to get some opportunistic
> resources when the cluster is free.
> The current logic will also prevent this type of use cases as when you enable
> queue overriding, users cannot use these "adhoc" queues any more. They will
> always be routed to the dedicated organization queues.
>
[jira] [Commented] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421674#comment-15421674
]
Ellen Hui commented on YARN-5519:
-
The test failed on a timing issue, since the {{MemoryFederationStateStore}}
updates the lastHeartbeat field of a {{SubClusterInfo}} during registration,
causing the stored value and the expected one to differ slightly. I will remove
the line in {{MemoryFederationStateStore}} that modifies the timestamp, so that
the timestamp passed in through the {{SubClusterRegisterRequest}} is stored
unchanged.
> Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
>
>
> Key: YARN-5519
> URL: https://issues.apache.org/jira/browse/YARN-5519
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Ellen Hui
> Attachments: YARN-5519-YARN-2915.v1.patch,
> YARN-5519-YARN-2915.v2.patch, YARN-5519-YARN-2915.v3.patch
>
>
> This JIRA tracks the addition of SubClusterId into
> AddApplicationHomeSubClusterResponse.
> in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659],
> to handle better fail-over scenario the response needs SubclusterId as field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-5304: --- Comment: was deleted (was: Think I found the cause of the unit test failures. In one case I created a now config rather than taking the one passed in. Also realized that in the main for creating schemas the new behavior was different, so that should be fixed now. Problem was that with default (empty) config, HBase tries to open ports already in use.) > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5265) Make HBase configuration for the timeline service configurable
[ https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-5265: --- Attachment: YARN-5265-YARN-5355.09.patch Think I found the cause of the unit test failures. In one case I created a now config rather than taking the one passed in. Also realized that in the main for creating schemas the new behavior was different, so that should be fixed now. Problem was that with default (empty) config, HBase tries to open ports already in use. > Make HBase configuration for the timeline service configurable > -- > > Key: YARN-5265 > URL: https://issues.apache.org/jira/browse/YARN-5265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > Attachments: ATS v2 cluster deployment v1.png, > YARN-5265-YARN-2928.01.patch, YARN-5265-YARN-2928.02.patch, > YARN-5265-YARN-2928.03.patch, YARN-5265-YARN-2928.04.patch, > YARN-5265-YARN-2928.05.patch, YARN-5265-YARN-5355.06.patch, > YARN-5265-YARN-5355.07.patch, YARN-5265-YARN-5355.08.patch, > YARN-5265-YARN-5355.09.patch > > > Currently we create "default" HBase configurations, this works as long as the > user places the appropriate configuration on the classpath. > This works fine for a standalone Hadoop cluster. > However, if a user wants to monitor an HBase cluster and has a separate ATS > HBase cluster, then it can become tricky to create the right classpath for > the nodemanagers and still have tasks have their separate configs. > It will be much easier to add a yarn configuration to let cluster admins > configure which HBase to connect to to write ATS metrics to. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-5304: --- Attachment: YARN-5265-YARN-5355.09.patch Think I found the cause of the unit test failures. In one case I created a now config rather than taking the one passed in. Also realized that in the main for creating schemas the new behavior was different, so that should be fixed now. Problem was that with default (empty) config, HBase tries to open ports already in use. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-5304: --- Attachment: (was: YARN-5265-YARN-5355.09.patch) > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421616#comment-15421616
]
Subru Krishnan commented on YARN-5519:
--
[~ellenfkh], I just have one last request :) - can you fix the Javadoc of
{{AddApplicationHomeSubClusterResponse}} to say:
bq. If a mapping for the application already existed, the {@code SubClusterId}
in this response will return the existing mapping which might be different from
that in the {@code AddApplicationHomeSubClusterRequest}.
Also include the same comment in {{FederationApplicationHomeSubClusterStore}}.
> Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
>
>
> Key: YARN-5519
> URL: https://issues.apache.org/jira/browse/YARN-5519
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Ellen Hui
> Attachments: YARN-5519-YARN-2915.v1.patch,
> YARN-5519-YARN-2915.v2.patch, YARN-5519-YARN-2915.v3.patch
>
>
> This JIRA tracks the addition of SubClusterId into
> AddApplicationHomeSubClusterResponse.
> in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659],
> to handle better fail-over scenario the response needs SubclusterId as field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421593#comment-15421593
]
Hadoop QA commented on YARN-5519:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 2 new + 0 unchanged - 2 fixed = 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 51s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823744/YARN-5519-YARN-2915.v3.patch
|
| JIRA Issue | YARN-5519 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux cd2deaeaae16 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / b689f55 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12773/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12773/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12773/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add SubClusterId in AddApplicationHomeSubCl
[jira] [Updated] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ellen Hui updated YARN-5519: Attachment: YARN-5519-YARN-2915.v3.patch Pick the fix for FederationStateStoreBaseTest from YARN-3672 > Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover > > > Key: YARN-5519 > URL: https://issues.apache.org/jira/browse/YARN-5519 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Ellen Hui > Attachments: YARN-5519-YARN-2915.v1.patch, > YARN-5519-YARN-2915.v2.patch, YARN-5519-YARN-2915.v3.patch > > > This JIRA tracks the addition of SubClusterId into > AddApplicationHomeSubClusterResponse. > in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], > to handle better fail-over scenario the response needs SubclusterId as field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-4566:
-
Fix Version/s: (was: 2.9.0)
2.8.0
Thanks, [~bwtakacy]! I committed this to branch-2.8 as well.
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Fix For: 2.8.0
>
> Attachments: YARN-4566.1.patch
>
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5507) Inconsistent logging content and logging level for server nodemanager
[
https://issues.apache.org/jira/browse/YARN-5507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421518#comment-15421518
]
Vrushali C commented on YARN-5507:
--
+1 for merging as [~Naganarasimha] suggested.
[~chenfsd] please go ahead and merge the yarn jiras, I can put up a combined
patch then.
> Inconsistent logging content and logging level for server nodemanager
> -
>
> Key: YARN-5507
> URL: https://issues.apache.org/jira/browse/YARN-5507
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>Assignee: Vrushali C
> Labels: easyfix, easytest
> Attachments: YARN-5507.001.patch
>
>
> Similar to a fix for MAPREDUCE-2907, in file:
> hadoop-rel-release-2.7.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java
> In line 96, the logging text indicates this is a DEBUG level log, but the
> level is set to info.
> {code:borderStyle=solid}
> LOG.info("DEBUG: " + req + ":" + rsrcReqs.getContainer().getContainerId());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4916) TestNMProxy.tesNMProxyRPCRetry fails.
[
https://issues.apache.org/jira/browse/YARN-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-4916:
-
Fix Version/s: (was: 2.9.0)
2.8.0
Thanks, [~tibor.k...@gmail.com]! I committed this to branch-2.8 since
HADOOP-11212 was also fixed in 2.8.0.
> TestNMProxy.tesNMProxyRPCRetry fails.
> -
>
> Key: YARN-4916
> URL: https://issues.apache.org/jira/browse/YARN-4916
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: OS X 10.11 with Oracle JDK 1.7.0_79
> Windows Server 2012 with Oracle JDK 1.7.0_79
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: YARN-4916.01.patch, YARN-4916.02-WiP.patch
>
>
> The test ensures that java.net.SocketException is thrown from
> NMProxy.startContainers() without the RPC Request being retried.
> With Oracle JDK 1.7 on OS X & Windows BindException is thrown from
> startContainers().
> The testcase expects that SocketException is thrown - which is
> BindException's superclass.
> The exception type check is implemented using string compare and not
> reflection, therefore the thrown BindException is not accepted.
> {noformat}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.149 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy
> testNMProxyRPCRetry(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy)
> Time elapsed: 0.211 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:191)
> {noformat}
> Actual exception:
> {noformat}
> 2016-04-02 21:25:13,311 WARN [Thread-93] ipc.Client
> (Client.java:handleConnectionFailure(880)) - Failed to connect to server:
> 1234/0.0.4.210:0: retries get failed due to exceeded maximum allowed retries
> java.net.BindException: Can't assign requested address
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:634)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1413)
> at org.apache.hadoop.ipc.Client.call(Client.java:1328)
> at org.apache.hadoop.ipc.Client.call(Client.java:1306)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy10.startContainers(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5523) Yarn live log aggregation runs out of memory
[
https://issues.apache.org/jira/browse/YARN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421511#comment-15421511
]
Varun Saxena commented on YARN-5523:
Thanks [~xgong] for the patch. As such the patch looks fine.
A couple of comments :
* ClientResponse#getClientResponseStatus is deprecated.
ClientResponse#getStatusInfo can be used instead.
* If response status is not 200 OK, we print below now. I think response from
server can be printed here, as we were doing earlier. NMWebServices seems to
pass on the exception message too which will give more details.
{code}
out.println("Can not get any logs for the log file: " + logFile);
{code}
> Yarn live log aggregation runs out of memory
>
>
> Key: YARN-5523
> URL: https://issues.apache.org/jira/browse/YARN-5523
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Xuan Gong
> Attachments: YARN-5523.1.patch
>
>
> Fetching a 256MB log from container caused OOM on client
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
> 16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address:
> http://ctr-e20-1468887904486-0007-01-03.hwx.site:8188/ws/v1/timeline/
> 16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at
> ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:8050
> 16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History
> server at ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:10200
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_01 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_02 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_03 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_04 within the
> application: application_1470931023753_0001
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
> at java.lang.StringBuilder.append(StringBuilder.java:190)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
> at
> com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5523) Yarn live log aggregation runs out of memory
[
https://issues.apache.org/jira/browse/YARN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421508#comment-15421508
]
Hadoop QA commented on YARN-5523:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client generated 1
new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 2s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823730/YARN-5523.1.patch |
| JIRA Issue | YARN-5523 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 68ea68d9121f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2424911 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/12772/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12772/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12772/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12772/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
[jira] [Commented] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421506#comment-15421506 ] Subru Krishnan commented on YARN-5519: -- [~ellenfkh], I have fix for test case failure in my patch for YARN-3672. Can you include the fix here, thanks. > Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover > > > Key: YARN-5519 > URL: https://issues.apache.org/jira/browse/YARN-5519 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Ellen Hui > Attachments: YARN-5519-YARN-2915.v1.patch, > YARN-5519-YARN-2915.v2.patch > > > This JIRA tracks the addition of SubClusterId into > AddApplicationHomeSubClusterResponse. > in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], > to handle better fail-over scenario the response needs SubclusterId as field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421497#comment-15421497
]
Vrushali C commented on YARN-5524:
--
Hmm, I get your point about pulling huge logs. Alright, let me think over this
and see if I can add the "unrecognized option" check.
Also, fixing this means it would break existing behavior for folks who perhaps
have been running (unintentionally) with incorrect/extra options.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421492#comment-15421492
]
Hudson commented on YARN-5521:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10273 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10273/])
YARN-5521. Fix random failure of (varunsaxena: rev
24249115bff3162c4202387da5bdd8eba13e6961)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Fix For: 2.9.0
>
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421486#comment-15421486
]
Prasanth Jayachandran commented on YARN-5524:
-
In hive LLAP, we have long running daemons that generate lot of logs. If user
explicitly specifies a log file option then the user's intention is fetch the
specific logs and not all logs. Also it is easy to provide to wrong option or
some typo in options which should not end up pulling all the logs which could
be huge. I agree that we should fetch all logs if no options are provided. But
if explicit option is provided, it should at least validate the provided option
and say "Unrecognized option". I am pretty sure, if we mistype "yarn
application -kill" it's not going to kill all applications.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421472#comment-15421472
]
Vrushali C commented on YARN-5524:
--
Based on what I am seeing, I don't think this should throw an exception. This
was intentionally added as part of YARN-5363 by [~vinodkv]
In patch
https://issues.apache.org/jira/secure/attachment/12817883/YARN-5363-2016-07-13.1.txt
you can see that in LogsCLI# fetchAllLogFiles, the default is set to fetch all
logs if no option is specified. The fact that the option is spelled differently
does not imply the meaning.
And for what it's worth, looks like the option changed (correctly) from
"-logFiles" to "-log_files" in this patch, so perhaps your command was present
in history and you happened to run it.
I think this should be closed as "Not a problem". What do you think Prashanth?
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421465#comment-15421465
]
Vrushali C commented on YARN-5524:
--
Looking into this.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5523) Yarn live log aggregation runs out of memory
[
https://issues.apache.org/jira/browse/YARN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421459#comment-15421459
]
Xuan Gong commented on YARN-5523:
-
We should use stream output instead of directly calling println which can avoid
this oom issue.
The patch did not add any unit test cases given that we did not have any
functional unit tests for fetching logs from running container. But I did the
manual test to make sure the fix works.
1) Run sleep job
2) override YARN_CLIENT_OPTS and set a very low Xmx value
3) run yarn logs command (without the fix), and we could see the exact oom
exception as shown in the description
4) run yarn logs command (with the fix), and we could not see the oom exception
anymore.
> Yarn live log aggregation runs out of memory
>
>
> Key: YARN-5523
> URL: https://issues.apache.org/jira/browse/YARN-5523
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Xuan Gong
> Attachments: YARN-5523.1.patch
>
>
> Fetching a 256MB log from container caused OOM on client
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
> 16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address:
> http://ctr-e20-1468887904486-0007-01-03.hwx.site:8188/ws/v1/timeline/
> 16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at
> ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:8050
> 16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History
> server at ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:10200
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_01 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_02 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_03 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_04 within the
> application: application_1470931023753_0001
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
> at java.lang.StringBuilder.append(StringBuilder.java:190)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
> at
> com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C reassigned YARN-5524:
Assignee: Vrushali C
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5523) Yarn live log aggregation runs out of memory
[
https://issues.apache.org/jira/browse/YARN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-5523:
Attachment: YARN-5523.1.patch
> Yarn live log aggregation runs out of memory
>
>
> Key: YARN-5523
> URL: https://issues.apache.org/jira/browse/YARN-5523
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Xuan Gong
> Attachments: YARN-5523.1.patch
>
>
> Fetching a 256MB log from container caused OOM on client
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
> 16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address:
> http://ctr-e20-1468887904486-0007-01-03.hwx.site:8188/ws/v1/timeline/
> 16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at
> ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:8050
> 16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History
> server at ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:10200
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_01 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_02 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_03 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_04 within the
> application: application_1470931023753_0001
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
> at java.lang.StringBuilder.append(StringBuilder.java:190)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
> at
> com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421448#comment-15421448
]
Varun Saxena commented on YARN-5521:
Committed to trunk, branch-2.
Thanks [~sandflee] for your contribution and [~sunilg], [~bibinchundatt] for
reviews.
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Fix For: 2.9.0
>
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service
[
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421428#comment-15421428
]
Hadoop QA commented on YARN-679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 22 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 48s {color}
| {color:red} root generated 8 new + 709 unchanged - 0 fixed = 717 total (was
709) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-common-project/hadoop-common: The patch generated
43 new + 112 unchanged - 33 fixed = 155 total (was 145) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 73 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 32s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823719/YARN-679-011.patch |
| JIRA Issue | YARN-679 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 22c05489ee7f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9f29f42 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/12771/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12771/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/12771/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12771/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12771/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/1277
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421440#comment-15421440
]
Hadoop QA commented on YARN-5521:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 49s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 4s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823711/YARN-5521.01.patch |
| JIRA Issue | YARN-5521 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 278edabf28b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9f29f42 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12770/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12770/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capac
[jira] [Commented] (YARN-5375) invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures
[ https://issues.apache.org/jira/browse/YARN-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421434#comment-15421434 ] Varun Saxena commented on YARN-5375: [~sandflee], after YARN-5436, the patch here doesnt apply cleanly. Can you rebase it ? > invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures > -- > > Key: YARN-5375 > URL: https://issues.apache.org/jira/browse/YARN-5375 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5375.01.patch, YARN-5375.03.patch, > YARN-5375.04.patch, YARN-5375.05.patch > > > seen many test failures related to RMApp/RMAppattempt comes to some state but > some event are not processed in rm event queue or scheduler event queue, > cause test failure, seems we could implicitly invokes drainEvents(should also > drain sheduler event) in some mockRM method like waitForState -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5523) Yarn live log aggregation runs out of memory
[
https://issues.apache.org/jira/browse/YARN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong reassigned YARN-5523:
---
Assignee: Xuan Gong
> Yarn live log aggregation runs out of memory
>
>
> Key: YARN-5523
> URL: https://issues.apache.org/jira/browse/YARN-5523
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Xuan Gong
>
> Fetching a 256MB log from container caused OOM on client
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
> 16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address:
> http://ctr-e20-1468887904486-0007-01-03.hwx.site:8188/ws/v1/timeline/
> 16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at
> ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:8050
> 16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History
> server at ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:10200
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_01 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_02 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_03 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_04 within the
> application: application_1470931023753_0001
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
> at java.lang.StringBuilder.append(StringBuilder.java:190)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
> at
> com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1394) RM to inform AMs when a container completed due to NM going offline -planned or unplanned
[ https://issues.apache.org/jira/browse/YARN-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421385#comment-15421385 ] Steve Loughran commented on YARN-1394: -- to be honest, I don't see anybody really asking for this feature. It's related to YARN-3244, so yes, lets close as a wontfix. > RM to inform AMs when a container completed due to NM going offline -planned > or unplanned > - > > Key: YARN-1394 > URL: https://issues.apache.org/jira/browse/YARN-1394 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Steve Loughran >Assignee: Rohith Sharma K S > > YARN-914 proposes graceful decommission of an NM, and NMs already have the > right to go offline. > If AMs could be told that a container completed from an NM option -offline vs > decommission, the AM could use that in its future blacklisting and placement > policy. > This matters in long-lived services which may like to place new instances > where they were placed before, and track hosts failure rates -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-1394) RM to inform AMs when a container completed due to NM going offline -planned or unplanned
[ https://issues.apache.org/jira/browse/YARN-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-1394. -- Resolution: Won't Fix > RM to inform AMs when a container completed due to NM going offline -planned > or unplanned > - > > Key: YARN-1394 > URL: https://issues.apache.org/jira/browse/YARN-1394 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Steve Loughran >Assignee: Rohith Sharma K S > > YARN-914 proposes graceful decommission of an NM, and NMs already have the > right to go offline. > If AMs could be told that a container completed from an NM option -offline vs > decommission, the AM could use that in its future blacklisting and placement > policy. > This matters in long-lived services which may like to place new instances > where they were placed before, and track hosts failure rates -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
Prasanth Jayachandran created YARN-5524:
---
Summary: Yarn live log aggregation does not throw if command line
arg is wrong
Key: YARN-5524
URL: https://issues.apache.org/jira/browse/YARN-5524
Project: Hadoop YARN
Issue Type: Bug
Components: log-aggregation
Affects Versions: 2.9.0
Reporter: Prasanth Jayachandran
When we used wrong command line arg for specify log file pattern, yarn did not
throw any exception instead it pulled entire log onto the client.
{code}
[hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
{code}
NOTE: we are using -logFiles instead of -log_files
This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5523) Yarn live log aggregation runs out of memory
Prasanth Jayachandran created YARN-5523:
---
Summary: Yarn live log aggregation runs out of memory
Key: YARN-5523
URL: https://issues.apache.org/jira/browse/YARN-5523
Project: Hadoop YARN
Issue Type: Bug
Components: log-aggregation
Affects Versions: 2.9.0
Reporter: Prasanth Jayachandran
Fetching a 256MB log from container caused OOM on client
{code}
[hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address:
http://ctr-e20-1468887904486-0007-01-03.hwx.site:8188/ws/v1/timeline/
16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at
ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:8050
16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History
server at ctr-e20-1468887904486-0007-01-03.hwx.site/172.27.8.192:10200
Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for
the container: container_e04_1470931023753_0001_01_01 within the
application: application_1470931023753_0001
Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for
the container: container_e04_1470931023753_0001_01_02 within the
application: application_1470931023753_0001
Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for
the container: container_e04_1470931023753_0001_01_03 within the
application: application_1470931023753_0001
Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for
the container: container_e04_1470931023753_0001_01_04 within the
application: application_1470931023753_0001
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at
com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
at
com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
at
com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
at
com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
at
com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
at
org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
at
org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-679) add an entry point that can start any Yarn service
[ https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-679: Attachment: YARN-679-011.patch Patch 011; in sync with recent changes to GenericOptionsParser > add an entry point that can start any Yarn service > -- > > Key: YARN-679 > URL: https://issues.apache.org/jira/browse/YARN-679 > Project: Hadoop YARN > Issue Type: New Feature > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-679-001.patch, YARN-679-002.patch, > YARN-679-002.patch, YARN-679-003.patch, YARN-679-004.patch, > YARN-679-005.patch, YARN-679-006.patch, YARN-679-007.patch, > YARN-679-008.patch, YARN-679-009.patch, YARN-679-010.patch, > YARN-679-011.patch, org.apache.hadoop.servic...mon 3.0.0-SNAPSHOT API).pdf > > Time Spent: 72h > Remaining Estimate: 0h > > There's no need to write separate .main classes for every Yarn service, given > that the startup mechanism should be identical: create, init, start, wait for > stopped -with an interrupt handler to trigger a clean shutdown on a control-c > interrupt. > Provide one that takes any classname, and a list of config files/options -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421324#comment-15421324
]
Varun Saxena commented on YARN-5521:
LGTM. Will commit it pending Jenkins.
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421284#comment-15421284
]
Sunil G commented on YARN-5521:
---
Yes. Analysis makes sense to me.
I think its high time we need to make progress in YARN-5375, I will help in
reviewing the same. cc/[~rohithsharma]
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandflee updated YARN-5521:
---
Attachment: YARN-5521.01.patch
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Attachments: Failure.txt, YARN-5521.01.patch
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421202#comment-15421202
]
sandflee commented on YARN-5521:
thanks [~bibinchundatt], seems caused by app went to KILLED state, but
APP_ATTEMPT_REMOVED event not processed by scheduler dispatcher. this could
simple fixed by
{code}
rm.waitForState(app.getApplicationId(), RMAppState.KILLED);
rm.waitForAppRemovedFromScheduler(app.getApplicationId());
appsInRoot = scheduler.getAppsInQueue("root");
{code}
and YARN-5375 introduce a more general way , cc [~sunilg] [~rohithsharma]
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Attachments: Failure.txt
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandflee reassigned YARN-5521:
--
Assignee: sandflee
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: sandflee
> Attachments: Failure.txt
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5521) TestCapacityScheduler#testKillAllAppsInQueue fails randomly
[
https://issues.apache.org/jira/browse/YARN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-5521:
---
Attachment: Failure.txt
Was able to reproduce the same after multiple runs.
{noformat}
2016-08-15 19:08:40,755 INFO [main] resourcemanager.MockRM
(MockRM.java:waitForState(194)) - App State is : KILLED
2016-08-15 19:08:40,755 INFO [SchedulerEventDispatcher:Event Processor]
scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(136)) -
Application application_1471268320166_0001 requests cleared
2016-08-15 19:08:40,756 DEBUG [main] service.AbstractService
(AbstractService.java:enterState(452)) - Service: ResourceManager entered state
STOPPED
2016-08-15 19:08:40,755 DEBUG [AsyncDispatcher event handler]
resourcemanager.RMAppManager (RMAppManager.java:handle(494)) - RMAppManager
processing event for application_1471268320166_0001 of type APP_COMPLETED
2016-08-15 19:08:40,757 DEBUG [main] service.CompositeService
(CompositeService.java:serviceStop(129)) - ResourceManager: stopping services,
size=3
2016-08-15 19:08:40,756 INFO [SchedulerEventDispatcher:Event Processor]
capacity.LeafQueue (LeafQueue.java:removeApplicationAttempt(789)) - Application
removed - appId: application_1471268320166_0001 user: user_0 queue: a1
#user-pending-applications: 0 #user-active-applications: 0
#queue-pending-applications: 0 #queue-active-applications: 0
{noformat}
The queue metrics is getting updated after the app state is set to KILLED. So
during below check
{code}
appsInRoot = scheduler.getAppsInQueue("root");
assertTrue(appsInRoot.isEmpty());
{code}
Application in queue is still 1 .
> TestCapacityScheduler#testKillAllAppsInQueue fails randomly
> ---
>
> Key: YARN-5521
> URL: https://issues.apache.org/jira/browse/YARN-5521
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
> Attachments: Failure.txt
>
>
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.922 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testKillAllAppsInQueue(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.146 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testKillAllAppsInQueue(TestCapacityScheduler.java:2188)
> Results :
> Failed tests:
> TestCapacityScheduler.testKillAllAppsInQueue:2188 null
> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
[
https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420796#comment-15420796
]
sandflee commented on YARN-5479:
will do, thx
> FairScheduler: Scheduling performance improvement
> -
>
> Key: YARN-5479
> URL: https://issues.apache.org/jira/browse/YARN-5479
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for
> scheduling. As number of nodes grows, more events need to be processed in
> time in FairScheduler. Also, increased number of applications & queues slows
> down processing of each single event.
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds.
> This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of
> some nodes cannot get processed before next heartbeat. This brings
> inefficiency handling burst requests (i.e. newly submitted MapReduce
> application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single
> cluster is scaled to several K of nodes (even with {{assignmultiple}}
> enabled).
> This issue proposes to perform several optimization towards performance in
> FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped
> (or should this be called 'delayed sorting'?). we can either start another
> dedicated thread to do the sorting & updating, or actually perform sorting
> after current result have been used several times (say sort once in every 100
> calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at
> least 2 objects ({{ResourceImpl}} and its proto builder) is created each time
> (using 'immutable' apis). the overhead can be eliminated with a
> light-weighted implementation of Resource, which do not instantiate a builder
> until necessary, because most instances are used as intermediate result in
> scheduler instead of being exchanged via IPC. Also, {{createResource}} is
> using reflection, which can be replaced by a plain {{new}} (for scheduler
> usage only). furthermore, perhaps we could 'intern' resource to avoid
> allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to
> {{update}}, making root queue metrics eventual consistent (which may
> satisfies most of the needs). or introduce counters to {{getResourceUsage}}
> and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5514) Clarify DecommissionType.FORCEFUL comment
[
https://issues.apache.org/jira/browse/YARN-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5514:
---
Priority: Minor (was: Major)
> Clarify DecommissionType.FORCEFUL comment
> -
>
> Key: YARN-5514
> URL: https://issues.apache.org/jira/browse/YARN-5514
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Vrushali C
>Priority: Minor
> Labels: newbie
> Attachments: YARN-5514.001.patch, YARN-5514.002.patch
>
>
> The comment for
> {{org.apache.hadoop.yarn.api.records.DecommissionType.FORCEFUL}} is a little
> unclear. It says:
> {code}
> /** Forceful decommissioning of nodes which are already in progress **/
> {code}
> It's not exactly clear of what the nodes are in progress. It should say
> something like "Forceful decommissioning of nodes which are already in
> progress of decommissioning".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5514) Clarify DecommissionType.FORCEFUL comment
[
https://issues.apache.org/jira/browse/YARN-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5514:
---
Assignee: Vrushali C
> Clarify DecommissionType.FORCEFUL comment
> -
>
> Key: YARN-5514
> URL: https://issues.apache.org/jira/browse/YARN-5514
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Vrushali C
> Labels: newbie
> Attachments: YARN-5514.001.patch, YARN-5514.002.patch
>
>
> The comment for
> {{org.apache.hadoop.yarn.api.records.DecommissionType.FORCEFUL}} is a little
> unclear. It says:
> {code}
> /** Forceful decommissioning of nodes which are already in progress **/
> {code}
> It's not exactly clear of what the nodes are in progress. It should say
> something like "Forceful decommissioning of nodes which are already in
> progress of decommissioning".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5513) Move Java only tests from slider develop to yarn-native-services
[
https://issues.apache.org/jira/browse/YARN-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420690#comment-15420690
]
Hadoop QA commented on YARN-5513:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 37 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s {color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed.
{color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s {color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core
generated 1 new + 35 unchanged - 0 fixed = 36 total (was 35) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core:
The patch generated 292 new + 0 unchanged - 0 fixed = 292 total (was 0)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 18s
{color} | {color:red} The patch generated 8 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823660/YARN-5513-yarn-native-services.002.patch
|
| JIRA Issue | YARN-5513 |
| Optional Tests | asflicense findbugs xml compile javac javadoc
mvninstall mvnsite unit checkstyle |
| uname | Linux dac548e61338 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 1c391db |
| Default Java | 1.8.0_101 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/12769/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/12769/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-ya
[jira] [Updated] (YARN-5513) Move Java only tests from slider develop to yarn-native-services
[ https://issues.apache.org/jira/browse/YARN-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-5513: Attachment: YARN-5513-yarn-native-services.002.patch > Move Java only tests from slider develop to yarn-native-services > > > Key: YARN-5513 > URL: https://issues.apache.org/jira/browse/YARN-5513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Gour Saha > Attachments: YARN-5513-yarn-native-services.001.patch, > YARN-5513-yarn-native-services.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
