[jira] [Commented] (YARN-4023) Publish Application Priority to TimelineServer
[ https://issues.apache.org/jira/browse/YARN-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14669880#comment-14669880 ] Sunil G commented on YARN-4023: --- HI [~rohithsharma] Pre-findbugs warnings are not related with this patch. There is one item shown in WebPageUtils.java which is fine for now. As we display priority in FairScheduler page, we can update a new column value and it will be vanished. Checkstyle warnings are for already existing code. AppReport newInstance has more that 7 items as arguements. I feel it can be suppressed. As for tests, YARN-4034 will handle test failures for TestRMAdminService. other test cases are passing locally. Publish Application Priority to TimelineServer -- Key: YARN-4023 URL: https://issues.apache.org/jira/browse/YARN-4023 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-4023.patch, 0001-YARN-4023.patch, ApplicationPage.png, TimelineserverMainpage.png Publish Application priority details to Timeline Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3887) Support for changing Application priority during runtime
[ https://issues.apache.org/jira/browse/YARN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-3887: -- Attachment: 0006-YARN-3887.patch Thank you very much [~jianhe] and [~rohithsharma] for the comments. As discussed with Jian offline, we are keeping current implementation in the patch to avoid more locks within ordering policy. I have addressed the comment from Rohith by providing a null check within the policy. Thank you. Uploading a new version of patch. please help to check the same and share the comments. Support for changing Application priority during runtime Key: YARN-3887 URL: https://issues.apache.org/jira/browse/YARN-3887 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler, resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-3887.patch, 0002-YARN-3887.patch, 0003-YARN-3887.patch, 0004-YARN-3887.patch, 0005-YARN-3887.patch, 0006-YARN-3887.patch After YARN-2003, adding support to change priority of an application after submission. This ticket will handle the server side implementation for same. A new RMAppEvent will be created to handle this, and will be common for all schedulers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3250) Support admin cli interface in for Application Priority
[
https://issues.apache.org/jira/browse/YARN-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14669887#comment-14669887
]
Sunil G commented on YARN-3250:
---
HI [~rohithsharma]
Thank you for sharing the patch. Overall patch looks good.
Few comments:
1. NumberFormatException will be thrown from {{conf.getInt()}}. So
{{getClusterMaxPriority()}} can be updated by throwing this exception, and can
be given back to client side
2. Could we have a test case to see priority is updated from config. As we do
not have a getter to see this change from client side, cud we verify it in
server side itself.
Support admin cli interface in for Application Priority
---
Key: YARN-3250
URL: https://issues.apache.org/jira/browse/YARN-3250
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Rohith Sharma K S
Attachments: 0001-YARN-3250-V1.patch
Current Application Priority Manager supports only configuration via file.
To support runtime configurations for admin cli and REST, a common management
interface has to be added which can be shared with NodeLabelsManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3887) Support for changing Application priority during runtime
[
https://issues.apache.org/jira/browse/YARN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679087#comment-14679087
]
Hadoop QA commented on YARN-3887:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated 2
new checkstyle issues (total was 266, now 268). |
| {color:red}-1{color} | whitespace | 0m 7s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 53m 19s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 38s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749454/0006-YARN-3887.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8804/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8804/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8804/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8804/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8804/console |
This message was automatically generated.
Support for changing Application priority during runtime
Key: YARN-3887
URL: https://issues.apache.org/jira/browse/YARN-3887
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler, resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-3887.patch, 0002-YARN-3887.patch,
0003-YARN-3887.patch, 0004-YARN-3887.patch, 0005-YARN-3887.patch,
0006-YARN-3887.patch
After YARN-2003, adding support to change priority of an application after
submission. This ticket will handle the server side implementation for same.
A new RMAppEvent will be created to handle this, and will be common for all
schedulers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3232) Some application states are not necessarily exposed to users
[ https://issues.apache.org/jira/browse/YARN-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3232: --- Attachment: YARN-3232.01.patch Some application states are not necessarily exposed to users Key: YARN-3232 URL: https://issues.apache.org/jira/browse/YARN-3232 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Varun Saxena Attachments: YARN-3232.01.patch application NEW_SAVING and SUBMITTED states are not necessarily exposed to users as they mostly internal to the system, transient and not user-facing. We may deprecate these two states and remove them from the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[
https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14674220#comment-14674220
]
Sunil G commented on YARN-4014:
---
Hi [~rohithsharma]
Thank you for sharing the patch. As discussed in YARN-3250, I also feel {{yarn
application --appId Applicationid --setPriority value}} is fine.
Few small nits:
1. In ApplicationCLI, {{public static final String SET_PRIORITY =
setPriority;}}
Cud we use UPDATE_PRIORITY instead of SET_PRIORITY?
2. In future {{--appId}} can be used with other parameters also, correct? So
below option may not be suitable.
{code}
opts.addOption(APP_ID, true, Updates the priority of an application);
{code}
3. updateApplicationPriority can throw NumberFormatException.
4. ClientRMService.java has few commented code.
Support user cli interface in for Application Priority
--
Key: YARN-4014
URL: https://issues.apache.org/jira/browse/YARN-4014
Project: Hadoop YARN
Issue Type: Sub-task
Components: client, resourcemanager
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S
Attachments: 0001-YARN-4014-V1.patch
Track the changes for user-RM client protocol i.e ApplicationClientProtocol
changes and discussions in this jira.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3232) Some application states are not necessarily exposed to users
[ https://issues.apache.org/jira/browse/YARN-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3232: --- Attachment: (was: YARN-3232.01.patch) Some application states are not necessarily exposed to users Key: YARN-3232 URL: https://issues.apache.org/jira/browse/YARN-3232 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Varun Saxena application NEW_SAVING and SUBMITTED states are not necessarily exposed to users as they mostly internal to the system, transient and not user-facing. We may deprecate these two states and remove them from the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3232) Some application states are not necessarily exposed to users
[ https://issues.apache.org/jira/browse/YARN-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3232: --- Attachment: YARN-3232.01.patch Some application states are not necessarily exposed to users Key: YARN-3232 URL: https://issues.apache.org/jira/browse/YARN-3232 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Varun Saxena Attachments: YARN-3232.01.patch application NEW_SAVING and SUBMITTED states are not necessarily exposed to users as they mostly internal to the system, transient and not user-facing. We may deprecate these two states and remove them from the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3232) Some application states are not necessarily exposed to users
[
https://issues.apache.org/jira/browse/YARN-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679146#comment-14679146
]
Hadoop QA commented on YARN-3232:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 10s | Findbugs (version 3.0.0)
appears to be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 3m 8s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 7m 27s | The patch appears to introduce 7
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 44s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 106m 17s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 6m 53s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 0m 28s | Tests passed in
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests | 52m 30s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 0m 29s | Tests passed in
hadoop-yarn-server-sharedcachemanager. |
| | | 217m 16s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-common |
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749458/YARN-3232.01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-sharedcachemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/artifact/patchprocess/testrun_hadoop-yarn-server-sharedcachemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8805/console |
This message was automatically generated.
Some application states are not necessarily exposed to users
Key: YARN-3232
URL: https://issues.apache.org/jira/browse/YARN-3232
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Jian He
Assignee: Varun Saxena
Attachments: YARN-3232.01.patch
[jira] [Commented] (YARN-2273) NPE in ContinuousScheduling thread when we lose a node
[
https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679170#comment-14679170
]
StefanLee commented on YARN-2273:
-
Hi Wei Yan, thank you for taking this JIRA,but after this NPE,why containers
were not assigned? i have dumped RM`s jstack file and find no dead lock
etc.thank you.
NPE in ContinuousScheduling thread when we lose a node
--
Key: YARN-2273
URL: https://issues.apache.org/jira/browse/YARN-2273
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, resourcemanager
Affects Versions: 2.3.0, 2.4.1
Environment: cdh5.0.2 wheezy
Reporter: Andy Skelton
Assignee: Wei Yan
Fix For: 2.6.0
Attachments: YARN-2273-5.patch, YARN-2273-replayException.patch,
YARN-2273.patch, YARN-2273.patch, YARN-2273.patch, YARN-2273.patch
One DN experienced memory errors and entered a cycle of rebooting and
rejoining the cluster. After the second time the node went away, the RM
produced this:
{code}
2014-07-09 21:47:36,571 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application attempt appattempt_1404858438119_4352_01 released container
container_1404858438119_4352_01_04 on node: host:
node-A16-R09-19.hadoop.dfw.wordpress.com:8041 #containers=0
available=memory:8192, vCores:8 used=memory:0, vCores:0 with event: KILL
2014-07-09 21:47:36,571 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Removed node node-A16-R09-19.hadoop.dfw.wordpress.com:8041 cluster capacity:
memory:335872, vCores:328
2014-07-09 21:47:36,571 ERROR
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[ContinuousScheduling,5,main] threw an Exception.
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$NodeAvailableResourceComparator.compare(FairScheduler.java:1044)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$NodeAvailableResourceComparator.compare(FairScheduler.java:1040)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:329)
at java.util.TimSort.sort(TimSort.java:203)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousScheduling(FairScheduler.java:1012)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.access$600(FairScheduler.java:124)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$2.run(FairScheduler.java:1306)
at java.lang.Thread.run(Thread.java:744)
{code}
A few cycles later YARN was crippled. The RM was running and jobs could be
submitted but containers were not assigned and no progress was made.
Restarting the RM resolved it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4037) Hadoop - failed redirect for container
Gagan created YARN-4037: --- Summary: Hadoop - failed redirect for container Key: YARN-4037 URL: https://issues.apache.org/jira/browse/YARN-4037 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.1 Environment: Windows 7, Apache Hadoop 2.7.1 Reporter: Gagan I believe this issue has been addressed earlier in https://issues.apache.org/jira/browse/YARN-1473 though I am not sure because the description of the JIRA does not talk about the following message Failed while trying to construct the redirect url to the log server. Log Server url may not be configured java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all. Could some one look at the same and provide detail on the root cause and resolution ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
Varun Saxena created YARN-4036: -- Summary: Findbugs warnings in hadoop-yarn-server-common Key: YARN-4036 URL: https://issues.apache.org/jira/browse/YARN-4036 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1473) Exception from container-launch(Apache Hadoop 2.2.0)
[
https://issues.apache.org/jira/browse/YARN-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679132#comment-14679132
]
Gagan commented on YARN-1473:
-
Hi,
I am running hadoop 2.7.1 and facing a similar issue. The details are
Failed redirect for container_1439014904092_0001_02_01
Failed while trying to construct the redirect url to the log server. Log Server
url may not be configured
java.lang.Exception: Unknown container. Container either has not started or has
already completed or doesn't belong to this node at all.
Diagnostics: Exception from container-launch.
Container id: container_1439014904092_0001_02_01
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:
722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.la
unchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.C
ontainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.C
ontainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:744)
Shell output: 1 file(s) moved.
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
15/08/08 11:55:51 INFO mapreduce.Job: Counters: 0
I tried to understand the root cause but could not. Can some one help me out to
at least understand the reason of this issue and the resolution ?
Exception from container-launch(Apache Hadoop 2.2.0)
Key: YARN-1473
URL: https://issues.apache.org/jira/browse/YARN-1473
Project: Hadoop YARN
Issue Type: Bug
Environment: CentOS5.8 and Apache Hadoop 2.2.0
Reporter: Joy Xu
Attachments: yarn-site.xml
Hello all,
I have meet a exception from container-launch when I run the built-in
wordcount program .and the error messge as follow:
{code}
13/12/05 00:17:31 INFO mapreduce.Job: Job job_1386171829089_0003 failed with
state FAILED due to: Application application_1386171829089_0003 failed 2
times due to AM Container for appattempt_1386171829089_0003_02 exited
with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
.Failing this attempt.. Failing the application.
13/12/05 00:17:31 INFO mapreduce.Job: Counters: 0
{code}
Hope someone can Help. Thx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4036: --- Attachment: findbugs_report.html Attached the findbugs report (7 new warnings added) Findbugs warnings in hadoop-yarn-server-common -- Key: YARN-4036 URL: https://issues.apache.org/jira/browse/YARN-4036 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Attachments: findbugs_report.html Refer to https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3986) getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface instead
[
https://issues.apache.org/jira/browse/YARN-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679273#comment-14679273
]
Hadoop QA commented on YARN-3986:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 17s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 48s | The applied patch generated 2
new checkstyle issues (total was 52, now 53). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 52m 30s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 90m 45s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749476/YARN-3986.01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8807/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8807/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8807/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8807/console |
This message was automatically generated.
getTransferredContainers in AbstractYarnScheduler should be present in
YarnScheduler interface instead
--
Key: YARN-3986
URL: https://issues.apache.org/jira/browse/YARN-3986
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.7.0
Reporter: Varun Saxena
Assignee: Varun Saxena
Attachments: YARN-3986.01.patch
Currently getTransferredContainers is present in {{AbstractYarnScheduler}}.
*But in ApplicationMasterService, while registering AM, we are calling this
method by typecasting it to AbstractYarnScheduler, which is incorrect.*
This method should be moved to YarnScheduler.
Because if a custom scheduler is to be added, it will implement
YarnScheduler, not AbstractYarnScheduler.
As ApplicationMasterService is calling getTransferredContainers by
typecasting it to AbstractYarnScheduler, it is imposing an indirect
dependency on AbstractYarnScheduler for any pluggable custom scheduler.
We can move the method to YarnScheduler and leave the definition in
AbstractYarnScheduler as it is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3232) Some application states are not necessarily exposed to users
[ https://issues.apache.org/jira/browse/YARN-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3232: --- Attachment: YARN-3232.02.patch Some application states are not necessarily exposed to users Key: YARN-3232 URL: https://issues.apache.org/jira/browse/YARN-3232 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Jian He Assignee: Varun Saxena Attachments: YARN-3232.01.patch, YARN-3232.02.patch application NEW_SAVING and SUBMITTED states are not necessarily exposed to users as they mostly internal to the system, transient and not user-facing. We may deprecate these two states and remove them from the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4037) Hadoop - failed redirect for container
[
https://issues.apache.org/jira/browse/YARN-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679222#comment-14679222
]
Xuan Gong commented on YARN-4037:
-
[~gagan.ban...@gmail.com]
Have you configured the
{code}
nameyarn.log.server.url/name
{code}
in the yarn-site.xml for container log redirection for the case you want to
check the container log when the app finished ?
Hadoop - failed redirect for container
--
Key: YARN-4037
URL: https://issues.apache.org/jira/browse/YARN-4037
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.7.1
Environment: Windows 7, Apache Hadoop 2.7.1
Reporter: Gagan
I believe this issue has been addressed earlier in
https://issues.apache.org/jira/browse/YARN-1473 though I am not sure because
the description of the JIRA does not talk about the following message
Failed while trying to construct the redirect url to the log server. Log
Server url may not be configured
java.lang.Exception: Unknown container. Container either has not started or
has already completed or doesn't belong to this node at all.
Could some one look at the same and provide detail on the root cause and
resolution ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3986) getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface instead
[
https://issues.apache.org/jira/browse/YARN-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-3986:
---
Attachment: YARN-3986.01.patch
getTransferredContainers in AbstractYarnScheduler should be present in
YarnScheduler interface instead
--
Key: YARN-3986
URL: https://issues.apache.org/jira/browse/YARN-3986
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.7.0
Reporter: Varun Saxena
Assignee: Varun Saxena
Attachments: YARN-3986.01.patch
Currently getTransferredContainers is present in {{AbstractYarnScheduler}}.
*But in ApplicationMasterService, while registering AM, we are calling this
method by typecasting it to AbstractYarnScheduler, which is incorrect.*
This method should be moved to YarnScheduler.
Because if a custom scheduler is to be added, it will implement
YarnScheduler, not AbstractYarnScheduler.
As ApplicationMasterService is calling getTransferredContainers by
typecasting it to AbstractYarnScheduler, it is imposing an indirect
dependency on AbstractYarnScheduler for any pluggable custom scheduler.
We can move the method to YarnScheduler and leave the definition in
AbstractYarnScheduler as it is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4036: --- Attachment: (was: findbugs_report.html) Findbugs warnings in hadoop-yarn-server-common -- Key: YARN-4036 URL: https://issues.apache.org/jira/browse/YARN-4036 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Refer to https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4014) Support user cli interface in for Application Priority
[ https://issues.apache.org/jira/browse/YARN-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679367#comment-14679367 ] Allen Wittenauer commented on YARN-4014: Could we *please* stop adding camel case arguments to shell commands? Support user cli interface in for Application Priority -- Key: YARN-4014 URL: https://issues.apache.org/jira/browse/YARN-4014 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S Attachments: 0001-YARN-4014-V1.patch Track the changes for user-RM client protocol i.e ApplicationClientProtocol changes and discussions in this jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.2.b2.patch Attaching patch WebAppProxyServlet should not redirect to RM page if AHS is enabled --- Key: YARN-3975 URL: https://issues.apache.org/jira/browse/YARN-3975 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.1 Reporter: Mit Desai Assignee: Mit Desai Attachments: YARN-3975.2.b2.patch WebAppProxyServlet should be updated to handle the case when the appreport doesn't have a tracking URL and the Application History Server is eanbled. As we would have already tried the RM and got the ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679389#comment-14679389
]
Hadoop QA commented on YARN-3975:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 14s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 39s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 42s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 36s | The applied patch generated 2
new checkstyle issues (total was 16, now 18). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 52s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 54s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-server-web-proxy. |
| | | 46m 29s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-common |
| Failed unit tests | hadoop.yarn.util.TestRackResolver |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749490/YARN-3975.2.b2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-web-proxy test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8808/console |
This message was automatically generated.
WebAppProxyServlet should not redirect to RM page if AHS is enabled
---
Key: YARN-3975
URL: https://issues.apache.org/jira/browse/YARN-3975
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Mit Desai
Assignee: Mit Desai
Attachments: YARN-3975.2.b2.patch
WebAppProxyServlet should be updated to handle the case when the appreport
doesn't have a tracking URL and the Application History Server is eanbled.
As we would have already tried the RM and got the
ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3232) Some application states are not necessarily exposed to users
[
https://issues.apache.org/jira/browse/YARN-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679294#comment-14679294
]
Hadoop QA commented on YARN-3232:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 22m 10s | Findbugs (version 3.0.0)
appears to be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 57s | Site still builds. |
| {color:green}+1{color} | checkstyle | 3m 25s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 7m 32s | The patch appears to introduce 7
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:red}-1{color} | mapreduce tests | 123m 32s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | yarn tests | 0m 28s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 6m 58s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 0m 30s | Tests passed in
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests | 52m 51s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests | 0m 30s | Tests passed in
hadoop-yarn-server-sharedcachemanager. |
| | | 241m 33s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-common |
| Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| Timed out tests |
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749469/YARN-3232.02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 8f73bdd |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-sharedcachemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/artifact/patchprocess/testrun_hadoop-yarn-server-sharedcachemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8806/console |
This message was automatically generated.
Some application states are not necessarily exposed to users
Key: YARN-3232
URL: https://issues.apache.org/jira/browse/YARN-3232
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Jian He
Assignee: Varun Saxena
Attachments: YARN-3232.01.patch,
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679355#comment-14679355 ] Ben Podgursky commented on YARN-2154: - Hi, I've been watching this ticket since this has been a big problem on our cluster.Karthik, your last comment concerned me because it seems like it will violate one of the things FairScheduler is supposed to promise: Unlike the default Hadoop scheduler, which forms a queue of jobs, this lets short jobs finish in reasonable time while not starving long jobs.' and When other jobs are submitted, tasks slots that free up are assigned to the new jobs, so that each job gets roughly the same amount of CPU time. If a job can't even get an AM, it surely won't be able to get an equitable portion of slots for Maps and Reduces -- being able to get small jobs through quickly even when the pool has huge jobs running is the reason we use FairScheduler. Sorry if I'm misunderstanding that comment. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4037) Hadoop - failed redirect for container
[ https://issues.apache.org/jira/browse/YARN-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679515#comment-14679515 ] Gagan commented on YARN-4037: - Hi Xuan, Yes I forgot to mention it, I did try to both in the yarn-site and mapred but couldn't get success either. I will retry with yarn-site. Could you tell me is the following value correct for nameyarn.log.server.url/name valuelocalhost:jobhistory port/value I think I might have put in the incorrect port earlier. What would be the correct port to put in and where to get it from ? Also when I googled about this error, one of the links mentioned the creation of stderr file, where is it created ? Hadoop - failed redirect for container -- Key: YARN-4037 URL: https://issues.apache.org/jira/browse/YARN-4037 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.1 Environment: Windows 7, Apache Hadoop 2.7.1 Reporter: Gagan I believe this issue has been addressed earlier in https://issues.apache.org/jira/browse/YARN-1473 though I am not sure because the description of the JIRA does not talk about the following message Failed while trying to construct the redirect url to the log server. Log Server url may not be configured java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all. Could some one look at the same and provide detail on the root cause and resolution ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3999) Add a timeout when drain the dispatcher
[ https://issues.apache.org/jira/browse/YARN-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-3999: -- Attachment: YARN-3999.2.patch Add a timeout when drain the dispatcher --- Key: YARN-3999 URL: https://issues.apache.org/jira/browse/YARN-3999 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-3999.1.patch, YARN-3999.2.patch, YARN-3999.2.patch, YARN-3999.patch, YARN-3999.patch If external systems like ATS, or ZK becomes very slow, draining all the events take a lot of time. If this time becomes larger than 10 mins, all applications will expire. We can add a timeout and stop the dispatcher even if not all events are drained. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4024) YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat
[ https://issues.apache.org/jira/browse/YARN-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679444#comment-14679444 ] Hong Zhiguo commented on YARN-4024: --- We've did this one year ago in our 5k+ cluster. Can I take this issue? YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat -- Key: YARN-4024 URL: https://issues.apache.org/jira/browse/YARN-4024 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Currently, YARN RM NodesListManager will resolve IP address every time when node doing heartbeat. When DNS server becomes slow, NM heartbeat will be blocked and cannot make progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4024) YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat
[ https://issues.apache.org/jira/browse/YARN-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4024: - Assignee: Hong Zhiguo YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat -- Key: YARN-4024 URL: https://issues.apache.org/jira/browse/YARN-4024 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Hong Zhiguo Currently, YARN RM NodesListManager will resolve IP address every time when node doing heartbeat. When DNS server becomes slow, NM heartbeat will be blocked and cannot make progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4024) YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat
[ https://issues.apache.org/jira/browse/YARN-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679471#comment-14679471 ] Wangda Tan commented on YARN-4024: -- [~zhiguohong], sure, please go ahead! YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat -- Key: YARN-4024 URL: https://issues.apache.org/jira/browse/YARN-4024 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Currently, YARN RM NodesListManager will resolve IP address every time when node doing heartbeat. When DNS server becomes slow, NM heartbeat will be blocked and cannot make progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679538#comment-14679538 ] Rohith Sharma K S commented on YARN-4036: - I don't see any findbugs warnings listed in the report. It is displaying as zero. [~varun_saxena] are you able to see it? It might be problem with report generation too!! Findbugs warnings in hadoop-yarn-server-common -- Key: YARN-4036 URL: https://issues.apache.org/jira/browse/YARN-4036 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Refer to https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679602#comment-14679602 ] Varun Saxena commented on YARN-4036: [~rohithsharma], you will get the findbugs warnings if you run it locally. There is some problem in QA report which leads to findbugs HTML report showing 0 warnings despite having errors(and QA report showing -1 for it). Raised issue for that as well Findbugs warnings in hadoop-yarn-server-common -- Key: YARN-4036 URL: https://issues.apache.org/jira/browse/YARN-4036 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Refer to https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4036) Findbugs warnings in hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4036: --- Description: Refer to https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 Findbugs warnings in hadoop-yarn-server-common -- Key: YARN-4036 URL: https://issues.apache.org/jira/browse/YARN-4036 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Refer to https://issues.apache.org/jira/browse/YARN-3232?focusedCommentId=14679146page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14679146 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
