[jira] [Commented] (YARN-941) RM Should have a way to update the tokens it has for a running application
[ https://issues.apache.org/jira/browse/YARN-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13815946#comment-13815946 ] Steve Loughran commented on YARN-941: - Thinking some more, if the AM just terminated itself with a special error code when tokens stopped working, then when it exits the RM could restart it and not file this as a failure -which would push back most of the token expiry logic into the AM -the RM wouldn't need to see when tokens had expired on running containers and restart them, just wait for container stopped events as usual and react slightly differently to the exit codes. This strategy would also give the AM the opportunity do cleaner shutdowns, send out notifications &c. If there was a way to determine the expiry time in advance then it could even display its remaining life to management tools/web UIs > RM Should have a way to update the tokens it has for a running application > -- > > Key: YARN-941 > URL: https://issues.apache.org/jira/browse/YARN-941 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Joseph Evans > > When an application is submitted to the RM it includes with it a set of > tokens that the RM will renew on behalf of the application, that will be > passed to the AM when the application is launched, and will be used when > launching the application to access HDFS to download files on behalf of the > application. > For long lived applications/services these tokens can expire, and then the > tokens that the AM has will be invalid, and the tokens that the RM had will > also not work to launch a new AM. > We need to provide an API that will allow the RM to replace the current > tokens for this application with a new set. To avoid any real race issues, I > think this API should be something that the AM calls, so that the client can > connect to the AM with a new set of tokens it got using kerberos, then the AM > can inform the RM of the new set of tokens and quickly update its tokens > internally to use these new ones. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-80) Support delay scheduling for node locality in MR2's capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13815950#comment-13815950 ] Mck SembWever commented on YARN-80: --- How can one debug this process? It was easy before with just `grep "Choosing" hadoop-xxx-jobtracker.log`. I can't find any similar information in YARN log files. Background: I just upgraded to YARN (hadoop-2.2.0). And despite setting yarn.scheduler.capacity.node-locality-delay=3 in capacity-scheduler.xml data-locality is poor. (It was 100% with hadoop-0.22 and fair-scheduler). > Support delay scheduling for node locality in MR2's capacity scheduler > -- > > Key: YARN-80 > URL: https://issues.apache.org/jira/browse/YARN-80 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Reporter: Todd Lipcon >Assignee: Arun C Murthy > Fix For: 2.0.2-alpha, 0.23.6 > > Attachments: YARN-80.patch, YARN-80.patch > > > The capacity scheduler in MR2 doesn't support delay scheduling for achieving > node-level locality. So, jobs exhibit poor data locality even if they have > good rack locality. Especially on clusters where disk throughput is much > better than network capacity, this hurts overall job performance. We should > optionally support node-level delay scheduling heuristics similar to what the > fair scheduler implements in MR1. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-647) historyServer can't show container's log when aggregation is not enabled
[ https://issues.apache.org/jira/browse/YARN-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Liang updated YARN-647: - Assignee: shenhong > historyServer can't show container's log when aggregation is not enabled > > > Key: YARN-647 > URL: https://issues.apache.org/jira/browse/YARN-647 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.7, 2.0.4-alpha > Environment: yarn.log-aggregation-enable=false , HistoryServer will > show like this: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 >Reporter: shenhong >Assignee: shenhong > Fix For: 2.2.1 > > Attachments: yarn-647.patch > > > When yarn.log-aggregation-enable is seted to false, after a MR_App complete, > we can't view the container's log from the HistoryServer, it shows message > like: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 > Since we don't want to aggregate the container's log, because it will be a > pressure to namenode. but sometimes we also want to take a look at > container's log. > Should we show the container's log across HistoryServer even if > yarn.log-aggregation-enable is seted to false. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-647) historyServer can't show container's log when aggregation is not enabled
[ https://issues.apache.org/jira/browse/YARN-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Liang updated YARN-647: - Fix Version/s: 2.2.1 > historyServer can't show container's log when aggregation is not enabled > > > Key: YARN-647 > URL: https://issues.apache.org/jira/browse/YARN-647 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.7, 2.0.4-alpha > Environment: yarn.log-aggregation-enable=false , HistoryServer will > show like this: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 >Reporter: shenhong > Fix For: 2.2.1 > > Attachments: yarn-647.patch > > > When yarn.log-aggregation-enable is seted to false, after a MR_App complete, > we can't view the container's log from the HistoryServer, it shows message > like: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 > Since we don't want to aggregate the container's log, because it will be a > pressure to namenode. but sometimes we also want to take a look at > container's log. > Should we show the container's log across HistoryServer even if > yarn.log-aggregation-enable is seted to false. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-647) historyServer can't show container's log when aggregation is not enabled
[ https://issues.apache.org/jira/browse/YARN-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Liang updated YARN-647: - Affects Version/s: 2.2.0 > historyServer can't show container's log when aggregation is not enabled > > > Key: YARN-647 > URL: https://issues.apache.org/jira/browse/YARN-647 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 0.23.7, 2.0.4-alpha, 2.2.0 > Environment: yarn.log-aggregation-enable=false , HistoryServer will > show like this: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 >Reporter: shenhong >Assignee: shenhong > Attachments: yarn-647.patch > > > When yarn.log-aggregation-enable is seted to false, after a MR_App complete, > we can't view the container's log from the HistoryServer, it shows message > like: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 > Since we don't want to aggregate the container's log, because it will be a > pressure to namenode. but sometimes we also want to take a look at > container's log. > Should we show the container's log across HistoryServer even if > yarn.log-aggregation-enable is seted to false. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-647) historyServer can't show container's log when aggregation is not enabled
[ https://issues.apache.org/jira/browse/YARN-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Liang updated YARN-647: - Component/s: (was: documentation) > historyServer can't show container's log when aggregation is not enabled > > > Key: YARN-647 > URL: https://issues.apache.org/jira/browse/YARN-647 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 0.23.7, 2.0.4-alpha, 2.2.0 > Environment: yarn.log-aggregation-enable=false , HistoryServer will > show like this: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 >Reporter: shenhong >Assignee: shenhong > Attachments: yarn-647.patch > > > When yarn.log-aggregation-enable is seted to false, after a MR_App complete, > we can't view the container's log from the HistoryServer, it shows message > like: > Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669 > Since we don't want to aggregate the container's log, because it will be a > pressure to namenode. but sometimes we also want to take a look at > container's log. > Should we show the container's log across HistoryServer even if > yarn.log-aggregation-enable is seted to false. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-647) historyServer can't show container's log when aggregation is not enabled
[
https://issues.apache.org/jira/browse/YARN-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816005#comment-13816005
]
Hadoop QA commented on YARN-647:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581885/yarn-647.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2390//console
This message is automatically generated.
> historyServer can't show container's log when aggregation is not enabled
>
>
> Key: YARN-647
> URL: https://issues.apache.org/jira/browse/YARN-647
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 0.23.7, 2.0.4-alpha, 2.2.0
> Environment: yarn.log-aggregation-enable=false , HistoryServer will
> show like this:
> Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
>Reporter: shenhong
>Assignee: shenhong
> Attachments: yarn-647.patch
>
>
> When yarn.log-aggregation-enable is seted to false, after a MR_App complete,
> we can't view the container's log from the HistoryServer, it shows message
> like:
> Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
> Since we don't want to aggregate the container's log, because it will be a
> pressure to namenode. but sometimes we also want to take a look at
> container's log.
> Should we show the container's log across HistoryServer even if
> yarn.log-aggregation-enable is seted to false.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-647) historyServer can't show container's log when aggregation is not enabled
[
https://issues.apache.org/jira/browse/YARN-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816216#comment-13816216
]
Zhijie Shen commented on YARN-647:
--
{code}
+if (!aggregation) {
+ logsLink = join(HttpConfig.getSchemePrefix(), nodeHttpAddr,
+ "/node", "/containerlogs/",
+ containerIdString, "/", app.getJob().getUserName());
{code}
I'm afraid this link will not work as well. When a container is stopped, NM
local logs will be deleted and not be accessible via webUI or service again.
However, I'm afraid "Try the nodemanager at ..." is a bit misleading as well,
which makes users think that the logs are available via NM webUI. The fact is
that for debugging purpose, we can config
"yarn.nodemanager.delete.debug-delay-sec" to delay the deletion of NM local
logs, but it's not user-oriented, and anyway, they're not accessible via web.
Maybe we want to remove misleading words here?
> historyServer can't show container's log when aggregation is not enabled
>
>
> Key: YARN-647
> URL: https://issues.apache.org/jira/browse/YARN-647
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 0.23.7, 2.0.4-alpha, 2.2.0
> Environment: yarn.log-aggregation-enable=false , HistoryServer will
> show like this:
> Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
>Reporter: shenhong
>Assignee: shenhong
> Attachments: yarn-647.patch
>
>
> When yarn.log-aggregation-enable is seted to false, after a MR_App complete,
> we can't view the container's log from the HistoryServer, it shows message
> like:
> Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
> Since we don't want to aggregate the container's log, because it will be a
> pressure to namenode. but sometimes we also want to take a look at
> container's log.
> Should we show the container's log across HistoryServer even if
> yarn.log-aggregation-enable is seted to false.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-90) NodeManager should identify failed disks becoming good back again
[
https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816232#comment-13816232
]
Ravi Prakash commented on YARN-90:
--
Thanks for updating the patch Song! With almost the same changes as nigel, I
was able to get the originally invalid directories to be used again. So the
src/main code looks good to me. The one nit I had was that
{code} } catch (IOException e2) {
Assert.fail("should not throw an exception");
Shell.execCommand(Shell.getSetPermissionCommand("755", false,
testDir.getAbsolutePath()));
throw e2;
} {code}, {code} catch (InterruptedException e1) {
} {code} , {code} } catch (IOException e2) {
Assert.fail("should not throw an exception");
throw e2;
} {code} and {code} } catch (IOException e) {
Assert.fail("Service should have thrown an exception while closing");
throw e;
} {code} can simply be removed.
Other than that, the patch looks good to me. +1. Thanks a lot Nigel and Song!
> NodeManager should identify failed disks becoming good back again
> -
>
> Key: YARN-90
> URL: https://issues.apache.org/jira/browse/YARN-90
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Ravi Gummadi
> Attachments: YARN-90.1.patch, YARN-90.patch, YARN-90.patch,
> YARN-90.patch
>
>
> MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes
> down, it is marked as failed forever. To reuse that disk (after it becomes
> good), NodeManager needs restart. This JIRA is to improve NodeManager to
> reuse good disks(which could be bad some time back).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-584) In fair scheduler web UI, queues unexpand on refresh
[ https://issues.apache.org/jira/browse/YARN-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshit Daga updated YARN-584: -- Attachment: YARN-584-branch-2.2.0.patch A patch to fix this issue. > In fair scheduler web UI, queues unexpand on refresh > > > Key: YARN-584 > URL: https://issues.apache.org/jira/browse/YARN-584 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza > Labels: newbie > Attachments: YARN-584-branch-2.2.0.patch > > > In the fair scheduler web UI, you can expand queue information. Refreshing > the page causes the expansions to go away, which is annoying for someone who > wants to monitor the scheduler page and needs to reopen all the queues they > care about each time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-584) In fair scheduler web UI, queues unexpand on refresh
[
https://issues.apache.org/jira/browse/YARN-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816444#comment-13816444
]
Sandy Ryza commented on YARN-584:
-
Thanks the picking this up, Harshit. It seems like it would be difficult to
write unit tests for this, so I'm ok with not including them. Can you mention
the steps you've taken to manually test the patch?
As the added code is very similar for both schedulers, are we able to share it
in a central location? We could create a SchedulerPageUtil.
Can we change the names of addRemoveQueueToQuery and the other function names
to make it clear that they're related to expand / unexpand?
A few stylistic comments:
* Lines should be broken up so that they fit within close to 80 characters.
* There should be a space between "if" and "for" and the corresponding open
parentheses.
* There should be spaces between the operators and operands in conditions, e.g
(a == b), not (a==b)
* There should be spaces after the semicolons in for loops.
* There should be spaces in between "else" and the surrounding curly braces.
* Closing curly braces should line up vertically with the beginning of the
statement like so:
{code}
if (condition) {
dostuff;
}
not
if (condition) {
dostuff; }
{code}
> In fair scheduler web UI, queues unexpand on refresh
>
>
> Key: YARN-584
> URL: https://issues.apache.org/jira/browse/YARN-584
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
> Labels: newbie
> Attachments: YARN-584-branch-2.2.0.patch
>
>
> In the fair scheduler web UI, you can expand queue information. Refreshing
> the page causes the expansions to go away, which is annoying for someone who
> wants to monitor the scheduler page and needs to reopen all the queues they
> care about each time.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1387) RMWebServices should use ClientRMService for filtering applications
[ https://issues.apache.org/jira/browse/YARN-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1387: --- Description: YARN's REST API allows filtering applications, this should be moved to ClientRMService to allow Java API also support the same functionality. (was: YARN's REST API allows filtering applications, but the Java API doesn't. Ideally, the Java API should implement this and the REST implementation should reuse it.) > RMWebServices should use ClientRMService for filtering applications > --- > > Key: YARN-1387 > URL: https://issues.apache.org/jira/browse/YARN-1387 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN's REST API allows filtering applications, this should be moved to > ClientRMService to allow Java API also support the same functionality. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1387) RMWebServices should use ClientRMService for filtering applications
[ https://issues.apache.org/jira/browse/YARN-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1387: --- Summary: RMWebServices should use ClientRMService for filtering applications (was: Add Java API to filter RM applications) > RMWebServices should use ClientRMService for filtering applications > --- > > Key: YARN-1387 > URL: https://issues.apache.org/jira/browse/YARN-1387 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN's REST API allows filtering applications, but the Java API doesn't. > Ideally, the Java API should implement this and the REST implementation > should reuse it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1387) RMWebServices should use ClientRMService for filtering applications
[ https://issues.apache.org/jira/browse/YARN-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1387: --- Attachment: yarn-1387-1.patch Here is a first-cut implementation. ClientRMService implements all the filtering and RMWebServices directly uses it. TestRMWebServices,TestRMWebServicesApps pass - so, I think this doesn't introduce any regressions as such. Any early feedback on the overall approach would greatly help. TODO: Manual testing on a pseudo-dist cluster. > RMWebServices should use ClientRMService for filtering applications > --- > > Key: YARN-1387 > URL: https://issues.apache.org/jira/browse/YARN-1387 > Project: Hadoop YARN > Issue Type: Improvement > Components: api >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-1387-1.patch > > > YARN's REST API allows filtering applications, this should be moved to > ClientRMService to allow Java API also support the same functionality. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-584) In fair scheduler web UI, queues unexpand on refresh
[
https://issues.apache.org/jira/browse/YARN-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816757#comment-13816757
]
Hadoop QA commented on YARN-584:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612691/YARN-584-branch-2.2.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2392//console
This message is automatically generated.
> In fair scheduler web UI, queues unexpand on refresh
>
>
> Key: YARN-584
> URL: https://issues.apache.org/jira/browse/YARN-584
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
> Labels: newbie
> Attachments: YARN-584-branch-2.2.0.patch
>
>
> In the fair scheduler web UI, you can expand queue information. Refreshing
> the page causes the expansions to go away, which is annoying for someone who
> wants to monitor the scheduler page and needs to reopen all the queues they
> care about each time.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp reassigned YARN-691: Assignee: Daryn Sharp > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp >Assignee: Daryn Sharp > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1387) RMWebServices should use ClientRMService for filtering applications
[
https://issues.apache.org/jira/browse/YARN-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816789#comment-13816789
]
Hadoop QA commented on YARN-1387:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612726/yarn-1387-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2391//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2391//console
This message is automatically generated.
> RMWebServices should use ClientRMService for filtering applications
> ---
>
> Key: YARN-1387
> URL: https://issues.apache.org/jira/browse/YARN-1387
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 2.2.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-1387-1.patch
>
>
> YARN's REST API allows filtering applications, this should be moved to
> ClientRMService to allow Java API also support the same functionality.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated YARN-691: - Assignee: (was: Daryn Sharp) > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816792#comment-13816792 ] Chen He commented on YARN-691: -- Please assign to me ! > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816799#comment-13816799 ] Chen He commented on YARN-691: -- I think JIRA has a bug for this issue. I can assign myself Mapreduce-5052 but can not assign myself this task. I also asked some committer to help, still the same problem. Looks like this is a BUG for JIRA!!! > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1376) NM need to notify the log aggregation status to RM through Node heartbeat
[ https://issues.apache.org/jira/browse/YARN-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1376: Attachment: YARN-1376.2.patch > NM need to notify the log aggregation status to RM through Node heartbeat > - > > Key: YARN-1376 > URL: https://issues.apache.org/jira/browse/YARN-1376 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1376.1.patch, YARN-1376.2.patch > > > Expose a client API to allow clients to figure if log aggregation is > complete. The ticket is used to track the changes on NM side -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1242) AHS start as independent process
[
https://issues.apache.org/jira/browse/YARN-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816816#comment-13816816
]
Zhijie Shen commented on YARN-1242:
---
1. Please fix yarn.cmd as well, which is for windows.
2. As we decide AHS to be independent. These params should be those of RM,
right?
{code}
+ YARN_OPTS="$YARN_OPTS $YARN_RESOURCEMANAGER_OPTS"
+ if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then
+JAVA_HEAP_MAX="-Xmx""$YARN_RESOURCEMANAGER_HEAPSIZE""m"
+ fi
{code}
> AHS start as independent process
>
>
> Key: YARN-1242
> URL: https://issues.apache.org/jira/browse/YARN-1242
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: YARN-1242-1.patch
>
>
> Maybe we should include AHS classes as well (for developer usage) in yarn and
> yarn.cmd
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1242) AHS start as independent process
[ https://issues.apache.org/jira/browse/YARN-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1242: -- Description: Add the command in yarn and yarn.cmd to start and stop AHS (was: Maybe we should include AHS classes as well (for developer usage) in yarn and yarn.cmd) > AHS start as independent process > > > Key: YARN-1242 > URL: https://issues.apache.org/jira/browse/YARN-1242 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: YARN-1242-1.patch > > > Add the command in yarn and yarn.cmd to start and stop AHS -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1279) Expose a client API to allow clients to figure if log aggregation is complete
[ https://issues.apache.org/jira/browse/YARN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816833#comment-13816833 ] Xuan Gong commented on YARN-1279: - bq. Does it make sense to send an event to RMApp to process the app log status, instead of explicitly creating an update api of RMApp? We can do that. Add a new RMAppEvent (RMAppLogAggregationStatusUpdateEvent) in FINISHED state and KILLED state bq. why is it possible for a single node to first get log aggregation succeeded then Failed ? Actually, I spend more time to think about this question. I made some changes in NM side. The ApplicationLogAggregationStatus will be set into NMContext when the applicationImpl receives the APPLICATION_LOG_HANDLING_FINISHED event. In that case, we can make sure that NM only send the logAggregationStatus out if the RMApp is finished/killed. This also makes sure that we will not receive two different logAggregationStatus from the same NM. bq. I think we can have separate maps, one for completed succeeded node aggregation, the other for failed node aggregation. Then we don't need two more counters for counting succeeded or failed nodes and those increment/decrement logic. Make sense. Changed bq. It's good to append failed log aggregation node info and also the diagnostics coming with ApplicationLogStatus to the diagnostics of the app. Added bq. Do we need a separate Timeout state? is it good to append the timeout diagnostics and return the state as Failed ? this code logic can be simplified to say, if exceeds timeout period return FAILED or Timeout, otherwise return In_Progress. And so we can remove the logAggregationTimeOutDisabled boolean. About this, I still prefer to keep the status something like TIME_OUT. Because it is different when we say logAggregation is time_out between it is failed. For my understanding, the process of Log aggregation should be quick(At nm side). But it will definitely add lots of delay to notify the RMApp. So, the time_out means "we are waitting for a long time. I do not the current state right now". It may be better to simply say the log aggregation is failed. bq. ApplicationLogStatus is better to be named as ApplicationLogAggregationStatus renamed bq. containerLogAggregationFail doesn't need to be atmoicBoolean changed > Expose a client API to allow clients to figure if log aggregation is complete > - > > Key: YARN-1279 > URL: https://issues.apache.org/jira/browse/YARN-1279 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Arun C Murthy >Assignee: Xuan Gong > Attachments: YARN-1279.1.patch, YARN-1279.2.patch, YARN-1279.2.patch, > YARN-1279.3.patch, YARN-1279.3.patch, YARN-1279.4.patch, YARN-1279.4.patch, > YARN-1279.5.patch, YARN-1279.6.patch, YARN-1279.7.patch, YARN-1279.8.patch > > > Expose a client API to allow clients to figure if log aggregation is complete -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1279) Expose a client API to allow clients to figure if log aggregation is complete
[ https://issues.apache.org/jira/browse/YARN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1279: Attachment: YARN-1279.8.patch > Expose a client API to allow clients to figure if log aggregation is complete > - > > Key: YARN-1279 > URL: https://issues.apache.org/jira/browse/YARN-1279 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Arun C Murthy >Assignee: Xuan Gong > Attachments: YARN-1279.1.patch, YARN-1279.2.patch, YARN-1279.2.patch, > YARN-1279.3.patch, YARN-1279.3.patch, YARN-1279.4.patch, YARN-1279.4.patch, > YARN-1279.5.patch, YARN-1279.6.patch, YARN-1279.7.patch, YARN-1279.8.patch > > > Expose a client API to allow clients to figure if log aggregation is complete -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1376) NM need to notify the log aggregation status to RM through Node heartbeat
[ https://issues.apache.org/jira/browse/YARN-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1376: Attachment: YARN-1376.2.patch > NM need to notify the log aggregation status to RM through Node heartbeat > - > > Key: YARN-1376 > URL: https://issues.apache.org/jira/browse/YARN-1376 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1376.1.patch, YARN-1376.2.patch, YARN-1376.2.patch > > > Expose a client API to allow clients to figure if log aggregation is > complete. The ticket is used to track the changes on NM side -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (YARN-691) Invalid NaN values in Hadoop REST API JSON response
[ https://issues.apache.org/jira/browse/YARN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned YARN-691: --- Assignee: Chen He The problem is you were not listed in the contributors list for the YARN project. I added you, so you will now be able to assign YARN JIRAs to yourself. > Invalid NaN values in Hadoop REST API JSON response > --- > > Key: YARN-691 > URL: https://issues.apache.org/jira/browse/YARN-691 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.6, 2.0.4-alpha >Reporter: Kendall Thrapp >Assignee: Chen He > > I've been occasionally coming across instances where Hadoop's Cluster > Applications REST API > (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API) > has returned JSON that PHP's json_decode function failed to parse. I've > tracked the syntax error down to the presence of the unquoted word NaN > appearing as a value in the JSON. For example: > "progress":NaN, > NaN is not part of the JSON spec, so its presence renders the whole JSON > string invalid. Hadoop needs to return something other than NaN in this case > -- perhaps an empty string or the quoted string "NaN". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1279) Expose a client API to allow clients to figure if log aggregation is complete
[
https://issues.apache.org/jira/browse/YARN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816841#comment-13816841
]
Hadoop QA commented on YARN-1279:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612740/YARN-1279.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2393//console
This message is automatically generated.
> Expose a client API to allow clients to figure if log aggregation is complete
> -
>
> Key: YARN-1279
> URL: https://issues.apache.org/jira/browse/YARN-1279
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.2.0
>Reporter: Arun C Murthy
>Assignee: Xuan Gong
> Attachments: YARN-1279.1.patch, YARN-1279.2.patch, YARN-1279.2.patch,
> YARN-1279.3.patch, YARN-1279.3.patch, YARN-1279.4.patch, YARN-1279.4.patch,
> YARN-1279.5.patch, YARN-1279.6.patch, YARN-1279.7.patch, YARN-1279.8.patch
>
>
> Expose a client API to allow clients to figure if log aggregation is complete
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1279) Expose a client API to allow clients to figure if log aggregation is complete
[ https://issues.apache.org/jira/browse/YARN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1279: Attachment: YARN-1279.8.patch > Expose a client API to allow clients to figure if log aggregation is complete > - > > Key: YARN-1279 > URL: https://issues.apache.org/jira/browse/YARN-1279 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Arun C Murthy >Assignee: Xuan Gong > Attachments: YARN-1279.1.patch, YARN-1279.2.patch, YARN-1279.2.patch, > YARN-1279.3.patch, YARN-1279.3.patch, YARN-1279.4.patch, YARN-1279.4.patch, > YARN-1279.5.patch, YARN-1279.6.patch, YARN-1279.7.patch, YARN-1279.8.patch, > YARN-1279.8.patch > > > Expose a client API to allow clients to figure if log aggregation is complete -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1279) Expose a client API to allow clients to figure if log aggregation is complete
[
https://issues.apache.org/jira/browse/YARN-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816846#comment-13816846
]
Hadoop QA commented on YARN-1279:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12612745/YARN-1279.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2394//console
This message is automatically generated.
> Expose a client API to allow clients to figure if log aggregation is complete
> -
>
> Key: YARN-1279
> URL: https://issues.apache.org/jira/browse/YARN-1279
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.2.0
>Reporter: Arun C Murthy
>Assignee: Xuan Gong
> Attachments: YARN-1279.1.patch, YARN-1279.2.patch, YARN-1279.2.patch,
> YARN-1279.3.patch, YARN-1279.3.patch, YARN-1279.4.patch, YARN-1279.4.patch,
> YARN-1279.5.patch, YARN-1279.6.patch, YARN-1279.7.patch, YARN-1279.8.patch,
> YARN-1279.8.patch
>
>
> Expose a client API to allow clients to figure if log aggregation is complete
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-955) [YARN-321] Implementation of ApplicationHistoryProtocol
[
https://issues.apache.org/jira/browse/YARN-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816872#comment-13816872
]
Zhijie Shen commented on YARN-955:
--
1. It's not necessary, and shouldn't be public.
{code}
+public ApplicationAttemptReport getApplicationAttempt(
+ApplicationAttemptId appAttemptId) throws IOException {
+ return history.getApplicationAttempt(appAttemptId);
+}
{code}
2. It's not a recommended way to construct pb instances. Please use
.newInstance() for all the responses here.
{code}
+ GetApplicationAttemptReportResponse response = Records
+ .newRecord(GetApplicationAttemptReportResponse.class);
{code}
3. Why the reference of the implementation is used? Should we use
ApplicationHistoryManager, the interface, instead?
{code}
+ ApplicationHistoryManagerImpl historyService;
{code}
4. In TestApplicationHistoryClientService, please write multiple instances, and
test gets methods as well.
5. Add \@VisibleForTesting
{code}
+ @Private
+ public ApplicationHistoryClientService getClientService() {
+return this.ahsClientService;
+ }
{code}
6. Unwrap the method and put it directly in ApplicationHistoryServer.main
{code}
+ static ApplicationHistoryServer launchAppHistoryServer(String[] args) {
{code}
> [YARN-321] Implementation of ApplicationHistoryProtocol
> ---
>
> Key: YARN-955
> URL: https://issues.apache.org/jira/browse/YARN-955
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Mayank Bansal
> Attachments: YARN-955-1.patch, YARN-955-2.patch, YARN-955-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (YARN-1289) Configuration "yarn.nodemanager.aux-services" should have default value for mapreduce_shuffle.
[ https://issues.apache.org/jira/browse/YARN-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du resolved YARN-1289. -- Resolution: Invalid > Configuration "yarn.nodemanager.aux-services" should have default value for > mapreduce_shuffle. > -- > > Key: YARN-1289 > URL: https://issues.apache.org/jira/browse/YARN-1289 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: wenwupeng > Attachments: YARN-1289.patch > > > Failed to run benchmark when not configure yarn.nodemanager.aux-services > value in yarn-site.xml', it is better to configure default value. > 13/10/09 22:19:23 INFO mapreduce.Job: Task Id : > attempt_1381371516570_0001_m_00_1, Status : FAILED > Container launch failed for container_1381371516570_0001_01_05 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-974) RMContainer should collect more useful information to be recorded in Application-History
[ https://issues.apache.org/jira/browse/YARN-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-974: - Attachment: YARN-974.3.patch Updated the patch: 1. Rebase according to the latest ContainerHistoryData 2. Simplify the way to collect the information. No need to change the Event object any more. 3. Update the test cases Please note that the logURL should be updated to point to the AHS web page, which shows the aggregated logs. This could be updated whenever YARN-954 is done. > RMContainer should collect more useful information to be recorded in > Application-History > > > Key: YARN-974 > URL: https://issues.apache.org/jira/browse/YARN-974 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-974.1.patch, YARN-974.2.patch, YARN-974.3.patch > > > To record the history of a container, users may be also interested in the > following information: > 1. Start Time > 2. Stop Time > 3. Diagnostic Information > 4. URL to the Log File > 5. Actually Allocated Resource > 6. Actually Assigned Node > These should be remembered during the RMContainer's life cycle. -- This message was sent by Atlassian JIRA (v6.1#6144)
