[jira] [Assigned] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik reassigned MAPREDUCE-3863: - Assignee: Konstantin Boudnik I am going to assign the ticket to myself, as I don't see Antony in the list contributors yet. > 0.22 branch mvn deploy is not publishing hadoop-streaming JAR > - > > Key: MAPREDUCE-3863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0, 0.22.1 >Reporter: Alejandro Abdelnur >Assignee: Konstantin Boudnik >Priority: Critical > Fix For: 0.22.1 > > Attachments: MAPREDUCE-3863.patch, MAPREDUCE-3863.patch, > MAPREDUCE-3863.patch > > > Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-3863. --- Resolution: Fixed Fix Version/s: 0.22.1 Hadoop Flags: Reviewed I have just committed it. Thank you, Antony! > 0.22 branch mvn deploy is not publishing hadoop-streaming JAR > - > > Key: MAPREDUCE-3863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0, 0.22.1 >Reporter: Alejandro Abdelnur >Priority: Critical > Fix For: 0.22.1 > > Attachments: MAPREDUCE-3863.patch, MAPREDUCE-3863.patch, > MAPREDUCE-3863.patch > > > Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261328#comment-13261328 ] Konstantin Boudnik commented on MAPREDUCE-3863: --- Patch looks good. Unfortunately, I wasn't been able to validate it is working as expected because I can't compile gridmix on my system as it aborts on some generics complication issues. According to a multiple confirmations from other ppl. looks like that the patch actually fixes the problem in some production environments. So, I am committing this with an official +1 > 0.22 branch mvn deploy is not publishing hadoop-streaming JAR > - > > Key: MAPREDUCE-3863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0, 0.22.1 >Reporter: Alejandro Abdelnur >Priority: Critical > Attachments: MAPREDUCE-3863.patch, MAPREDUCE-3863.patch, > MAPREDUCE-3863.patch > > > Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4124) Job Priority is not changing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261290#comment-13261290 ] xieguiming commented on MAPREDUCE-4124: --- Robert, you are right. the schedulers do not use the priority currently. I think we can change the FIFOschedulers to support the priority firstly. Currently, the FIFOschedulers only sort by submiting time. and we can change to sort by priority firstly, and then by submiting time. > Job Priority is not changing > - > > Key: MAPREDUCE-4124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0 >Reporter: Nishan Shetty >Priority: Critical > > 1.Submit job > 2.Change the job priority using setPriority() or CLI command ./mapred > job-set-priority > Observe that Job priority is not changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261280#comment-13261280
]
Devaraj K commented on MAPREDUCE-3173:
--
The above test failures are not related to the patch.
> MRV2 UI doesn't work properly without internet
> --
>
> Key: MAPREDUCE-3173
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz,
> jquery.min-1.6.4.js.gz
>
>
> When we try access the MRV2 UI, it is always giving the below message in the
> UI even if the java script enabled in the browser.
> {code:xml}
> This page works best with javascript enabled.
> {code}
> It is trying to download these below css/js files from internet and finally
> ending up with the above message. For loading the page also it is taking long
> time.
> {code:title=JQueryUI.java|borderStyle=solid}
> html.
>
> link(join("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/";,
> getTheme(), "/jquery-ui.css")).
> link("/static/dt-1.7.5/css/jui-dt.css").
>
> script("https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js";).
>
> script("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js";).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261275#comment-13261275
]
Devaraj K commented on MAPREDUCE-4189:
--
The above test failures are not related to the patch.
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261214#comment-13261214 ] Hadoop QA commented on MAPREDUCE-3921: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524088/MAPREDUCE-3921-7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 494 javac compiler warnings (more than the trunk's current 492 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2305//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2305//console This message is automatically generated. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-7.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261209#comment-13261209 ] Jonathan Eagles commented on MAPREDUCE-4194: The approach for this bug was to prevent concurrent access modification issues by using a structure that supports versioning of the data. Another approach was to give a copy of the data structure at a cost of performance. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261206#comment-13261206 ] Jonathan Eagles commented on MAPREDUCE-4194: MAPREDUCE-4189 covers TestContainerManagerSecurity and also shows the other test failures and preexisting. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261170#comment-13261170 ] Hadoop QA commented on MAPREDUCE-4193: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524086/MAPREDUCE-4193.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2304//console This message is automatically generated. > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Attachment: MAPREDUCE-3921-7.patch Changed assignedRequests maps to use Container as the value instead of ContainerId > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-7.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Status: Patch Available (was: Open) > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-7.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261167#comment-13261167 ] Bikas Saha commented on MAPREDUCE-3921: --- I will change the map to store Container directly. Return ContainerId and NodeId in the getters from the container object. About the OBSOLETE part. I get how it is used. What I dont get is why we are marking a previously successful task as obsolete and invalid upon the completion of a new task without first checking if the new task was itself successful or not. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-4193: Status: Patch Available (was: Open) > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-4193: Attachment: MAPREDUCE-4193.patch Fixed the link, re-generated the site and it looks good. > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261135#comment-13261135 ] Siddharth Seth commented on MAPREDUCE-3921: --- bq. Yes, the removed entry is marked OBSOLETE. But I still dont understand why that would be done if the current entry is not successful. Why lose the previously successful entry when the current one is not successful itself? This should be done only when the current entry is successful. This list is sent over to reduce tasks - which do consider the OBSOLETE state when deciding on which map outputs need to be fetched. bq. I like the change to store NodeId's instead of ContainerId's in the AssignedRequests map. I would like to make it a separate change and not merge it with this one. There might be other gotchas to doing that. Both NodeId and ContainerId can be stored (I believe containerId is required). That should be a reasonably simple change - and will allow TA_KILLs to be sent directly. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Attachment: MAPREDUCE-3921-6.patch Review comments fixed. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Status: Open (was: Patch Available) > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261127#comment-13261127 ] Bikas Saha commented on MAPREDUCE-3921: --- I see what you are saying about fetch failures and bad nodes. I am open to both approaches. The way its currently done is based off discussions I had with Vinod long ago. Changed JobImpl to ignore JOB_UPDATED_NODE events in NEW and INITED states. Changed to TaskId.getTaskType() Yes, the removed entry is marked OBSOLETE. But I still dont understand why that would be done if the current entry is not successful. Why lose the previously successful entry when the current one is not successful itself? This should be done only when the current entry is successful. I like the change to store NodeId's instead of ContainerId's in the AssignedRequests map. I would like to make it a separate change and not merge it with this one. There might be other gotchas to doing that. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261126#comment-13261126 ] Hadoop QA commented on MAPREDUCE-4194: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524066/MAPREDUCE-4194.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2303//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2303//console This message is automatically generated. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-3674:
-
Attachment: MAPREDUCE-3674.patch
Fix to src/webapps/job/jobqueue_details.jsp
to avoid NPE when invalid queue name, such as null or non-existing queue name
is specified. It avoids null schedInfo.
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
> MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
> hadoop-findbugs-report.html
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.h
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261094#comment-13261094 ] Siddharth Seth commented on MAPREDUCE-3921: --- bq. Yes, I think. Based on our experience, here we are pre-emptively taking action on a task that might actually be ok. And it should be an infrequent action. bq. My understanding of existing behavior in mrv1 was that only maps are pre-emptively terminated for performance reasons. I think 'fetch failure' / 'node unhealthy' should be considered in the same way - at least for the purpose of counting towards the allowed_task_failure limit. Ideally for the tasks state as well. There's currently no way to distinguish between a task causing a node to go unhealthy versus other problems. My guess is 'fetch failures' are more often than not caused by a bad tracker, as against a bad task. WRT killing reduce tasks on unhealthy node - I'm not sure what was done in 20 (From a quick look, couldn't find the code which kills map tasks either). It'd be best if Vinod or others with more knowledge and history about how and why 20 deals with this pitch in. bq. My understanding was that scheduling happens when the job moves from INIT to RUNNING state via the StartTransition(). Unless allocate is called on RM it will not return any unhealthy machines. So I thought that JOB_UPDATED_EVENT can never come until the job moves into the RUNNING state. Can you please point out the scenario you are thinking about? Calls to allocate() start once the RMCommunicator service is started - which happens before a JOB_START event is sent. Very unlikely - but there's an extremely remote possibility of an allocate call completing before a job moves into the START state. bq. Unless you really want this, I would prefer it the way its currently written. I prefer not to depend on string name encodings. It's safe to use TaskId.getTaskType() - don't need to explicitly depend on string name encoding. Avoids the extra task lookups. bq. That was a question I had and put it in the comments. It seems that for a TaskAttemptCompletedEventTransition the code removes the previous successful entry from successAttemptCompletionEventNoMap. It then checks if the current attempt is successful, and in that case adds it to the successAttemptCompletionEventNoMap. But what if the current attempt is not successful. We have now removed the previous successful attempt too. Is that the desired behavior. This question is independent of this jira. It also marks the removed entry as OBSOLETE - so the taskAttemptCompletionEvents list doesn't have any SUCCESSFUL attempts for the specific taskId. bq. I have moved the finishedTask increment out of that function and made it explicit in every transition that requires it to be that way. In the same context I have a question in comments in MapRetroactiveFailureTransition. Why is this not calling handleAttemptCompletion. My understanding is that handleAttemptCompletion is used to notify reducers about changes in map outputs. So if a map was failed after success then reducers should know about it so that they can abandon its outputs before getting too many fetch failures. Is that not so? It is calling it via AttemptFailedTransition.transition(). That's the bit which also counts the failure towards the allowed_failure_limit. bq. Sorry I did not find getContainerManagerAddress(). The map in AssignedRequests stores ContainerId and its not possible to get nodeId from it. What are you proposing? Correction - called getAssignedContainerMgrAddress. IAC, was proposing storing the containers NodeId with the AssignedRequest - that completely removes the need to fetch the actual task. bq. It does not look like it but there may be race conditions I have not thought of. But looking further, it seems that the action on this event checks for job completion in TaskCompletedTransition. TaskCompletedTransition increments job.completedTaskCount irrespective of whether the task has succeeded/killed or failed. Now, TaskCompletedTransition.checkJobCompleteSuccess() checks job.completedTaskCount == job.tasks.size() for completion. How is this working? Wont enough killed tasks/failed + completed tasks trigger job completion? Or is that expected behavior? It checks for failure before attempting the SUCCESS check - so that should work. Unless I'm missing something - Tasks could complete after a Job moves to state FAILED - which would end up generating this event. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug >
[jira] [Commented] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
[ https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261088#comment-13261088 ] Ravi Prakash commented on MAPREDUCE-4157: - bq. The RM then gives it so many seconds for the AM container to exit on its own. Jason! Wouldn't yarn.nodemanager.process-kill-wait.ms and yarn.nodemanager.sleep-delay-before-sigkill.ms achieve the same effect? > ResourceManager should not kill apps that are well behaved > -- > > Key: MAPREDUCE-4157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4157.patch > > > Currently when the ApplicationMaster unregisters with the ResourceManager, > the RM kills (via the NMs) all the active containers for an application. > This introduces a race where the AM may be trying to clean up and may not > finish before it is killed. The RM should give the AM a chance to exit > cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261069#comment-13261069 ] Jonathan Eagles commented on MAPREDUCE-4194: I have verified that the newly added test fails with ConcurrentModificationException without the changes and succeeds with the changes. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4194: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Patch Available (was: Open) > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4194: --- Attachment: MAPREDUCE-4194.patch > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
Jonathan Eagles created MAPREDUCE-4194: -- Summary: ConcurrentModificationError in DirectoryCollection Key: MAPREDUCE-4194 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles As found as part of work on MAPREDUCE-4169, it is possible for a ConcurrentModificationException to be thrown upon disk failure. DirectoryCollection hands out its internal list structure that is accessed across multiple threads. Upon disk failure its internal list is modified, invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3002) Delink History Context from AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned MAPREDUCE-3002: -- Assignee: Robert Joseph Evans > Delink History Context from AppContext > -- > > Key: MAPREDUCE-3002 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3002 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mrv2 >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > Currently the JobHistory Server has a HistoryContext that pretends to be a > Map Reduce ApplicationMaster's AppContext so that UI pages can be shared > between the two. This is not ideal because the UIs have already diverged a > lot, and we have to translate the native History Server's data into > implementations of Job to provide the same interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4154) streaming MR job succeeds even if the streaming command fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-4154: -- Target Version/s: 1.0.3 (was: 1.0.3, 1.1.0) Fix Version/s: (was: 1.1.0) +1. Thanks, Owen and Devaraj. > streaming MR job succeeds even if the streaming command fails > - > > Key: MAPREDUCE-4154 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4154 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.2 >Reporter: Thejas M Nair >Assignee: Devaraj Das > Fix For: 1.0.3 > > Attachments: streaming.patch > > > Hadoop 1.0.1 behaves as expected - The task fails for streaming MR job if the > streaming command fails. But it succeeds in hadoop 1.0.2 . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261001#comment-13261001 ] Hadoop QA commented on MAPREDUCE-3921: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524035/MAPREDUCE-3921-5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 494 javac compiler warnings (more than the trunk's current 492 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2302//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2302//console This message is automatically generated. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Status: Patch Available (was: Open) > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Status: Open (was: Patch Available) > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260860#comment-13260860 ] Bikas Saha commented on MAPREDUCE-3921: --- added new patch for comments above. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Attachment: MAPREDUCE-3921-5.patch > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260834#comment-13260834 ] Bikas Saha commented on MAPREDUCE-3921: --- bq.Does 'node unhealthy' need to be treated differently from 'TooManyFetchFailures' ? Killed versus Failed. This ends up with NodeFailures not counting towards the limit on task attempts. Yes, I think. Based on our experience, here we are pre-emptively taking action on a task that might actually be ok. And it should be an infrequent action. bq. JOB_UPDATED_NODES needs to be handled in the JOB_INIT state. Very small chance of hitting this. My understanding was that scheduling happens when the job moves from INIT to RUNNING state via the StartTransition(). Unless allocate is called on RM it will not return any unhealthy machines. So I thought that JOB_UPDATED_EVENT can never come until the job moves into the RUNNING state. Can you please point out the scenario you are thinking about? I can make the change for safety reasons, just in case. bq. Minor: JobImpl.actOnUsableNode can get the task type from the id itself. It doesn't need to fetch the actual task. Unless you really want this, I would prefer it the way its currently written. I prefer not to depend on string name encodings. bq. Minor: "if this attempt is not successful" this comment in JobImpl can be removed. It's removing an entry from a successfulAttempt index. That was a question I had and put it in the comments. It seems that for a TaskAttemptCompletedEventTransition the code removes the previous successful entry from successAttemptCompletionEventNoMap. It then checks if the current attempt is successful, and in that case adds it to the successAttemptCompletionEventNoMap. But what if the current attempt is not successful. We have now removed the previous successful attempt too. Is that the desired behavior. This question is independent of this jira. bq. In KilledAfterSuccessTransition - createJobCounterUpdateEventTAFailed should be createJobCounterUpdateEventTAKilled Done. bq. TaskImpl.handleAttemptCompletion - finishedAttempts - this will end up double counting the same task attempt. It's used in some other transition. I have moved the finishedTask increment out of that function and made it explicit in every transition that requires it to be that way. In the same context I have a question in comments in MapRetroactiveFailureTransition. Why is this not calling handleAttemptCompletion. My understanding is that handleAttemptCompletion is used to notify reducers about changes in map outputs. So if a map was failed after success then reducers should know about it so that they can abandon its outputs before getting too many fetch failures. Is that not so? bq. Does the JobHistoryParser need some more changes - to unset fields which may have been set previously by Map/ReduceAttemptSuccessfulEvents and TaskFinishedEvent Done. Reset all fields set in handleTaskFinishedEvent. Others are already handled in the existing code. bq. For running tasks - shouldn't running Reduce attempts also be killed ? My understanding of existing behavior in mrv1 was that only maps are pre-emptively terminated for performance reasons. bq. RMContainerAllocator.handleUpdatedNodes - instead of fetching the nodeId via appContext, job etc - the nodeId can be stored with the AssignedRequest. 1) getTask, getAttempt require readLocks - can avoid these calls every second. 2) There's an unlikely race where the nodeId may not be assigned in the TaskAttempt (if the dispatcher thread is backlogged). TaskAttemptId.getNodeId() can be avoided. getContainerManagerAddress can be used instead. Sorry I did not find getContainerManagerAddress(). The map in AssignedRequests stores ContainerId and its not possible to get nodeId from it. What are you proposing? bq. Not related to this patch. Does JOB_TASK_COMPLETED need to be handled (ignored) in additional states? It does not look like it but there may be race conditions I have not thought of. But looking further, it seems that the action on this event checks for job completion in TaskCompletedTransition. TaskCompletedTransition increments job.completedTaskCount irrespective of whether the task has succeeded/killed or failed. Now, TaskCompletedTransition.checkJobCompleteSuccess() checks job.completedTaskCount == job.tasks.size() for completion. How is this working? Wont enough killed tasks/failed + completed tasks trigger job completion? Or is that expected behavior? > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug >
[jira] [Commented] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260805#comment-13260805
]
Hadoop QA commented on MAPREDUCE-4189:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524012/MAPREDUCE-4189.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test
files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2301//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2301//console
This message is automatically generated.
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260738#comment-13260738
]
Daryn Sharp commented on MAPREDUCE-4189:
+1 Looks good!
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4189:
-
Status: Patch Available (was: Open)
Thanks a lot Daryn.
I have updated the patch with the change.
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4189:
-
Attachment: MAPREDUCE-4189.patch
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4189:
-
Status: Open (was: Patch Available)
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260724#comment-13260724
]
Hadoop QA commented on MAPREDUCE-3173:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524002/MAPREDUCE-3173.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2300//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2300//console
This message is automatically generated.
> MRV2 UI doesn't work properly without internet
> --
>
> Key: MAPREDUCE-3173
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz,
> jquery.min-1.6.4.js.gz
>
>
> When we try access the MRV2 UI, it is always giving the below message in the
> UI even if the java script enabled in the browser.
> {code:xml}
> This page works best with javascript enabled.
> {code}
> It is trying to download these below css/js files from internet and finally
> ending up with the above message. For loading the page also it is taking long
> time.
> {code:title=JQueryUI.java|borderStyle=solid}
> html.
>
> link(join("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/";,
> getTheme(), "/jquery-ui.css")).
> link("/static/dt-1.7.5/css/jui-dt.css").
>
> script("https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js";).
>
> script("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js";).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4158) Port mapreduce.MapFileOutputFormat to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-4158: - Attachment: MAPREDUCE-4158.patch > Port mapreduce.MapFileOutputFormat to branch-1 > -- > > Key: MAPREDUCE-4158 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4158 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 1.0.2 >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4158.patch > > > MapFileOutputFormat was missed in MAPREDUCE-3607. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3173:
-
Affects Version/s: (was: 0.24.0)
(was: 0.23.0)
3.0.0
2.0.0
Status: Patch Available (was: Open)
> MRV2 UI doesn't work properly without internet
> --
>
> Key: MAPREDUCE-3173
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz,
> jquery.min-1.6.4.js.gz
>
>
> When we try access the MRV2 UI, it is always giving the below message in the
> UI even if the java script enabled in the browser.
> {code:xml}
> This page works best with javascript enabled.
> {code}
> It is trying to download these below css/js files from internet and finally
> ending up with the above message. For loading the page also it is taking long
> time.
> {code:title=JQueryUI.java|borderStyle=solid}
> html.
>
> link(join("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/";,
> getTheme(), "/jquery-ui.css")).
> link("/static/dt-1.7.5/css/jui-dt.css").
>
> script("https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js";).
>
> script("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js";).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3173:
-
Attachment: MAPREDUCE-3173.patch
jquery-ui.min-1.8.16.js.gz
jquery.min-1.6.4.js.gz
> MRV2 UI doesn't work properly without internet
> --
>
> Key: MAPREDUCE-3173
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz,
> jquery.min-1.6.4.js.gz
>
>
> When we try access the MRV2 UI, it is always giving the below message in the
> UI even if the java script enabled in the browser.
> {code:xml}
> This page works best with javascript enabled.
> {code}
> It is trying to download these below css/js files from internet and finally
> ending up with the above message. For loading the page also it is taking long
> time.
> {code:title=JQueryUI.java|borderStyle=solid}
> html.
>
> link(join("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/";,
> getTheme(), "/jquery-ui.css")).
> link("/static/dt-1.7.5/css/jui-dt.css").
>
> script("https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js";).
>
> script("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js";).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5) Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
[ https://issues.apache.org/jira/browse/MAPREDUCE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260660#comment-13260660 ] Han Qiu commented on MAPREDUCE-5: - We are getting the same exception in shuffle and it causes all reducers to failed, attached please find the exception. 2012-04-24 15:03:52,163 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.112.195.128:50060, dest: 10.242.102.100:35678, bytes: 65536, op: MAPRED_SHUFFLE, cliID: attempt_201204202043_0002_m_000328_0, duration: 11772847 2012-04-24 15:03:52,163 ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3718) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:824) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Shuffle's getMapOutput() fails with EofException, followed by > IllegalStateException > --- > > Key: MAPREDUCE-5 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 > Environment: Sun Java 1.6.0_13, OpenSolaris, running on a SunFire > 4150 (x64) 10 node cluster >Reporter: George Porter > > During the shuffle phase, I'm seeing a large sequence of the following > actions: > 1) WARN org.apache.hadoop.mapred.TaskTracker: > getMapOutput(attempt_200905181452_0002_m_10_0,0) failed : > org.mortbay.jetty.EofException > 2) WARN org.mortbay.log: Committed before 410 > getMapOutput(attempt_200905181452_0002_m_10_0,0) failed : > org.mortbay.jetty.EofException > 3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException: > Committed > The map phase completes with 100%, and then the reduce phase crawls along > with the above errors in each of the TaskTracker logs. None of the > tasktrackers get lost. When I run non-data jobs like the 'pi' test from the > example jar, everything works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4154) streaming MR job succeeds even if the streaming command fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-4154: - Target Version/s: 1.1.0, 1.0.3 (was: 1.0.3, 1.1.0) Fix Version/s: 1.0.3 I just committed this back to 1.0.3 and moved the log in the branch-1/CHANGES.txt. > streaming MR job succeeds even if the streaming command fails > - > > Key: MAPREDUCE-4154 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4154 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.2 >Reporter: Thejas M Nair >Assignee: Devaraj Das > Fix For: 1.1.0, 1.0.3 > > Attachments: streaming.patch > > > Hadoop 1.0.1 behaves as expected - The task fails for streaming MR job if the > streaming command fails. But it succeeds in hadoop 1.0.2 . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260635#comment-13260635
]
Daryn Sharp commented on MAPREDUCE-4189:
Please use {{SecurityUtil.setTokenService(appToken, schedulerAddr)}}.
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260614#comment-13260614
]
Ravi Prakash commented on MAPREDUCE-4135:
-
Hi Devaraj! I've assigned this JIRA to myself. I hope you don't mind.
AFTER applying the latest patch, inside MRAppMaster.stop(), super.stop() is
throwing an InterruptedException
{noformat}
2012-04-24 09:25:20,104 WARN [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Interrupted Exception while
stopping
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1186)
at java.lang.Thread.join(Thread.java:1239)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:104)
at
org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
at
org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1016)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:418)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:386)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This is causing the subsequent statement (FileSystem.closeAll() ) to be
skipped. So to fix this JIRA, I'll see what I can do to fix the
InterruptedException.
> MRAppMaster throws IllegalStateException while shutting down
>
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
> MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting
> down. It doesn't look good having this exception in the stderr file of
> MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in
> progress
> at
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4124) Job Priority is not changing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260589#comment-13260589 ] Robert Joseph Evans commented on MAPREDUCE-4124: I thought that the problem was deeper then that. I do not believe that any of the schedulers actually use the priority currently, so even just getting the priority over to the RM will not really do anything yet, at least not until the schedulers are updated to use priority. > Job Priority is not changing > - > > Key: MAPREDUCE-4124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0 >Reporter: Nishan Shetty >Priority: Critical > > 1.Submit job > 2.Change the job priority using setPriority() or CLI command ./mapred > job-set-priority > Observe that Job priority is not changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260539#comment-13260539 ] Hudson commented on MAPREDUCE-4079: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2140 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2140/]) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329694) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329694 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/YarnRPC.java > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash reassigned MAPREDUCE-4135:
---
Assignee: Ravi Prakash (was: Devaraj K)
> MRAppMaster throws IllegalStateException while shutting down
>
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
> MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting
> down. It doesn't look good having this exception in the stderr file of
> MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in
> progress
> at
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes, fix allocation configurations and document them
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260526#comment-13260526 ] Hudson commented on MAPREDUCE-3812: --- Integrated in Hadoop-Mapreduce-trunk #1059 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/]) MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Lower default allocation sizes, fix allocation configurations and document > them > --- > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Harsh J > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-3812-20120205.txt, > MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. > [P.s.]: > - Currently min/max alloc is set at a per-scheduler config level, which makes > no sense as there's no way to run multiple schedulers anyway. Switch configs > to use a generic RM-config. > - The min/max alloc configs aren't documented and we ought to document it > (i.e. MAPREDUCE-4027) > - 1 GB is perhaps too high for a slot's minimum. While job defaults can be > left at such values, we should lower the minimum alloc to 128 MB to allow > special requests of low memory out of the box itself. Shouldn't impact MR App > in any way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4133) MR over viewfs is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260527#comment-13260527
]
Hudson commented on MAPREDUCE-4133:
---
Integrated in Hadoop-Mapreduce-trunk #1059 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/])
MAPREDUCE-4133. MR over viewfs is broken (John George via bobby) (Revision
1329403)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/DistributedFSCheck.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java
> MR over viewfs is broken
>
>
> Key: MAPREDUCE-4133
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4133
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.0
>Reporter: John George
>Assignee: John George
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MR-4133.patch, MR-4133.patch, MR-4133.patch
>
>
> After the changes in HADOOP-8014 went in, MR programs using viewfs broke.
> This is because, viewfs now expects getDefaultBlockSize,
> getDefaultReplication, and getServerDefaults to pass in a {{path}} as an
> argument. In the existing MR source, these are called with no arguments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260528#comment-13260528 ] Hudson commented on MAPREDUCE-4141: --- Integrated in Hadoop-Mapreduce-trunk #1059 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/]) MAPREDUCE-4141. clover integration broken, also mapreduce poms are pulling in clover as a dependency. (phunt via tucu) (Revision 1329493) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329493 Files : * /hadoop/common/trunk/hadoop-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4141.patch > > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4190) Improve web UI for task attempts userlog link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260529#comment-13260529 ] Hudson commented on MAPREDUCE-4190: --- Integrated in Hadoop-Mapreduce-trunk #1059 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/]) MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves via bobby) (Revision 1329398) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329398 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java > Improve web UI for task attempts userlog link > -- > > Key: MAPREDUCE-4190 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4190 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, webapps >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4190.patch > > > Users have reported being confused about the user logs link on the MR app > master and history server task attempts page. The logs link shows up in the > same column as the node link and the users didn't realize it was a separate > link. > job history web address for this page is like: > jobhistoryserver:19888/jobhistory/task/task_1334686840316_0003_m_00 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260519#comment-13260519 ] Hudson commented on MAPREDUCE-4079: --- Integrated in Hadoop-Hdfs-trunk-Commit #2198 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2198/]) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329694) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329694 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/YarnRPC.java > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260522#comment-13260522 ] Hudson commented on MAPREDUCE-4079: --- Integrated in Hadoop-Common-trunk-Commit #2124 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2124/]) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329694) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329694 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/YarnRPC.java > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4079: - Resolution: Fixed Fix Version/s: 0.23.3 Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0, 0.23.3) Status: Resolved (was: Patch Available) I committed this to trunk, branch-2, and branch-0.23. > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260509#comment-13260509 ] Thomas Graves commented on MAPREDUCE-4079: -- +1 Thanks Bobby! > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes, fix allocation configurations and document them
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260500#comment-13260500 ] Hudson commented on MAPREDUCE-3812: --- Integrated in Hadoop-Hdfs-trunk #1024 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/]) MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Lower default allocation sizes, fix allocation configurations and document > them > --- > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Harsh J > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-3812-20120205.txt, > MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. > [P.s.]: > - Currently min/max alloc is set at a per-scheduler config level, which makes > no sense as there's no way to run multiple schedulers anyway. Switch configs > to use a generic RM-config. > - The min/max alloc configs aren't documented and we ought to document it > (i.e. MAPREDUCE-4027) > - 1 GB is perhaps too high for a slot's minimum. While job defaults can be > left at such values, we should lower the minimum alloc to 128 MB to allow > special requests of low memory out of the box itself. Shouldn't impact MR App > in any way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260502#comment-13260502 ] Hudson commented on MAPREDUCE-4141: --- Integrated in Hadoop-Hdfs-trunk #1024 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/]) MAPREDUCE-4141. clover integration broken, also mapreduce poms are pulling in clover as a dependency. (phunt via tucu) (Revision 1329493) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329493 Files : * /hadoop/common/trunk/hadoop-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4141.patch > > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4190) Improve web UI for task attempts userlog link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260503#comment-13260503 ] Hudson commented on MAPREDUCE-4190: --- Integrated in Hadoop-Hdfs-trunk #1024 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/]) MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves via bobby) (Revision 1329398) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329398 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java > Improve web UI for task attempts userlog link > -- > > Key: MAPREDUCE-4190 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4190 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, webapps >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4190.patch > > > Users have reported being confused about the user logs link on the MR app > master and history server task attempts page. The logs link shows up in the > same column as the node link and the users didn't realize it was a separate > link. > job history web address for this page is like: > jobhistoryserver:19888/jobhistory/task/task_1334686840316_0003_m_00 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4133) MR over viewfs is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260501#comment-13260501
]
Hudson commented on MAPREDUCE-4133:
---
Integrated in Hadoop-Hdfs-trunk #1024 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/])
MAPREDUCE-4133. MR over viewfs is broken (John George via bobby) (Revision
1329403)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/DistributedFSCheck.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java
> MR over viewfs is broken
>
>
> Key: MAPREDUCE-4133
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4133
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.0
>Reporter: John George
>Assignee: John George
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MR-4133.patch, MR-4133.patch, MR-4133.patch
>
>
> After the changes in HADOOP-8014 went in, MR programs using viewfs broke.
> This is because, viewfs now expects getDefaultBlockSize,
> getDefaultReplication, and getServerDefaults to pass in a {{path}} as an
> argument. In the existing MR source, these are called with no arguments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4133) MR over viewfs is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260478#comment-13260478
]
Hudson commented on MAPREDUCE-4133:
---
Integrated in Hadoop-Hdfs-0.23-Build #237 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/237/])
svn merge -c 1329403. FIXES: MAPREDUCE-4133. MR over viewfs is broken (John
George via bobby) (Revision 1329406)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329406
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/DistributedFSCheck.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java
> MR over viewfs is broken
>
>
> Key: MAPREDUCE-4133
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4133
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.0
>Reporter: John George
>Assignee: John George
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MR-4133.patch, MR-4133.patch, MR-4133.patch
>
>
> After the changes in HADOOP-8014 went in, MR programs using viewfs broke.
> This is because, viewfs now expects getDefaultBlockSize,
> getDefaultReplication, and getServerDefaults to pass in a {{path}} as an
> argument. In the existing MR source, these are called with no arguments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes, fix allocation configurations and document them
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260477#comment-13260477 ] Hudson commented on MAPREDUCE-3812: --- Integrated in Hadoop-Hdfs-0.23-Build #237 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/237/]) svn merge -c 1329282 FIXES: MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329285) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329285 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Lower default allocation sizes, fix allocation configurations and document > them > --- > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Harsh J > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-3812-20120205.txt, > MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. > [P.s.]: > - Currently min/max alloc is set at a per-scheduler config level, which makes > no sense as there's no way to run multiple schedulers anyway. Switch configs > to use a generic RM-config. > - The min/max alloc configs aren't documented and we ought to document it > (i.e. MAPREDUCE-4027) > - 1 GB is perhaps too high for a slot's minimum. While job defaults can be > left at such values, we should lower the minimum alloc to 128 MB to allow > special requests of low memory out of the box itself. Shouldn't impact MR App > in any way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4190) Improve web UI for task attempts userlog link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260479#comment-13260479 ] Hudson commented on MAPREDUCE-4190: --- Integrated in Hadoop-Hdfs-0.23-Build #237 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/237/]) svn merge -c 1329398 FIXES: MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves via bobby) (Revision 1329402) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java > Improve web UI for task attempts userlog link > -- > > Key: MAPREDUCE-4190 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4190 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, webapps >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4190.patch > > > Users have reported being confused about the user logs link on the MR app > master and history server task attempts page. The logs link shows up in the > same column as the node link and the users didn't realize it was a separate > link. > job history web address for this page is like: > jobhistoryserver:19888/jobhistory/task/task_1334686840316_0003_m_00 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4124) Job Priority is not changing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260408#comment-13260408
]
xieguiming commented on MAPREDUCE-4124:
---
1, When jobclient create the ApplicationSubmissionContext, set the priority to
the ApplicationSubmissionContext.
{color:red}
Priority pri = Records.newRecord(Priority.class); // Job Priority.
pri.setPriority(jobConf.getInt(JobContext.PRIORITY,
YarnConfiguration.DEFAULT_PRIORITY));
appContext.setPriority(pri);
{color}
appContext.setAMContainerSpec(amContainer); // AM Container
2, Add one interface function in the ClientRMProtocol. and implement the
function:
public void setJobPriority(JobID arg0, String arg1) throws IOException,
InterruptedException {
{color:red}
// call the ClientRMProtocol.setApplicationPriority()
{color}
return;
}
> Job Priority is not changing
> -
>
> Key: MAPREDUCE-4124
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4124
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0
>Reporter: Nishan Shetty
>Priority: Critical
>
> 1.Submit job
> 2.Change the job priority using setPriority() or CLI command ./mapred
> job-set-priority
> Observe that Job priority is not changed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4052) Windows eclpise can not submit the job
[
https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260395#comment-13260395
]
xieguiming commented on MAPREDUCE-4052:
---
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
The faillure of org.apache.hadoop.yarn.server.TestContainerManagerSecurity has
not relation with my testcase.
> Windows eclpise can not submit the job
> --
>
> Key: MAPREDUCE-4052
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Affects Versions: 0.23.1
> Environment: client on the Windows, the the cluster on the suse
>Reporter: xieguiming
> Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.patch
>
>
> when I use the eclipse on the windows to submit the job. and the
> applicationmaster throw the exception:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> The reasion is :
> class Apps addToEnvironment function, use the
> private static final String SYSTEM_PATH_SEPARATOR =
> System.getProperty("path.separator");
> and will result the MRApplicationMaster classpath use the ";" separator.
> I suggest that nodemanger do the replace.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
