[jira] [Commented] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255355#comment-13255355
]
Hadoop QA commented on MAPREDUCE-4074:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522906/MAPREDUCE-4074-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test
files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
org.apache.hadoop.yarn.server.TestDiskFailures
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2233//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2233//console
This message is automatically generated.
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc
[jira] [Created] (MAPREDUCE-4158) Port mapreduce.MapFileOutputFormat to branch-1
Port mapreduce.MapFileOutputFormat to branch-1 -- Key: MAPREDUCE-4158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4158 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 1.0.2 Reporter: Tom White Assignee: Tom White MapFileOutputFormat was missed in MAPREDUCE-3607. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xieguiming updated MAPREDUCE-4074:
--
Status: Patch Available (was: Open)
add 3 unit test cases.
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
> 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
> 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
> 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
> 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
> 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xieguiming updated MAPREDUCE-4074:
--
Attachment: MAPREDUCE-4074-1.patch
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
> 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
> 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
> 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
> 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
> 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4074:
-
Status: Open (was: Patch Available)
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Attachments: MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
> 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
> 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
> 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
> 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
> 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255254#comment-13255254 ] Hadoop QA commented on MAPREDUCE-3942: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522894/MR-3942.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2232//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2232//console This message is automatically generated. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt, > MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3942: -- Status: Patch Available (was: Open) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt, > MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3942: -- Attachment: MR-3942.txt Fixing the findbugs warning. Waiting for jenkins again ... > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt, > MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn use fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Status: Open (was: Patch Available) > MiniMRYarn/MiniYarn use fixed ports > --- > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3942: -- Status: Open (was: Patch Available) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3972) Locking and exception issues in JobHistory Server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3972: -- Status: Open (was: Patch Available) Changes look good. Thanks Bobby! One more fix required though - there's a race in TestJobHistoryParsing - since it always tries to read the file from the intermediate location, and the file may have been moved meanwhile. > Locking and exception issues in JobHistory Server. > -- > > Key: MAPREDUCE-3972 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3972 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-3972.txt, MR-3972.txt, MR-3972.txt, MR-3972.txt, > MR-3972.txt > > > The JobHistory server's locking is inconsistent and wrong in some cases. > This is not super critical because the issues would only show up if a job is > being cleaned up or moved from intermediate done to done, at the same time it > is being parsed into a CompletedJob. However the locking is slowing down the > server in some cases, and is a ticking time bomb that needs to be addressed. > As part of this too we need to be sure that the Cleaner and Intermediate to > Done migration threads handle exceptions properly. Now it appears that the > exception is logged, and the thread just shuts down. This means that the > history server could still be up and running for weeks and never remove old > jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3867) MiniMRYarn/MiniYarn use fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255223#comment-13255223 ] Hadoop QA commented on MAPREDUCE-3867: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522881/MAPREDUCE-3867.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2230//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2230//console This message is automatically generated. > MiniMRYarn/MiniYarn use fixed ports > --- > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255219#comment-13255219 ] Hadoop QA commented on MAPREDUCE-3942: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522882/MR-3942.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2231//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2231//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2231//console This message is automatically generated. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn use fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Assignee: Alejandro Abdelnur Status: Patch Available (was: Open) Creating a ServerSocket on port 0 to get a random avail socket for the RM & HS rpc & web ports. cannot use :0 as they don't push back to the config their ports. > MiniMRYarn/MiniYarn use fixed ports > --- > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn use fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Attachment: MAPREDUCE-3867.patch > MiniMRYarn/MiniYarn use fixed ports > --- > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3942: -- Status: Patch Available (was: Open) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3942: -- Attachment: MR-3942.txt Same as the previous patch with a small fix for the unit test failure. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt, MR-3942.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3942: -- Fix Version/s: (was: 0.23.2) Target Version/s: 0.23.3 Status: Open (was: Patch Available) +1. The patch looks good. Also tried it on a secure cluster. Will upload an updated patch for the unit test failure shortly - and commit this once Jenkins returns. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255160#comment-13255160 ] Benoy Antony commented on MAPREDUCE-3863: - The patch involves the following : 1. correcting the name of the jar so that version and build number comes after streaming. 2. adding a template for streaming pom file 3. changes to mapreduce/build.xml to publish the jar to repo. > 0.22 branch mvn deploy is not publishing hadoop-streaming JAR > - > > Key: MAPREDUCE-3863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0, 0.22.1 >Reporter: Alejandro Abdelnur >Priority: Critical > Attachments: MAPREDUCE-3863.patch > > > Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-3863: Attachment: MAPREDUCE-3863.patch > 0.22 branch mvn deploy is not publishing hadoop-streaming JAR > - > > Key: MAPREDUCE-3863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0, 0.22.1 >Reporter: Alejandro Abdelnur >Priority: Critical > Attachments: MAPREDUCE-3863.patch > > > Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255114#comment-13255114 ] Alejandro Abdelnur commented on MAPREDUCE-4141: --- +1 > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4141.patch > > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255086#comment-13255086 ] Hudson commented on MAPREDUCE-4156: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2094 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2094/]) MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves) (Revision 1326795) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326795 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistoryParsing.java > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255054#comment-13255054 ] Hudson commented on MAPREDUCE-4156: --- Integrated in Hadoop-Common-trunk-Commit #2080 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2080/]) MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves) (Revision 1326795) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326795 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistoryParsing.java > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255050#comment-13255050 ] Hudson commented on MAPREDUCE-4156: --- Integrated in Hadoop-Hdfs-trunk-Commit #2153 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2153/]) MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves) (Revision 1326795) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326795 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistoryParsing.java > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4156: - Resolution: Fixed Fix Version/s: 0.23.3 Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2, and branch-0.23. Thanks for the reviews Bobby and Eric. > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254982#comment-13254982 ] Robert Joseph Evans commented on MAPREDUCE-4156: Then feel free to commit it +1 > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
[ https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4157: -- Affects Version/s: (was: 0.23.3) 2.0.0 > ResourceManager should not kill apps that are well behaved > -- > > Key: MAPREDUCE-4157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > > Currently when the ApplicationMaster unregisters with the ResourceManager, > the RM kills (via the NMs) all the active containers for an application. > This introduces a race where the AM may be trying to clean up and may not > finish before it is killed. The RM should give the AM a chance to exit > cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
ResourceManager should not kill apps that are well behaved -- Key: MAPREDUCE-4157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Currently when the ApplicationMaster unregisters with the ResourceManager, the RM kills (via the NMs) all the active containers for an application. This introduces a race where the AM may be trying to clean up and may not finish before it is killed. The RM should give the AM a chance to exit cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4144) ResourceManager NPE while handling NODE_UPDATE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254943#comment-13254943
]
Jason Lowe commented on MAPREDUCE-4144:
---
Do we want to allocate a node local container if there's no rack local to go
with it? I thought the change in MAPREDUCE-3005 was specifically trying to
avoid allocating in that case. Does the fact that the container is reserved
change that somehow?
> ResourceManager NPE while handling NODE_UPDATE
> --
>
> Key: MAPREDUCE-4144
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4144
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Fix For: 0.23.3
>
> Attachments: MAPREDUCE-4144-testcase.patch, MAPREDUCE-4144.patch
>
>
> The RM on one of our clusters has exited twice in the past few days because
> of an NPE while trying to handle a NODE_UPDATE:
> {noformat}
> 2012-04-12 02:09:01,672 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> [ResourceManager Event Processor]java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:261)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:223)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:246)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1078)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1048)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:859)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:756)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:573)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:622)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:78)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:302)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> This is very similar to the failure reported in MAPREDUCE-3005.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4144) ResourceManager NPE while handling NODE_UPDATE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254921#comment-13254921
]
Arun C Murthy commented on MAPREDUCE-4144:
--
Sorry for coming late to this...
I'm not sure this is the right fix, we could just handle the NPE with a null
check in AppSchedulingInfo.allocateNodeLocal?
> ResourceManager NPE while handling NODE_UPDATE
> --
>
> Key: MAPREDUCE-4144
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4144
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Fix For: 0.23.3
>
> Attachments: MAPREDUCE-4144-testcase.patch, MAPREDUCE-4144.patch
>
>
> The RM on one of our clusters has exited twice in the past few days because
> of an NPE while trying to handle a NODE_UPDATE:
> {noformat}
> 2012-04-12 02:09:01,672 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_UPDATE to the scheduler
> [ResourceManager Event Processor]java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:261)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:223)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.allocate(SchedulerApp.java:246)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1078)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1048)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:859)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:756)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:573)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:622)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:78)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:302)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> This is very similar to the failure reported in MAPREDUCE-3005.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254887#comment-13254887 ] Hadoop QA commented on MAPREDUCE-3942: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522808/MAPREDUCE-3942-20120416.1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2229//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2229//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2229//console This message is automatically generated. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4151) RM scheduler web page should filter apps to those that are relevant to scheduling
[ https://issues.apache.org/jira/browse/MAPREDUCE-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254883#comment-13254883 ] Hadoop QA commented on MAPREDUCE-4151: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522811/MAPREDUCE-4151.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2228//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2228//console This message is automatically generated. > RM scheduler web page should filter apps to those that are relevant to > scheduling > - > > Key: MAPREDUCE-4151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4151 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, webapps >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4151.patch > > > On the ResourceManager's scheduler web page, the bottom of the page shows the > apps block. When the cluster has run a lot of applications (e.g.: 10,000+) > loading the apps table can take a long time, and that prolongs the plotting > of the queue status which is the most interesting portion of the page. > If the user is bothering to go to the scheduler page, they're probably not > interested in apps that are not affecting what the scheduler is doing (e.g.: > FINISHED, FAILED, KILLED, etc.). Having the RM filter the apps for this page > should significantly reduce the time it takes to load this page on the > client, and it also helps reduce the amount of apps the user has to sift > through when looking for the apps that are affecting what the scheduler is > doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2742) [MR-279] [Security] All tokens in YARN + MR should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-2742. Resolution: Duplicate Being fixed in parts via MAPREDUCE-3940, MAPREDUCE-3943. > [MR-279] [Security] All tokens in YARN + MR should have an expiry interval > --- > > Key: MAPREDUCE-2742 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2742 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Critical > > Right now none of the tokens and secret-keys have an expiry interval. This > needs to be fixed. > This ticket should also handle how to renew the tickets when necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4008) ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
[
https://issues.apache.org/jira/browse/MAPREDUCE-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254874#comment-13254874
]
Hudson commented on MAPREDUCE-4008:
---
Integrated in Hadoop-Common-trunk-Commit #2076 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2076/])
MAPREDUCE-4008. ResourceManager throws MetricsException on start up saying
QueueMetrics MBean already exists (Devaraj K via tgraves) (Revision 1326707)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326707
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
> ResourceManager throws MetricsException on start up saying QueueMetrics MBean
> already exists
>
>
> Key: MAPREDUCE-4008
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4008
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4008.patch
>
>
> {code:xml}
> 2012-03-14 15:22:23,089 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:117)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
> at $Proxy6.postStart(Unknown Source)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:183)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:155)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:454)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:588)
> Caused by: org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:113)
> ... 19 more
> 2012-03-14 15:22:23,090 WARN org.apache.hadoop.metrics2.util.MBeans: Failed
> to register MBean "null"
> javax.management.RuntimeOperationsException: Exception occurred trying to
> register the MBean
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:969)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
>
[jira] [Commented] (MAPREDUCE-4008) ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
[
https://issues.apache.org/jira/browse/MAPREDUCE-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254877#comment-13254877
]
Hudson commented on MAPREDUCE-4008:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #2090 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2090/])
MAPREDUCE-4008. ResourceManager throws MetricsException on start up saying
QueueMetrics MBean already exists (Devaraj K via tgraves) (Revision 1326707)
Result = FAILURE
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326707
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
> ResourceManager throws MetricsException on start up saying QueueMetrics MBean
> already exists
>
>
> Key: MAPREDUCE-4008
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4008
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4008.patch
>
>
> {code:xml}
> 2012-03-14 15:22:23,089 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:117)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
> at $Proxy6.postStart(Unknown Source)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:183)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:155)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:454)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:588)
> Caused by: org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:113)
> ... 19 more
> 2012-03-14 15:22:23,090 WARN org.apache.hadoop.metrics2.util.MBeans: Failed
> to register MBean "null"
> javax.management.RuntimeOperationsException: Exception occurred trying to
> register the MBean
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:969)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:
[jira] [Commented] (MAPREDUCE-4008) ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
[
https://issues.apache.org/jira/browse/MAPREDUCE-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254870#comment-13254870
]
Hudson commented on MAPREDUCE-4008:
---
Integrated in Hadoop-Hdfs-trunk-Commit #2149 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2149/])
MAPREDUCE-4008. ResourceManager throws MetricsException on start up saying
QueueMetrics MBean already exists (Devaraj K via tgraves) (Revision 1326707)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326707
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
> ResourceManager throws MetricsException on start up saying QueueMetrics MBean
> already exists
>
>
> Key: MAPREDUCE-4008
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4008
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4008.patch
>
>
> {code:xml}
> 2012-03-14 15:22:23,089 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:117)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
> at $Proxy6.postStart(Unknown Source)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:183)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:155)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:454)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:588)
> Caused by: org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:113)
> ... 19 more
> 2012-03-14 15:22:23,090 WARN org.apache.hadoop.metrics2.util.MBeans: Failed
> to register MBean "null"
> javax.management.RuntimeOperationsException: Exception occurred trying to
> register the MBean
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:969)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
>
[jira] [Updated] (MAPREDUCE-4008) ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
[
https://issues.apache.org/jira/browse/MAPREDUCE-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated MAPREDUCE-4008:
-
Resolution: Fixed
Fix Version/s: 2.0.0
Target Version/s: 2.0.0
Status: Resolved (was: Patch Available)
+1 thanks Devaraj! I committed this to trunk and branch-2
> ResourceManager throws MetricsException on start up saying QueueMetrics MBean
> already exists
>
>
> Key: MAPREDUCE-4008
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4008
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4008.patch
>
>
> {code:xml}
> 2012-03-14 15:22:23,089 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:117)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
> at $Proxy6.postStart(Unknown Source)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:183)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:155)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:454)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:588)
> Caused by: org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:113)
> ... 19 more
> 2012-03-14 15:22:23,090 WARN org.apache.hadoop.metrics2.util.MBeans: Failed
> to register MBean "null"
> javax.management.RuntimeOperationsException: Exception occurred trying to
> register the MBean
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:969)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Metho
[jira] [Updated] (MAPREDUCE-3943) RM-NM secret-keys should be randomly generated and rolled every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3943: --- Attachment: MAPREDUCE-3943-20120416.txt Attaching patch that implements the proposal in the description. The patch is big enough already, and deserves specific tests which I'm planning to do in follow up tickets. > RM-NM secret-keys should be randomly generated and rolled every so often > > > Key: MAPREDUCE-3943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3943 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3943-20120416.txt > > > - RM should generate the master-key randomly > - The master-key should roll every so often > - NM should remember old expired keys so that already doled out > container-requests can be satisfied. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254830#comment-13254830
]
Hadoop QA commented on MAPREDUCE-4135:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522790/MAPREDUCE-4135.branch-0.23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
org.apache.hadoop.yarn.server.TestDiskFailures
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2225//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2225//console
This message is automatically generated.
> MRAppMaster throws IllegalStateException while shutting down
>
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
> MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting
> down. It doesn't look good having this exception in the stderr file of
> MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Attachment: MAPREDUCE-3940-20120416.txt Patch that applies over latest trunk. > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt, MAPREDUCE-3940-20120416.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4151) RM scheduler web page should filter apps to those that are relevant to scheduling
[ https://issues.apache.org/jira/browse/MAPREDUCE-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4151: -- Attachment: MAPREDUCE-4151.patch Patch that limits the applications on the scheduler page to those in NEW, SUBMITTED, ACCEPTED, or RUNNING. > RM scheduler web page should filter apps to those that are relevant to > scheduling > - > > Key: MAPREDUCE-4151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4151 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, webapps >Reporter: Jason Lowe > Attachments: MAPREDUCE-4151.patch > > > On the ResourceManager's scheduler web page, the bottom of the page shows the > apps block. When the cluster has run a lot of applications (e.g.: 10,000+) > loading the apps table can take a long time, and that prolongs the plotting > of the queue status which is the most interesting portion of the page. > If the user is bothering to go to the scheduler page, they're probably not > interested in apps that are not affecting what the scheduler is doing (e.g.: > FINISHED, FAILED, KILLED, etc.). Having the RM filter the apps for this page > should significantly reduce the time it takes to load this page on the > client, and it also helps reduce the amount of apps the user has to sift > through when looking for the apps that are affecting what the scheduler is > doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4151) RM scheduler web page should filter apps to those that are relevant to scheduling
[ https://issues.apache.org/jira/browse/MAPREDUCE-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4151: -- Assignee: Jason Lowe Target Version/s: 0.23.3, 2.0.0 Status: Patch Available (was: Open) > RM scheduler web page should filter apps to those that are relevant to > scheduling > - > > Key: MAPREDUCE-4151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4151 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, webapps >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4151.patch > > > On the ResourceManager's scheduler web page, the bottom of the page shows the > apps block. When the cluster has run a lot of applications (e.g.: 10,000+) > loading the apps table can take a long time, and that prolongs the plotting > of the queue status which is the most interesting portion of the page. > If the user is bothering to go to the scheduler page, they're probably not > interested in apps that are not affecting what the scheduler is doing (e.g.: > FINISHED, FAILED, KILLED, etc.). Having the RM filter the apps for this page > should significantly reduce the time it takes to load this page on the > client, and it also helps reduce the amount of apps the user has to sift > through when looking for the apps that are affecting what the scheduler is > doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Status: Open (was: Patch Available) > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Attachment: MAPREDUCE-3942-20120416.1.txt Updated patch. bq. Is the additional synchronization in ApplicationTokenIdentifier really required ? IIRC, I added this for thread safety. Removed it for now to be consistent with other IDs. > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Status: Patch Available (was: Open) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt, MAPREDUCE-3942-20120416.1.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3942: --- Status: Open (was: Patch Available) > Randomize master key generation for ApplicationTokenSecretManager and roll it > every so often > > > Key: MAPREDUCE-3942 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3942-20120307.1.txt, > MAPREDUCE-3942-20120307.txt > > > - Master key for authentication of AMs need to be automatically generated. > - The key needs to be rolled every so often but AMs with old keys should > continue to be able to talk to the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254808#comment-13254808 ] Thomas Graves commented on MAPREDUCE-4156: -- I ran TestJobHistoryParsing and TestJobInProgress and both passed. > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254801#comment-13254801 ] Hadoop QA commented on MAPREDUCE-4156: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522788/MAPREDUCE-4156.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2226//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2226//console This message is automatically generated. > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254797#comment-13254797 ] Hadoop QA commented on MAPREDUCE-4141: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522800/MAPREDUCE-4141.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2227//console This message is automatically generated. > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4141.patch > > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-4141: Status: Patch Available (was: Open) > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4141.patch > > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated MAPREDUCE-4141: Attachment: MAPREDUCE-4141.patch need both this patch and HADOOP-8284 to address the issue, please commit both at the same time. I've tested this out with a full build and clover test. It was able to generate clover output and view in a browser fine. Fixed the issue with pb changes (itself has code gen phase) by pushing clover into process-sources phase - which makes more sense anyway. I also tested generating the distro and it gen'd that fine as well (w/o including clover jar) > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4141.patch > > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4141) clover integration broken, also mapreduce poms are pulling in clover as a dependency
[ https://issues.apache.org/jira/browse/MAPREDUCE-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt reassigned MAPREDUCE-4141: --- Assignee: Patrick Hunt > clover integration broken, also mapreduce poms are pulling in clover as a > dependency > > > Key: MAPREDUCE-4141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: trunk >Reporter: Patrick Hunt >Assignee: Patrick Hunt > > Some of the poms are specifying clover as a dependency, rather than > exclusively as a plugin. > Also I tried running with the clover profile on trunk and the build is > failing due to some issue with protobufs code generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254790#comment-13254790 ] Avner BenHanoch commented on MAPREDUCE-4049: Hi Mariappan, thanks for your comment. I just checked out trunk and started to learn Hadoop 3.0. I am looking for the the place to plug ShuffleProvider there :) My next patch will be to the trunk. However, I think it will take me few weeks to reach there. > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, task, tasktracker >Affects Versions: 1.0.3 >Reporter: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, > HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle > Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, > mapred.diff, src.tgz, test.diff > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254786#comment-13254786 ] Robert Joseph Evans commented on MAPREDUCE-4156: I am a +1 too (binding) assuming that you ran the impacted tests and they pass. > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254780#comment-13254780 ] Eric Payne commented on MAPREDUCE-4156: --- +1 (non-binding) Thanks Tom. I took a look at the patch and it looks good. > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254777#comment-13254777
]
Mariappan Asokan commented on MAPREDUCE-4049:
-
Hi Avner,
I have a minor comment:
* The {{ShuffleProviderPlugin}} and {{ShuffleConsumerPlugin}} can probably
implement {{java.io.Closeable}}. In that case you need not declare {{abstract
void close()}} method in them.
I am waiting for your patch on top of the trunk version.
Thanks.
> plugin for generic shuffle service
> --
>
> Key: MAPREDUCE-4049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task, tasktracker
>Affects Versions: 1.0.3
>Reporter: Avner BenHanoch
> Labels: merge, plugin, rdma, shuffle
> Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch,
> HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle
> Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml,
> mapred.diff, src.tgz, test.diff
>
>
> Support generic shuffle service as set of two plugins: ShuffleProvider &
> ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on
> shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
> or Infiniband) instead of using the current HTTP shuffle. Based on the fast
> RDMA shuffle, the plugin can also utilize a suitable merge approach during
> the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
> dependency of NodeManager with a specific version of mapreduce shuffle
> (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
> from Auburn University with others,
> [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins
> (currently, based on 1.0 branch)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-4135:
Attachment: MAPREDUCE-4135.branch-0.23.patch
Wait! I thought of another way.
3. We can @Override stop() in MRAppMaster to call super.stop() and then do the
FileSystem.closeAll(). Set a flag to denote that stop() has already been
called. In the MRAppMasterShutdownHook, if the flag is set, NOT call the
FileSystem.closeAll(). Sheesh! Now I feel stupid for not thinking about this
earlier :(
What do all y'all think of this way?
> MRAppMaster throws IllegalStateException while shutting down
>
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
> MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting
> down. It doesn't look good having this exception in the stderr file of
> MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in
> progress
> at
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4156: - Status: Patch Available (was: Open) > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4156: - Attachment: MAPREDUCE-4156.patch fixes compilation. > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4156.patch > > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4079: --- Attachment: MR-4079-trunk.txt MR-4079-full-branch-0.23.txt Updated to match latest patch for HADOOP-8227 > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4156) ant build fails compiling JobInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned MAPREDUCE-4156: Assignee: Thomas Graves > ant build fails compiling JobInProgress > --- > > Key: MAPREDUCE-4156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > > The ant build fails trying to compile jobInProgress. Looks like > TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we > probably need to update the old mrv1 source so it atleast compiles. > compile-mapred-classes: > [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache > (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File > /WEB-INF/web.xml not found > [javac] Compiling 86 source files to > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes > [javac] > /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: > cannot find symbol > [javac] symbol : constructor > TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) > [javac] location: class > org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent > [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4156) ant build fails compiling JobInProgress
ant build fails compiling JobInProgress --- Key: MAPREDUCE-4156 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4156 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.3 Reporter: Thomas Graves The ant build fails trying to compile jobInProgress. Looks like TaskFinishedEvent has a new parameter (added in MAPREDUCE-4128) so we probably need to update the old mrv1 source so it atleast compiles. compile-mapred-classes: [jsp-compile] 2012-04-13 08:43:33,175 WARN [main] compiler.TldLocationsCache (TldLocationsCache.java:processWebDotXml(284)) - Internal Error: File /WEB-INF/web.xml not found [javac] Compiling 86 source files to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/build/classes [javac] /home/y/var/builds/thread2/workspace/Cloud-Hadoop-All-0.23.3-Secondary/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:2712: cannot find symbol [javac] symbol : constructor TaskFinishedEvent(org.apache.hadoop.mapred.TaskID,long,org.apache.hadoop.mapreduce.TaskType,java.lang.String,org.apache.hadoop.mapreduce.Counters) [javac] location: class org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent [javac] TaskFinishedEvent tfe = new TaskFinishedEvent(tip.getTIPId(), [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 1 error -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4149) Rumen fails to parse certain counter strings
[
https://issues.apache.org/jira/browse/MAPREDUCE-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254647#comment-13254647
]
Amar Kamat commented on MAPREDUCE-4149:
---
+1 for the branch-1 patch.
> Rumen fails to parse certain counter strings
>
>
> Key: MAPREDUCE-4149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4149
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Reporter: Ravi Gummadi
>Assignee: Ravi Gummadi
> Attachments: 4149.branch-1.v1.patch, 4149.branch-1.v2.patch,
> 4149.patch
>
>
> If a counter name contains "{" or "}", Rumen is not able to parse it and
> throws ParseException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254620#comment-13254620 ] Hadoop QA commented on MAPREDUCE-4049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522757/HADOOP-1.0.x.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2224//console This message is automatically generated. > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, task, tasktracker >Affects Versions: 1.0.3 >Reporter: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, > HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle > Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, > mapred.diff, src.tgz, test.diff > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4155) Provide support for these yarn commands
Provide support for these yarn commands --- Key: MAPREDUCE-4155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0 Reporter: Devaraj K Assignee: Devaraj K 1. status 2. kill (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 1.0.3 (was: 1.0.3, 0.23.3, 0.23.2, 1.1.0, 0.24.0) Affects Version/s: (was: 1.0.1) (was: 0.23.1) 1.0.3 Status: Patch Available (was: Open) > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, task, tasktracker >Affects Versions: 1.0.3 >Reporter: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, > HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle > Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, > mapred.diff, src.tgz, test.diff > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Attachment: HADOOP-1.0.x.patch attached patch patch for the live 1.0.x branch (http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0) > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, task, tasktracker >Affects Versions: 0.23.1, 1.0.1 >Reporter: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, > HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle > Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, > mapred.diff, src.tgz, test.diff > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 0.23.2, 1.1.0, 0.24.0) Status: Open (was: Patch Available) > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, task, tasktracker >Affects Versions: 1.0.1, 0.23.1 >Reporter: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle > Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, > MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, > test.diff > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4149) Rumen fails to parse certain counter strings
[
https://issues.apache.org/jira/browse/MAPREDUCE-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-4149:
Attachment: 4149.branch-1.v2.patch
Attaching updated patch for branch-1
> Rumen fails to parse certain counter strings
>
>
> Key: MAPREDUCE-4149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4149
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Reporter: Ravi Gummadi
>Assignee: Ravi Gummadi
> Attachments: 4149.branch-1.v1.patch, 4149.branch-1.v2.patch,
> 4149.patch
>
>
> If a counter name contains "{" or "}", Rumen is not able to parse it and
> throws ParseException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
