[jira] [Created] (MAPREDUCE-4174) Clean up hadoop-mapreduce-client-common
Harsh J created MAPREDUCE-4174: -- Summary: Clean up hadoop-mapreduce-client-common Key: MAPREDUCE-4174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4174 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4175) Clean up hadoop-mapreduce-client-core
Harsh J created MAPREDUCE-4175: -- Summary: Clean up hadoop-mapreduce-client-core Key: MAPREDUCE-4175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-core module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reassigned MAPREDUCE-4173: -- Assignee: Harsh J Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4173: --- Target Version/s: trunk (was: 3.0.0) Affects Version/s: (was: 3.0.0) trunk Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4173: --- Attachment: 0001-MR-Client-App-Cleanup.patch Complete cleanup Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0001-MR-Client-App-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4173: --- Status: Patch Available (was: Open) Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0001-MR-Client-App-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4174) Clean up hadoop-mapreduce-client-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4174: --- Attachment: 0002-MR-Client-Common-Cleanup.patch Complete cleanup (but some proto-generated files carry some warnings which I can't do anything about - will probably get fixed auto in a protobuf update) Clean up hadoop-mapreduce-client-common --- Key: MAPREDUCE-4174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4174 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0002-MR-Client-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4174) Clean up hadoop-mapreduce-client-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4174: --- Status: Patch Available (was: Open) Clean up hadoop-mapreduce-client-common --- Key: MAPREDUCE-4174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4174 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0002-MR-Client-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4175) Clean up hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4175: --- Attachment: 0003-MapReduce-Client-Core-Cleanup.patch Almost a complete clean up. I didn't touch JobConf (There are too many changes here, I'll do it later as another subtask), JobClient (Couldn't yet figure out how to switch to Token#renew from existing deprecated calls) and a couple of non-inheritance warnings in some client API packages that need a ticket of their own. I'll create their tickets after this is reviewed (when I revisit). Clean up hadoop-mapreduce-client-core - Key: MAPREDUCE-4175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0003-MapReduce-Client-Core-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-core module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4175) Clean up hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4175: --- Status: Patch Available (was: Open) Clean up hadoop-mapreduce-client-core - Key: MAPREDUCE-4175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0003-MapReduce-Client-Core-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-core module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4172) Clean up java warnings in the hadoop-mapreduce-project sub projects
[ https://issues.apache.org/jira/browse/MAPREDUCE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4172: --- Target Version/s: trunk (was: 3.0.0) Affects Version/s: (was: 3.0.0) trunk Clean up java warnings in the hadoop-mapreduce-project sub projects --- Key: MAPREDUCE-4172 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4172 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J There are lots of warnings (unused imports, deprecation warnings where an alternative isn't clear, unchecked calls, raw types, etc.) in the hadoop-mapreduce-project presently. We can clear almost all of this away. I'm going to open one sub-task per sub-project we have, with patches attached to them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4174) Clean up hadoop-mapreduce-client-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259037#comment-13259037 ] Hadoop QA commented on MAPREDUCE-4174: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523669/0002-MR-Client-Common-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2272//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2272//console This message is automatically generated. Clean up hadoop-mapreduce-client-common --- Key: MAPREDUCE-4174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4174 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0002-MR-Client-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4175) Clean up hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259038#comment-13259038 ] Hadoop QA commented on MAPREDUCE-4175: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523670/0003-MapReduce-Client-Core-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 4 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2271//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2271//console This message is automatically generated. Clean up hadoop-mapreduce-client-core - Key: MAPREDUCE-4175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0003-MapReduce-Client-Core-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-core module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259057#comment-13259057 ] Avner BenHanoch commented on MAPREDUCE-4049: Hi Mariappan, thanks for your thorough comments. It will take me time to provide you full answer (hopefully by the end of next week). Hence, I'll start with something short and complete it later: 1. In some cases it make sense to use consumer provider plugins as pair (for example, I want the Shuffle to be over RDMA instead of HTTP; hence, I must provide both sides). However, it is not mandatory. The interface (and configuration) allow any combinations of plugins, including one from 3rd party and one from vanilla (the default). 23. It could be that you are right. At this phase I am not 100% sure about that (please allow me additional time). My merge doesn't work on shuffled segments, but on streamed of segments during streaming (perhaps it is described in the technical paper). Hence, I need to check deeper. I hope to give you complete answer by the end of next week. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.3 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Affects Version/s: 3.0.0 2.0.0 1.1.0 plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.1.0, 1.0.3, 2.0.0, 3.0.0 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4176) Clean up hadoop-mapreduce-client-hs
Harsh J created MAPREDUCE-4176: -- Summary: Clean up hadoop-mapreduce-client-hs Key: MAPREDUCE-4176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4176 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-hs module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4177) Clean up hadoop-mapreduce-client-jobclient
Harsh J created MAPREDUCE-4177: -- Summary: Clean up hadoop-mapreduce-client-jobclient Key: MAPREDUCE-4177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4177 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-jobclient module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4172) Clean up java warnings in the hadoop-mapreduce-project sub projects
[ https://issues.apache.org/jira/browse/MAPREDUCE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4172: --- Description: There are lots of warnings (unused imports, unused variables, for loops that can be replaced with while instead to save a variable, unused methods, deprecation warnings where an alternative isn't clear, unchecked calls, raw types, etc.) in the hadoop-mapreduce-project presently. We can clear almost all of this away. I'm going to open one sub-task per sub-project we have, with patches attached to them. was: There are lots of warnings (unused imports, deprecation warnings where an alternative isn't clear, unchecked calls, raw types, etc.) in the hadoop-mapreduce-project presently. We can clear almost all of this away. I'm going to open one sub-task per sub-project we have, with patches attached to them. Clean up java warnings in the hadoop-mapreduce-project sub projects --- Key: MAPREDUCE-4172 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4172 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J There are lots of warnings (unused imports, unused variables, for loops that can be replaced with while instead to save a variable, unused methods, deprecation warnings where an alternative isn't clear, unchecked calls, raw types, etc.) in the hadoop-mapreduce-project presently. We can clear almost all of this away. I'm going to open one sub-task per sub-project we have, with patches attached to them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4172) Clean up java warnings in the hadoop-mapreduce-project sub projects
[ https://issues.apache.org/jira/browse/MAPREDUCE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4172: --- Description: There are lots of warnings in the hadoop-mapreduce-project presently. We can clear almost all of this away: * Unused imports * Unused variables ** For loops that can be replaced with while instead to save an unused variable * Unused methods * Deprecation warnings where an alternative can be used (Especially SequenceFile reader/writer usage and MiniDFSCluster usage) * Deprecation warnings where an alternative isn't clear (Especially MiniMRCluster usage and DistributedCache API usage where a Job object may not be available) * Unchecked conversions * Raw type usage * (etc.) I'm going to open one sub-task per sub-project we have, with patches attached to them. was: There are lots of warnings (unused imports, unused variables, for loops that can be replaced with while instead to save a variable, unused methods, deprecation warnings where an alternative isn't clear, unchecked calls, raw types, etc.) in the hadoop-mapreduce-project presently. We can clear almost all of this away. I'm going to open one sub-task per sub-project we have, with patches attached to them. Clean up java warnings in the hadoop-mapreduce-project sub projects --- Key: MAPREDUCE-4172 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4172 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J There are lots of warnings in the hadoop-mapreduce-project presently. We can clear almost all of this away: * Unused imports * Unused variables ** For loops that can be replaced with while instead to save an unused variable * Unused methods * Deprecation warnings where an alternative can be used (Especially SequenceFile reader/writer usage and MiniDFSCluster usage) * Deprecation warnings where an alternative isn't clear (Especially MiniMRCluster usage and DistributedCache API usage where a Job object may not be available) * Unchecked conversions * Raw type usage * (etc.) I'm going to open one sub-task per sub-project we have, with patches attached to them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4179) Clean up hadoop-mapreduce-examples
Harsh J created MAPREDUCE-4179: -- Summary: Clean up hadoop-mapreduce-examples Key: MAPREDUCE-4179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4179 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-mapreduce-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4178) Clean up hadoop-rumen
Harsh J created MAPREDUCE-4178: -- Summary: Clean up hadoop-rumen Key: MAPREDUCE-4178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4178 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4180) Clean up hadoop-streaming
Harsh J created MAPREDUCE-4180: -- Summary: Clean up hadoop-streaming Key: MAPREDUCE-4180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4180 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4176) Clean up hadoop-mapreduce-client-hs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4176: --- Attachment: 0004-MapReduce-Client-HistoryServer-Cleanup.patch This patch should take care of it. Clean up hadoop-mapreduce-client-hs --- Key: MAPREDUCE-4176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4176 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0004-MapReduce-Client-HistoryServer-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-hs module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4176) Clean up hadoop-mapreduce-client-hs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4176: --- Status: Patch Available (was: Open) Clean up hadoop-mapreduce-client-hs --- Key: MAPREDUCE-4176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4176 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0004-MapReduce-Client-HistoryServer-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-hs module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4177) Clean up hadoop-mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4177: --- Attachment: 0005-MapReduce-Client-JobClient-Cleanup.patch Attached patch cleans up almost all of the code and tests save for some MiniMRCluster and DistributedCache references for which its best to revisit and open a new JIRA (will do upon parent completion). Clean up hadoop-mapreduce-client-jobclient -- Key: MAPREDUCE-4177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4177 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0005-MapReduce-Client-JobClient-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-jobclient module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4177) Clean up hadoop-mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4177: --- Status: Patch Available (was: Open) Clean up hadoop-mapreduce-client-jobclient -- Key: MAPREDUCE-4177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4177 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0005-MapReduce-Client-JobClient-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-jobclient module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4178) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4178: --- Status: Patch Available (was: Open) Clean up hadoop-rumen - Key: MAPREDUCE-4178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4178 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4179) Clean up hadoop-mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4179: --- Status: Patch Available (was: Open) Clean up hadoop-mapreduce-examples -- Key: MAPREDUCE-4179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4179 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0006-MapReduce-Examples-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4179) Clean up hadoop-mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4179: --- Attachment: 0006-MapReduce-Examples-Cleanup.patch Attached patch should clean up entire project. Clean up hadoop-mapreduce-examples -- Key: MAPREDUCE-4179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4179 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0006-MapReduce-Examples-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4180) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4180: --- Attachment: 0008-MapReduce-Streaming-Cleanup.patch Attached patch should clean up everything save for MiniMRCluster usage (see parent). Clean up hadoop-streaming - Key: MAPREDUCE-4180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4180 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4178) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4178: --- Attachment: 0007-MapReduce-Rumen-Cleanup.patch Attached patch should address almost everything. Clean up hadoop-rumen - Key: MAPREDUCE-4178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4178 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4180) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4180: --- Status: Patch Available (was: Open) Clean up hadoop-streaming - Key: MAPREDUCE-4180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4180 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4178) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259175#comment-13259175 ] Hadoop QA commented on MAPREDUCE-4178: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523693/0007-MapReduce-Rumen-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2276//console This message is automatically generated. Clean up hadoop-rumen - Key: MAPREDUCE-4178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4178 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4176) Clean up hadoop-mapreduce-client-hs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259187#comment-13259187 ] Hadoop QA commented on MAPREDUCE-4176: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523691/0004-MapReduce-Client-HistoryServer-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2275//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2275//console This message is automatically generated. Clean up hadoop-mapreduce-client-hs --- Key: MAPREDUCE-4176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4176 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0004-MapReduce-Client-HistoryServer-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-hs module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4177) Clean up hadoop-mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259188#comment-13259188 ] Hadoop QA commented on MAPREDUCE-4177: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523692/0005-MapReduce-Client-JobClient-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 130 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestMapRed org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2274//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2274//console This message is automatically generated. Clean up hadoop-mapreduce-client-jobclient -- Key: MAPREDUCE-4177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4177 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0005-MapReduce-Client-JobClient-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-jobclient module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4178) Clean up hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4178: --- Resolution: Duplicate Status: Resolved (was: Patch Available) The above QA run failed cause this has to be on the common-side. Opened HADOOP-8302 with this patch instead. Resolving as duplicate. Clean up hadoop-rumen - Key: MAPREDUCE-4178 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4178 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Attachments: 0007-MapReduce-Rumen-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-rumen module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4179) Clean up hadoop-mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259191#comment-13259191 ] Hadoop QA commented on MAPREDUCE-4179: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523694/0006-MapReduce-Examples-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2278//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2278//console This message is automatically generated. Clean up hadoop-mapreduce-examples -- Key: MAPREDUCE-4179 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4179 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0006-MapReduce-Examples-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4180) Clean up hadoop-streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4180: --- Resolution: Duplicate Status: Resolved (was: Patch Available) This needs to be on Hadoop Common for QA Bot to run it successfully. Hence, I've opened HADOOP-8303 with the same patch moved to that project. Resolving as duplicate. Clean up hadoop-streaming - Key: MAPREDUCE-4180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4180 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0008-MapReduce-Streaming-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-streaming module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4181) Clean up yarn-api
Harsh J created MAPREDUCE-4181: -- Summary: Clean up yarn-api Key: MAPREDUCE-4181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4181 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-api module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4183) Clean up yarn-common
Harsh J created MAPREDUCE-4183: -- Summary: Clean up yarn-common Key: MAPREDUCE-4183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4183 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4182) Clean up yarn-applications-distributedshell
Harsh J created MAPREDUCE-4182: -- Summary: Clean up yarn-applications-distributedshell Key: MAPREDUCE-4182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4182 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-applications-distributedshell module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4184) Clean up yarn-server-common
Harsh J created MAPREDUCE-4184: -- Summary: Clean up yarn-server-common Key: MAPREDUCE-4184 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4184 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-server-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4185) Clean up yarn-server-nodemanager
Harsh J created MAPREDUCE-4185: -- Summary: Clean up yarn-server-nodemanager Key: MAPREDUCE-4185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4185 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-server-nodemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4181) Clean up yarn-api
[ https://issues.apache.org/jira/browse/MAPREDUCE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4181: --- Status: Patch Available (was: Open) Clean up yarn-api - Key: MAPREDUCE-4181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4181 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0009-YARN-API-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-api module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4182) Clean up yarn-applications-distributedshell
[ https://issues.apache.org/jira/browse/MAPREDUCE-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4182: --- Attachment: 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch Minor patch that removes some of the warnings left in DistributedShell app example. Clean up yarn-applications-distributedshell --- Key: MAPREDUCE-4182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4182 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-applications-distributedshell module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4182) Clean up yarn-applications-distributedshell
[ https://issues.apache.org/jira/browse/MAPREDUCE-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4182: --- Status: Patch Available (was: Open) Clean up yarn-applications-distributedshell --- Key: MAPREDUCE-4182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4182 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-applications-distributedshell module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4181) Clean up yarn-api
[ https://issues.apache.org/jira/browse/MAPREDUCE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4181: --- Attachment: 0009-YARN-API-Cleanup.patch Patch that cleans up all the warnings. Some warnings still may exist in generated sources but I can't do anything about those - perhaps a protobuf update may solve it automatically. Clean up yarn-api - Key: MAPREDUCE-4181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4181 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0009-YARN-API-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-api module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4183) Clean up yarn-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4183: --- Attachment: 0011-YARN-Common-Cleanup.patch Patch that cleans up all warnings from hadoop-yarn-common Clean up yarn-common Key: MAPREDUCE-4183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4183 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0011-YARN-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4184) Clean up yarn-server-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4184: --- Attachment: 0012-YARN-Server-Common-Cleanup.patch Patch that cleans up all warnings from hadoop-yarn-server-common. Clean up yarn-server-common --- Key: MAPREDUCE-4184 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4184 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0012-YARN-Server-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4185) Clean up yarn-server-nodemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4185: --- Attachment: 0013-YARN-Server-NodeManager-Cleanup.patch Javac warning cleanup patch for the hadoop-yarn-server-nodemanager project. Clean up yarn-server-nodemanager Key: MAPREDUCE-4185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4185 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0013-YARN-Server-NodeManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-nodemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4185) Clean up yarn-server-nodemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4185: --- Status: Patch Available (was: Open) Clean up yarn-server-nodemanager Key: MAPREDUCE-4185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4185 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0013-YARN-Server-NodeManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-nodemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4186) Clean up yarn-server-resourcemanager
Harsh J created MAPREDUCE-4186: -- Summary: Clean up yarn-server-resourcemanager Key: MAPREDUCE-4186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4186 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-server-resourcemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4187) Clean up yarn-server-tests
Harsh J created MAPREDUCE-4187: -- Summary: Clean up yarn-server-tests Key: MAPREDUCE-4187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4187 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-server-tests module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4186) Clean up yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4186: --- Status: Patch Available (was: Open) Patch that cleans up almost all the javac warnings. Clean up yarn-server-resourcemanager Key: MAPREDUCE-4186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4186 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0014-YARN-Server-ResourceManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-resourcemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4186) Clean up yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4186: --- Attachment: 0014-YARN-Server-ResourceManager-Cleanup.patch Clean up yarn-server-resourcemanager Key: MAPREDUCE-4186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4186 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0014-YARN-Server-ResourceManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-resourcemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4188) Clean up yarn-server-web-proxy
Harsh J created MAPREDUCE-4188: -- Summary: Clean up yarn-server-web-proxy Key: MAPREDUCE-4188 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4188 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Clean up a bunch of existing javac warnings in hadoop-yarn-server-web-proxy module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4172) Clean up java warnings in the hadoop-mapreduce-project sub projects
[ https://issues.apache.org/jira/browse/MAPREDUCE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259207#comment-13259207 ] Harsh J commented on MAPREDUCE-4172: The only thing automated about these patches is the removal of unused imports (Eclipse code cleaner helped there). All the other changes were hand-done. Clean up java warnings in the hadoop-mapreduce-project sub projects --- Key: MAPREDUCE-4172 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4172 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J There are lots of warnings in the hadoop-mapreduce-project presently. We can clear almost all of this away: * Unused imports * Unused variables ** For loops that can be replaced with while instead to save an unused variable * Unused methods * Deprecation warnings where an alternative can be used (Especially SequenceFile reader/writer usage and MiniDFSCluster usage) * Deprecation warnings where an alternative isn't clear (Especially MiniMRCluster usage and DistributedCache API usage where a Job object may not be available) * Unchecked conversions * Raw type usage * (etc.) I'm going to open one sub-task per sub-project we have, with patches attached to them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4188) Clean up yarn-server-web-proxy
[ https://issues.apache.org/jira/browse/MAPREDUCE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4188: --- Status: Patch Available (was: Open) Clean up yarn-server-web-proxy -- Key: MAPREDUCE-4188 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4188 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0016-YARN-Server-Web-Proxy-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-web-proxy module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4187) Clean up yarn-server-tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4187: --- Status: Patch Available (was: Open) Clean up yarn-server-tests -- Key: MAPREDUCE-4187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4187 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0015-YARN-Server-Tests-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-tests module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4188) Clean up yarn-server-web-proxy
[ https://issues.apache.org/jira/browse/MAPREDUCE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4188: --- Attachment: 0016-YARN-Server-Web-Proxy-Cleanup.patch Minor patch that cleans up javac warnings from yarn-server-web-proxy submodule. Clean up yarn-server-web-proxy -- Key: MAPREDUCE-4188 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4188 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0016-YARN-Server-Web-Proxy-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-web-proxy module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4187) Clean up yarn-server-tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4187: --- Attachment: 0015-YARN-Server-Tests-Cleanup.patch Patch that cleans up the tests submodule of yarn-server. Clean up yarn-server-tests -- Key: MAPREDUCE-4187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4187 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0015-YARN-Server-Tests-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-tests module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4181) Clean up yarn-api
[ https://issues.apache.org/jira/browse/MAPREDUCE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259214#comment-13259214 ] Hadoop QA commented on MAPREDUCE-4181: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523702/0009-YARN-API-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2282//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2282//console This message is automatically generated. Clean up yarn-api - Key: MAPREDUCE-4181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4181 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0009-YARN-API-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-api module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4184) Clean up yarn-server-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259215#comment-13259215 ] Hadoop QA commented on MAPREDUCE-4184: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523705/0012-YARN-Server-Common-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2279//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2279//console This message is automatically generated. Clean up yarn-server-common --- Key: MAPREDUCE-4184 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4184 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0012-YARN-Server-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4188) Clean up yarn-server-web-proxy
[ https://issues.apache.org/jira/browse/MAPREDUCE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259216#comment-13259216 ] Hadoop QA commented on MAPREDUCE-4188: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523709/0016-YARN-Server-Web-Proxy-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 493 javac compiler warnings (more than the trunk's current 492 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2283//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2283//console This message is automatically generated. Clean up yarn-server-web-proxy -- Key: MAPREDUCE-4188 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4188 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0016-YARN-Server-Web-Proxy-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-web-proxy module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4188) Clean up yarn-server-web-proxy
[
https://issues.apache.org/jira/browse/MAPREDUCE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259219#comment-13259219
]
Harsh J commented on MAPREDUCE-4188:
The only change here is:
{code}
---
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
+++
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
@@ -144,7 +144,6 @@ public class WebAppProxyServlet extends HttpServlet {
config.setLocalAddress(localAddress);
HttpMethod method = new GetMethod(uri.getEscapedURI());
-@SuppressWarnings(unchecked)
EnumerationString names = req.getHeaderNames();
while(names.hasMoreElements()) {
String name = names.nextElement();
{code}
And with my local JDK 1.6 (and Eclipse 3.7.x), the warning ought to reduce. I'm
not sure why we need an unchecked suppressor there, as the signature for
{{req.getHeaderNames()}} is typed in itself:
{{EnumerationString javax.servlet.http.HttpServletRequest.getHeaderNames()}}
Clean up yarn-server-web-proxy
--
Key: MAPREDUCE-4188
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4188
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0016-YARN-Server-Web-Proxy-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-yarn-server-web-proxy
module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4183) Clean up yarn-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259226#comment-13259226 ] Hadoop QA commented on MAPREDUCE-4183: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523704/0011-YARN-Common-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServer org.apache.hadoop.yarn.server.TestDiskFailures org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.yarn.webapp.TestWebApp org.apache.hadoop.yarn.TestRpcFactoryProvider org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.security.TestJHSSecurity org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2280//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2280//console This message is automatically generated. Clean up yarn-common Key: MAPREDUCE-4183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4183 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0011-YARN-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4187) Clean up yarn-server-tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259242#comment-13259242 ] Hadoop QA commented on MAPREDUCE-4187: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523708/0015-YARN-Server-Tests-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2284//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2284//console This message is automatically generated. Clean up yarn-server-tests -- Key: MAPREDUCE-4187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4187 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0015-YARN-Server-Tests-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-tests module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4174) Clean up hadoop-mapreduce-client-common
[
https://issues.apache.org/jira/browse/MAPREDUCE-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259258#comment-13259258
]
Harsh J commented on MAPREDUCE-4174:
The {{org.apache.hadoop.yarn.server.TestContainerManagerSecurity}} failure is
unrelated to these changes.
Clean up hadoop-mapreduce-client-common
---
Key: MAPREDUCE-4174
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4174
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0002-MR-Client-Common-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-common
module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4173: --- Attachment: 0001-MR-Client-App-Cleanup.patch Resubmitting the same patch to get QA Bot to pick it up as it looks like it didn't. Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0001-MR-Client-App-Cleanup.patch, 0001-MR-Client-App-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4183) Clean up yarn-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4183: --- Attachment: 0011-YARN-Common-Cleanup.patch A bad cast from my changes in the reflection areas of some factories and the Router class caused most of these failures. Those bad casts are fixed in this patch now. Clean up yarn-common Key: MAPREDUCE-4183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4183 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0011-YARN-Common-Cleanup.patch, 0011-YARN-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4186) Clean up yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259259#comment-13259259 ] Hadoop QA commented on MAPREDUCE-4186: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523707/0014-YARN-Server-ResourceManager-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2285//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2285//console This message is automatically generated. Clean up yarn-server-resourcemanager Key: MAPREDUCE-4186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4186 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0014-YARN-Server-ResourceManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-resourcemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4175) Clean up hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4175: --- Attachment: 0003-MapReduce-Client-Core-Cleanup.patch The org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter failure was cause of MapFileOutputFormat changes. I ported the MapFile Writer API to the new Options format but it had a small issue in it (key class matched the value class). This issue has been resolved in this new patch. Clean up hadoop-mapreduce-client-core - Key: MAPREDUCE-4175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0003-MapReduce-Client-Core-Cleanup.patch, 0003-MapReduce-Client-Core-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-core module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4175) Clean up hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259265#comment-13259265 ] Harsh J commented on MAPREDUCE-4175: (Rest of the failures are not related to patch, and are appearing intermittently over all tests, save for org.apache.hadoop.yarn.server.TestContainerManagerSecurity which is appearing always) Clean up hadoop-mapreduce-client-core - Key: MAPREDUCE-4175 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4175 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0003-MapReduce-Client-Core-Cleanup.patch, 0003-MapReduce-Client-Core-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-core module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4176) Clean up hadoop-mapreduce-client-hs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259266#comment-13259266
]
Harsh J commented on MAPREDUCE-4176:
The failure on {{org.apache.hadoop.yarn.server.TestContainerManagerSecurity}}
is unrelated to this patch.
Clean up hadoop-mapreduce-client-hs
---
Key: MAPREDUCE-4176
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4176
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0004-MapReduce-Client-HistoryServer-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-hs
module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4186) Clean up yarn-server-resourcemanager
[
https://issues.apache.org/jira/browse/MAPREDUCE-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259282#comment-13259282
]
Harsh J commented on MAPREDUCE-4186:
Failing test {{org.apache.hadoop.yarn.server.TestContainerManagerSecurity}}
isn't related to this patch.
Clean up yarn-server-resourcemanager
Key: MAPREDUCE-4186
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4186
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0014-YARN-Server-ResourceManager-Cleanup.patch
Clean up a bunch of existing javac warnings in
hadoop-yarn-server-resourcemanager module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4187) Clean up yarn-server-tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259281#comment-13259281 ] Harsh J commented on MAPREDUCE-4187: None of these failures are related to this patch. This patch merely removes one unused import under the tests module. Clean up yarn-server-tests -- Key: MAPREDUCE-4187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4187 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0015-YARN-Server-Tests-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-tests module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4183) Clean up yarn-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259286#comment-13259286 ] Hadoop QA commented on MAPREDUCE-4183: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523716/0011-YARN-Common-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2288//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2288//console This message is automatically generated. Clean up yarn-common Key: MAPREDUCE-4183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4183 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0011-YARN-Common-Cleanup.patch, 0011-YARN-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259285#comment-13259285 ] Hadoop QA commented on MAPREDUCE-4173: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523717/0001-MR-Client-App-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2287//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2287//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2287//console This message is automatically generated. Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0001-MR-Client-App-Cleanup.patch, 0001-MR-Client-App-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4173) Clean up hadoop-mapreduce-client-app
[ https://issues.apache.org/jira/browse/MAPREDUCE-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259290#comment-13259290 ] Harsh J commented on MAPREDUCE-4173: These failing tests are unrelated to the changes introduced by the patch. Clean up hadoop-mapreduce-client-app Key: MAPREDUCE-4173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4173 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0001-MR-Client-App-Cleanup.patch, 0001-MR-Client-App-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-mapreduce-client-app module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4183) Clean up yarn-common
[
https://issues.apache.org/jira/browse/MAPREDUCE-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259291#comment-13259291
]
Harsh J commented on MAPREDUCE-4183:
{{org.apache.hadoop.yarn.server.TestContainerManagerSecurity}} failure appears
unrelated to this clean up patch.
Clean up yarn-common
Key: MAPREDUCE-4183
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4183
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0011-YARN-Common-Cleanup.patch,
0011-YARN-Common-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-yarn-common module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4177) Clean up hadoop-mapreduce-client-jobclient
[
https://issues.apache.org/jira/browse/MAPREDUCE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-4177:
---
Attachment: 0005-MapReduce-Client-JobClient-Cleanup.patch
The relevant test failures here are:
{{org.apache.hadoop.mapred.TestMapRed}}
This failed cause I fixed the reduce() signature in MyReducer (which extended
IdentityReducer). Doing so activated it to run in real. All these months
(years?) its been running just the identity signature cause the provided
implementation had no @Override. The method in reduce() also was old - from a
time back when SequenceFile was used as the intermediate format (it tests if
the sequence file is compressed if map-output compression is on). I got rid of
this check as there are checks elsewhere that guarantee this now. The reducer
now used is purely identity and hence the test shall no longer fail.
{{org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat}}
This was cause one of the DummyRecordReader constructors was removed (Cause it
was marked as unused by javac). I added it back and test passes again, so I
threw in a comment on why we should keep it and suppressed the unused warning.
Other test failures are irrelevant to the patch and have been failing
intermittently across all MAPREDUCE-4172 (and others) QA builds.
Newly attached patch should fix all the real issues that arose here (as
described above).
Clean up hadoop-mapreduce-client-jobclient
--
Key: MAPREDUCE-4177
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4177
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0005-MapReduce-Client-JobClient-Cleanup.patch,
0005-MapReduce-Client-JobClient-Cleanup.patch
Clean up a bunch of existing javac warnings in
hadoop-mapreduce-client-jobclient module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4179) Clean up hadoop-mapreduce-examples
[
https://issues.apache.org/jira/browse/MAPREDUCE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259371#comment-13259371
]
Harsh J commented on MAPREDUCE-4179:
The failing test {{org.apache.hadoop.yarn.server.TestContainerManagerSecurity}}
is unrelated to the changes in this patch. This patch only fixes javac warnings
of several kinds in the hadoop-mapreduce-examples project.
Clean up hadoop-mapreduce-examples
--
Key: MAPREDUCE-4179
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4179
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0006-MapReduce-Examples-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-mapreduce-examples
module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4182) Clean up yarn-applications-distributedshell
[ https://issues.apache.org/jira/browse/MAPREDUCE-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4182: --- Attachment: 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch Re-uploading patch as previous didn't trigger a QA build for some reason. Patch is ditto as previous. Clean up yarn-applications-distributedshell --- Key: MAPREDUCE-4182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4182 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch, 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-applications-distributedshell module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4181) Clean up yarn-api
[
https://issues.apache.org/jira/browse/MAPREDUCE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259373#comment-13259373
]
Harsh J commented on MAPREDUCE-4181:
The failure of {{org.apache.hadoop.yarn.server.TestContainerManagerSecurity}}
is unrelated to this patch. Its been failing for a while across all builds.
Clean up yarn-api
-
Key: MAPREDUCE-4181
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4181
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0009-YARN-API-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-yarn-api module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4184) Clean up yarn-server-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259376#comment-13259376 ] Harsh J commented on MAPREDUCE-4184: I checked them each carefully and these failing tests are all from an intermittent set of failures that are unrelated to the changes introduced in this patch for yarn-server-common. Clean up yarn-server-common --- Key: MAPREDUCE-4184 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4184 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0012-YARN-Server-Common-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-common module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4185) Clean up yarn-server-nodemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4185: --- Attachment: 0013-YARN-Server-NodeManager-Cleanup.patch Re-uploading a ditto patch as it looks like the QA bot didn't pick it up yet. Clean up yarn-server-nodemanager Key: MAPREDUCE-4185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4185 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0013-YARN-Server-NodeManager-Cleanup.patch, 0013-YARN-Server-NodeManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-nodemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4188) Clean up yarn-server-web-proxy
[
https://issues.apache.org/jira/browse/MAPREDUCE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259378#comment-13259378
]
Harsh J commented on MAPREDUCE-4188:
The above comment is for this message:
{{-1 javac. The applied patch generated 493 javac compiler warnings (more than
the trunk's current 492 warnings).}}
[Perhaps the build machines are using an old version of the http jar that has a
different signature?)
Clean up yarn-server-web-proxy
--
Key: MAPREDUCE-4188
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4188
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: build
Affects Versions: trunk
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Attachments: 0016-YARN-Server-Web-Proxy-Cleanup.patch
Clean up a bunch of existing javac warnings in hadoop-yarn-server-web-proxy
module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4185) Clean up yarn-server-nodemanager
[ https://issues.apache.org/jira/browse/MAPREDUCE-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259397#comment-13259397 ] Hadoop QA commented on MAPREDUCE-4185: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523749/0013-YARN-Server-NodeManager-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.TestDeletionService org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2290//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2290//console This message is automatically generated. Clean up yarn-server-nodemanager Key: MAPREDUCE-4185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4185 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0013-YARN-Server-NodeManager-Cleanup.patch, 0013-YARN-Server-NodeManager-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-server-nodemanager module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4182) Clean up yarn-applications-distributedshell
[ https://issues.apache.org/jira/browse/MAPREDUCE-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259398#comment-13259398 ] Hadoop QA commented on MAPREDUCE-4182: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523748/0010-YARN-Applications-DistributedShell-Example-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2292//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2292//console This message is automatically generated. Clean up yarn-applications-distributedshell --- Key: MAPREDUCE-4182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4182 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: trunk Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch, 0010-YARN-Applications-DistributedShell-Example-Cleanup.patch Clean up a bunch of existing javac warnings in hadoop-yarn-applications-distributedshell module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-278) Proposal for redesign/refactoring of the JobTracker and TaskTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated MAPREDUCE-278:
Attachment: TestUTF8AndStringGetBytes.java
Proposal for redesign/refactoring of the JobTracker and TaskTracker
---
Key: MAPREDUCE-278
URL: https://issues.apache.org/jira/browse/MAPREDUCE-278
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Sharad Agarwal
Attachments: Job_Tracker_FSM.pdf, TestUTF8AndStringGetBytes.java,
mapred_as_dfa.patch
During discussions on HADOOP-815 wrt some hard-to-maintain code on the
JobTracker we all agreed that the current state-of-affairs there is brittle
and merits some rework.
Case in point: there are back-calls from TaskInProgress to JobTracker and
from JobInProgress to JobTracker which mean that synchronization is quite
involved and brittle, leading to issues like HADOOP-600. Also one is forced
to lock several data-structures individually before certain operations
(taskTrackers, trackerExpiryQueue, jobs etc.)
Hence I'd like to present some early thoughts (which have undergone a quick
iteration) on how we could do slightly better by a bit of
redesign/refactoring, also during discussions with Owen on the same we agreed
that HADOOP-554 is an integral part along the same direction... and I also
feel that a good candidate to be done along with this is HADOOP-398 (mapred
package refactoring).
Context:
-
a) The unit of communication between the JobTracker TaskTracker is a 'task'.
b) Due to (a) the JobTracker maintains a bunch of information related on the
'taskid' i.e. taskidToTipMap, taskidToTrackerMap etc. and hence we need to
update the JobTracker's data-structures via back-calls from TaskInProgress
JobInProgress where the context is available (complete/failed task,
already-completed task etc.)
c) This implies that we have a fairly elaborate and hard to maintain locking
structures and also some redundant information in the JobTracker; making it
harder to maintain.
Overall at both the JobTracker TaskTracker the concept of a 'job' is
overshadowed by the 'task'; which I propose we fix.
Proposal:
--
Here is the main flow of control:
JobTracker - JobInProgress - TaskInProgress - task_attempt
The main idea is to break the existing nexus between the JobTracker
TaskInProgress/taskid by (I've put code for illustrative purposes only, and
ignored pieces irrelevant to this discussion):
a) Making the 'job' the primary unit of communication between JobTracker
TaskTracker.
b) TaskTrackerStatus now looks like this:
class TaskTrackerStatus {
ListJobStatus jobStatuses; // the status of the 'jobs' running on a
TaskTracker
String getTrackerName();
}
class JobStatus {
ListTaskStatus taskStatuses; // the status of the 'tasks' belonging to
a job
JobId getJobId();
}
c) The JobTracker maintains only a single map of jobid - JobInProgress, and
mapping from taskTracker - ListJobInProgress
MapJobId, JobInProgress allJobs;
MapString, ListJobInProgress trackerToJobsMap;
d) The JobTracker delegates a bunch of responsibilities to the JobInProgress
to reflect the fact the primary 'concept' in map/reduce is the 'job', thus
empowering the JobInProgress class:
class JobInProgress {
TaskInProgress[] mapTasks;
TaskInProgress[] reduceTasks;
MapString, ListTaskInProgress trackerToTasksMap; // tracker - tasks
running
MapString, ListTaskAttempt trackerToMarkedTasksMap; // tracker -
completed (success/failed/killed) task-attempt,
//but the tracker doesn't know it yet
void updateStatus(JobStatus jobStatus);
MapOutputLocation[] getMapOutputLocations(int[] mapTasksNeeded, int
reduce);
TaskAttempt getTaskToRun(String taskTracker);
ListTaskTrackerAction getTaskToKill(String taskTracker);
}
d) On receipt of TaskTrackerStatus from a tracker, the processeing of
heartbeat looks like this:
for (JobStatus jobStatus : taskTrackerStatus.getJobStatuses()) {
JobInProgress job = allJobs.get(jobId);
synchronized (job) {
job.updateStatus(jobStatus);
return (HeartbeatResponse(repsonseId,
job.getTaskAttemptToRun(trackerName),
job.getTaskToKill(trackerName)
));
}
}
The big change is that the JobTracker delegates a lot of responsibility to
the JobInProgress, we get away from all the complicated synchronization
constructs: simply lock the JobInProgress object at all places via
allJobs/trackerToJobsMap and we are
[jira] [Created] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
Devaraj K created MAPREDUCE-4189:
Summary: TestContainerManagerSecurity is failing
Key: MAPREDUCE-4189
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: trunk
Reporter: Devaraj K
Priority: Critical
{code:xml}
---
T E S T S
---
Running org.apache.hadoop.yarn.server.TestDiskFailures
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
FAILURE!
Results :
Tests in error:
testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
