[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383385#comment-14383385 ] Rohith commented on MAPREDUCE-6291: --- I think it is better to strict to [~qwertymaniac] suggestion, all Hadoop script help message changes can go all together in one jira. I believe this will reduce work. Correct mapred queue usage command -- Key: MAPREDUCE-6291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, MAPREDUCE-6291-002.patch *Currently it is like following..* Usage: JobQueueClient command args *It should be* Usage: queue command args *For more Details check following* {noformat} hdfs@host1:/hadoop/bin ./mapred queue Usage: JobQueueClient command args [-list] [-info job-queue-name [-showJobs]] [-showacls] {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[ https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383462#comment-14383462 ] Hadoop QA commented on MAPREDUCE-6165: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707697/MAPREDUCE-6165-003.patch against trunk revision af618f2. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapreduce.v2.TestMRJobs Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5340//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5340//console This message is automatically generated. [JDK8] TestCombineFileInputFormat failed on JDK8 Key: MAPREDUCE-6165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Wei Yan Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-reproduce.patch The error msg: {noformat} testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) Time
[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[ https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6165: -- Attachment: MAPREDUCE-6165-003.patch The results of tests looks strange. Attaching v3 patch again to kick Jenkins. [JDK8] TestCombineFileInputFormat failed on JDK8 Key: MAPREDUCE-6165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Wei Yan Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-reproduce.patch The error msg: {noformat} testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) Time elapsed: 2.487 sec FAILURE! junit.framework.AssertionFailedError: expected:2 but was:1 at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911) testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) Time elapsed: 0.985 sec FAILURE! junit.framework.AssertionFailedError: expected:2 but was:1 at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383662#comment-14383662 ] Hadoop QA commented on MAPREDUCE-5608: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707730/MAPREDUCE-5608-002.patch against trunk revision af618f2. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5342//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5342//console This message is automatically generated. Replace and deprecate mapred.tasktracker.indexcache.mb -- Key: MAPREDUCE-5608 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Akira AJISAKA Labels: configuration, newbie Attachments: MAPREDUCE-5608-002.patch, MAPREDUCE-5608.patch In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size of the shuffle service index cache. As the tasktracker no longer exists, we should replace this with something like mapreduce.shuffle.indexcache.mb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5762) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5762: - Attachment: MAPREDUCE-5762-branch-2-002.patch Rebased for the latest branch-2. Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2 Key: MAPREDUCE-5762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5762 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: MAPREDUCE-5762-branch-2-002.patch, MAPREDUCE-5762-branch-2.patch MRv1 configs are removed in trunk, but they are not removed in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5762) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5762: - Priority: Minor (was: Major) Target Version/s: 2.8.0 (was: 2.4.0) Issue Type: Improvement (was: Bug) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2 Key: MAPREDUCE-5762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5762 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5762-branch-2-002.patch, MAPREDUCE-5762-branch-2.patch MRv1 configs are removed in trunk, but they are not removed in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5608: - Attachment: MAPREDUCE-5608-002.patch Rebased. Replace and deprecate mapred.tasktracker.indexcache.mb -- Key: MAPREDUCE-5608 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Akira AJISAKA Labels: configuration, newbie Attachments: MAPREDUCE-5608-002.patch, MAPREDUCE-5608.patch In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size of the shuffle service index cache. As the tasktracker no longer exists, we should replace this with something like mapreduce.shuffle.indexcache.mb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5762) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383604#comment-14383604 ] Hadoop QA commented on MAPREDUCE-5762: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707722/MAPREDUCE-5762-branch-2-002.patch against trunk revision af618f2. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5341//console This message is automatically generated. Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2 Key: MAPREDUCE-5762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5762 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5762-branch-2-002.patch, MAPREDUCE-5762-branch-2.patch MRv1 configs are removed in trunk, but they are not removed in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6294) Remove an extra parameter described in Javadoc of TockenCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa moved HADOOP-11759 to MAPREDUCE-6294: Affects Version/s: (was: 2.6.0) (was: 3.0.0) 3.0.0 2.6.0 Key: MAPREDUCE-6294 (was: HADOOP-11759) Project: Hadoop Map/Reduce (was: Hadoop Common) Remove an extra parameter described in Javadoc of TockenCache - Key: MAPREDUCE-6294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6294 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0, 3.0.0 Reporter: Chen He Assignee: Brahma Reddy Battula Priority: Trivial Labels: newbie++ Attachments: HADOOP-11759.patch /** * get delegation token for a specific FS * @param fs * @param credentials * @param p * @param conf * @throws IOException */ static void obtainTokensForNamenodesInternal(FileSystem fs, Credentials credentials, Configuration conf) throws IOException { -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6294) Remove an extra parameter described in Javadoc of TockenCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6294: -- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~brahmareddy] for the contribution and thanks [~airbots] for your reporting! Remove an extra parameter described in Javadoc of TockenCache - Key: MAPREDUCE-6294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6294 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.6.0 Reporter: Chen He Assignee: Brahma Reddy Battula Priority: Trivial Labels: newbie++ Fix For: 2.8.0 Attachments: HADOOP-11759.patch /** * get delegation token for a specific FS * @param fs * @param credentials * @param p * @param conf * @throws IOException */ static void obtainTokensForNamenodesInternal(FileSystem fs, Credentials credentials, Configuration conf) throws IOException { -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5762) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384047#comment-14384047 ] Harsh J commented on MAPREDUCE-5762: Went over the entire property list, and the removals look fine to me. +1, please commit. Many thanks for the work! Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2 Key: MAPREDUCE-5762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5762 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5762-branch-2-002.patch, MAPREDUCE-5762-branch-2.patch MRv1 configs are removed in trunk, but they are not removed in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6294) Remove an extra parameter described in Javadoc of TockenCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384007#comment-14384007 ] Brahma Reddy Battula commented on MAPREDUCE-6294: - Thanks a lot [~ozawa]!!! Remove an extra parameter described in Javadoc of TockenCache - Key: MAPREDUCE-6294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6294 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.6.0 Reporter: Chen He Assignee: Brahma Reddy Battula Priority: Trivial Labels: newbie++ Fix For: 2.8.0 Attachments: HADOOP-11759.patch /** * get delegation token for a specific FS * @param fs * @param credentials * @param p * @param conf * @throws IOException */ static void obtainTokensForNamenodesInternal(FileSystem fs, Credentials credentials, Configuration conf) throws IOException { -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6294) Remove an extra parameter described in Javadoc of TockenCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383977#comment-14383977 ] Hudson commented on MAPREDUCE-6294: --- FAILURE: Integrated in Hadoop-trunk-Commit #7448 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7448/]) MAPREDUCE-6294. Remove an extra parameter described in Javadoc of TockenCache. Contributed by Brahma Reddy Battula. (ozawa: rev 05499b1093ea6ba6a39a1354d67b0a46a2982824) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java Remove an extra parameter described in Javadoc of TockenCache - Key: MAPREDUCE-6294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6294 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.6.0 Reporter: Chen He Assignee: Brahma Reddy Battula Priority: Trivial Labels: newbie++ Fix For: 2.8.0 Attachments: HADOOP-11759.patch /** * get delegation token for a specific FS * @param fs * @param credentials * @param p * @param conf * @throws IOException */ static void obtainTokensForNamenodesInternal(FileSystem fs, Credentials credentials, Configuration conf) throws IOException { -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384132#comment-14384132 ] Hadoop QA commented on MAPREDUCE-5036: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587173/MAPREDUCE-5036-2.patch against trunk revision 05499b1. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5344//console This message is automatically generated. Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384339#comment-14384339 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5036: I think changing it to an ephemeral port is good in general. But the recent feature of NM-recovery (YARN-1336) can make this a problem. - ShuffleHandler starts on port A - MR AM launches a container, gets port A as the target location for the intermediate output, passes this to the reducer - NM restarts and ShuffleHandler starts on port B now. - Reducer fails to perform shuffle. Either we don't allow ephemeral ports here. Or have a mechanism for YARN RM to send changed aux-service port info to all apps when a node restarts. /cc [~jlowe], [~djp] Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5496) Document mapreduce.cluster.administrators in mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5496: - Description: {{mapreduce.cluster.administrators}} is not documented anywhere. We should document it in mapred-default.xml. (was: Two issues: # {{mapreduce.tasktracker.group}} is mentioned in [http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml], but is no longer used. # {{mapreduce.cluster.administrators}} is not documented anywhere.) Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) Summary: Document mapreduce.cluster.administrators in mapred-default.xml (was: Documentation should be updated for two MR2 properties) Updated the description. Document mapreduce.cluster.administrators in mapred-default.xml --- Key: MAPREDUCE-5496 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5496 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Priority: Minor {{mapreduce.cluster.administrators}} is not documented anywhere. We should document it in mapred-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384364#comment-14384364 ] Harsh J commented on MAPREDUCE-5036: Thanks Vinod, As it exists, is a static port config proving as a trouble worthy enough that we would wanna support ephemeral ports? Also, can this be done on a new JIRA, abandoning the follow-up patch that this seems to be kept open for? On Sat, Mar 28, 2015 at 12:09 AM, Vinod Kumar Vavilapalli (JIRA) -- Harsh J Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036-2.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384384#comment-14384384 ] Karthik Kambatla commented on MAPREDUCE-6288: - On a cluster that had this issue, Robert added executable bit to the appropriate directories to verify that alone is enough to fix the issue. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490)
[jira] [Updated] (MAPREDUCE-5762) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5762: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed v2 patch to branch-2. Thanks [~qwertymaniac] for review! Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2 Key: MAPREDUCE-5762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5762 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5762-branch-2-002.patch, MAPREDUCE-5762-branch-2.patch MRv1 configs are removed in trunk, but they are not removed in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5496) Documentation should be updated for two MR2 properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384340#comment-14384340 ] Akira AJISAKA commented on MAPREDUCE-5496: -- {{mapreduce.tasktracker.group}} was removed from branch-2 by MAPREDUCE-5762, so we need to remove {{mapreduce.cluster.administrators}} only. Documentation should be updated for two MR2 properties -- Key: MAPREDUCE-5496 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5496 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Two issues: # {{mapreduce.tasktracker.group}} is mentioned in [http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml], but is no longer used. # {{mapreduce.cluster.administrators}} is not documented anywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384360#comment-14384360 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-6288: Okay. I am good doing this. Can you please add the permissions correction to the patch? Thanks. [~revans2], as clarified immediately above by [~rkanter], his patch doesn't open up read permissions of the files, nor does it let users list all files. We are going ahead with this unless there are more objections. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384673#comment-14384673 ] Robert Kanter commented on MAPREDUCE-6288: -- I'll post an updated patch later today that makes the JHS update the permissions on startup. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490) at
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6279: -- Status: Patch Available (was: Open) AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6279: -- Attachment: MAPREDUCE-6279.v1.txt [~jlowe], are unit tests necessary for this change? AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384871#comment-14384871 ] Hadoop QA commented on MAPREDUCE-6279: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707896/MAPREDUCE-6279.v1.txt against trunk revision 3836ad6. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup org.apache.hadoop.mapreduce.v2.app.TestJobEndNotifier Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5345//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5345//console This message is automatically generated. AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384879#comment-14384879 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-6288: Thanks Robert. Assuming it is close, I'd like to commit it later today or over the weekend so that I can roll an RC for 2.7.0. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384888#comment-14384888 ] Bowen Zhang commented on MAPREDUCE-6288: [~rkanter], this is breaking all TestJavaActionExecutor unit tests on Oozie with hadoop-2.7 mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490) at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384893#comment-14384893 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-6288: [~rkanter] / [~jira.shegalov], Sigh, I see another big issue with the patch at MAPREDUCE-5875. Now with every Cluster.Job getJob(JobID jobId) call, we create a new JobConf and open the HDFS file in order to load it later. That can become a huge perf hit for the job as well as the Namenode. Even the previous code was adding the remote conf file, but it never gets read till somebody does an explicit conf.get() - which doesn't happen in the usual wait-for-completion code path. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384925#comment-14384925 ] Robert Kanter commented on MAPREDUCE-6288: -- What if we do this fix for now and add a followup JIRA to add a cache? This shouldn't be a that huge of a performance hit. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490) at
[jira] [Updated] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6288: - Attachment: MAPREDUCE-6288.002.patch The MAPREDUCE-6288.002.patch builds on the previous MAPREDUCE-6288.patch I originally made. Besides using 771 instead of 770, it also makes the JHS check and correct the permissions on startup. I added a unit test and also verified in a cluster. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385005#comment-14385005 ] zhihai xu commented on MAPREDUCE-6288: -- I double checked the code in Job.java, JobClient.java, JobSubmitter.java and Cluster.java. Cluster#getJob is not used by Job.java, JobSubmitter.java and Cluster.java. The only place where Cluster#getJob is used is from JobClient#getJobUsingCluster. JobClient#getJobUsingCluster is used by JobClient#getJob, JobClient#displayTask, JobClient#getMapTaskReports, JobClient#getReduceTaskReports, JobClient#getCleanupTaskReports and JobClient#getSetupTaskReports. All these functions(JobClient#getJob, JobClient#displayTask, JobClient#getMapTaskReports, JobClient#getReduceTaskReports, JobClient#getCleanupTaskReports and JobClient#getSetupTaskReports) are public APIs, which are currently only used by applications for example CLI. So It looks like Cluster#getJob will be only used by applications, the performance hit won't be that much if the applications don't call these functions too many times. If we can add cache to improve the performance, that will be better. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385061#comment-14385061 ] Hadoop QA commented on MAPREDUCE-6288: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707951/MAPREDUCE-6288.002.patch against trunk revision 3836ad6. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5346//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5346//console This message is automatically generated. mapred job -status fails with AccessControlException - Key: MAPREDUCE-6288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Robert Kanter Assignee: Robert Kanter Priority: Blocker Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch, MAPREDUCE-6288.patch After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred job -status job_1427080398288_0001}} {noformat} Exception in thread main org.apache.hadoop.security.AccessControlException: Permission denied: user=jenkins, access=EXECUTE, inode=/user/history/done:mapred:hadoop:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)