[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381475#comment-14381475 ] Harsh J commented on MAPREDUCE-6291: Thanks again! {code} - System.out.println("bin/hadoop pipes"); + System.out.println("pipes"); {code} Just like was done for 'archives', can this carry "Usage: " prefix too? Other than this it looks good to me. Do we have identified YARN/etc. changes we want to also do? We could target them all in one go to reduce work, and move the JIRA to HADOOP? > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[ https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6165: - Attachment: MAPREDUCE-6165-003.patch Updated the patch. > [JDK8] TestCombineFileInputFormat failed on JDK8 > > > Key: MAPREDUCE-6165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wei Yan >Assignee: Akira AJISAKA >Priority: Minor > Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch, > MAPREDUCE-6165-003.patch, MAPREDUCE-6165-reproduce.patch > > > The error msg: > {noformat} > testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 2.487 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<2> but was:<1> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:234) > at junit.framework.Assert.assertEquals(Assert.java:241) > at junit.framework.TestCase.assertEquals(TestCase.java:409) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911) > testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 0.985 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<2> but was:<1> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:234) > at junit.framework.Assert.assertEquals(Assert.java:241) > at junit.framework.TestCase.assertEquals(TestCase.java:409) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[ https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6165: - Status: Patch Available (was: Open) > [JDK8] TestCombineFileInputFormat failed on JDK8 > > > Key: MAPREDUCE-6165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wei Yan >Assignee: Akira AJISAKA >Priority: Minor > Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch, > MAPREDUCE-6165-003.patch, MAPREDUCE-6165-reproduce.patch > > > The error msg: > {noformat} > testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 2.487 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<2> but was:<1> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:234) > at junit.framework.Assert.assertEquals(Assert.java:241) > at junit.framework.TestCase.assertEquals(TestCase.java:409) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911) > testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 0.985 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<2> but was:<1> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:234) > at junit.framework.Assert.assertEquals(Assert.java:241) > at junit.framework.TestCase.assertEquals(TestCase.java:409) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[ https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381550#comment-14381550 ] Hadoop QA commented on MAPREDUCE-6165: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707447/MAPREDUCE-6165-003.patch against trunk revision b4b4fe9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapred.TestJobCounters Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5338//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5338//console This message is automatically generated. > [JDK8] TestCombineFileInputFormat failed on JDK8 > > > Key: MAPREDUCE-6165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wei Yan >Assignee: Akira AJISAKA >Priority: Minor > Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch, > MAPREDUCE-6165-003.patch, MAPREDUCE-6165-reproduce.patch > > > The error msg: > {noformat} > testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 2.
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382038#comment-14382038 ] Robert Joseph Evans commented on MAPREDUCE-6288: [~kasha], No one doubts that MAPREDUCE-5875 fixed an issue that has existed in MRv2 since the beginning. It is mostly a question of priorities. This JIRA with mapred job -status failing horribly is considered by the filer to be a BLOCKER. MAPREDUCE-5875 that introduced this bug was only considered to be MAJOR. I am biased because I run hadoop with security and would not be able to deploy the current codebase because of this issue. For me I would rather get the code base back into a stable place quickly, and then we can think about the correct way to re-implement MAPREDUCE-5875 without this issue. If you have a good solution/patch to the problem now then I would be fine with just merging that in, but the current proposed solution does not actually fix the problem on a properly secured cluster. You will still get an exception, except this time it will be coming from the restful web service not HDFS. Changing the execute bit will not expose anything proprietary, but it also will not fix the problem. HDFS does not tell you all of the ways that you are violating permissions, it tells you the first one it runs into. The file you are trying to read is owned by a headless user and should not have world read permissions. If on a secure cluster it does have global read permissions I would file another BLOCKER to fix it. > mapred job -status fails with AccessControlException > - > > Key: MAPREDUCE-6288 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch > > > After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred > job -status job_1427080398288_0001}} > {noformat} > Exception in thread "main" org.apache.hadoop.security.AccessControlException: > Permission denied: user=jenkins, access=EXECUTE, > inode="/user/history/done":mapred:hadoop:drwxrwx--- > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInfo
[jira] [Commented] (MAPREDUCE-6293) uberized job fails with the job classloader enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382044#comment-14382044 ] Sangjin Lee commented on MAPREDUCE-6293: The test failures are caused by HADOOP-11754. I would greatly appreciate your review on the patch. Thanks! > uberized job fails with the job classloader enabled > --- > > Key: MAPREDUCE-6293 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-6293.001.patch > > > An uberized job fails if the job classloader is enabled and the job needs to > use the thread context classloader to load a class. Some example error in the > log: > {quote} > 2015-03-23 23:28:34,675 INFO [main\] > org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader > ... > 2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\] > cascading.provider.ServiceLoader: unable to find service class: > cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception: > java.lang.ClassNotFoundException: > cascading.tuple.hadoop.collect.HadoopTupleMapFactory > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6291: Attachment: MAPREDUCE-6291-002.patch > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, > MAPREDUCE-6291-002.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382158#comment-14382158 ] Brahma Reddy Battula commented on MAPREDUCE-6291: - I did not find for remaining commands and updated the patch for pipes usage.. > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, > MAPREDUCE-6291-002.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382239#comment-14382239 ] Hadoop QA commented on MAPREDUCE-6291: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707524/MAPREDUCE-6291-002.patch against trunk revision 2228456. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-archives. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5339//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5339//console This message is automatically generated. > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, > MAPREDUCE-6291-002.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6293) uberized job fails with the job classloader enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382336#comment-14382336 ] Sangjin Lee commented on MAPREDUCE-6293: Github link: https://github.com/sjlee/hadoop/commit/259766e63d8fd739a02ee0c59990270bda78c7d4 > uberized job fails with the job classloader enabled > --- > > Key: MAPREDUCE-6293 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: MAPREDUCE-6293.001.patch > > > An uberized job fails if the job classloader is enabled and the job needs to > use the thread context classloader to load a class. Some example error in the > log: > {quote} > 2015-03-23 23:28:34,675 INFO [main\] > org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader > ... > 2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\] > cascading.provider.ServiceLoader: unable to find service class: > cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception: > java.lang.ClassNotFoundException: > cascading.tuple.hadoop.collect.HadoopTupleMapFactory > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382573#comment-14382573 ] Gera Shegalov commented on MAPREDUCE-6288: -- bq. The file you are trying to read is owned by a headless user and should not have world read permissions. If on a secure cluster it does have global read permissions I would file another BLOCKER to fix it. [~revans2], at least in the scenarios we were dealing with it's the owner of the job who is trying to get to her conf file. So the file itself does not have to be world-readable as long as the hosting filesystem permits traversing the path. > mapred job -status fails with AccessControlException > - > > Key: MAPREDUCE-6288 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch > > > After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred > job -status job_1427080398288_0001}} > {noformat} > Exception in thread "main" org.apache.hadoop.security.AccessControlException: > Permission denied: user=jenkins, access=EXECUTE, > inode="/user/history/done":mapred:hadoop:drwxrwx--- > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) > at > org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201) > at > org.apache.hadoop.hdfs.DFSClient.getL
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382626#comment-14382626 ] Robert Kanter commented on MAPREDUCE-6288: -- Exactly. That's what my original patch does. It doesn't change the permissions of the jobconf file itself. It only adds executable permissions to the parent directories. Users will still be prevented from listing the directories or other operations they shouldn't be doing. It simply allows users who own files in that hierarchy the ability to access those files, which they previously couldn't (which in hindsight, seems incorrect regardless). {quote}I quickly checked, and there seems be another not-so-minor problem of this patch not working for existing clusters without needing admins to explicitly go and change the permissions on the directory. History server doesn't seem to correct the permissions if the directory already exists.{quote} It should be pretty easy to make the JHS correct the permissions on startup. > mapred job -status fails with AccessControlException > - > > Key: MAPREDUCE-6288 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.patch > > > After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred > job -status job_1427080398288_0001}} > {noformat} > Exception in thread "main" org.apache.hadoop.security.AccessControlException: > Permission denied: user=jenkins, access=EXECUTE, > inode="/user/history/done":mapred:hadoop:drwxrwx--- > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.ipc.Remo
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382921#comment-14382921 ] Ray Chiang commented on MAPREDUCE-3971: --- I just ran across this JIRA. I know this one is quite old, but I thought I'd see if anyone here thinks this has overlap with MAPREDUCE-6222. > Job History web services need to have limits on the number of items they can > return. > > > Key: MAPREDUCE-3971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > > The Job History web services canput a very large load on the job history > server. We should put in a limit on the number of entries that can be > returned by the web service, and also add in the ability to modify the > starting location in the list, so that all entries can still be downlaoded. > Just not all at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6271) org.apache.hadoop.mapreduce.Cluster GetJob() display warn log
[ https://issues.apache.org/jira/browse/MAPREDUCE-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382958#comment-14382958 ] Gera Shegalov commented on MAPREDUCE-6271: -- Awaiting the outcome of the MAPREDUCE-6288 discussion before deciding how to proceed here. > org.apache.hadoop.mapreduce.Cluster GetJob() display warn log > - > > Key: MAPREDUCE-6271 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6271 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.7.0 >Reporter: Peng Zhang >Assignee: Peng Zhang > Attachments: MAPREDUCE-6271.v2.patch, MR-6271.patch > > > When using getJob() with MapReduce 2.7, warn log caused by configuration > loaded twice is displayed every time. And when job completed, this function > will display warn log of "java.io.FileNotFoundException" > And I think this is related with MAPREDUCE-5875, the change in GetJob() seems > to be not needed, cause it's only for test. > {noformat} > 15/03/04 13:41:23 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:23 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:24 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:24 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:25 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:25 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:26 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:26 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:27 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:27 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:28 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:28 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:29 WARN conf.Configuration: > hdfsG://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:29 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/03/04 13:41:29 INFO exec.Task: 2015-03-04 13:41:29,853 Stage-1 map = 100%, > reduce = 0%, Cumulative CPU 2.37 sec > 15/03/04 13:41:30 WARN conf.Configuration: > hdfs://example/yarn/example2/staging/test_user/.staging/job_1425388652704_0116/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/03/04 13:41:30 WARN conf.Configuration: > hdfs://e
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383266#comment-14383266 ] Rohith commented on MAPREDUCE-6291: --- If no commands need to change then I think YARN-3398 can closed as 'Not a Problem' if you dont have any concern. > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, > MAPREDUCE-6291-002.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383340#comment-14383340 ] Brahma Reddy Battula commented on MAPREDUCE-6291: - YARN-3398 for unification of usage commands not for other commands correction...:) > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, > MAPREDUCE-6291-002.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[ https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6165: -- Attachment: MAPREDUCE-6165-003.patch The results of tests looks strange. Attaching v3 patch again to kick Jenkins. > [JDK8] TestCombineFileInputFormat failed on JDK8 > > > Key: MAPREDUCE-6165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wei Yan >Assignee: Akira AJISAKA >Priority: Minor > Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch, > MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, > MAPREDUCE-6165-reproduce.patch > > > The error msg: > {noformat} > testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 2.487 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<2> but was:<1> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:234) > at junit.framework.Assert.assertEquals(Assert.java:241) > at junit.framework.TestCase.assertEquals(TestCase.java:409) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911) > testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat) > Time elapsed: 0.985 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<2> but was:<1> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:234) > at junit.framework.Assert.assertEquals(Assert.java:241) > at junit.framework.TestCase.assertEquals(TestCase.java:409) > at > org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[ https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383385#comment-14383385 ] Rohith commented on MAPREDUCE-6291: --- I think it is better to strict to [~qwertymaniac] suggestion, all Hadoop script help message changes can go all together in one jira. I believe this will reduce work. > Correct mapred queue usage command > -- > > Key: MAPREDUCE-6291 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch, > MAPREDUCE-6291-002.patch > > > *Currently it is like following..* > Usage: JobQueueClient > *It should be* > Usage: queue > *For more Details check following* > {noformat} > hdfs@host1:/hadoop/bin> ./mapred queue > Usage: JobQueueClient > [-list] > [-info [-showJobs]] > [-showacls] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)