[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Attachment: MAPREDUCE-6296.patch
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Attachments: MAPREDUCE-6296.patch
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated MAPREDUCE-6296:
Status: Patch Available (was: Open)
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Attachments: MAPREDUCE-6296.patch
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6087) MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6087: - Fix Version/s: 2.6.0 MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong Key: MAPREDUCE-6087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6087 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jian He Assignee: Akira AJISAKA Labels: newbie Fix For: 2.6.0 Attachments: MAPREDUCE-6087.2.patch, MAPREDUCE-6087.patch The config name for MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS now has double prefix as yarn.app.mapreduce. + yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6296) A better way to deal with InterruptedException on waitForCompletion
[
https://issues.apache.org/jira/browse/MAPREDUCE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386312#comment-14386312
]
Hadoop QA commented on MAPREDUCE-6296:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708102/MAPREDUCE-6296.patch
against trunk revision 1ed9fb7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5351//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5351//console
This message is automatically generated.
A better way to deal with InterruptedException on waitForCompletion
---
Key: MAPREDUCE-6296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6296
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Yang Hao
Assignee: Yang Hao
Attachments: MAPREDUCE-6296.patch
Some code in method waitForCompletion of Job class is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
{code}
but a better way to deal with InterruptException is
{code:title=Job.java|borderStyle=solid}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
}
}
return isSuccessful();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6295) Fix MR resource counter to handle negative value for getting memory resource after YARN-3304
[ https://issues.apache.org/jira/browse/MAPREDUCE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6295: -- Attachment: YARN-3304-v8.patch Update patch to address issues as comments: https://issues.apache.org/jira/browse/YARN-3304?focusedCommentId=14386735page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14386735 Fix MR resource counter to handle negative value for getting memory resource after YARN-3304 Key: MAPREDUCE-6295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6295 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Junping Du Assignee: Junping Du Priority: Critical Attachments: YARN-3304-v6-no-rename.patch, YARN-3304-v7.patch, YARN-3304-v8.patch After YARN-3304, we will get negative value for memory resource if resource data is unavailable. MR resource counter shouldn't put negative value there so a simple fix is required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6295) Fix MR resource counter to handle negative value for getting memory resource after YARN-3304
[
https://issues.apache.org/jira/browse/MAPREDUCE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386890#comment-14386890
]
Hadoop QA commented on MAPREDUCE-6295:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708162/YARN-3304-v8.patch
against trunk revision ae3e8c6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5352//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5352//console
This message is automatically generated.
Fix MR resource counter to handle negative value for getting memory resource
after YARN-3304
Key: MAPREDUCE-6295
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6295
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical
Attachments: YARN-3304-v6-no-rename.patch, YARN-3304-v7.patch,
YARN-3304-v8.patch
After YARN-3304, we will get negative value for memory resource if resource
data is unavailable. MR resource counter shouldn't put negative value there
so a simple fix is required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387022#comment-14387022
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-6288:
The initialization code in HistoryManager and the entire JHS code-base itself
has become a little hairy, sigh, but I wouldn't touch it now.
The patch looks good to me. +1. Checking this in.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at
[jira] [Updated] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-6288:
---
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-2 and branch-2.7. Thanks Robert. Tx for the
discussion, everyone else.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at
[jira] [Updated] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6297: --- Attachment: 58CCA024-7455-4A87-BCFD-C88054FF841B.png Task Id of the failed task in diagnostics should link to the task page -- Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Assignee: Siqi Li Priority: Minor Attachments: 58CCA024-7455-4A87-BCFD-C88054FF841B.png, MAPREDUCE-6297.v1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6297: --- Description: Currently we have to copy it and search in the task list. Task Id of the failed task in diagnostics should link to the task page -- Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Assignee: Siqi Li Priority: Minor Attachments: 58CCA024-7455-4A87-BCFD-C88054FF841B.png, MAPREDUCE-6297.v1.patch Currently we have to copy it and search in the task list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
Siqi Li created MAPREDUCE-6297: -- Summary: Task Id of the failed task in diagnostics should link to the task page Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[
https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387081#comment-14387081
]
Hadoop QA commented on MAPREDUCE-6297:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708202/58CCA024-7455-4A87-BCFD-C88054FF841B.png
against trunk revision 5358b83.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5353//console
This message is automatically generated.
Task Id of the failed task in diagnostics should link to the task page
--
Key: MAPREDUCE-6297
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Minor
Attachments: 58CCA024-7455-4A87-BCFD-C88054FF841B.png,
MAPREDUCE-6297.v1.patch
Currently we have to copy it and search in the task list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6295) Fix MR resource counter to handle negative value for getting memory resource after YARN-3304
[ https://issues.apache.org/jira/browse/MAPREDUCE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6295: -- Resolution: Fixed Fix Version/s: 2.7.0 Status: Resolved (was: Patch Available) v8 patch in YARN-3304 get committed (with MR code), so we can mark this JIRA as resolved. Fix MR resource counter to handle negative value for getting memory resource after YARN-3304 Key: MAPREDUCE-6295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6295 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Junping Du Assignee: Junping Du Priority: Critical Fix For: 2.7.0 Attachments: YARN-3304-v6-no-rename.patch, YARN-3304-v7.patch, YARN-3304-v8.patch After YARN-3304, we will get negative value for memory resource if resource data is unavailable. MR resource counter shouldn't put negative value there so a simple fix is required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reopened MAPREDUCE-6288:
I still don't consider this fixed. I'm willing to let MR shoot itself in the
foot and expose the HDFS layout of the history server. [~jlowe] my disagree
with me because he still has to support this code, but please test this fix
with a user that has ACL permissions to read the job status, but did not launch
the job.
If the job is still up and running or not, It will fail with a permission
denied error, because the original job owner is the only one in HDFS that has
permissions to read the config file.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
[jira] [Updated] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6297: --- Attachment: (was: MAPREDUCE-6297.v1.patch) Task Id of the failed task in diagnostics should link to the task page -- Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Assignee: Siqi Li Priority: Minor Attachments: 58CCA024-7455-4A87-BCFD-C88054FF841B.png Currently we have to copy it and search in the task list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387040#comment-14387040
]
Robert Kanter commented on MAPREDUCE-6288:
--
Thanks everyone.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490)
at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387041#comment-14387041
]
Hudson commented on MAPREDUCE-6288:
---
FAILURE: Integrated in Hadoop-trunk-Commit #7462 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7462/])
MAPREDUCE-6288. Changed permissions on JobHistory server's done directory so
that user's client can load the conf files directly. Contributed by Robert
Kanter. (vinodkv: rev 5358b8316a7108b32c9900fb0d01ca0fe961)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestHistoryFileManager.java
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387066#comment-14387066
]
Robert Joseph Evans commented on MAPREDUCE-6288:
-1
Please stop trying to open up the permissions for the files and directories the
history server manages. We should not be exposing an implementation detail
like how files are laid out on HDFS in the history server to end users. If we
do expose it we will not be able to change it in the future, like when/if we go
to the timeline server. Also it does not solve the issue for anyone except the
original user that launched the MR job. It will still blow up if another user
has permissions, because of Job ACLs, to check the status of the job, but did
not launch the job. There is also technically a race condition here too. The
history server moves the files asynchronously from done_intermediate to done.
You may try to read the file before the history server has finished moving it.
There is synchronization within the history server to prevent this from
happening if you go through the proper APIs.
If you want the job conf create a new RPC call for it in the history server and
in the MR AM, then have the client download it through those RPC APIs. This
will not expose any internal detail and will support the ACLs correctly across
the board.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[jira] [Updated] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6297: --- Assignee: Siqi Li Status: Patch Available (was: Open) Task Id of the failed task in diagnostics should link to the task page -- Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Assignee: Siqi Li Priority: Minor Attachments: MAPREDUCE-6297.v1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6297: --- Attachment: MAPREDUCE-6297.v1.patch Task Id of the failed task in diagnostics should link to the task page -- Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Priority: Minor Attachments: MAPREDUCE-6297.v1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6297: --- Attachment: MAPREDUCE-6297.v1.patch Task Id of the failed task in diagnostics should link to the task page -- Key: MAPREDUCE-6297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siqi Li Assignee: Siqi Li Priority: Minor Attachments: 58CCA024-7455-4A87-BCFD-C88054FF841B.png, MAPREDUCE-6297.v1.patch Currently we have to copy it and search in the task list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6293) uberized job fails with the job classloader enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387192#comment-14387192
]
Sangjin Lee commented on MAPREDUCE-6293:
Restarted the jenkins build after the fix for HADOOP-11754:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5355/
uberized job fails with the job classloader enabled
---
Key: MAPREDUCE-6293
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-6293.001.patch
An uberized job fails if the job classloader is enabled and the job needs to
use the thread context classloader to load a class. Some example error in the
log:
{quote}
2015-03-23 23:28:34,675 INFO [main\]
org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader
...
2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\]
cascading.provider.ServiceLoader: unable to find service class:
cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception:
java.lang.ClassNotFoundException:
cascading.tuple.hadoop.collect.HadoopTupleMapFactory
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5465) Container killed before hprof dumps profile.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387119#comment-14387119 ] Ray Chiang commented on MAPREDUCE-5465: --- [~mingma], can you give me an idea when you can get an rebased patch uploaded? Thanks. Container killed before hprof dumps profile.out --- Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6297) Task Id of the failed task in diagnostics should link to the task page
[
https://issues.apache.org/jira/browse/MAPREDUCE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387174#comment-14387174
]
Hadoop QA commented on MAPREDUCE-6297:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708205/MAPREDUCE-6297.v1.patch
against trunk revision 1feb956.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5354//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5354//console
This message is automatically generated.
Task Id of the failed task in diagnostics should link to the task page
--
Key: MAPREDUCE-6297
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6297
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Minor
Attachments: 58CCA024-7455-4A87-BCFD-C88054FF841B.png,
MAPREDUCE-6297.v1.patch
Currently we have to copy it and search in the task list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6298) Job#toString throws an exception when not in state RUNNING
Lars Francke created MAPREDUCE-6298:
---
Summary: Job#toString throws an exception when not in state RUNNING
Key: MAPREDUCE-6298
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6298
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Lars Francke
Priority: Minor
Job#toString calls {{ensureState(JobState.RUNNING);}} as the very first thing.
That method causes an Exception to be thrown which is not nice.
One thing this breaks is usage of Job on the Scala (e.g. Spark) REPL as that
calls toString after every invocation and that fails every time.
I'll attach a patch that checks state and if it's RUNNING prints the original
message and if not prints something else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387502#comment-14387502
]
Hudson commented on MAPREDUCE-6288:
---
FAILURE: Integrated in Hadoop-trunk-Commit #7470 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7470/])
Revert MAPREDUCE-6288. Changed permissions on JobHistory server's done
directory so that user's client can load the conf files directly. Contributed
by Robert Kanter. (vinodkv: rev cc0a01c50308fe599b4bdd5a80068d6244062e4f)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestHistoryFileManager.java
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
[jira] [Updated] (MAPREDUCE-6263) Configurable timeout between YARNRunner terminate the application and forcefully kill.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6263: Fix Version/s: 2.7.0 Configurable timeout between YARNRunner terminate the application and forcefully kill. -- Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.7.0 Attachments: MAPREDUCE-6263.v1.txt, MAPREDUCE-6263.v2.txt YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387576#comment-14387576
]
Karthik Kambatla commented on MAPREDUCE-6288:
-
bq. We should not be exposing an implementation detail like how files are laid
out on HDFS in the history server to end users.
These files are intended to be accessed only through the MapReduce API - REST,
Java or Client. Do we honor compatibility around the file permissions and
locations internal YARN/MR logic? If we are undecided, I would like for us to
explicitly state that we will not be honoring them.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387577#comment-14387577
]
Karthik Kambatla commented on MAPREDUCE-6288:
-
Let me know if I am missing anything.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:257)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1490)
at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387439#comment-14387439
]
zhihai xu commented on MAPREDUCE-6288:
--
[~revans2], Oh, That is a good catch for the race condition at JHS! I reviewed
the patch at MAPREDUCE-5875, It looks like the patch at MAPREDUCE-5875 not just
fixed the issue Make Counter limits consistent across JobClient, MRAppMaster,
and YarnChild.
To fix the issue Make Counter limits consistent across JobClient, MRAppMaster,
and YarnChild,
We don't really need this change at Cluster#getJob:
This change at Cluster#getJob is to help multiple Job clients to get the same
configuration, which looks like a different issue it fixed.
{code}
final JobConf conf = new JobConf();
final Path jobPath = new Path(client.getFilesystemName(),
status.getJobFile());
final FileSystem fs = FileSystem.get(jobPath.toUri(), getConf());
try {
conf.addResource(fs.open(jobPath), jobPath.toString());
} catch (FileNotFoundException fnf) {
if (LOG.isWarnEnabled()) {
LOG.warn(Job conf missing on cluster, fnf);
}
}
{code}
The above code causes the current issue(MAPREDUCE-6288) and it also causes
MAPREDUCE-6271.
IMHO, Can we just revert the code change at Cluster#getJob? then we can create
a following up JIRA to support multiple Job clients to get the same
configuration from the Job configuration file.
The proposed patch
[MR-6271.patch|https://issues.apache.org/jira/secure/attachment/12703587/MR-6271.patch]
at MAPREDUCE-6271 exactly did the same thing which revert the change at
Cluster#getJob and fix the test failure.
I also find two other very old JIRAs MAPREDUCE-4443 and MAPREDUCE-5149 (more
than two years old) which also try to fix the same issue as MAPREDUCE-5875
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild.
The patch at MAPREDUCE-5875 without the code change at Cluster#getJob can fix
both issues MAPREDUCE-4443 and MAPREDUCE-5149. So we really don't need the
above code change at Cluster#getJob to fix both MAPREDUCE-4443 and
MAPREDUCE-5149.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387563#comment-14387563
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-6288:
bq. IMHO, Can we just revert the code change at Cluster#getJob? then we can
create a following up JIRA to support multiple Job clients to get the same
configuration from the Job configuration file.
bq. I am inclined towards reverting MAPREDUCE-5875 from 2.7.0. Will see how
much of an effort that is.
I found out what is needed: reverting MAPREDUCE-6286 - MAPREDUCE-6282 -
MAPREDUCE-6199 - MAPREDUCE-5875 in that order works.
Depending on how other 2.7.0 blockers go, I'll revert these from branch-2.7
first later in the evening.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387496#comment-14387496
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-6288:
Reverted the patch while the discussion continues.
I am inclined towards reverting MAPREDUCE-5875 from 2.7.0. Will see how much of
an effort that is.
mapred job -status fails with AccessControlException
-
Key: MAPREDUCE-6288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
Fix For: 2.7.0
Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
MAPREDUCE-6288.patch
After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
job -status job_1427080398288_0001}}
{noformat}
Exception in thread main org.apache.hadoop.security.AccessControlException:
Permission denied: user=jenkins, access=EXECUTE,
inode=/user/history/done:mapred:hadoop:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at
[jira] [Commented] (MAPREDUCE-5) Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
[ https://issues.apache.org/jira/browse/MAPREDUCE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387686#comment-14387686 ] Jipeng Tan commented on MAPREDUCE-5: Can you let me know which parameter control reducer memory cache limit? Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException --- Key: MAPREDUCE-5 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 1.1.1 Environment: Sun Java 1.6.0_13, OpenSolaris, running on a SunFire 4150 (x64) 10 node cluster Reporter: George Porter Attachments: temp.rar During the shuffle phase, I'm seeing a large sequence of the following actions: 1) WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_200905181452_0002_m_10_0,0) failed : org.mortbay.jetty.EofException 2) WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_200905181452_0002_m_10_0,0) failed : org.mortbay.jetty.EofException 3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException: Committed The map phase completes with 100%, and then the reduce phase crawls along with the above errors in each of the TaskTracker logs. None of the tasktrackers get lost. When I run non-data jobs like the 'pi' test from the example jar, everything works fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6293) uberized job fails with the job classloader enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387694#comment-14387694
]
Hadoop QA commented on MAPREDUCE-6293:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12707416/MAPREDUCE-6293.001.patch
against trunk revision 90e07d5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestMRTimelineEventHandling
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
org.apache.hadoop.mapred.pipes.TestPipeApplication
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapreduce.TestLargeSort
org.apache.hadoop.mapreduce.TestMRJobClient
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRChildTask
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5355//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5355//console
This message is automatically generated.
uberized job fails with the job classloader enabled
---
Key: MAPREDUCE-6293
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-6293.001.patch
An uberized job fails if the job classloader is enabled and the job needs to
use the thread context classloader to load a class. Some example error in the
log:
{quote}
2015-03-23 23:28:34,675 INFO [main\]
org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader
...
2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\]
cascading.provider.ServiceLoader: unable to find service class:
cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception:
java.lang.ClassNotFoundException:
cascading.tuple.hadoop.collect.HadoopTupleMapFactory
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6299) bzip2 codec read duplicate rows
Keith Ly created MAPREDUCE-6299: --- Summary: bzip2 codec read duplicate rows Key: MAPREDUCE-6299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6299 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.0 Reporter: Keith Ly Assignee: Jason Lowe Priority: Critical select count(*) from bzip_table shows 36 rows count when there are 18 actual rows in bzip_table. Create table bzip_table2 as select * from bzip_table results in 36 rows in bzip_table2 and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4844) Counters.java doesn't obey Java Memory Model
[
https://issues.apache.org/jira/browse/MAPREDUCE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388048#comment-14388048
]
Gera Shegalov commented on MAPREDUCE-4844:
--
Thanks for working on this old JIRA, [~brahmareddy]! Changes to
{{mapred.Counters}} look fine to me. Can you also add a final qualifier to a
bunch maps towards the top of the {{counters.AbstractCounters}} class?
Counters.java doesn't obey Java Memory Model
Key: MAPREDUCE-4844
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4844
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.6.0
Reporter: Gera Shegalov
Assignee: Brahma Reddy Battula
Attachments: MAPREDUCE-4844-002.patch, MAPREDUCE-4844-branch-1.patch
Counters have a number of immutable fields that have not been declared
'final'.
For example, the field groups is not final. It is, however, accessed in a
couple of methods that are declared 'synchronized'. While there is a
happens-before relationship between these methods calls, there is none
between the Counters object initialization and these synchronized methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6293) uberized job fails with the job classloader enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388019#comment-14388019
]
Sangjin Lee commented on MAPREDUCE-6293:
Kicking off the jenkins build one more time. I suspect these test failures are
spurious, as there are no failures related with classloading issues (they're
mostly timeouts and other varierty). The patch is a no-op if the job
classloader is not set.
uberized job fails with the job classloader enabled
---
Key: MAPREDUCE-6293
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6293
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.6.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: MAPREDUCE-6293.001.patch
An uberized job fails if the job classloader is enabled and the job needs to
use the thread context classloader to load a class. Some example error in the
log:
{quote}
2015-03-23 23:28:34,675 INFO [main\]
org.apache.hadoop.mapreduce.v2.util.MRApps: Creating job classloader
...
2015-03-23 23:28:42,096 ERROR [uber-SubtaskRunner\]
cascading.provider.ServiceLoader: unable to find service class:
cascading.tuple.hadoop.collect.HadoopTupleMapFactory, with exception:
java.lang.ClassNotFoundException:
cascading.tuple.hadoop.collect.HadoopTupleMapFactory
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
