[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654643#comment-13654643
]
Alejandro Abdelnur commented on MAPREDUCE-5217:
---
Sidd, you are correct, the patch for branch-1 is needed (as branch-1 has distcp
v1), for branch-2 it is not needed. THX
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654767#comment-13654767
]
Daryn Sharp commented on MAPREDUCE-5217:
Boilerplate code like this is _wrong_, at least for versions later than 1.x.
It's unreasonable to expect every tool that might be run within the context of
a job to be aware of the internal implementation details of both the UGI (its
env var) and mapreduce.job.credentials.binary. I'd personally like to see the
conf key removed.
If a tools wants to control its token set, my opinion is it should create a new
UGI for the submission. The vast majority of tools, like distcp, have no need
for special logic just in case its running within a job.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654854#comment-13654854
]
Alejandro Abdelnur commented on MAPREDUCE-5217:
---
[~daryn], yep this is for hadoop 1, in Hadoop 2 distcp works just fine.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654910#comment-13654910
]
Siddharth Seth commented on MAPREDUCE-5217:
---
+1 for the branch-1 patch. Committing based on the comments. Thanks Venkat for
the patch, and Akejandro, Daryn and Jitendra for taking a look.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652783#comment-13652783
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582430/MAPREDUCE-5217-branch-1.patch.3
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3603//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653059#comment-13653059
]
Alejandro Abdelnur commented on MAPREDUCE-5217:
---
Venkat, Oozie works with distcp on secure clusters. Please check OOZIE-611
which is adding the missing documentation for it. Please check if this solves
your problem.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13653557#comment-13653557
]
Siddharth Seth commented on MAPREDUCE-5217:
---
Alejandro, does Oozie work on secure clusters with both branch-1 and branch-2 ?
Looking at the code, it looks like branch-2 will work - since credentials are
copied over from the current ugi by the Job class in branch-2 (Job(JobConf
conf)). However, branch-1 is missing this code.
I'm in favour of committing this to branch-1, rather than making changes to
branch-1 client code - since there's a reasonable difference between the client
code on the two branches, and this isn't necessarily the correct behaviour - if
Jobs are trying to control their token set. Will create a separate jira for
this.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652162#comment-13652162
]
Jitendra Nath Pandey commented on MAPREDUCE-5217:
-
1. Please use the LOG object for logging instead of System.out.
2. Minor: 'delegationToken' variable name should be renamed to tokenFile,
because it actually represents the file name that contains tokens, and could
also contain tokens other than delegation tokens.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652168#comment-13652168
]
Jitendra Nath Pandey commented on MAPREDUCE-5217:
-
Also, the job conf is being set twice.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652233#comment-13652233
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
Thanks for the comments Jitendra. I have modified the patch with all the
comments and uploaded a new patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652250#comment-13652250
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582343/MAPREDUCE-5217-trunk.patch.2
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3596//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3596//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652317#comment-13652317
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
This fix has been manually test with a secure cluster with branch-1
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652321#comment-13652321
]
Jitendra Nath Pandey commented on MAPREDUCE-5217:
-
+1
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652327#comment-13652327
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582358/MAPREDUCE-5217-branch-1.patch.2
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3597//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652416#comment-13652416
]
Siddharth Seth commented on MAPREDUCE-5217:
---
Venkat, the patch looks good. Couple of minor nits though.
- In the trunk patch, the credentials constant can refer to
MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY
- Rename delegationTokenFiles to delegationTokenFile
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652545#comment-13652545
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
Thanks [~sseth]. I have updated the trunk patch as you suggested and also
changed the variable name to tokenFile as suggested by [~jnp].
I am uploading the trunk patch first and will upload the branch-1 patch with
new variable name after precommit tests with trunk patch are done
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652580#comment-13652580
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582389/MAPREDUCE-5217-trunk.patch.3
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3600//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3600//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650873#comment-13650873
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582093/MAPREDUCE-5217.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3579//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651376#comment-13651376
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582170/MAPREDUCE-5217-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3581//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651448#comment-13651448
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582196/MAPREDUCE-5217-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3582//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3582//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651553#comment-13651553
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
This patch checks an environment variable and sets the
mapreduce.job.credentials.binary property. It has been validated by running
oozie distcp action on a secure cluster
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651558#comment-13651558
]
Hadoop QA commented on MAPREDUCE-5217:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582240/MAPREDUCE-5217-branch-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3586//console
This message is automatically generated.
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
