[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654910#comment-13654910 ] Siddharth Seth commented on MAPREDUCE-5217: --- +1 for the branch-1 patch. Committing based on the comments. Thanks Venkat for the patch, and Akejandro, Daryn and Jitendra for taking a look. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654854#comment-13654854 ] Alejandro Abdelnur commented on MAPREDUCE-5217: --- [~daryn], yep this is for hadoop 1, in Hadoop 2 distcp works just fine. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654767#comment-13654767 ] Daryn Sharp commented on MAPREDUCE-5217: Boilerplate code like this is _wrong_, at least for versions later than 1.x. It's unreasonable to expect every tool that might be run within the context of a job to be aware of the internal implementation details of both the UGI (its env var) and mapreduce.job.credentials.binary. I'd personally like to see the conf key removed. If a tools wants to control its token set, my opinion is it should create a new UGI for the submission. The vast majority of tools, like distcp, have no need for special logic just in case its running within a job. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654643#comment-13654643 ] Alejandro Abdelnur commented on MAPREDUCE-5217: --- Sidd, you are correct, the patch for branch-1 is needed (as branch-1 has distcp v1), for branch-2 it is not needed. THX > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653557#comment-13653557 ] Siddharth Seth commented on MAPREDUCE-5217: --- Alejandro, does Oozie work on secure clusters with both branch-1 and branch-2 ? Looking at the code, it looks like branch-2 will work - since credentials are copied over from the current ugi by the Job class in branch-2 (Job(JobConf conf)). However, branch-1 is missing this code. I'm in favour of committing this to branch-1, rather than making changes to branch-1 client code - since there's a reasonable difference between the client code on the two branches, and this isn't necessarily the correct behaviour - if Jobs are trying to control their token set. Will create a separate jira for this. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653059#comment-13653059 ] Alejandro Abdelnur commented on MAPREDUCE-5217: --- Venkat, Oozie works with distcp on secure clusters. Please check OOZIE-611 which is adding the missing documentation for it. Please check if this solves your problem. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652783#comment-13652783 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582430/MAPREDUCE-5217-branch-1.patch.3 against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3603//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652580#comment-13652580 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582389/MAPREDUCE-5217-trunk.patch.3 against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-distcp. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3600//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3600//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2, > MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652545#comment-13652545 ] Venkat Ranganathan commented on MAPREDUCE-5217: --- Thanks [~sseth]. I have updated the trunk patch as you suggested and also changed the variable name to tokenFile as suggested by [~jnp]. I am uploading the trunk patch first and will upload the branch-1 patch with new variable name after precommit tests with trunk patch are done > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2, > MAPREDUCE-5217-trunk.patch.3 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652416#comment-13652416 ] Siddharth Seth commented on MAPREDUCE-5217: --- Venkat, the patch looks good. Couple of minor nits though. - In the trunk patch, the credentials constant can refer to MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY - Rename delegationTokenFiles to delegationTokenFile > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652327#comment-13652327 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582358/MAPREDUCE-5217-branch-1.patch.2 against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3597//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652321#comment-13652321 ] Jitendra Nath Pandey commented on MAPREDUCE-5217: - +1 > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652317#comment-13652317 ] Venkat Ranganathan commented on MAPREDUCE-5217: --- This fix has been manually test with a secure cluster with branch-1 > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652250#comment-13652250 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582343/MAPREDUCE-5217-trunk.patch.2 against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-distcp. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3596//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3596//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652233#comment-13652233 ] Venkat Ranganathan commented on MAPREDUCE-5217: --- Thanks for the comments Jitendra. I have modified the patch with all the comments and uploaded a new patch > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch, > MAPREDUCE-5217-trunk.patch.2 > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652168#comment-13652168 ] Jitendra Nath Pandey commented on MAPREDUCE-5217: - Also, the job conf is being set twice. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2, 2.0.4-alpha > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13652162#comment-13652162 ] Jitendra Nath Pandey commented on MAPREDUCE-5217: - 1. Please use the LOG object for logging instead of System.out. 2. Minor: 'delegationToken' variable name should be renamed to tokenFile, because it actually represents the file name that contains tokens, and could also contain tokens other than delegation tokens. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651558#comment-13651558 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582240/MAPREDUCE-5217-branch-1.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3586//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651553#comment-13651553 ] Venkat Ranganathan commented on MAPREDUCE-5217: --- This patch checks an environment variable and sets the mapreduce.job.credentials.binary property. It has been validated by running oozie distcp action on a secure cluster > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-branch-1.patch, > MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651448#comment-13651448 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582196/MAPREDUCE-5217-trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-distcp. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3582//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3582//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651376#comment-13651376 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582170/MAPREDUCE-5217-trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3581//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp, security >Affects Versions: trunk, 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217-trunk.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650873#comment-13650873 ] Hadoop QA commented on MAPREDUCE-5217: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582093/MAPREDUCE-5217.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3579//console This message is automatically generated. > DistCp fails when launched by Oozie in a secure cluster > --- > > Key: MAPREDUCE-5217 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 1.1.2 > Environment: Hadoop secure cluster >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: MAPREDUCE-5217.patch > > > As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in > in the main launcher for Pig, Hive, MR and Sqoop actions. > if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) { > jobConf.set("mapreduce.job.credentials.binary", > System.getenv("HADOOP_TOKEN_FILE_LOCATION")); > } > For Java action, which does not have a main launcher in oozie, the above > codecan be added by the user as the user purportedly has the code that is > launched. > But for DistCp action, the user has no such luxury. The solution attempted in > MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it > would break MAPREDUCE-3727. So, we have to fix DistCp and > add the same boilerplate code so that DistCp action can be launched by Oozie > in a secure cluster. > The code added checks for an System env. variable to be set which is not > typically set in normal command line execution of DistCp, DistCp runs fine > with commnad line usage both in secure and non-secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira