[jira] [Resolved] (MAPREDUCE-5249) Oozie delegation token renewal fails for MR tokens in branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkat Ranganathan resolved MAPREDUCE-5249. --- Resolution: Duplicate Oozie delegation token renewal fails for MR tokens in branch-1 -- Key: MAPREDUCE-5249 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5249 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, security Affects Versions: 1.1.2 Reporter: Venkat Ranganathan When Oozie java action is executed, the following shows up in the job tracker log. 013-05-14 21:51:45,643 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65 64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93 e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: 192.168.56.101:50300. Not rescheduled org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client jt/condor-sec.venkat@venkat.org tries to renew a token with renewer specified as mapred at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267) at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source) at org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578) at org.apache.hadoop.security.token.Token.renew(Token.java:309) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Setting the renewer to Kerberos Local name does not help because AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but JobTracker.renewDelegationToken uses the fullName. This essentially causes the renewal to fail -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5375) Delegation Token renewal exception in jobtracker logs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700373#comment-13700373
]
Venkat Ranganathan commented on MAPREDUCE-5375:
---
Thanks [~vinodkv] and [~sseth] for help with understanding and fixing this
issue. I had created MAPREDUCE-5249 - Resolved that was duplicate.
Delegation Token renewal exception in jobtracker logs
-
Key: MAPREDUCE-5375
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5375
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Priority: Critical
Attachments: MRDelegationIssue-MR-5375.patch
Filing on behalf of [~venkatnrangan] who found this originally and provided a
patch.
Saw this in the JT logs while oozie tests were running with Hadoop.
When Oozie java action is executed, the following shows up in the job tracker
log.
{code}
ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65
64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e
6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93
e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: ip:50300. Not
rescheduled
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.security.AccessControlException: Client jt/h...@domain.com
tries to renew a token with renewer specified as mapred
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267)
at
org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown
Source)
at
org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578)
at org.apache.hadoop.security.token.Token.renew(Token.java:309)
at
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221)
at
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
{code}
Setting the renewer to Kerberos Local name does not help because
AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but
JobTracker.renewDelegationToken uses the fullName. This essentially causes
the renewal to fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA,
[jira] [Assigned] (MAPREDUCE-5249) Oozie delegation token renewal fails for MR tokens in branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkat Ranganathan reassigned MAPREDUCE-5249: - Assignee: Venkat Ranganathan Oozie delegation token renewal fails for MR tokens in branch-1 -- Key: MAPREDUCE-5249 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5249 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, security Affects Versions: 1.1.2 Reporter: Venkat Ranganathan Assignee: Venkat Ranganathan When Oozie java action is executed, the following shows up in the job tracker log. 013-05-14 21:51:45,643 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65 64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93 e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: 192.168.56.101:50300. Not rescheduled org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client jt/condor-sec.venkat@venkat.org tries to renew a token with renewer specified as mapred at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267) at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source) at org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578) at org.apache.hadoop.security.token.Token.renew(Token.java:309) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Setting the renewer to Kerberos Local name does not help because AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but JobTracker.renewDelegationToken uses the fullName. This essentially causes the renewal to fail -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5249) Oozie delegation token renewal fails for MR tokens in branch-1
Venkat Ranganathan created MAPREDUCE-5249: - Summary: Oozie delegation token renewal fails for MR tokens in branch-1 Key: MAPREDUCE-5249 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5249 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, security Affects Versions: 1.1.2 Reporter: Venkat Ranganathan When Oozie java action is executed, the following shows up in the job tracker log. 013-05-14 21:51:45,643 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 07 68 64 70 75 73 65 72 06 6d 61 70 72 65 64 26 6f 6f 7a 69 65 2f 63 6f 6e 64 6f 72 2d 73 65 63 2e 76 65 6e 6b 61 74 2e 6f 72 67 40 76 65 6e 6b 61 74 2e 6f 72 67 8a 01 3e a6 87 5e 5b 8a 01 3e ca 93 e2 5b 02 02, Kind: MAPREDUCE_DELEGATION_TOKEN, Service: 192.168.56.101:50300. Not rescheduled org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client jt/condor-sec.venkat@venkat.org tries to renew a token with renewer specified as mapred at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267) at org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3878) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at org.apache.hadoop.mapred.$Proxy8.renewDelegationToken(Unknown Source) at org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:578) at org.apache.hadoop.security.token.Token.renew(Token.java:309) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Setting the renewer to Kerberos Local name does not help because AbstractDelegationTokenIdentifier sets the renewer to Kerberos shortname but JobTracker.renewDelegationToken uses the fullName. This essentially causes the renewal to fail -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-branch-1.patch.3
Patch for branch1
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-branch-1.patch.3,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2, MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-trunk.patch.2
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652233#comment-13652233
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
Thanks for the comments Jitendra. I have modified the patch with all the
comments and uploaded a new patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652317#comment-13652317
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
This fix has been manually test with a secure cluster with branch-1
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-branch-1.patch.2
branch-1 patch after addressing Jitendra's comments
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13652545#comment-13652545
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
Thanks [~sseth]. I have updated the trunk patch as you suggested and also
changed the variable name to tokenFile as suggested by [~jnp].
I am uploading the trunk patch first and will upload the branch-1 patch with
new variable name after precommit tests with trunk patch are done
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-trunk.patch.3
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2, 2.0.4-alpha
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-branch-1.patch.2, MAPREDUCE-5217-trunk.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch.2,
MAPREDUCE-5217-trunk.patch.3
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
Venkat Ranganathan created MAPREDUCE-5217:
-
Summary: DistCp fails when launched by Oozie in a secure cluster
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it would
break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217.patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Status: Patch Available (was: Open)
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Status: Open (was: Patch Available)
Uploading trunk patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Component/s: security
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: (was: MAPREDUCE-5217.patch)
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-trunk.patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Affects Version/s: trunk
Status: Patch Available (was: Open)
The same issue is in trunk
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2, trunk
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Status: Open (was: Patch Available)
Uploaded wrong patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2, trunk
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-trunk.patch
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Status: Patch Available (was: Open)
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: 1.1.2, trunk
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651553#comment-13651553
]
Venkat Ranganathan commented on MAPREDUCE-5217:
---
This patch checks an environment variable and sets the
mapreduce.job.credentials.binary property. It has been validated by running
oozie distcp action on a secure cluster
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5217) DistCp fails when launched by Oozie in a secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venkat Ranganathan updated MAPREDUCE-5217:
--
Attachment: MAPREDUCE-5217-branch-1.patch
Patch for branch-1
DistCp fails when launched by Oozie in a secure cluster
---
Key: MAPREDUCE-5217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp, security
Affects Versions: trunk, 1.1.2
Environment: Hadoop secure cluster
Reporter: Venkat Ranganathan
Assignee: Venkat Ranganathan
Attachments: MAPREDUCE-5217-branch-1.patch,
MAPREDUCE-5217-trunk.patch, MAPREDUCE-5217-trunk.patch
As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
in the main launcher for Pig, Hive, MR and Sqoop actions.
if (System.getenv(HADOOP_TOKEN_FILE_LOCATION) != null) {
jobConf.set(mapreduce.job.credentials.binary,
System.getenv(HADOOP_TOKEN_FILE_LOCATION));
}
For Java action, which does not have a main launcher in oozie, the above
codecan be added by the user as the user purportedly has the code that is
launched.
But for DistCp action, the user has no such luxury. The solution attempted in
MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it
would break MAPREDUCE-3727. So, we have to fix DistCp and
add the same boilerplate code so that DistCp action can be launched by Oozie
in a secure cluster.
The code added checks for an System env. variable to be set which is not
typically set in normal command line execution of DistCp, DistCp runs fine
with commnad line usage both in secure and non-secure cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
