[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627226#comment-16627226 ] Steve Loughran commented on HADOOP-10016: - do we still want this? Or can it be closed as a WONTFIX? > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Sub-task > Components: tools/distcp >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Major > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[
https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629125#comment-16629125
]
Daryn Sharp commented on HADOOP-10016:
--
{quote}This functionality is important for operators to migrate data to a new
Hadoop installation.
{quote}
I don't know if the problem still occurs but the premise is questionable.
Users should be migrating from insecure to secure clusters which I believe
does/should work. My vote is won't fix.
> Distcp should support copy from a secure Hadoop 1 cluster to an insecure
> Hadoop 2 cluster
> -
>
> Key: HADOOP-10016
> URL: https://issues.apache.org/jira/browse/HADOOP-10016
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: tools/distcp
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Major
>
> Distcp should be able to copy from a secure cluster to an insecure cluster.
> This functionality is important for operators to migrate data to a new Hadoop
> installation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785778#comment-13785778 ] Haohui Mai commented on HADOOP-10016: - Indeed this is related to HADOOP-8828. After discussions with [~sureshms], [~jingzhao], [~sanjay.radia], we believe that it is a valid use case that distcp copies data from a secure cluster to an insecure cluster. That is, distcp runs inside the secure cluster and writes to the insecure cluster. The set up is the same as the one described in HADOOP-8828. It is particularly important to support the use case of copying data from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster, since this gives users a path to migrate data from a secure Hadoop 1 cluster to a new installation of insecure Hadoop 2 cluster. The problem here is that in this set up, both distcp and map-reduce try to ask for delegation tokens in order to authenticate with the insecure cluster, in which case the insecure cluster returns an error. Currently Hadoop 2 can mostly handle this case (see HADOOP-10017), since it uses RPCv9 which supports negotiation and fallback during authentication. For Hadoop 1, however, the above use case is fundamentally _broken_ since RPCv8 does not support negotiation at all. What it means is that you cannot write to the insecure Hadoop 2 cluster directly via HDFS. You cannot write to the cluster via WebHDFS either because neither distcp nor map-reduce can successfully get delegation tokens from the insecure cluster. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785783#comment-13785783 ] Haohui Mai commented on HADOOP-10016: - We've tested two approaches and they both work. (1) Fix the WebHDFS client and map-reduce so that they continue even they fail to get a delegation token. (2) In the insecure cluster, issue an artificial delegation token. There is no security hole in the approach (2), since the token will be ignored by the insecure cluster completely. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785795#comment-13785795 ] Haohui Mai commented on HADOOP-10016: - Both approaches might be considered imperfect with the respect to today's implementation. For (1), what we essentially try to achieve is to allow negotiation and fallbacks happen at the application level, but it would be difficult to do it in branch-1 without architectural changes. For (2) someone might consider issuing a token from an insecure cluster counter-intuitive. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786395#comment-13786395 ] Sanjay Radia commented on HADOOP-10016: --- (2) - artificial delegation token, is cleaner. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786400#comment-13786400 ] Sanjay Radia commented on HADOOP-10016: --- Add a config flag for this so as to minimize potential issues in case there are parts of the stack that depend on the old behavior (though I doubt there is). > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786455#comment-13786455 ] Daryn Sharp commented on HADOOP-10016: -- I don't understand the need for an artificial token. Servers are supposed to return null if asked for a token but they don't support tokens. The token acquisition handles this unless bugs were introduced. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[
https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786485#comment-13786485
]
Haohui Mai commented on HADOOP-10016:
-
On a insecure cluster, getDelegationToken returns IOException:
{noformat}
$ curl http://localhost:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN
{"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"Failed
to get the token for dr.who, user=dr.who"}}
{noformat}
which eventually fails this job. I agree the fact that returning null is not a
valid part of the contract of getDelegationToken(), but there are multiple bugs
in the code right now that hinder distcp running successfully.
> Distcp should support copy from a secure Hadoop 1 cluster to an insecure
> Hadoop 2 cluster
> -
>
> Key: HADOOP-10016
> URL: https://issues.apache.org/jira/browse/HADOOP-10016
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> Distcp should be able to copy from a secure cluster to an insecure cluster.
> This functionality is important for operators to migrate data to a new Hadoop
> installation.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786727#comment-13786727 ] Sanjay Radia commented on HADOOP-10016: --- Context: * Copying data from Secure 2.x to Insecure 2.x works because of rpc v9 (the reverse, I think will also works). * Want to copy data via DistCp from insecure 1.x to Secure 2.x - this does not work - the issue is similar to the next one. * Want to copy data via DistCp from secure 1.x to insecure 2.x - this fails as described below. Currently an insecure cluster returns *null* for getDelegationToken(). 2.x clients do freakout on this null token but this is fixed by Hadoop-10017. But the key problem is as follows. * Distcp job runs in Secure1.x cluster, and it tries connect to NN in Insecure2.x. * Because security is enabled (distcp is running in Secure1.x cluster), it sees that it has no tokens for that cluster (recall none were obtained because a null was returned); then it tries to do a kerberos based authentication; this fails because it has no kerberos credentials (it is running as a MR job) - even the fallback to insecure does not work because it fails *before* the RPC-connection. Solution - have the NN in Insecure2.x return an artificial-token. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
[ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900585#comment-13900585 ] Daryn Sharp commented on HADOOP-10016: -- bq. it (client) tries to do a kerberos based authentication; this fails because it has no kerberos credentials (it is running as a MR job) - even the fallback to insecure does not work because it fails before the RPC-connection. To clarify, this is the behavior of the v8 client, correct? v9 clients should not attempt kerberos unless requested. If the client conf claims security is enabled, it will fortunately create a ugi that claims it's kerberos even if there's no tgt. I think that causes the v9 client to attempt and fail kerberos in a task. I thought I fixed UGI to not return kerberos when it is really simple, but it was either undone or not committed. bq. Solution - have the NN in Insecure2.x return an artificial-token. Solution that doesn't involve adding more code paths - always enable the secret manager irregardless of security setting. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure > Hadoop 2 cluster > - > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. > This functionality is important for operators to migrate data to a new Hadoop > installation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
