[jira] [Commented] (HDFS-15916) Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff
[
https://issues.apache.org/jira/browse/HDFS-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308776#comment-17308776
]
Jason Wen commented on HDFS-15916:
--
Yes, it makes sense for sure if we can make it work with the mentioned fix.
In general I would like to see the compatibility between 3.x client to 2.x
server, but I am not sure if there is any more road blocks to achieve that. If
for this particular case we can make it work with few changes, it's definitely
worth fixing that.
> Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for
> snapshotdiff
>
>
> Key: HDFS-15916
> URL: https://issues.apache.org/jira/browse/HDFS-15916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.2
>Reporter: Srinivasu Majeti
>Priority: Major
>
> Looks like when using distcp diff options between two snapshots from a hadoop
> 3 cluster to hadoop 2 cluster , we get below exception and seems to be break
> backward compatibility due to new API introduction
> getSnapshotDiffReportListing.
>
> {code:java}
> hadoop distcp -diff s1 s2 -update src_cluster_path dst_cluster_path
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcNoSuchMethodException):
> Unknown method getSnapshotDiffReportListing called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15916) Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff
[
https://issues.apache.org/jira/browse/HDFS-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308711#comment-17308711
]
Ayush Saxena commented on HDFS-15916:
-
Well in any case if older client isn't able to connect to newer server, that
would be a big problem, but in this case also I don't think there is any harm
supporting a newer client to connect to an older server by means of a fallback.
DistCp is a tool commonly used for migration and usually the data copy is done
at the load(newer cluster) since the source cluster is serving the client
requests till the migration is complete.
And DistCp is a widely used util, supporting new client to old server should be
good, most of the commonly used APIs do work that way barring some.
IMO adding a fallback won't have any performance implications in the present
code, so no resistance that way, Just something like below in
\{{getSnapshotDiffReportInternal}} (I suppose)
{code:java}
try {
report = dfs.getSnapshotDiffReportListing(snapshotDir, fromSnapshot,
toSnapshot, startPath, index);
} catch (RpcNoSuchMethodException e) {
return dfs.getSnapshotDiffReport(snapshotDir, fromSnapshot, toSnapshot);
}{code}
There is a fallback for a case in the API today also:
{code:java}
// In case the diff needs to be computed between a snapshot and the current
// tree, we should not do iterative diffReport computation as the iterative
// approach might fail if in between the rpc calls the current tree
// changes in absence of the global fsn lock.
if (!isValidSnapshotName(fromSnapshot) || !isValidSnapshotName(
toSnapshot)) {
return dfs.getSnapshotDiffReport(snapshotDir, fromSnapshot, toSnapshot);
}{code}
[~zhenshan.wen] does that makes sense now?
> Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for
> snapshotdiff
>
>
> Key: HDFS-15916
> URL: https://issues.apache.org/jira/browse/HDFS-15916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.2
>Reporter: Srinivasu Majeti
>Priority: Major
>
> Looks like when using distcp diff options between two snapshots from a hadoop
> 3 cluster to hadoop 2 cluster , we get below exception and seems to be break
> backward compatibility due to new API introduction
> getSnapshotDiffReportListing.
>
> {code:java}
> hadoop distcp -diff s1 s2 -update src_cluster_path dst_cluster_path
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcNoSuchMethodException):
> Unknown method getSnapshotDiffReportListing called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15916) Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff
[
https://issues.apache.org/jira/browse/HDFS-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308221#comment-17308221
]
Jason Wen commented on HDFS-15916:
--
Is it supposed to be supported between Hadoop 3.x client and 2.x server?
In this case, distcp source is Hadoop 3 HDFS client and destination is Hadoop 2
HDFS server.
> Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for
> snapshotdiff
>
>
> Key: HDFS-15916
> URL: https://issues.apache.org/jira/browse/HDFS-15916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.2
>Reporter: Srinivasu Majeti
>Priority: Major
>
> Looks like when using distcp diff options between two snapshots from a hadoop
> 3 cluster to hadoop 2 cluster , we get below exception and seems to be break
> backward compatibility due to new API introduction
> getSnapshotDiffReportListing.
>
> {code:java}
> hadoop distcp -diff s1 s2 -update src_cluster_path dst_cluster_path
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcNoSuchMethodException):
> Unknown method getSnapshotDiffReportListing called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15916) Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff
[
https://issues.apache.org/jira/browse/HDFS-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307625#comment-17307625
]
Srinivasu Majeti commented on HDFS-15916:
-
Hi [~ayushtkn] , Backporting or modifying hadoop 2.x brach may not be helping
any easy way for newer products like CDP not supporting them . Fallback
mechanism what you said could be made to work in my opinion.
> Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for
> snapshotdiff
>
>
> Key: HDFS-15916
> URL: https://issues.apache.org/jira/browse/HDFS-15916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.2
>Reporter: Srinivasu Majeti
>Priority: Major
>
> Looks like when using distcp diff options between two snapshots from a hadoop
> 3 cluster to hadoop 2 cluster , we get below exception and seems to be break
> backward compatibility due to new API introduction
> getSnapshotDiffReportListing.
>
> {code:java}
> hadoop distcp -diff s1 s2 -update src_cluster_path dst_cluster_path
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcNoSuchMethodException):
> Unknown method getSnapshotDiffReportListing called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15916) Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff
[
https://issues.apache.org/jira/browse/HDFS-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307617#comment-17307617
]
Ayush Saxena commented on HDFS-15916:
-
Yeps, Seems to be an issue, I feel two ways to proceed, First is backport the
new Api to branch-2.10, this will make things work or secondly add a fallback
mechanism, if the new API throws {{RpcNoSuchMethodException}} use the old API
or something of that sort
> Backward compatibility - Distcp fails from Hadoop 3 to Hadoop 2 for
> snapshotdiff
>
>
> Key: HDFS-15916
> URL: https://issues.apache.org/jira/browse/HDFS-15916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.2
>Reporter: Srinivasu Majeti
>Priority: Major
>
> Looks like when using distcp diff options between two snapshots from a hadoop
> 3 cluster to hadoop 2 cluster , we get below exception and seems to be break
> backward compatibility due to new API introduction
> getSnapshotDiffReportListing.
>
> {code:java}
> hadoop distcp -diff s1 s2 -update src_cluster_path dst_cluster_path
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcNoSuchMethodException):
> Unknown method getSnapshotDiffReportListing called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
