[
https://issues.apache.org/jira/browse/HDFS-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781245#comment-16781245
]
Wei-Chiu Chuang edited comment on HDFS-14323 at 3/1/19 3:36 AM:
----------------------------------------------------------------
Thanks for reporting the issue, [~smajeti]. We ran distcp between a hadoop 3
and a hadoop 2 over webhdfs, but we didn't spot the issue you described.
[~zvenczel] could you take a look since you worked on HDFS-13582?
was (Author: jojochuang):
Thanks for reporting the issue, [~smajeti]. We ran distcp between a hadoop 3
and a hadoop 2 over webhdfs and we didn't spot the issue you described before.
[~zvenczel] could you take a look since you worked on HDFS-13582?
> Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters
> in hdfs file path
> -----------------------------------------------------------------------------------------------
>
> Key: HDFS-14323
> URL: https://issues.apache.org/jira/browse/HDFS-14323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 3.2.0
> Reporter: Srinivasu Majeti
> Priority: Major
>
> There was an enhancement to allow semicolon in source/target URLs for distcp
> use case as part of HDFS-13176 and backward compatibility fix as part of
> HDFS-13582 . Still there seems to be an issue when trying to trigger distcp
> from 3.x cluster to pull webhdfs data from 2.x hadoop cluster. We might need
> to deal with existing fix as described below by making sure if url is already
> encoded or not. That fixes it.
> diff --git
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
>
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> index 5936603c34a..dc790286aff 100644
> ---
> a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> +++
> b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
> @@ -609,7 +609,10 @@ URL toUrl(final HttpOpParam.Op op, final Path fspath,
> boolean pathAlreadyEncoded = false;
> try {
> fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
> - pathAlreadyEncoded = true;
> + if(!fspathUri.getPath().equals(fspathUriDecoded))
> + {
> + pathAlreadyEncoded = true;
> + }
> } catch (IllegalArgumentException ex) {
> LOG.trace("Cannot decode URL encoded file", ex);
> }
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
