Hello Quanlong Huang, Qifan Chen, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17316
to look at the new patch set (#2).
Change subject: IMPALA-10658: LOAD DATA INPATH silently fails between HDFS and
Azure ABFS
......................................................................
IMPALA-10658: LOAD DATA INPATH silently fails between HDFS and Azure ABFS
LOAD DATA INPATH silently fails when Impala tries to move files from
HDFS to ABFS. The problem is that we use FileSystem.makeQualified(Path)
to decide if path is on a given filesystem. We expect to get an
IllegalArgumentException if path is on a different filesystem. However,
the Azure FileSystem implementation doesn't throw this exception.
Because of that Impala thinks that an 'hdfs://' path and an 'abfs://'
path is on the same filesystem, so it tries to move files with
FileSystem.rename(). In case of errors rename() might throw an
Exception, or return false. Impala doesn't check the return value,
therefore if rename() returns false then the error remains silent.
This patch fixes Impala's isPathOnFileSystem() and also adds a check
for the return value of rename().
Testing:
* tested manually between HDFS and Azure ABFS.
* added JUnit test to FileSystemUtilTest
Change-Id: Id807e8a200b83283a09d3a917185cabab930017d
---
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
2 files changed, 53 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/17316/2
--
To view, visit http://gerrit.cloudera.org:8080/17316
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id807e8a200b83283a09d3a917185cabab930017d
Gerrit-Change-Number: 17316
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
