[
https://issues.apache.org/jira/browse/IMPALA-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-8581.
-----------------------------------
Fix Version/s: Not Applicable
Resolution: Duplicate
https://github.com/apache/impala/commit/ac87278b169422091af1c03fcd2101516372defb
> Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where
> applicable
> --------------------------------------------------------------------------------------
>
> Key: IMPALA-8581
> URL: https://issues.apache.org/jira/browse/IMPALA-8581
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.3.0
> Reporter: Joe McDonnell
> Priority: Minor
> Labels: newbie
> Fix For: Not Applicable
>
>
> When doing a put or copyFromLocal via the HDFS commandline, it puts the file
> into a temporary file and then moves it into place. The "-d" option skips the
> temporary file and copies the file directly into place. On S3, this avoids an
> extra copy and can also avoid consistency issues. We should look into using
> the "-d" option across our tests. Our usages of the HDFS commandline
> generally don't require atomicity of this file copy, so using "-d" is
> unlikely to impact HDFS tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
