[ https://issues.apache.org/jira/browse/IMPALA-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell resolved IMPALA-8581. ----------------------------------- Fix Version/s: Not Applicable Resolution: Duplicate https://github.com/apache/impala/commit/ac87278b169422091af1c03fcd2101516372defb > Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where > applicable > -------------------------------------------------------------------------------------- > > Key: IMPALA-8581 > URL: https://issues.apache.org/jira/browse/IMPALA-8581 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Affects Versions: Impala 3.3.0 > Reporter: Joe McDonnell > Priority: Minor > Labels: newbie > Fix For: Not Applicable > > > When doing a put or copyFromLocal via the HDFS commandline, it puts the file > into a temporary file and then moves it into place. The "-d" option skips the > temporary file and copies the file directly into place. On S3, this avoids an > extra copy and can also avoid consistency issues. We should look into using > the "-d" option across our tests. Our usages of the HDFS commandline > generally don't require atomicity of this file copy, so using "-d" is > unlikely to impact HDFS tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org