[
https://issues.apache.org/jira/browse/HIVE-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-15215:
--------------------------------
Summary: Investigate if staging data on S3 can always go under the scratch
dir (was: Files on S3 are deleted one by one in INSERT OVERWRITE queries)
> Investigate if staging data on S3 can always go under the scratch dir
> ---------------------------------------------------------------------
>
> Key: HIVE-15215
> URL: https://issues.apache.org/jira/browse/HIVE-15215
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Reporter: Sahil Takiar
>
> When running {{INSERT OVERWRITE}} queries the files to overwrite are deleted
> one by one. The reason is that, by default, hive.exec.stagingdir is inside
> the target table directory.
> Ideally Hive would just delete the entire table directory, but it can't do
> that since the staging data is also inside the directory. Instead it deletes
> each file one-by-one, which is very slow.
> There are a few ways to fix this:
> 1: Move the staging directory outside the table location. This can be done by
> setting hive.exec.stagingdir to a different location when running on S3. It
> would be nice if users didn't have to explicitly set this when running on S3
> and things just worked out-of-the-box. My understanding is that
> hive.exec.stagingdir was only added to support HDFS encryption zones. Since
> S3 doesn't have encryption zones, there should be no problem with using the
> value of hive.exec.scratchdir to store all intermediate data instead.
> 2: Multi-thread the delete operations
> 3: See if the {{S3AFileSystem}} can expose some type of bulk delete op
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
