[
https://issues.apache.org/jira/browse/HIVE-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Zhang updated HIVE-26758:
----------------------------
Description:
The query results are staged in stagingdir that is relative to the destination
path <destination_dir>/<staging_dir>/
during blobstorage optimzation HIVE-17620 final job is set to use stagingdir.
HIVE-15215 mentioned the possibility of using scratch for staging when write to
S3 but it was long time ago and no activity.
This is to allow final job to use hive.exec.scratchdir as the interim jobs,
with a configuration
hive.use.scratchdir.for.staging
This is useful for cross Filesystem, user can use local source filesystem
instead of remote filesystem for the staging.
was:
The query results are staged in stagingdir that is relative to the destination
path <destination_dir>/<staging_dir>/
It used to be able to change hive.exec.stagingdir for a different location, but
that is lost during blobstorage optimzation HIVE-17620.
HIVE-15215 mentioned the possibility of using scratch for staging when write to
S3 but it was long time ago and no activity.
This is to allow final job to use hive.exec.scratchdir as the interim jobs,
with a configuration
hive.use.scratchdir.for.staging
This is useful for cross Filesystem, user can use local source filesystem
instead of remote filesystem for the staging.
> Allow use scratchdir for staging final job
> ------------------------------------------
>
> Key: HIVE-26758
> URL: https://issues.apache.org/jira/browse/HIVE-26758
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
> Affects Versions: 4.0.0-alpha-2
> Reporter: Yi Zhang
> Assignee: Yi Zhang
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The query results are staged in stagingdir that is relative to the
> destination path <destination_dir>/<staging_dir>/
> during blobstorage optimzation HIVE-17620 final job is set to use stagingdir.
> HIVE-15215 mentioned the possibility of using scratch for staging when write
> to S3 but it was long time ago and no activity.
>
> This is to allow final job to use hive.exec.scratchdir as the interim jobs,
> with a configuration
> hive.use.scratchdir.for.staging
> This is useful for cross Filesystem, user can use local source filesystem
> instead of remote filesystem for the staging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
