[ https://issues.apache.org/jira/browse/HIVE-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yi Zhang updated HIVE-26758: ---------------------------- Description: The query results are staged in stagingdir that is relative to the destination path <destination_dir>/<staging_dir>/ It used to be able to change hive.exec.stagingdir for a different location, but that is lost during blobstorage optimzation HIVE-17620. This is to allow final job to use hive.exec.scratchdir as the interim jobs, with a configuration hive.use.scratchdir_for_staging This is useful for cross Filesystem, user can use local source filesystem instead of remote filesystem for the staging. main change: for dynamic partitions that has static partition it was <destination_dir>/<static_partition>/<staging_dir>/<dynamic_partition> changes to <destination_dir>/<staging_dir>/<static_partition>/<dynamic_partition> or in case of \{hive.use.scratchdir_for_staging} <scratch_dir>/<static_partition>/<dynamic_partition> the change is due to that hive relies on parsing the path to discover partitions. was: The query results are staged in stagingdir that is relative to the destination path <destination_dir>/<staging_dir>/ It used to be able to change hive.exec.stagingdir for a different location, but that is lost during blobstorage optimzation HIVE-17620. This is to allow final job to use hive.exec.scratchdir as the interim jobs, with a configuration hive.use.scratchdir_for_staging This is useful for cross Filesystem, user can use local source filesystem instead of remote filesystem for the staging. > Allow use scratchdir for staging > -------------------------------- > > Key: HIVE-26758 > URL: https://issues.apache.org/jira/browse/HIVE-26758 > Project: Hive > Issue Type: New Feature > Components: Query Planning > Affects Versions: 4.0.0-alpha-2 > Reporter: Yi Zhang > Assignee: Yi Zhang > Priority: Minor > > The query results are staged in stagingdir that is relative to the > destination path <destination_dir>/<staging_dir>/ > It used to be able to change hive.exec.stagingdir for a different location, > but that is lost during blobstorage optimzation HIVE-17620. > This is to allow final job to use hive.exec.scratchdir as the interim jobs, > with a configuration > hive.use.scratchdir_for_staging > This is useful for cross Filesystem, user can use local source filesystem > instead of remote filesystem for the staging. > main change: > for dynamic partitions that has static partition it was > <destination_dir>/<static_partition>/<staging_dir>/<dynamic_partition> > changes to > <destination_dir>/<staging_dir>/<static_partition>/<dynamic_partition> > or in case of \{hive.use.scratchdir_for_staging} > <scratch_dir>/<static_partition>/<dynamic_partition> > the change is due to that hive relies on parsing the path to discover > partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)