Zhen Wang created SPARK-40600:
---------------------------------
Summary: Support recursiveFileLookup for partitioned datasource
Key: SPARK-40600
URL: https://issues.apache.org/jira/browse/SPARK-40600
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.1
Environment: Spark: 3.1.1
Hive: 3.1.2
Reporter: Zhen Wang
I use hive tez engine to execute union statement and insert into partitioned
table may generate HIVE_UNION_SUBDIR subdirectory, and when I use spark sql to
read this partitioned table, the data below HIVE_UNION_SUBDIR is not read.
For non-partitioned table, I can read the subdirectories of the table when
setting recursiveFileLookup to true, but for partitioned table, it seems
impossible to set recursiveFileLookup to true.
So I want to support recursiveFileLookup for partitioned table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
