Huicheng Song created SPARK-38536:
-------------------------------------
Summary: Spark 3 can not read mixed format partitions
Key: SPARK-38536
URL: https://issues.apache.org/jira/browse/SPARK-38536
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.1, 3.0.0
Reporter: Huicheng Song
Spark 3.x reads partitions with table's input format, which fails when the
partition has a different input format than the table.
This is a regression introduced by SPARK-26630. Before that fix, Spark will use
Partition InputFormat when creating HadoopRDD. With that fix, Spark uses only
Table InputFormat when creating HadoopRDD, causing failures
Reading mixed format partitions is an import scenario, especially for format
migration. It is also well supported in query engines like Hive and Presto.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
