[
https://issues.apache.org/jira/browse/SPARK-61174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145217#comment-18145217
]
Hemanth Boyina commented on SPARK-61174:
----------------------------------------
Hi [~LuciferYang] , are you planning to work on this ? If not I would be
interested to pick this up. Please let me know if that would be okay.
> Built-in V2 file scans should report columnarSupportMode() instead of packing
> every file at planning
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-61174
> URL: https://issues.apache.org/jira/browse/SPARK-61174
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
>
> `FileScan` keeps the default `columnarSupportMode()`, `PARTITION_DEFINED`, so
> `DataSourceV2Strategy` asking `supportsColumnar` plans `inputPartitions`,
> which splits and packs every listed file at planning. With SPARK-30628, a
> scan that a runtime filter prunes packs the surviving files again at
> execution, so with DPP keeping 10 of 10,000 partitions of a table with a
> million files, the driver packs a million files at planning only to learn one
> boolean. V1 packs only the partitions it selects.
> The reader factories decide columnar reads from the schema and the conf, not
> from the partition: Parquet and ORC from their vectorized reader settings and
> the result schema, and CSV, JSON, text and Avro never read columnar. So
> `ParquetScan` and `OrcScan` could report `SUPPORTED` or `UNSUPPORTED` from
> the same check their factories use, and the other four `UNSUPPORTED`, leaving
> the default of `FileScan` for other subclasses.
> Found in the review of #55579 (SPARK-30628).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]