Gengliang Wang created SPARK-30627: -------------------------------------- Summary: Disable all the V2 file sources in Spark 3.0 by default Key: SPARK-30627 URL: https://issues.apache.org/jira/browse/SPARK-30627 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.0.0 Reporter: Gengliang Wang Assignee: Gengliang Wang
There are still some missing parts in the file source V2 framework: 1. It doesn't support reporting file scan metrics such as "numOutputRows"/"numFiles"/"fileSize" like `FileSourceScanExec`. 2. It doesn't support partition pruning with subqueries or dynamic partition pruning. As we are going to code freeze on Jan 31st, I suggest disabling all the V2 file sources in Spark 3.0 by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org