[ https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218827#comment-17218827 ]
Cheng Su commented on SPARK-33207: ---------------------------------- If you filter out empty `FilePartition` here, then `FileScanRDD` will have less partitions ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala#L210] ). Then this is not consistent with `outputPartitioning` ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L295-L296] ). For join operators, it needs to zip two RDDs from left and right side with same number of partitions. So I think this will cause issue if we join the bucketed table with another table. > Reduce the number of tasks launched after bucket pruning > -------------------------------------------------------- > > Key: SPARK-33207 > URL: https://issues.apache.org/jira/browse/SPARK-33207 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: Yuming Wang > Priority: Major > Attachments: image-2020-10-22-15-17-01-389.png, > image-2020-10-22-15-17-26-956.png > > > We only need to read 1 bucket, but it still launch 200 tasks. > {code:sql} > create table test_bucket using parquet clustered by (ID) sorted by (ID) into > 200 buckets AS (SELECT id FROM range(1000) cluster by id) > spark-sql> explain select * from test_bucket where id = 4; > == Physical Plan == > *(1) Project [id#7L] > +- *(1) Filter (isnotnull(id#7L) AND (id#7L = 4)) > +- *(1) ColumnarToRow > +- FileScan parquet default.test_bucket[id#7L] Batched: true, > DataFilters: [isnotnull(id#7L), (id#7L = 4)], Format: Parquet, Location: > InMemoryFileIndex[file:/root/spark-3.0.1-bin-hadoop3.2/spark-warehouse/test_bucket], > PartitionFilters: [], PushedFilters: [IsNotNull(id), EqualTo(id,4)], > ReadSchema: struct<id:bigint>, SelectedBucketsCount: 1 out of 200 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org