[ https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-33207: -------------------------------- Fix Version/s: 3.2.0 > Reduce the number of tasks launched after bucket pruning > -------------------------------------------------------- > > Key: SPARK-33207 > URL: https://issues.apache.org/jira/browse/SPARK-33207 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: Yuming Wang > Priority: Major > Fix For: 3.2.0 > > Attachments: Screen Shot 2021-02-05 at 11.44.12 AM.png, > image-2020-10-22-15-17-01-389.png, image-2020-10-22-15-17-26-956.png > > > We only need to read 1 bucket, but it still launch 200 tasks. > {code:sql} > create table test_bucket using parquet clustered by (ID) sorted by (ID) into > 200 buckets AS (SELECT id FROM range(1000) cluster by id) > spark-sql> explain select * from test_bucket where id = 4; > == Physical Plan == > *(1) Project [id#7L] > +- *(1) Filter (isnotnull(id#7L) AND (id#7L = 4)) > +- *(1) ColumnarToRow > +- FileScan parquet default.test_bucket[id#7L] Batched: true, > DataFilters: [isnotnull(id#7L), (id#7L = 4)], Format: Parquet, Location: > InMemoryFileIndex[file:/root/spark-3.0.1-bin-hadoop3.2/spark-warehouse/test_bucket], > PartitionFilters: [], PushedFilters: [IsNotNull(id), EqualTo(id,4)], > ReadSchema: struct<id:bigint>, SelectedBucketsCount: 1 out of 200 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org