[
https://issues.apache.org/jira/browse/SPARK-60702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-60702:
-------------------------------
Affects Version/s: 5.0.0
(was: 4.4.0)
> Don't skip a runtime bloom filter for a DPP filter that physical planning
> drops
> -------------------------------------------------------------------------------
>
> Key: SPARK-60702
> URL: https://issues.apache.org/jira/browse/SPARK-60702
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> With the default
> {{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}},
> {{PartitionPruning}} adds an {{onlyInBroadcast}} DPP filter on a join key
> that the scan offers for runtime filtering, such as a partition column.
> {{InjectRuntimeFilter}} skips a key that has a DPP filter, so that key gets
> no bloom filter. In a shuffle join there is no broadcast to reuse, so
> physical planning replaces the DPP filter with {{true}}. The key ends up with
> no runtime filter at all, and the application side is shuffled unfiltered.
> Measured on master with AQE on and off:
> * a V1 Parquet table partitioned by the join key, in a shuffle join;
> * a V2 scan that offers the source column of a {{bucket(N, id)}} partition
> transform for runtime filtering, as Iceberg's does, in a shuffle join and in
> a storage-partitioned join.
> In each case, turning DPP off brings the bloom filter back.
> A possible fix: an {{onlyInBroadcast}} DPP filter does not keep
> {{InjectRuntimeFilter}} from adding a bloom filter on a join that will not
> broadcast.
> Discussed on apache/spark#58635:
> https://github.com/apache/spark/pull/58635#issuecomment-6468432320
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
