Peter Toth created SPARK-60702:
----------------------------------
Summary: Don't skip a runtime bloom filter for a DPP filter that
physical planning drops
Key: SPARK-60702
URL: https://issues.apache.org/jira/browse/SPARK-60702
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
With the default
{{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}},
{{PartitionPruning}} adds an {{onlyInBroadcast}} DPP filter on a join key that
the scan offers for runtime filtering, such as a partition column.
{{InjectRuntimeFilter}} skips a key that has a DPP filter, so that key gets no
bloom filter. In a shuffle join there is no broadcast to reuse, so physical
planning replaces the DPP filter with {{true}}. The key ends up with no runtime
filter at all, and the application side is shuffled unfiltered.
Measured on master with AQE on and off:
* a V1 Parquet table partitioned by the join key, in a shuffle join;
* a V2 scan that offers the source column of a {{bucket(N, id)}} partition
transform for runtime filtering, as Iceberg's does, in a shuffle join and in a
storage-partitioned join.
In each case, turning DPP off brings the bloom filter back.
A possible fix: an {{onlyInBroadcast}} DPP filter does not keep
{{InjectRuntimeFilter}} from adding a bloom filter on a join that will not
broadcast.
Discussed on apache/spark#58635:
https://github.com/apache/spark/pull/58635#issuecomment-6468432320
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
