Artem Kupchinskiy created SPARK-52769:
-----------------------------------------
Summary: InjectRuntimeFilter is not applied when hints or join
type force the shuffle
Key: SPARK-52769
URL: https://issues.apache.org/jira/browse/SPARK-52769
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0, 3.5.6
Reporter: Artem Kupchinskiy
Currently, the logic of the InjectRuntimeFilter relies on shuffle detection in
its children's plans. Basically, it tries to predict whether a join is gonna be
broadcast or not. Since it uses only data size to determine it, the legit
shuffle cases caused by hints or join type (i.e left/right outer joins when
only key-preserved side fits in broadcast) are overlooked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
