[
https://issues.apache.org/jira/browse/SPARK-36630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408901#comment-17408901
]
gaoyajun02 commented on SPARK-36630:
------------------------------------
Found my issue is similar to https://issues.apache.org/jira/browse/SPARK-35264.
I can set the autoBroadcastThreshold to -1 to disable logical plan stats,
and set spark.sql.adaptive.autoBroadcastJoinThreshold to control the threshold
for physical stats
> Add the option to use physical statistics to avoid large tables being
> broadcast
> -------------------------------------------------------------------------------
>
> Key: SPARK-36630
> URL: https://issues.apache.org/jira/browse/SPARK-36630
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: gaoyajun02
> Priority: Major
>
> Currently when AQE's queryStage is not materialized, it uses the stats of the
> logical plan to estimate whether the plan can be converted to BHJ, and in
> some scenarios the estimated value is several orders of magnitude smaller
> than the actual broadcast data, which can lead to large tables being broadcast
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
