[
https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-38013:
------------------------------
Description:
An example to reproduce the bug.
{code:java}
create table t1 as select 1 c1, 2 c2;
create table t2 as select 1 c1, 2 c2;
create table t3 as select 1 c1, 2 c2;
set spark.sql.adaptive.autoBroadcastJoinThreshold=-1;
select /*+ merge(t3) */ * from t1
left join (
select c1 as c from t3
) t3 on t1.c1 = t3.c
left join (
select /*+ repartition(c1) */ c1 from t2
) t2 on t1.c1 = t2.c1;
{code}
The key to produce this bug is that a bhj convert to smj/shj without
introducing extra shuffe and AQE does not think the join can be planned as bhj.
was:
An example to reproduce the bug.
{code:java}
create table t1 as select 1 c1, 2 c2;
create table t2 as select 1 c1, 2 c2;
create table t3 as select 1 c1, 2 c2;
set spark.sql.adaptive.autoBroadcastJoinThreshold=-1;
select /*+ merge(t3) */ * from t1
left join (
select c1 as c from t3
) t3 on t1.c1 = t3.c
left join (
select /*+ repartition(c1) */ c1 from t2
) t2 on t1.c1 = t2.c1;
{code}
The key to produce this bug is that a bhj convert to smj/shj without
introducing extra shuffe.
> Fix AQE can change bhj to smj if no extra shuffle introduce
> -----------------------------------------------------------
>
> Key: SPARK-38013
> URL: https://issues.apache.org/jira/browse/SPARK-38013
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: XiDuo You
> Priority: Major
>
> An example to reproduce the bug.
> {code:java}
> create table t1 as select 1 c1, 2 c2;
> create table t2 as select 1 c1, 2 c2;
> create table t3 as select 1 c1, 2 c2;
> set spark.sql.adaptive.autoBroadcastJoinThreshold=-1;
> select /*+ merge(t3) */ * from t1
> left join (
> select c1 as c from t3
> ) t3 on t1.c1 = t3.c
> left join (
> select /*+ repartition(c1) */ c1 from t2
> ) t2 on t1.c1 = t2.c1;
> {code}
> The key to produce this bug is that a bhj convert to smj/shj without
> introducing extra shuffe and AQE does not think the join can be planned as
> bhj.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
