wuyi created SPARK-31384:
----------------------------
Summary: Fix NPE in OptimizeSkewedJoin
Key: SPARK-31384
URL: https://issues.apache.org/jira/browse/SPARK-31384
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: wuyi
When there's a inputRDD of a plan with 0 partitions, rule OptimizeSkewedJoin
can hit NPE.
The issue can be reproduced by below test:
{code:java}
withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
withTempView("t2") {
// create DataFrame with 0 partition
spark.createDataFrame(sparkContext.emptyRDD[Row], new StructType().add("b",
IntegerType))
.createOrReplaceTempView("t2")
// should run successfully without NPE
runAdaptiveAndVerifyResult("SELECT * FROM testData2 t1 left semi join t2 ON
t1.a=t2.b")
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
