Andy Grove created SPARK-34682: ---------------------------------- Summary: Regression in "operating on canonicalized plan" check in CustomShuffleReaderExec Key: SPARK-34682 URL: https://issues.apache.org/jira/browse/SPARK-34682 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.1.1 Reporter: Andy Grove Fix For: 3.2.0, 3.1.2
In Spark 3.0.2 if I attempt to execute on a canonicalized version of CustomShuffleReaderExec I get an error "operating on canonicalized plan", as expected. There is a regression in Spark 3.1.1 where this check can never be reached because of a new call to sendDriverMetrics that was added prior to the check. This method will fail if operating on a canonicalized plan because it assumes the existence of metrics that do not exist if this is a canonicalized plan. {code:java} private lazy val shuffleRDD: RDD[_] = { sendDriverMetrics() shuffleStage.map { stage => stage.shuffle.getShuffleRDD(partitionSpecs.toArray) }.getOrElse { throw new IllegalStateException("operating on canonicalized plan") } }{code} The specific error looks like this: {code:java} java.util.NoSuchElementException: key not found: numPartitions at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:101) at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:99) at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.sendDriverMetrics(CustomShuffleReaderExec.scala:122) at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.shuffleRDD$lzycompute(CustomShuffleReaderExec.scala:182) at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.shuffleRDD(CustomShuffleReaderExec.scala:181) at org.apache.spark.sql.execution.adaptive.CustomShuffleReaderExec.doExecuteColumnar(CustomShuffleReaderExec.scala:196) {code} I think the fix is simply to avoid calling sendDriverMetrics if the plan is canonicalized and I am planning on creating a PR to fix this. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org