Marc Arndt created SPARK-27850: ---------------------------------- Summary: Make SparkPlan#doExecuteBroadcast public Key: SPARK-27850 URL: https://issues.apache.org/jira/browse/SPARK-27850 Project: Spark Issue Type: Improvement Components: Optimizer, Spark Core, SQL Affects Versions: 2.4.3 Reporter: Marc Arndt
The handling of broadcasts of SparkPlan objects is handled inside the SparkPlan#executeBroadcast method. According to the documentation of SparkPlan to provide custom broadcast functionality the `doExecuteBroadcast` method should be overriden as indicated by the comment: {code:scala} /** * Returns the result of this query as a broadcast variable by delegating to `doExecuteBroadcast` * after preparations. * * Concrete implementations of SparkPlan should override `doExecuteBroadcast`. */ final def executeBroadcast[T](): broadcast.Broadcast[T] = executeQuery { if (isCanonicalizedPlan) { throw new IllegalStateException("A canonicalized plan is not supposed to be executed.") } doExecuteBroadcast() } {code} When looking at the definition of SparkPlan#doExecuteBroadcast: {code:scala} /** * Produces the result of the query as a broadcast variable. * * Overridden by concrete implementations of SparkPlan. */ protected[sql] def doExecuteBroadcast[T](): broadcast.Broadcast[T] = { throw new UnsupportedOperationException(s"$nodeName does not implement doExecuteBroadcast") } {code} it becomes apparent that it is not possible to override the method from user-defined SparkPlan implementations, because the method has been defined as package protected. To allow custom SparkPlan implementations to provide their own broadcast operations I ask to change the SparkPlan#doExecuteBroadcast to be a public method, so that all SparkPlan implementations, independent of the package they belong to, can override it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org