[
https://issues.apache.org/jira/browse/SPARK-54549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacek Laskowski resolved SPARK-54549.
-------------------------------------
Fix Version/s: 4.2.0
Resolution: Cannot Reproduce
Closing the issue as I can no longer reproduce in 4.2.0-SNAPSHOT built just now
(commit 54cde812c2657717651156636041a84556619a5b).
> [SDP] @dp.create_sink with no flows fails with NoSuchElementException
> ---------------------------------------------------------------------
>
> Key: SPARK-54549
> URL: https://issues.apache.org/jira/browse/SPARK-54549
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.1.0
> Reporter: Jacek Laskowski
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> Executing *spark-pipelines run --full-refresh-all* with *@dp.create_sink* and
> no flows fails with the following {*}NoSuchElementException{*}:
> {noformat}
> java.util.NoSuchElementException: key not found: `my_sink`
> at
> scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:674)
> at scala.collection.immutable.HashMap.apply(HashMap.scala:132)
> at
> org.apache.spark.sql.pipelines.graph.State$.$anonfun$findElementsToReset$5(State.scala:66)
> at scala.collection.immutable.List.flatMap(List.scala:283)
> at scala.collection.immutable.List.flatMap(List.scala:79)
> at
> org.apache.spark.sql.pipelines.graph.State$.findElementsToReset(State.scala:66)
> at org.apache.spark.sql.pipelines.graph.State$.reset(State.scala:75)
> at
> org.apache.spark.sql.pipelines.graph.PipelineExecution.startPipeline(PipelineExecution.scala:50)
> at
> org.apache.spark.sql.pipelines.graph.PipelineExecution.runPipeline(PipelineExecution.scala:67)
> at
> org.apache.spark.sql.connect.pipelines.PipelinesHandler$.$anonfun$startRun$1(PipelinesHandler.scala:382)
> at
> org.apache.spark.sql.connect.pipelines.PipelinesHandler$.$anonfun$startRun$1$adapted(PipelinesHandler.scala:344)
> at scala.util.Using$.resource(Using.scala:296)
> at
> org.apache.spark.sql.connect.pipelines.PipelinesHandler$.startRun(PipelinesHandler.scala:344)
> at
> org.apache.spark.sql.connect.pipelines.PipelinesHandler$.handlePipelinesCommand(PipelinesHandler.scala:118)
> at
> org.apache.spark.sql.connect.planner.SparkConnectPlanner.handlePipelineCommand(SparkConnectPlanner.scala:2898)
> at
> org.apache.spark.sql.connect.planner.SparkConnectPlanner.process(SparkConnectPlanner.scala:2868)
> at
> org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.handlePlan(SparkConnectPlanExecution.scala:95)
> at
> org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:224)
> at
> org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:196)
> at
> org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:394)
> at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
