Jacek Laskowski created SPARK-54549:
---------------------------------------
Summary: [SDP] @dp.create_sink with no flows fails with
NoSuchElementException
Key: SPARK-54549
URL: https://issues.apache.org/jira/browse/SPARK-54549
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Jacek Laskowski
Executing *spark-pipelines run --full-refresh-all* with *@dp.create_sink* and
no flows fails with the following {*}NoSuchElementException{*}:
{noformat}
java.util.NoSuchElementException: key not found: `my_sink`
at
scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:674)
at scala.collection.immutable.HashMap.apply(HashMap.scala:132)
at
org.apache.spark.sql.pipelines.graph.State$.$anonfun$findElementsToReset$5(State.scala:66)
at scala.collection.immutable.List.flatMap(List.scala:283)
at scala.collection.immutable.List.flatMap(List.scala:79)
at
org.apache.spark.sql.pipelines.graph.State$.findElementsToReset(State.scala:66)
at org.apache.spark.sql.pipelines.graph.State$.reset(State.scala:75)
at
org.apache.spark.sql.pipelines.graph.PipelineExecution.startPipeline(PipelineExecution.scala:50)
at
org.apache.spark.sql.pipelines.graph.PipelineExecution.runPipeline(PipelineExecution.scala:67)
at
org.apache.spark.sql.connect.pipelines.PipelinesHandler$.$anonfun$startRun$1(PipelinesHandler.scala:382)
at
org.apache.spark.sql.connect.pipelines.PipelinesHandler$.$anonfun$startRun$1$adapted(PipelinesHandler.scala:344)
at scala.util.Using$.resource(Using.scala:296)
at
org.apache.spark.sql.connect.pipelines.PipelinesHandler$.startRun(PipelinesHandler.scala:344)
at
org.apache.spark.sql.connect.pipelines.PipelinesHandler$.handlePipelinesCommand(PipelinesHandler.scala:118)
at
org.apache.spark.sql.connect.planner.SparkConnectPlanner.handlePipelineCommand(SparkConnectPlanner.scala:2898)
at
org.apache.spark.sql.connect.planner.SparkConnectPlanner.process(SparkConnectPlanner.scala:2868)
at
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.handlePlan(SparkConnectPlanExecution.scala:95)
at
org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:224)
at
org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:196)
at
org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:394)
at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
