Raghu Angadi created SPARK-45957: ------------------------------------ Summary: SQL on streaming Temp view fails Key: SPARK-45957 URL: https://issues.apache.org/jira/browse/SPARK-45957 Project: Spark Issue Type: Bug Components: Connect, Structured Streaming Affects Versions: 4.0.0 Reporter: Raghu Angadi Fix For: 4.0.0
The following code fails in the last step with Spark Connect. The root cause is that Connect server triggers physical plan on a streaming Dataframe [in SparkConnectPlanner.scala|https://github.com/apache/spark/blob/334d952f9555cbfad8ef84987d6f978eb6b37b9b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L2591]. Better to avoid that entirely, but at least for streaming it should be avoided since it cannot be done with a batch execution engine. {code:java} df = spark.readStream.format("rate").option("numPartitions", "1").load() df.createOrReplaceTempView("temp_view") view_df = spark.sql("SELECT * FROM temp_view") // FAILS{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org