Kris Mok created SPARK-43973:
--------------------------------
Summary: Structured Streaming UI should display failed queries
correctly
Key: SPARK-43973
URL: https://issues.apache.org/jira/browse/SPARK-43973
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 3.4.0, 3.3.0, 3.2.0, 3.1.0
Reporter: Kris Mok
The Structured Streaming UI is designed to be able to show a query's status
(active/finished/failed) and if failed, the error message.
Due to a bug in the implementation, the error message in
{{QueryTerminatedEvent}} isn't being tracked by the UI data, so in turn the UI
always shows failed queries as "finished".
Example:
{code:scala}
implicit val ctx = spark.sqlContext
import org.apache.spark.sql.execution.streaming.MemoryStream
spark.conf.set("spark.sql.ansi.enabled", "true")
val inputData = MemoryStream[(Int, Int)]
val df = inputData.toDF().selectExpr("_1 / _2 as a")
inputData.addData((1, 2), (3, 4), (5, 6), (7, 0))
val testQuery =
df.writeStream.format("memory").queryName("kristest").outputMode("append").start
testQuery.processAllAvailable()
{code}
Here we intentionally fail a query, but the Spark UI's Structured Streaming tab
would show this as "FINISHED" without any errors, which is wrong.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
