[ https://issues.apache.org/jira/browse/SPARK-43973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gengliang Wang reassigned SPARK-43973: -------------------------------------- Assignee: Kris Mok > Structured Streaming UI should display failed queries correctly > --------------------------------------------------------------- > > Key: SPARK-43973 > URL: https://issues.apache.org/jira/browse/SPARK-43973 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 3.1.0, 3.2.0, 3.3.0, 3.4.0 > Reporter: Kris Mok > Assignee: Kris Mok > Priority: Major > > The Structured Streaming UI is designed to be able to show a query's status > (active/finished/failed) and if failed, the error message. > Due to a bug in the implementation, the error message in > {{QueryTerminatedEvent}} isn't being tracked by the UI data, so in turn the > UI always shows failed queries as "finished". > Example: > {code:scala} > implicit val ctx = spark.sqlContext > import org.apache.spark.sql.execution.streaming.MemoryStream > spark.conf.set("spark.sql.ansi.enabled", "true") > val inputData = MemoryStream[(Int, Int)] > val df = inputData.toDF().selectExpr("_1 / _2 as a") > inputData.addData((1, 2), (3, 4), (5, 6), (7, 0)) > val testQuery = > df.writeStream.format("memory").queryName("kristest").outputMode("append").start > testQuery.processAllAvailable() > {code} > Here we intentionally fail a query, but the Spark UI's Structured Streaming > tab would show this as "FINISHED" without any errors, which is wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org