[ https://issues.apache.org/jira/browse/SPARK-43310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-43310: --------------------------------- Component/s: Structured Streaming > Dataset.observe is ignored when writing to Kafka with batch query > ----------------------------------------------------------------- > > Key: SPARK-43310 > URL: https://issues.apache.org/jira/browse/SPARK-43310 > Project: Spark > Issue Type: Bug > Components: SQL, Structured Streaming > Affects Versions: 3.3.2, 3.4.0 > Reporter: David Deuber > Priority: Major > > When writing to Kafka with a batch query, metrics defined with > {{Dataset.observe}} are not recorded. > For example, > {code:java} > import org.apache.spark.sql.execution.QueryExecution > import org.apache.spark.sql.util.QueryExecutionListener > spark.listenerManager.register(new QueryExecutionListener { > override def onSuccess(funcName: String, qe: QueryExecution, durationNs: > Long): Unit = { > println(qe.observedMetrics) > } > override def onFailure(funcName: String, qe: QueryExecution, exception: > Exception): Unit = { > //pass > } > }) > val df = Seq(("k", "v")).toDF("key", "value") > val observed = df.observe("my_observation", > lit("metric_value").as("some_metric")) > observed > .write > .format("kafka") > .option("kafka.bootstrap.servers", "host1:port1") > .option("topic", "topic1") > .save() > {code} > prints {{{}Map(){}}}. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org