Jungtaek Lim created SPARK-43183:
------------------------------------
Summary: Move update event on idleness in streaming query listener
to separate callback method
Key: SPARK-43183
URL: https://issues.apache.org/jira/browse/SPARK-43183
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.5.0
Reporter: Jungtaek Lim
People has been having a lot of confusions about update event on idleness; it’s
not only the matter of understanding but also comes up with various types of
complaints. For example, since we give the latest batch ID for update event on
idleness, if the listener implementation blindly performs upsert based on batch
ID, they are in risk to lose metrics.
This also complicates the logic because we have to memorize the execution for
the previous batch, which is arguably not necessary.
Because of this, we’d be better to move the idle event out of progress update
event and have separate callback method for this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
