Jose Torres created SPARK-23558:
-----------------------------------
Summary: clean up StreamWriter factory lifecycle
Key: SPARK-23558
URL: https://issues.apache.org/jira/browse/SPARK-23558
Project: Spark
Issue Type: Sub-task
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Jose Torres
Right now, StreamWriter and children have different lifecycles in continuous
processing and microbatch mode. Both execution modes impose significant
constraints on what that lifecycle must be, so the achievable consistent
semantic is:
* StreamWriter lasts for the duration of the query execution
* DataWriterFactory lasts for the duration of the query execution
* DataWriter (the task-level writer) has a lifecycle tied to each individual
epoch
This also allows us to restore the implicit semantic that
DataWriter.commit()/abort() terminates the lifecycle.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
