L. C. Hsieh created SPARK-34321:
-----------------------------------
Summary: Fix the guarantee of foreachBatch
Key: SPARK-34321
URL: https://issues.apache.org/jira/browse/SPARK-34321
Project: Spark
Issue Type: Documentation
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
Similar to SPARK-28650, {{foreachBatch}} API document also documents the
guarantee:
The batchId can be used to deduplicate and transactionally write the output
(that is, the provided Dataset) to external systems. The output Dataset is
guaranteed to be exactly the same for the same batchId
But like the reason of fixing the document of {{ForeachWriter}} in SPARK-28650,
it is not hard to break the guarantee by changing the partition number.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
