Hi everyone,
A commonly used functionality offered by Flink is the
"cancel-with-savepoint" operation. When applied to the current exactly-once
sinks, the current implementation of the feature can be problematic, as it
does not guarantee that side-effects will be committed by Flink to the 3rd
party storage system.
This discussion targets fixing this issue and proposes the addition of two
termination modes, namely:
1) SUSPEND, for temporarily stopping the job, e.g. for Flink version
upgrading in your cluster
2) TERMINATE, for terminal shut down which ends the stream and sends
MAX_WATERMARK time, and flushes any state associated with (event time)
timers
A google doc with the FLIP proposal can be found here:
https://docs.google.com/document/d/1EZf6pJMvqh_HeBCaUOnhLUr9JmkhfPgn6Mre_z6tgp8/edit?usp=sharing
And the page for the FLIP is here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212
The implementation sketch is far from complete, but it is worth having a
discussion on the semantics as soon as possible. The implementation section
is going to be updated soon.
Looking forward to the discussion,
Kostas
--
Kostas Kloudas | Software Engineer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
