Kezhu Wang created FLINK-21578:
----------------------------------
Summary: Closeable Sink Committer/GlobalCommitter were created to
function in onestep during job graph composition
Key: FLINK-21578
URL: https://issues.apache.org/jira/browse/FLINK-21578
Project: Flink
Issue Type: Improvement
Components: API / DataStream
Affects Versions: 1.13.0
Reporter: Kezhu Wang
Normally, functions/operators are created in job graph composition phase for
serialization and transmission. Them are "opened" in flink cluster to function.
This two steps procedure succeed in that there will be no resource-cleanup
requirement in job graph composition phase.
While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
they were created in job graph composition phase.
Following are fixes I could image if we converge to "this is problematic".
# Add {{open}} or similar method for these two classes.
# Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
{{createCommitter}} and others not optional(enforce this in runtime).
Personally, I am a bit preferring second approach for possible less code path
touching in job graph composition phase. But first approach has advantage that
it could be an no breaking change.
There might be other approaches though.
cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
