Hi Team, Did some experimenting and found the originally proposed solution to be a bit awkward for cases where WithPostCommitTopology was needed but we do not need the WithPreCommitTopology transformation. The flexibility of the new API would be better if we would use a mixin like approach. The interfaces would only be used to define the specific required methods, and they would not need to extend the original TwoPhaseCommittingSink interface too.
Since the interfaces WithPreCommitTopology and the WithPostCommitTopology interfaces are still Experimental, after talking to Gyula offline, I have updated the FLIP to use this new approach. Any comments, thoughts are welcome. Thanks, Peter Péter Váry <peter.vary.apa...@gmail.com> ezt írta (időpont: 2023. okt. 5., Cs, 16:04): > Hi Team, > > In my previous email[1] I have described our challenges migrating the > existing Iceberg SinkFunction based implementation, to the new SinkV2 based > implementation. > > As a result of the discussion around that topic, I have created the > FLIP-371 [2] to address the Committer related changes, and now I have > created a companion FLIP-372 [3] to address the WithPreCommitTopology > related issues. > > FLIP-372: Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the > type of the Committable > > The main goal of the FLIP-372 is to extend the currently existing > TwoPhaseCommittingSink API by adding the possibility to have a > PreCommitTopology where the input of and the output types of the pre commit > transformation are different. > > Here is the FLIP: FLIP-372: Allow TwoPhaseCommittingSink > WithPreCommitTopology to alter the type of the Committable > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable> > > Please use this thread to discuss the FLIP related questions, proposals, > and feedback. > > Thanks, > Peter > > - [1] https://lists.apache.org/thread/h3kg7jcgjstpvwlhnofq093vk93ylgsn > - [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink > - [3] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable >
