Ok, thanks for the clarification. Really appreciate your help Kostas On Thu 10 Jan, 2019, 6:19 PM Kostas Kloudas <k.klou...@da-platform.com wrote:
> Hi Taher, > > Well, I would say there is no single class that implements it. > In a nutshell, it is the StreamingFileSink that (through Buckets) tells > the responsible Bucket what to do at each step of the lifecycle of the > Flink operator > (mainly on element, on checkpoint, on checkpoint completed and on restore). > > So I would suggest that you should have a look in these classes. > In essence, the main steps of the "choreography" are similar to the ones > that the older BucketingSink was doing. > > Cheers, > Kostas > > On Thu, Jan 10, 2019 at 10:47 AM Taher Koitawala < > taher.koitaw...@gslab.com> wrote: > >> Hi Kostas, >> Thanks you for the clarification, also can you please >> point how StreamingFileSink uses TwoPhaseCommit. Can you also point out the >> implementing class for that? >> >> >> Regards, >> Taher Koitawala >> GS Lab Pune >> +91 8407979163 >> >> >> On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <kklou...@gmail.com> >> wrote: >> >>> That is correct. >>> The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction >>> because of some internal implementation details. >>> But this, does not mean that it does not implement a two phase commit >>> protocol (which is independent of the implementation). >>> >>> Cheers, >>> Kostas >>> >>> On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala < >>> taher.koitaw...@gslab.com> wrote: >>> >>>> StreamingFileSink extends RichSinkFunction and implements >>>> CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however >>>> TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had >>>> a question if the sink uses the TwoPhaseCommit protocol or not. >>>> >>>> Regards, >>>> Taher Koitawala >>>> >>>> On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kklou...@gmail.com wrote: >>>> >>>>> Hi Taher, >>>>> >>>>> The StreamingFileSink implements a version of TwoPhaseCommit. Can you >>>>> elaborate a bit on what do you mean by " TwoPhaseCommit is not being >>>>> used"? >>>>> >>>>> Cheers, >>>>> Kostas >>>>> >>>>> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala < >>>>> taher.koitaw...@gslab.com> wrote: >>>>> >>>>>> Hi All, >>>>>> As per my understanding and the API of StreamingFileSink, >>>>>> TwoPhaseCommit is not being used. Can someone please confirm is that's >>>>>> right? Also if StreamingFileSink does not support >>>>>> TwoPhaseCommits what is the best way to implement this? >>>>>> >>>>>> >>>>>> Regards, >>>>>> Taher Koitawala >>>>>> GS Lab Pune >>>>>> +91 8407979163 >>>>>> >>>>>
