Hi Biao, Thanks for driving this FLIP. In this FLIP, it introduces "int getFinishedAttempt(int subtaskIndex)" for OutputFormat to know which subtask attempt is the one marked as finished by JM and commit the right data. I have a question about this: does SinkV2 need to do the same thing? Should we/users do it in the committer? If yes, how does the commiter know which one is the right subtask attempt?
Best, Lijie yuxia <luoyu...@alumni.sjtu.edu.cn> 于2022年12月27日周二 10:01写道： > HI, Biao. > Thanks for driving this FLIP. > After quick look of this FLIP, I have a question about "expose the attempt > number which can be used to isolate data produced by speculative executions > with the same subtask id". > What the sink expect to do to isolate data produced by speculative > executions? IIUC, if the taks failover, it also generate a new attempt. > Does it make difference in isolating data produced? > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "Biao Liu" <mmyy1...@gmail.com> > 收件人: "dev" <dev@flink.apache.org> > 发送时间: 星期四, 2022年 12 月 22日 下午 8:16:21 > 主题: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job > > Hi everyone, > > I would like to start a discussion about making Sink support speculative > execution for batch jobs. This proposal is a follow up of "FLIP-168: > Speculative Execution For Batch Job"[1]. Speculative execution is very > meaningful for batch jobs. And it would be more complete after supporting > speculative execution of Sink. Please find more details in the FLIP > document > [2]. > > Looking forward to your feedback. > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-281+Sink+Supports+Speculative+Execution+For+Batch+Job > > Thanks, > Biao /'bɪ.aʊ/ >
