Hi, @Galen @Yuxia
Your points are valid. Speaking of removing deprecated API, I have the same concern. As a matter of fact, I have been raising it in the discussion thread of API deprecation process[1]. This is another example that we should care about more factors than the migration period, thanks for the hint! I will add one more update into that thread with the reference of this thread. In a nutshell, this thread is focusing on the graduation process. Your valid concerns should be taken care of by the deprecation process. Please don't hesitate to share your thoughts in that thread. Best regards, Jing [1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 On Sun, Jun 25, 2023 at 3:48 AM yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > Thanks Jing for briging this to dicuss. > I agree it's not a blocker for graduting the FileSink to @PublicEvolving > since the Sink which is the rootcause has marked as @PublicEvolving. > But I do also share the same concern with Galen. At least it should be a > blocker for removing StreamingFileSink. > Btw, seems it's really a big headache for migrating to Sink, we may need > to pay more attention to this ticket and try to fix it. > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "Galen Warren" <ga...@cvillewarrens.com.INVALID> > 收件人: "dev" <dev@flink.apache.org> > 发送时间: 星期五, 2023年 6 月 23日 下午 7:47:24 > 主题: Re: [DISCUSS] Graduate the FileSink to @PublicEvolving > > Thanks Jing. I can only offer my perspective on this, others may view it > differently. > > If FileSink is subject to data loss in the "stop-on-savepoint then restart" > scenario, that makes it unusable for me, and presumably for anyone who uses > it in a long-running streaming application and who cannot tolerate data > loss. I still use the (deprecated!) StreamingFileSink for this reason. > > The bigger picture here is that StreamingFileSink is deprecated and will > presumably ultimately be removed, to be replaced with FileSink. Graduating > the status of FileSink seems to be a step along that path; I'm concerned > about continuing down that path with such a critical issue present. > Ultimately, my concern is that FileSink will graduate fully and that > StreamingFileSink will be removed and that there will be no remaining > option to reliably stop/start streaming jobs that write to files without > incurring the risk of data loss. > > I'm sure I'd feel better about things if there were an ongoing effort to > address this FileSink issue and/or a commitment that StreamingFileSink > would not be removed until this issue is addressed. > > My two cents -- thanks. > > > On Fri, Jun 23, 2023 at 1:47 AM Jing Ge <j...@ververica.com.invalid> > wrote: > > > Hi Galen, > > > > Thanks for the hint which is helpful for us to have a clear big picture. > > Afaiac, this will not be a blocking issue for the graduation. There will > > always be some (potential) bugs in the implementation. The API is very > > stable from 2020. The timing is good to graduate. WDYT? > > Furthermore, I'd like to have more opinions. All opinions together will > > help the community build a mature API graduation process. > > > > Best regards, > > Jing > > > > On Tue, Jun 20, 2023 at 12:48 PM Galen Warren > > <ga...@cvillewarrens.com.invalid> wrote: > > > > > Is this issue still unresolved? > > > > > > > https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238 > > > > > > Based on prior discussion, I believe this could lead to data loss with > > > FileSink. > > > > > > > > > > > > On Tue, Jun 20, 2023, 5:41 AM Jing Ge <j...@ververica.com.invalid> > > wrote: > > > > > > > Hi all, > > > > > > > > The FileSink has been marked as @Experimental[1] since Oct. 2020. > > > > According to FLIP-197[2], I would like to propose to graduate it > > > > to @PublicEvloving in the upcoming 1.18 release. > > > > > > > > On the other hand, as a related topic, FileSource was marked > > > > as @PublicEvolving[3] 3 years ago. It deserves a graduation > discussion > > > too. > > > > To keep this discussion lean and efficient, let's focus on FlieSink > in > > > this > > > > thread. There will be another discussion thread for the FileSource. > > > > > > > > I was wondering if anyone might have any concerns. Looking forward to > > > > hearing from you. > > > > > > > > > > > > Best regards, > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java#L129 > > > > [2] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > > > > [3] > > > > > > > > > > > > > > https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/FileSource.java#L95 > > > > > > > > > >
