Re: GlobalCommitter in Flink's two-phase commit
Hi, 1. What are the general usage scenarios of GlobalCommitter? - GlobalCommitter is used for creating and committing an aggregated committable. It is part of a 2-phase-commit protocol. One use case is the compaction of small files. 2. Why should GlobalCommitter be removed in the new version of the api? - As FLIP-191 described, there are many different requirement from different downstream systems, e.g. Iceberg, Delta lake, Hive. One GlobalCommitter could not cover all of them. If you take a look at the SinkV1Adapter source code, you will see that StandardSinkTopologies#addGlobalCommitter, which is recommended to replace the usage of GlobalCommitter, is used to take care of the post commit topology. Best regards, Jing On Tue, May 24, 2022 at 9:11 AM di wu <676366...@qq.com> wrote: > Hello > Regarding the GlobalCommitter in Flink's two-phase commit, > I see it was introduced in FLIP-143, but it seems to have been removed > again in FLP-191 and marked as Deprecated in the source code. > I haven't found any relevant information about the use of GlobalCommitter. > > There are two questions I would like to ask: > 1. What are the general usage scenarios of GlobalCommitter? > 2. Why should GlobalCommitter be removed in the new version of the api? > > Thanks && Regards, > di.wu > >
Re: GlobalCommitter in Flink's two-phase commit
Hi, 1. What are the general usage scenarios of GlobalCommitter? - GlobalCommitter is used for creating and committing an aggregated committable. It is part of a 2-phase-commit protocol. One use case is the compaction of small files. 2. Why should GlobalCommitter be removed in the new version of the api? - As FLIP-191 described, there are many different requirement from different downstream systems, e.g. Iceberg, Delta lake, Hive. One GlobalCommitter could not cover all of them. If you take a look at the SinkV1Adapter source code, you will see that StandardSinkTopologies#addGlobalCommitter, which is recommended to replace the usage of GlobalCommitter, is used to take care of the post commit topology. Best regards, Jing On Tue, May 24, 2022 at 9:11 AM di wu <676366...@qq.com> wrote: > Hello > Regarding the GlobalCommitter in Flink's two-phase commit, > I see it was introduced in FLIP-143, but it seems to have been removed > again in FLP-191 and marked as Deprecated in the source code. > I haven't found any relevant information about the use of GlobalCommitter. > > There are two questions I would like to ask: > 1. What are the general usage scenarios of GlobalCommitter? > 2. Why should GlobalCommitter be removed in the new version of the api? > > Thanks && Regards, > di.wu > >
GlobalCommitter in Flink's two-phase commit
Hello Regarding the GlobalCommitter in Flink's two-phase commit, I see it was introduced in FLIP-143, but it seems to have been removed again in FLP-191 and marked as Deprecated in the source code. I haven't found any relevant information about the use of GlobalCommitter. There are two questions I would like to ask: 1. What are the general usage scenarios of GlobalCommitter? 2. Why should GlobalCommitter be removed in the new version of the api? Thanks Regards, di.wu
GlobalCommitter in Flink's two-phase commit
Hello Regarding the GlobalCommitter in Flink's two-phase commit, I see it was introduced in FLIP-143, but it seems to have been removed again in FLP-191 and marked as Deprecated in the source code. I haven't found any relevant information about the use of GlobalCommitter. There are two questions I would like to ask: 1. What are the general usage scenarios of GlobalCommitter? 2. Why should GlobalCommitter be removed in the new version of the api? Thanks Regards, di.wu
