Hi, Could you clarify what's the proposal that you have in mind? From the context I would understand that the newly extracted `BaseCoordinatorContext` would have to be marked as `@PublicEvolving` or `@Experimental`, since otherwise extracting it and keeping `@Internal` wouldn't change much? Such `@Internal` base class could have been removed at any point of time in the future. Having said that, it sounds to me like your proposal is a bit bigger than it looks at the first glance and you actually want to expose the operator coordinator concept to the public API?
AFAIK there were some discussions about that, and it was a bit of a conscious decision to NOT do that. I don't know those reasons however. Only now, I've just heard that there are for example some problems with checkpointing of hypothetical non source operator coordinators. Maybe someone else could shed some light on this? Conceptually I would be actually in favour of exposing operator coordinators if there is a good reason behind that, but it is a more difficult topic and might be a larger effort than it seems at the first glance. Best, Piotrek wt., 4 paź 2022 o 19:41 Steven Wu <stevenz...@gmail.com> napisał(a): > Jing, thanks a lot for your reply. The linked google doc is not for this > FLIP, which is fully documented in the wiki page. The linked google doc is > the design doc to introduce shuffling in Flink Iceberg sink, which > motivated this FLIP proposal so that the shuffle coordinator can leverage > the introduced BaseCoordinatorContext to avoid code duplication. > > On Tue, Oct 4, 2022 at 1:04 AM Jing Ge <j...@ververica.com> wrote: > > > Thanks for bringing this up. It looks overall good! One small thing, you > > might want to write all content on the wiki page instead of linking to a > > google doc. The reason is that some people might not be able to access > the > > google doc. > > > > Best regards, > > Jing > > > > On Tue, Oct 4, 2022 at 3:57 AM gang ye <yegang...@gmail.com> wrote: > > > >> Hi, > >> > >> We submit the Flip proposal > >> < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-264%3A+Extract+BaseCoordinatorContext > >> > > >> at Confluent to extract BaseCoordinatorContext from > >> SourceCoordinatorContext to reuse it for other coordinators E.g. in the > >> shuffling support of Flink Iceberg sink > >> < > >> > https://docs.google.com/document/d/13N8cMqPi-ZPSKbkXGOBMPOzbv2Fua59j8bIjjtxLWqo > >> > > >> > >> Could you help to take a look? > >> Thanks > >> > >> Gang > >> > > >