Hi Shammon, Thanks for the clarification. Just out of curiosity, if the context is not part of the event, why should it be the input parameter of each onEvent call?
Best regards, Jing On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu <xbjt...@gmail.com> wrote: > Thanks Shammon for the timely update, the updated FLIP looks good to me. > > Hope to see the vote thread and following FLIP-314 discussion thread. > > Best, > Leonard > > > On Jun 6, 2023, at 5:04 PM, Shammon FY <zjur...@gmail.com> wrote: > > > > Hi, > > > > Thanks for all the feedback. > > > > For @Jing Ge, > > I forget to update the demo code in the FLIP, the method is > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there > > is no `onEvent(CatalogModificationEvent)`. I have updated the code. > Context > > contains some additional information that is not part of an Event, but > > needs to be used in the listener, so we separate it from the event. > > > > For @Panagiotis, > > I think `ioExecutor` make sense to me and I have added it in > > `ContextModificationContext`, thanks > > > > For @Leonard, > > Thanks for your input. > > 1. I have updated `CatalogModificationContext` as an interface, as well > as > > Context in CatalogModificationListenerFactory > > 2. Configuration sounds good to me, I have updated the method name and > > getConfiguration in Context > > > > For @David, > > Yes, you're right. The listener will only be used on the client side and > > won't introduce a new code path for running per-job/per-session jobs. The > > listener will be created in `TableEnvironment` and `SqlGateway` which > can a > > `CatalogManager` with the listener. > > > > > > Best, > > Shammon FY > > > > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek <david.mora...@gmail.com> > > wrote: > > > >> Hi, > >> > >> Thanks for the FLIP! Data lineage is an important problem to tackle. > >> > >> Can you please expand on how this is planned to be wired into the > >> JobManager? As I understand, the listeners will be configured globally > (per > >> cluster), so this won't introduce a new code path for running per-job / > >> per-session user code. Is that correct? > >> > >> Best, > >> D > >> > >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu <xbjt...@gmail.com> wrote: > >> > >>> Thanks Shammon for driving this FLIP forward, I’ve several comments > about > >>> the updated FLIP. > >>> > >>> 1. CatalogModificationContext is introduced as a class instead of an > >>> interface, is it a typo? > >>> > >>> 2. The FLIP defined multiple Map<String, String> config(); methods in > >>> some Context classes, Could we use Configuration > >> getConfiguration()；Class > >>> org.apache.flink.configuration.Configuration is recommend as it’s > public > >>> API and offers more useful methods as well. > >>> > >>> 3. The Context of CatalogModificationListenerFactory should be an > >>> interface too, and getUserClassLoder() > >>> would be more aligned with flink’s naming style. > >>> > >>> > >>> Best, > >>> Leonard > >>> > >>>> On May 26, 2023, at 4:08 PM, Shammon FY <zjur...@gmail.com> wrote: > >>>> > >>>> Hi devs, > >>>> > >>>> We would like to bring up a discussion about FLIP-294: Support > >> Customized > >>>> Job Meta Data Listener[1]. We have had several discussions with Jark > >> Wu, > >>>> Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > >> and > >>>> interfaces, and thanks for their valuable advice. > >>>> The overall job and connector information is divided into metadata and > >>>> lineage, this FLIP focuses on metadata and lineage will be discussed > in > >>>> another FLIP in the future. In this FLIP we want to add a customized > >>>> listener in Flink to report catalog modifications to external metadata > >>>> systems such as datahub[2] or atlas[3]. Users can view the specific > >>>> information of connectors such as source and sink for Flink jobs in > >> these > >>>> systems, including fields, watermarks, partitions, etc. > >>>> > >>>> Looking forward to hearing from you, thanks. > >>>> > >>>> > >>>> [1] > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > >>>> [2] https://datahub.io/ > >>>> [3] https://atlas.apache.org/#/ > >>> > >>> > >> > >
