Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi all, Thanks for all the valuable feedback. If there's no further question or concern, I will start voting for FLIP-294 [1] later. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener Best, Shammon FY On Fri, Jun 9, 2023 at 9:07 AM Shammon FY wrote: > Thanks Jing, it makes sense to me and I have updated the FLIP > > > Best, > Shammon FY > > > On Thu, Jun 8, 2023 at 11:15 PM Jing Ge > wrote: > >> Hi Shammon, >> >> If we take a look at the JDK Event design as a reference, we can even add >> an Object into the event [1]. Back to the CatalogModificationEvent, >> everything related to the event could be defined in the Event. If we want >> to group some information into the Context, we could also consider adding >> the CatalogModificationContext into the Event and make the onEvent() >> method >> cleaner with only one input parameter CatalogModificationEvent, because >> the >> interface CatalogModificationListener is the most often used interface for >> users. Just my two cents. >> >> Best regards, >> Jing >> >> [1] >> >> http://www.java2s.com/example/java-src/pkg/java/util/eventobject-85298.html >> >> On Thu, Jun 8, 2023 at 7:50 AM Shammon FY wrote: >> >> > Hi, >> > >> > To @Jing Ge >> > > Thanks for the clarification. Just out of curiosity, if the context is >> > not part of the event, why should it be the input parameter of each >> onEvent >> > call? >> > >> > I think it's quite strange to put some information in an Event, such as >> a >> > factory identifier for catalog, but they will be used by the listener. >> I >> > place it in the context class and I think it is more suitable than >> directly >> > placing it in the event class. >> > >> > To @Mason >> > > 1. I'm also curious about default implementations. Would >> atlas/datahub be >> > supported by default? >> > >> > We won't do that and external systems such as atlas/datahub need to >> > implement the listener themselves. >> > >> > > 2. The FLIP title is confusing to me, especially in distinguishing it >> > from FLIP-314. Would a better FLIP title be "Support Catalog Metadata >> > Listener" or something alike? >> > >> > Thanks, I think "Support Catalog Modification Listener" will be >> > more suitable, I'll update the title to it. >> > >> > >> > Best, >> > Shammon FY >> > >> > >> > On Thu, Jun 8, 2023 at 12:25 PM Mason Chen >> wrote: >> > >> > > Hi Shammon, >> > > >> > > FLIP generally looks good and I'm excited to see this feature. >> > > >> > > 1. I'm also curious about default implementations. Would >> atlas/datahub be >> > > supported by default? >> > > 2. The FLIP title is confusing to me, especially in distinguishing it >> > from >> > > FLIP-314. Would a better FLIP title be "Support Catalog Metadata >> > Listener" >> > > or something alike? >> > > >> > > Best, >> > > Mason >> > > >> > > On Tue, Jun 6, 2023 at 3:33 AM Jing Ge >> > wrote: >> > > >> > > > Hi Shammon, >> > > > >> > > > Thanks for the clarification. Just out of curiosity, if the context >> is >> > > not >> > > > part of the event, why should it be the input parameter of each >> onEvent >> > > > call? >> > > > >> > > > Best regards, >> > > > Jing >> > > > >> > > > On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu >> wrote: >> > > > >> > > > > Thanks Shammon for the timely update, the updated FLIP looks good >> to >> > > me. >> > > > > >> > > > > Hope to see the vote thread and following FLIP-314 discussion >> thread. >> > > > > >> > > > > Best, >> > > > > Leonard >> > > > > >> > > > > > On Jun 6, 2023, at 5:04 PM, Shammon FY >> wrote: >> > > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Thanks for all the feedback. >> > > > > > >> > > > > > For @Jing Ge, >> > > > > > I forget to update the demo code in the FLIP, the method is >> > > > > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` >> and >> > > > there >> > > > > > is no `onEvent(CatalogModificationEvent)`. I have updated the >> code. >> > > > > Context >> > > > > > contains some additional information that is not part of an >> Event, >> > > but >> > > > > > needs to be used in the listener, so we separate it from the >> event. >> > > > > > >> > > > > > For @Panagiotis, >> > > > > > I think `ioExecutor` make sense to me and I have added it in >> > > > > > `ContextModificationContext`, thanks >> > > > > > >> > > > > > For @Leonard, >> > > > > > Thanks for your input. >> > > > > > 1. I have updated `CatalogModificationContext` as an interface, >> as >> > > well >> > > > > as >> > > > > > Context in CatalogModificationListenerFactory >> > > > > > 2. Configuration sounds good to me, I have updated the method >> name >> > > and >> > > > > > getConfiguration in Context >> > > > > > >> > > > > > For @David, >> > > > > > Yes, you're right. The listener will only be used on the client >> > side >> > > > and >> > > > > > won't introduce a new code path for running per-job/per-session >> > jobs. >> > > > The >> > > > > > listener will be created in `Table
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks Jing, it makes sense to me and I have updated the FLIP Best, Shammon FY On Thu, Jun 8, 2023 at 11:15 PM Jing Ge wrote: > Hi Shammon, > > If we take a look at the JDK Event design as a reference, we can even add > an Object into the event [1]. Back to the CatalogModificationEvent, > everything related to the event could be defined in the Event. If we want > to group some information into the Context, we could also consider adding > the CatalogModificationContext into the Event and make the onEvent() method > cleaner with only one input parameter CatalogModificationEvent, because the > interface CatalogModificationListener is the most often used interface for > users. Just my two cents. > > Best regards, > Jing > > [1] > http://www.java2s.com/example/java-src/pkg/java/util/eventobject-85298.html > > On Thu, Jun 8, 2023 at 7:50 AM Shammon FY wrote: > > > Hi, > > > > To @Jing Ge > > > Thanks for the clarification. Just out of curiosity, if the context is > > not part of the event, why should it be the input parameter of each > onEvent > > call? > > > > I think it's quite strange to put some information in an Event, such as a > > factory identifier for catalog, but they will be used by the listener. I > > place it in the context class and I think it is more suitable than > directly > > placing it in the event class. > > > > To @Mason > > > 1. I'm also curious about default implementations. Would atlas/datahub > be > > supported by default? > > > > We won't do that and external systems such as atlas/datahub need to > > implement the listener themselves. > > > > > 2. The FLIP title is confusing to me, especially in distinguishing it > > from FLIP-314. Would a better FLIP title be "Support Catalog Metadata > > Listener" or something alike? > > > > Thanks, I think "Support Catalog Modification Listener" will be > > more suitable, I'll update the title to it. > > > > > > Best, > > Shammon FY > > > > > > On Thu, Jun 8, 2023 at 12:25 PM Mason Chen > wrote: > > > > > Hi Shammon, > > > > > > FLIP generally looks good and I'm excited to see this feature. > > > > > > 1. I'm also curious about default implementations. Would atlas/datahub > be > > > supported by default? > > > 2. The FLIP title is confusing to me, especially in distinguishing it > > from > > > FLIP-314. Would a better FLIP title be "Support Catalog Metadata > > Listener" > > > or something alike? > > > > > > Best, > > > Mason > > > > > > On Tue, Jun 6, 2023 at 3:33 AM Jing Ge > > wrote: > > > > > > > Hi Shammon, > > > > > > > > Thanks for the clarification. Just out of curiosity, if the context > is > > > not > > > > part of the event, why should it be the input parameter of each > onEvent > > > > call? > > > > > > > > Best regards, > > > > Jing > > > > > > > > On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu > wrote: > > > > > > > > > Thanks Shammon for the timely update, the updated FLIP looks good > to > > > me. > > > > > > > > > > Hope to see the vote thread and following FLIP-314 discussion > thread. > > > > > > > > > > Best, > > > > > Leonard > > > > > > > > > > > On Jun 6, 2023, at 5:04 PM, Shammon FY > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > Thanks for all the feedback. > > > > > > > > > > > > For @Jing Ge, > > > > > > I forget to update the demo code in the FLIP, the method is > > > > > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` > and > > > > there > > > > > > is no `onEvent(CatalogModificationEvent)`. I have updated the > code. > > > > > Context > > > > > > contains some additional information that is not part of an > Event, > > > but > > > > > > needs to be used in the listener, so we separate it from the > event. > > > > > > > > > > > > For @Panagiotis, > > > > > > I think `ioExecutor` make sense to me and I have added it in > > > > > > `ContextModificationContext`, thanks > > > > > > > > > > > > For @Leonard, > > > > > > Thanks for your input. > > > > > > 1. I have updated `CatalogModificationContext` as an interface, > as > > > well > > > > > as > > > > > > Context in CatalogModificationListenerFactory > > > > > > 2. Configuration sounds good to me, I have updated the method > name > > > and > > > > > > getConfiguration in Context > > > > > > > > > > > > For @David, > > > > > > Yes, you're right. The listener will only be used on the client > > side > > > > and > > > > > > won't introduce a new code path for running per-job/per-session > > jobs. > > > > The > > > > > > listener will be created in `TableEnvironment` and `SqlGateway` > > which > > > > > can a > > > > > > `CatalogManager` with the listener. > > > > > > > > > > > > > > > > > > Best, > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek < > > > david.mora...@gmail.com> > > > > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> Thanks for the FLIP! Data lineage is an important problem to > > tackle. > > > > > >> > > > > > >> Can you please expand on how this is pl
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi Shammon, If we take a look at the JDK Event design as a reference, we can even add an Object into the event [1]. Back to the CatalogModificationEvent, everything related to the event could be defined in the Event. If we want to group some information into the Context, we could also consider adding the CatalogModificationContext into the Event and make the onEvent() method cleaner with only one input parameter CatalogModificationEvent, because the interface CatalogModificationListener is the most often used interface for users. Just my two cents. Best regards, Jing [1] http://www.java2s.com/example/java-src/pkg/java/util/eventobject-85298.html On Thu, Jun 8, 2023 at 7:50 AM Shammon FY wrote: > Hi, > > To @Jing Ge > > Thanks for the clarification. Just out of curiosity, if the context is > not part of the event, why should it be the input parameter of each onEvent > call? > > I think it's quite strange to put some information in an Event, such as a > factory identifier for catalog, but they will be used by the listener. I > place it in the context class and I think it is more suitable than directly > placing it in the event class. > > To @Mason > > 1. I'm also curious about default implementations. Would atlas/datahub be > supported by default? > > We won't do that and external systems such as atlas/datahub need to > implement the listener themselves. > > > 2. The FLIP title is confusing to me, especially in distinguishing it > from FLIP-314. Would a better FLIP title be "Support Catalog Metadata > Listener" or something alike? > > Thanks, I think "Support Catalog Modification Listener" will be > more suitable, I'll update the title to it. > > > Best, > Shammon FY > > > On Thu, Jun 8, 2023 at 12:25 PM Mason Chen wrote: > > > Hi Shammon, > > > > FLIP generally looks good and I'm excited to see this feature. > > > > 1. I'm also curious about default implementations. Would atlas/datahub be > > supported by default? > > 2. The FLIP title is confusing to me, especially in distinguishing it > from > > FLIP-314. Would a better FLIP title be "Support Catalog Metadata > Listener" > > or something alike? > > > > Best, > > Mason > > > > On Tue, Jun 6, 2023 at 3:33 AM Jing Ge > wrote: > > > > > Hi Shammon, > > > > > > Thanks for the clarification. Just out of curiosity, if the context is > > not > > > part of the event, why should it be the input parameter of each onEvent > > > call? > > > > > > Best regards, > > > Jing > > > > > > On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu wrote: > > > > > > > Thanks Shammon for the timely update, the updated FLIP looks good to > > me. > > > > > > > > Hope to see the vote thread and following FLIP-314 discussion thread. > > > > > > > > Best, > > > > Leonard > > > > > > > > > On Jun 6, 2023, at 5:04 PM, Shammon FY wrote: > > > > > > > > > > Hi, > > > > > > > > > > Thanks for all the feedback. > > > > > > > > > > For @Jing Ge, > > > > > I forget to update the demo code in the FLIP, the method is > > > > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and > > > there > > > > > is no `onEvent(CatalogModificationEvent)`. I have updated the code. > > > > Context > > > > > contains some additional information that is not part of an Event, > > but > > > > > needs to be used in the listener, so we separate it from the event. > > > > > > > > > > For @Panagiotis, > > > > > I think `ioExecutor` make sense to me and I have added it in > > > > > `ContextModificationContext`, thanks > > > > > > > > > > For @Leonard, > > > > > Thanks for your input. > > > > > 1. I have updated `CatalogModificationContext` as an interface, as > > well > > > > as > > > > > Context in CatalogModificationListenerFactory > > > > > 2. Configuration sounds good to me, I have updated the method name > > and > > > > > getConfiguration in Context > > > > > > > > > > For @David, > > > > > Yes, you're right. The listener will only be used on the client > side > > > and > > > > > won't introduce a new code path for running per-job/per-session > jobs. > > > The > > > > > listener will be created in `TableEnvironment` and `SqlGateway` > which > > > > can a > > > > > `CatalogManager` with the listener. > > > > > > > > > > > > > > > Best, > > > > > Shammon FY > > > > > > > > > > > > > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek < > > david.mora...@gmail.com> > > > > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> Thanks for the FLIP! Data lineage is an important problem to > tackle. > > > > >> > > > > >> Can you please expand on how this is planned to be wired into the > > > > >> JobManager? As I understand, the listeners will be configured > > globally > > > > (per > > > > >> cluster), so this won't introduce a new code path for running > > per-job > > > / > > > > >> per-session user code. Is that correct? > > > > >> > > > > >> Best, > > > > >> D > > > > >> > > > > >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu > > wrote: > > > > >> > > > > >>> Thanks Shammon for driving this FLIP forward,
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, To @Jing Ge > Thanks for the clarification. Just out of curiosity, if the context is not part of the event, why should it be the input parameter of each onEvent call? I think it's quite strange to put some information in an Event, such as a factory identifier for catalog, but they will be used by the listener. I place it in the context class and I think it is more suitable than directly placing it in the event class. To @Mason > 1. I'm also curious about default implementations. Would atlas/datahub be supported by default? We won't do that and external systems such as atlas/datahub need to implement the listener themselves. > 2. The FLIP title is confusing to me, especially in distinguishing it from FLIP-314. Would a better FLIP title be "Support Catalog Metadata Listener" or something alike? Thanks, I think "Support Catalog Modification Listener" will be more suitable, I'll update the title to it. Best, Shammon FY On Thu, Jun 8, 2023 at 12:25 PM Mason Chen wrote: > Hi Shammon, > > FLIP generally looks good and I'm excited to see this feature. > > 1. I'm also curious about default implementations. Would atlas/datahub be > supported by default? > 2. The FLIP title is confusing to me, especially in distinguishing it from > FLIP-314. Would a better FLIP title be "Support Catalog Metadata Listener" > or something alike? > > Best, > Mason > > On Tue, Jun 6, 2023 at 3:33 AM Jing Ge wrote: > > > Hi Shammon, > > > > Thanks for the clarification. Just out of curiosity, if the context is > not > > part of the event, why should it be the input parameter of each onEvent > > call? > > > > Best regards, > > Jing > > > > On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu wrote: > > > > > Thanks Shammon for the timely update, the updated FLIP looks good to > me. > > > > > > Hope to see the vote thread and following FLIP-314 discussion thread. > > > > > > Best, > > > Leonard > > > > > > > On Jun 6, 2023, at 5:04 PM, Shammon FY wrote: > > > > > > > > Hi, > > > > > > > > Thanks for all the feedback. > > > > > > > > For @Jing Ge, > > > > I forget to update the demo code in the FLIP, the method is > > > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and > > there > > > > is no `onEvent(CatalogModificationEvent)`. I have updated the code. > > > Context > > > > contains some additional information that is not part of an Event, > but > > > > needs to be used in the listener, so we separate it from the event. > > > > > > > > For @Panagiotis, > > > > I think `ioExecutor` make sense to me and I have added it in > > > > `ContextModificationContext`, thanks > > > > > > > > For @Leonard, > > > > Thanks for your input. > > > > 1. I have updated `CatalogModificationContext` as an interface, as > well > > > as > > > > Context in CatalogModificationListenerFactory > > > > 2. Configuration sounds good to me, I have updated the method name > and > > > > getConfiguration in Context > > > > > > > > For @David, > > > > Yes, you're right. The listener will only be used on the client side > > and > > > > won't introduce a new code path for running per-job/per-session jobs. > > The > > > > listener will be created in `TableEnvironment` and `SqlGateway` which > > > can a > > > > `CatalogManager` with the listener. > > > > > > > > > > > > Best, > > > > Shammon FY > > > > > > > > > > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek < > david.mora...@gmail.com> > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> Thanks for the FLIP! Data lineage is an important problem to tackle. > > > >> > > > >> Can you please expand on how this is planned to be wired into the > > > >> JobManager? As I understand, the listeners will be configured > globally > > > (per > > > >> cluster), so this won't introduce a new code path for running > per-job > > / > > > >> per-session user code. Is that correct? > > > >> > > > >> Best, > > > >> D > > > >> > > > >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu > wrote: > > > >> > > > >>> Thanks Shammon for driving this FLIP forward, I’ve several comments > > > about > > > >>> the updated FLIP. > > > >>> > > > >>> 1. CatalogModificationContext is introduced as a class instead of > an > > > >>> interface, is it a typo? > > > >>> > > > >>> 2. The FLIP defined multiple Map config(); > methods > > in > > > >>> some Context classes, Could we use Configuration > > > >> getConfiguration()；Class > > > >>> org.apache.flink.configuration.Configuration is recommend as it’s > > > public > > > >>> API and offers more useful methods as well. > > > >>> > > > >>> 3. The Context of CatalogModificationListenerFactory should be an > > > >>> interface too, and getUserClassLoder() > > > >>> would be more aligned with flink’s naming style. > > > >>> > > > >>> > > > >>> Best, > > > >>> Leonard > > > >>> > > > On May 26, 2023, at 4:08 PM, Shammon FY > wrote: > > > > > > Hi devs, > > > > > > We would like to bring up a discussion about FLIP-294: Support > > > >> Customized > > > Job Meta Da
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi Shammon, FLIP generally looks good and I'm excited to see this feature. 1. I'm also curious about default implementations. Would atlas/datahub be supported by default? 2. The FLIP title is confusing to me, especially in distinguishing it from FLIP-314. Would a better FLIP title be "Support Catalog Metadata Listener" or something alike? Best, Mason On Tue, Jun 6, 2023 at 3:33 AM Jing Ge wrote: > Hi Shammon, > > Thanks for the clarification. Just out of curiosity, if the context is not > part of the event, why should it be the input parameter of each onEvent > call? > > Best regards, > Jing > > On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu wrote: > > > Thanks Shammon for the timely update, the updated FLIP looks good to me. > > > > Hope to see the vote thread and following FLIP-314 discussion thread. > > > > Best, > > Leonard > > > > > On Jun 6, 2023, at 5:04 PM, Shammon FY wrote: > > > > > > Hi, > > > > > > Thanks for all the feedback. > > > > > > For @Jing Ge, > > > I forget to update the demo code in the FLIP, the method is > > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and > there > > > is no `onEvent(CatalogModificationEvent)`. I have updated the code. > > Context > > > contains some additional information that is not part of an Event, but > > > needs to be used in the listener, so we separate it from the event. > > > > > > For @Panagiotis, > > > I think `ioExecutor` make sense to me and I have added it in > > > `ContextModificationContext`, thanks > > > > > > For @Leonard, > > > Thanks for your input. > > > 1. I have updated `CatalogModificationContext` as an interface, as well > > as > > > Context in CatalogModificationListenerFactory > > > 2. Configuration sounds good to me, I have updated the method name and > > > getConfiguration in Context > > > > > > For @David, > > > Yes, you're right. The listener will only be used on the client side > and > > > won't introduce a new code path for running per-job/per-session jobs. > The > > > listener will be created in `TableEnvironment` and `SqlGateway` which > > can a > > > `CatalogManager` with the listener. > > > > > > > > > Best, > > > Shammon FY > > > > > > > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek > > > wrote: > > > > > >> Hi, > > >> > > >> Thanks for the FLIP! Data lineage is an important problem to tackle. > > >> > > >> Can you please expand on how this is planned to be wired into the > > >> JobManager? As I understand, the listeners will be configured globally > > (per > > >> cluster), so this won't introduce a new code path for running per-job > / > > >> per-session user code. Is that correct? > > >> > > >> Best, > > >> D > > >> > > >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu wrote: > > >> > > >>> Thanks Shammon for driving this FLIP forward, I’ve several comments > > about > > >>> the updated FLIP. > > >>> > > >>> 1. CatalogModificationContext is introduced as a class instead of an > > >>> interface, is it a typo? > > >>> > > >>> 2. The FLIP defined multiple Map config(); methods > in > > >>> some Context classes, Could we use Configuration > > >> getConfiguration()；Class > > >>> org.apache.flink.configuration.Configuration is recommend as it’s > > public > > >>> API and offers more useful methods as well. > > >>> > > >>> 3. The Context of CatalogModificationListenerFactory should be an > > >>> interface too, and getUserClassLoder() > > >>> would be more aligned with flink’s naming style. > > >>> > > >>> > > >>> Best, > > >>> Leonard > > >>> > > On May 26, 2023, at 4:08 PM, Shammon FY wrote: > > > > Hi devs, > > > > We would like to bring up a discussion about FLIP-294: Support > > >> Customized > > Job Meta Data Listener[1]. We have had several discussions with Jark > > >> Wu, > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > > >> and > > interfaces, and thanks for their valuable advice. > > The overall job and connector information is divided into metadata > and > > lineage, this FLIP focuses on metadata and lineage will be discussed > > in > > another FLIP in the future. In this FLIP we want to add a customized > > listener in Flink to report catalog modifications to external > metadata > > systems such as datahub[2] or atlas[3]. Users can view the specific > > information of connectors such as source and sink for Flink jobs in > > >> these > > systems, including fields, watermarks, partitions, etc. > > > > Looking forward to hearing from you, thanks. > > > > > > [1] > > > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > [2] https://datahub.io/ > > [3] https://atlas.apache.org/#/ > > >>> > > >>> > > >> > > > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi Shammon, Thanks for the clarification. Just out of curiosity, if the context is not part of the event, why should it be the input parameter of each onEvent call? Best regards, Jing On Tue, Jun 6, 2023 at 11:58 AM Leonard Xu wrote: > Thanks Shammon for the timely update, the updated FLIP looks good to me. > > Hope to see the vote thread and following FLIP-314 discussion thread. > > Best, > Leonard > > > On Jun 6, 2023, at 5:04 PM, Shammon FY wrote: > > > > Hi, > > > > Thanks for all the feedback. > > > > For @Jing Ge, > > I forget to update the demo code in the FLIP, the method is > > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there > > is no `onEvent(CatalogModificationEvent)`. I have updated the code. > Context > > contains some additional information that is not part of an Event, but > > needs to be used in the listener, so we separate it from the event. > > > > For @Panagiotis, > > I think `ioExecutor` make sense to me and I have added it in > > `ContextModificationContext`, thanks > > > > For @Leonard, > > Thanks for your input. > > 1. I have updated `CatalogModificationContext` as an interface, as well > as > > Context in CatalogModificationListenerFactory > > 2. Configuration sounds good to me, I have updated the method name and > > getConfiguration in Context > > > > For @David, > > Yes, you're right. The listener will only be used on the client side and > > won't introduce a new code path for running per-job/per-session jobs. The > > listener will be created in `TableEnvironment` and `SqlGateway` which > can a > > `CatalogManager` with the listener. > > > > > > Best, > > Shammon FY > > > > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek > > wrote: > > > >> Hi, > >> > >> Thanks for the FLIP! Data lineage is an important problem to tackle. > >> > >> Can you please expand on how this is planned to be wired into the > >> JobManager? As I understand, the listeners will be configured globally > (per > >> cluster), so this won't introduce a new code path for running per-job / > >> per-session user code. Is that correct? > >> > >> Best, > >> D > >> > >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu wrote: > >> > >>> Thanks Shammon for driving this FLIP forward, I’ve several comments > about > >>> the updated FLIP. > >>> > >>> 1. CatalogModificationContext is introduced as a class instead of an > >>> interface, is it a typo? > >>> > >>> 2. The FLIP defined multiple Map config(); methods in > >>> some Context classes, Could we use Configuration > >> getConfiguration()；Class > >>> org.apache.flink.configuration.Configuration is recommend as it’s > public > >>> API and offers more useful methods as well. > >>> > >>> 3. The Context of CatalogModificationListenerFactory should be an > >>> interface too, and getUserClassLoder() > >>> would be more aligned with flink’s naming style. > >>> > >>> > >>> Best, > >>> Leonard > >>> > On May 26, 2023, at 4:08 PM, Shammon FY wrote: > > Hi devs, > > We would like to bring up a discussion about FLIP-294: Support > >> Customized > Job Meta Data Listener[1]. We have had several discussions with Jark > >> Wu, > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > >> and > interfaces, and thanks for their valuable advice. > The overall job and connector information is divided into metadata and > lineage, this FLIP focuses on metadata and lineage will be discussed > in > another FLIP in the future. In this FLIP we want to add a customized > listener in Flink to report catalog modifications to external metadata > systems such as datahub[2] or atlas[3]. Users can view the specific > information of connectors such as source and sink for Flink jobs in > >> these > systems, including fields, watermarks, partitions, etc. > > Looking forward to hearing from you, thanks. > > > [1] > > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > [2] https://datahub.io/ > [3] https://atlas.apache.org/#/ > >>> > >>> > >> > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks Shammon for the timely update, the updated FLIP looks good to me. Hope to see the vote thread and following FLIP-314 discussion thread. Best, Leonard > On Jun 6, 2023, at 5:04 PM, Shammon FY wrote: > > Hi, > > Thanks for all the feedback. > > For @Jing Ge, > I forget to update the demo code in the FLIP, the method is > `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there > is no `onEvent(CatalogModificationEvent)`. I have updated the code. Context > contains some additional information that is not part of an Event, but > needs to be used in the listener, so we separate it from the event. > > For @Panagiotis, > I think `ioExecutor` make sense to me and I have added it in > `ContextModificationContext`, thanks > > For @Leonard, > Thanks for your input. > 1. I have updated `CatalogModificationContext` as an interface, as well as > Context in CatalogModificationListenerFactory > 2. Configuration sounds good to me, I have updated the method name and > getConfiguration in Context > > For @David, > Yes, you're right. The listener will only be used on the client side and > won't introduce a new code path for running per-job/per-session jobs. The > listener will be created in `TableEnvironment` and `SqlGateway` which can a > `CatalogManager` with the listener. > > > Best, > Shammon FY > > > On Tue, Jun 6, 2023 at 3:33 PM David Morávek > wrote: > >> Hi, >> >> Thanks for the FLIP! Data lineage is an important problem to tackle. >> >> Can you please expand on how this is planned to be wired into the >> JobManager? As I understand, the listeners will be configured globally (per >> cluster), so this won't introduce a new code path for running per-job / >> per-session user code. Is that correct? >> >> Best, >> D >> >> On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu wrote: >> >>> Thanks Shammon for driving this FLIP forward, I’ve several comments about >>> the updated FLIP. >>> >>> 1. CatalogModificationContext is introduced as a class instead of an >>> interface, is it a typo? >>> >>> 2. The FLIP defined multiple Map config(); methods in >>> some Context classes, Could we use Configuration >> getConfiguration()；Class >>> org.apache.flink.configuration.Configuration is recommend as it’s public >>> API and offers more useful methods as well. >>> >>> 3. The Context of CatalogModificationListenerFactory should be an >>> interface too, and getUserClassLoder() >>> would be more aligned with flink’s naming style. >>> >>> >>> Best, >>> Leonard >>> On May 26, 2023, at 4:08 PM, Shammon FY wrote: Hi devs, We would like to bring up a discussion about FLIP-294: Support >> Customized Job Meta Data Listener[1]. We have had several discussions with Jark >> Wu, Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions >> and interfaces, and thanks for their valuable advice. The overall job and connector information is divided into metadata and lineage, this FLIP focuses on metadata and lineage will be discussed in another FLIP in the future. In this FLIP we want to add a customized listener in Flink to report catalog modifications to external metadata systems such as datahub[2] or atlas[3]. Users can view the specific information of connectors such as source and sink for Flink jobs in >> these systems, including fields, watermarks, partitions, etc. Looking forward to hearing from you, thanks. [1] >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener [2] https://datahub.io/ [3] https://atlas.apache.org/#/ >>> >>> >>
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, Thanks for all the feedback. For @Jing Ge, I forget to update the demo code in the FLIP, the method is `onEvent(CatalogModificationEvent, CatalogModificationContext)` and there is no `onEvent(CatalogModificationEvent)`. I have updated the code. Context contains some additional information that is not part of an Event, but needs to be used in the listener, so we separate it from the event. For @Panagiotis, I think `ioExecutor` make sense to me and I have added it in `ContextModificationContext`, thanks For @Leonard, Thanks for your input. 1. I have updated `CatalogModificationContext` as an interface, as well as Context in CatalogModificationListenerFactory 2. Configuration sounds good to me, I have updated the method name and getConfiguration in Context For @David, Yes, you're right. The listener will only be used on the client side and won't introduce a new code path for running per-job/per-session jobs. The listener will be created in `TableEnvironment` and `SqlGateway` which can a `CatalogManager` with the listener. Best, Shammon FY On Tue, Jun 6, 2023 at 3:33 PM David Morávek wrote: > Hi, > > Thanks for the FLIP! Data lineage is an important problem to tackle. > > Can you please expand on how this is planned to be wired into the > JobManager? As I understand, the listeners will be configured globally (per > cluster), so this won't introduce a new code path for running per-job / > per-session user code. Is that correct? > > Best, > D > > On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu wrote: > > > Thanks Shammon for driving this FLIP forward, I’ve several comments about > > the updated FLIP. > > > > 1. CatalogModificationContext is introduced as a class instead of an > > interface, is it a typo? > > > > 2. The FLIP defined multiple Map config(); methods in > > some Context classes, Could we use Configuration > getConfiguration()；Class > > org.apache.flink.configuration.Configuration is recommend as it’s public > > API and offers more useful methods as well. > > > > 3. The Context of CatalogModificationListenerFactory should be an > > interface too, and getUserClassLoder() > > would be more aligned with flink’s naming style. > > > > > > Best, > > Leonard > > > > > On May 26, 2023, at 4:08 PM, Shammon FY wrote: > > > > > > Hi devs, > > > > > > We would like to bring up a discussion about FLIP-294: Support > Customized > > > Job Meta Data Listener[1]. We have had several discussions with Jark > Wu, > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > and > > > interfaces, and thanks for their valuable advice. > > > The overall job and connector information is divided into metadata and > > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > > another FLIP in the future. In this FLIP we want to add a customized > > > listener in Flink to report catalog modifications to external metadata > > > systems such as datahub[2] or atlas[3]. Users can view the specific > > > information of connectors such as source and sink for Flink jobs in > these > > > systems, including fields, watermarks, partitions, etc. > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > > [2] https://datahub.io/ > > > [3] https://atlas.apache.org/#/ > > > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, Thanks for the FLIP! Data lineage is an important problem to tackle. Can you please expand on how this is planned to be wired into the JobManager? As I understand, the listeners will be configured globally (per cluster), so this won't introduce a new code path for running per-job / per-session user code. Is that correct? Best, D On Tue, Jun 6, 2023 at 9:17 AM Leonard Xu wrote: > Thanks Shammon for driving this FLIP forward, I’ve several comments about > the updated FLIP. > > 1. CatalogModificationContext is introduced as a class instead of an > interface, is it a typo? > > 2. The FLIP defined multiple Map config(); methods in > some Context classes, Could we use Configuration getConfiguration()；Class > org.apache.flink.configuration.Configuration is recommend as it’s public > API and offers more useful methods as well. > > 3. The Context of CatalogModificationListenerFactory should be an > interface too, and getUserClassLoder() > would be more aligned with flink’s naming style. > > > Best, > Leonard > > > On May 26, 2023, at 4:08 PM, Shammon FY wrote: > > > > Hi devs, > > > > We would like to bring up a discussion about FLIP-294: Support Customized > > Job Meta Data Listener[1]. We have had several discussions with Jark Wu, > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and > > interfaces, and thanks for their valuable advice. > > The overall job and connector information is divided into metadata and > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > another FLIP in the future. In this FLIP we want to add a customized > > listener in Flink to report catalog modifications to external metadata > > systems such as datahub[2] or atlas[3]. Users can view the specific > > information of connectors such as source and sink for Flink jobs in these > > systems, including fields, watermarks, partitions, etc. > > > > Looking forward to hearing from you, thanks. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > [2] https://datahub.io/ > > [3] https://atlas.apache.org/#/ > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks Shammon for driving this FLIP forward, I’ve several comments about the updated FLIP. 1. CatalogModificationContext is introduced as a class instead of an interface, is it a typo? 2. The FLIP defined multiple Map config(); methods in some Context classes, Could we use Configuration getConfiguration()；Class org.apache.flink.configuration.Configuration is recommend as it’s public API and offers more useful methods as well. 3. The Context of CatalogModificationListenerFactory should be an interface too, and getUserClassLoder() would be more aligned with flink’s naming style. Best, Leonard > On May 26, 2023, at 4:08 PM, Shammon FY wrote: > > Hi devs, > > We would like to bring up a discussion about FLIP-294: Support Customized > Job Meta Data Listener[1]. We have had several discussions with Jark Wu, > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and > interfaces, and thanks for their valuable advice. > The overall job and connector information is divided into metadata and > lineage, this FLIP focuses on metadata and lineage will be discussed in > another FLIP in the future. In this FLIP we want to add a customized > listener in Flink to report catalog modifications to external metadata > systems such as datahub[2] or atlas[3]. Users can view the specific > information of connectors such as source and sink for Flink jobs in these > systems, including fields, watermarks, partitions, etc. > > Looking forward to hearing from you, thanks. > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > [2] https://datahub.io/ > [3] https://atlas.apache.org/#/
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hey Shammon, Thanks for the FLIP and the discussion! Pretty useful stuff -- also looking forward to the lineage feature! Just my 2 cents: if we want to enable users to perform heavy async operations, should we pass an ioExecutor to the listeners? Also, would it make sense to provide a default implementation? Cheers, Panagiotis On Mon, Jun 5, 2023 at 10:43 PM Shammon FY wrote: > Hi devs: > > Thanks for all the feedback, and if there are no more comments, I will > start a vote on FLIP-294 [1] later. Thanks again. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > Best, > Shammon FY > > On Tue, Jun 6, 2023 at 1:40 PM Shammon FY wrote: > > > Hi Martijn, > > > > Thanks for your attention, I will soon initiate a discussion about > > FLIP-314. > > > > Best, > > Shammon FY > > > > > > On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser > > wrote: > > > >> Hi Shammon, > >> > >> Just wanted to chip-in that I like the overall FLIP. Will be interesting > >> to > >> see the follow-up discussion on FLIP-314. > >> > >> Best regards, > >> > >> Martijn > >> > >> On Thu, Jun 1, 2023 at 5:45 AM yuxia > wrote: > >> > >> > Thanks for explanation. Make sense to me. > >> > > >> > Best regards, > >> > Yuxia > >> > > >> > - 原始邮件 - > >> > 发件人: "Shammon FY" > >> > 收件人: "dev" > >> > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 > >> > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > >> > > >> > Thanks yuxia, you're right and I'll add the new database to > >> > AlterDatabaseEvent. > >> > > >> > I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a > >> > parameter in the `Catalog.alterDatabase` method. Although this value > is > >> > currently always false in `AlterDatabaseOperation`, I think it's > better > >> > to stay consistent with `Catalog.alterDatabase`. What do you think? > >> > > >> > Best, > >> > Shammon FY > >> > > >> > On Thu, Jun 1, 2023 at 10:25 AM yuxia > >> wrote: > >> > > >> > > Hi, Shammon. > >> > > I mean do we need to contain the new database after alter in > >> > > AlterDatabaseEvent? So that the listener can know what has been > >> modified > >> > > for the database. Or the listener don't need to care about the > actual > >> > > modification. > >> > > Also, I'm wondering whether AlterDatabaseEven need to include > >> > > ignoreIfNotExists method since alter database operation don't have > >> such > >> > > syntax like 'alter database if exists xxx'. > >> > > > >> > > Best regards, > >> > > Yuxia > >> > > > >> > > - 原始邮件 - > >> > > 发件人: "Shammon FY" > >> > > 收件人: "dev" > >> > > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 > >> > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data > Listener > >> > > > >> > > Hi yuxia > >> > > > >> > > Thanks for your input. The `AlterDatabaseEvent` extends > >> > > `DatabaseModificationEvent` which has the original database. > >> > > > >> > > Best, > >> > > Shammon FY > >> > > > >> > > On Wed, May 31, 2023 at 2:24 PM yuxia > >> > wrote: > >> > > > >> > > > Thanks Shammon for driving it. > >> > > > The FLIP generally looks good to me. I only have one question. > >> > > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database > name > >> > and > >> > > > the new CatalogDatabase after modified. Is it enough only pass the > >> > origin > >> > > > database name? Will it be better to contain the origin > >> CatalogDatabase > >> > so > >> > > > that listener have ways to know what changes? > >> > > > > >> > > > Best regards, > >> > > > Yuxia > >> > > > > >> > > > - 原始邮件 - > >> > > > 发件人: "ron9 liu" > >> > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi Shammon, Thanks for driving it! It is a really interesting proposal. Looking forward to the follow-up FLIP for the lineage feature, users will love it :-) There are some inconsistencies in the content. In the very below example, listener.onEvent(CatalogModificationEvent) is called, while in the CatalogModificationListener interface definition, only onEvent(CatalogModificationEvent, CatalogModificationContext) has been defined. I was wondering(NIT): 1. should there be another overloading method onEvent(CatalogModificationEvent) alongside onEvent(CatalogModificationEvent, CatalogModificationContext) ? 2. Since onEvent(CatalogModificationEvent) could be used, do we really need CatalogModificationContext? API design example as reference: [1] Best regards, Jing [1] http://www.java2s.com/example/java-src/pkg/java/awt/event/actionlistener-add27.html On Tue, Jun 6, 2023 at 7:43 AM Shammon FY wrote: > Hi devs: > > Thanks for all the feedback, and if there are no more comments, I will > start a vote on FLIP-294 [1] later. Thanks again. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > Best, > Shammon FY > > On Tue, Jun 6, 2023 at 1:40 PM Shammon FY wrote: > > > Hi Martijn, > > > > Thanks for your attention, I will soon initiate a discussion about > > FLIP-314. > > > > Best, > > Shammon FY > > > > > > On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser > > wrote: > > > >> Hi Shammon, > >> > >> Just wanted to chip-in that I like the overall FLIP. Will be interesting > >> to > >> see the follow-up discussion on FLIP-314. > >> > >> Best regards, > >> > >> Martijn > >> > >> On Thu, Jun 1, 2023 at 5:45 AM yuxia > wrote: > >> > >> > Thanks for explanation. Make sense to me. > >> > > >> > Best regards, > >> > Yuxia > >> > > >> > - 原始邮件 - > >> > 发件人: "Shammon FY" > >> > 收件人: "dev" > >> > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 > >> > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > >> > > >> > Thanks yuxia, you're right and I'll add the new database to > >> > AlterDatabaseEvent. > >> > > >> > I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a > >> > parameter in the `Catalog.alterDatabase` method. Although this value > is > >> > currently always false in `AlterDatabaseOperation`, I think it's > better > >> > to stay consistent with `Catalog.alterDatabase`. What do you think? > >> > > >> > Best, > >> > Shammon FY > >> > > >> > On Thu, Jun 1, 2023 at 10:25 AM yuxia > >> wrote: > >> > > >> > > Hi, Shammon. > >> > > I mean do we need to contain the new database after alter in > >> > > AlterDatabaseEvent? So that the listener can know what has been > >> modified > >> > > for the database. Or the listener don't need to care about the > actual > >> > > modification. > >> > > Also, I'm wondering whether AlterDatabaseEven need to include > >> > > ignoreIfNotExists method since alter database operation don't have > >> such > >> > > syntax like 'alter database if exists xxx'. > >> > > > >> > > Best regards, > >> > > Yuxia > >> > > > >> > > - 原始邮件 - > >> > > 发件人: "Shammon FY" > >> > > 收件人: "dev" > >> > > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 > >> > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data > Listener > >> > > > >> > > Hi yuxia > >> > > > >> > > Thanks for your input. The `AlterDatabaseEvent` extends > >> > > `DatabaseModificationEvent` which has the original database. > >> > > > >> > > Best, > >> > > Shammon FY > >> > > > >> > > On Wed, May 31, 2023 at 2:24 PM yuxia > >> > wrote: > >> > > > >> > > > Thanks Shammon for driving it. > >> > > > The FLIP generally looks good to me. I only have one question. > >> > > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database > name > >> > and > >> > > > the new CatalogDatabase after mo
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi devs: Thanks for all the feedback, and if there are no more comments, I will start a vote on FLIP-294 [1] later. Thanks again. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener Best, Shammon FY On Tue, Jun 6, 2023 at 1:40 PM Shammon FY wrote: > Hi Martijn, > > Thanks for your attention, I will soon initiate a discussion about > FLIP-314. > > Best, > Shammon FY > > > On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser > wrote: > >> Hi Shammon, >> >> Just wanted to chip-in that I like the overall FLIP. Will be interesting >> to >> see the follow-up discussion on FLIP-314. >> >> Best regards, >> >> Martijn >> >> On Thu, Jun 1, 2023 at 5:45 AM yuxia wrote: >> >> > Thanks for explanation. Make sense to me. >> > >> > Best regards, >> > Yuxia >> > >> > ----- 原始邮件 - >> > 发件人: "Shammon FY" >> > 收件人: "dev" >> > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 >> > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener >> > >> > Thanks yuxia, you're right and I'll add the new database to >> > AlterDatabaseEvent. >> > >> > I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a >> > parameter in the `Catalog.alterDatabase` method. Although this value is >> > currently always false in `AlterDatabaseOperation`, I think it's better >> > to stay consistent with `Catalog.alterDatabase`. What do you think? >> > >> > Best, >> > Shammon FY >> > >> > On Thu, Jun 1, 2023 at 10:25 AM yuxia >> wrote: >> > >> > > Hi, Shammon. >> > > I mean do we need to contain the new database after alter in >> > > AlterDatabaseEvent? So that the listener can know what has been >> modified >> > > for the database. Or the listener don't need to care about the actual >> > > modification. >> > > Also, I'm wondering whether AlterDatabaseEven need to include >> > > ignoreIfNotExists method since alter database operation don't have >> such >> > > syntax like 'alter database if exists xxx'. >> > > >> > > Best regards, >> > > Yuxia >> > > >> > > - 原始邮件 - >> > > 发件人: "Shammon FY" >> > > 收件人: "dev" >> > > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 >> > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener >> > > >> > > Hi yuxia >> > > >> > > Thanks for your input. The `AlterDatabaseEvent` extends >> > > `DatabaseModificationEvent` which has the original database. >> > > >> > > Best, >> > > Shammon FY >> > > >> > > On Wed, May 31, 2023 at 2:24 PM yuxia >> > wrote: >> > > >> > > > Thanks Shammon for driving it. >> > > > The FLIP generally looks good to me. I only have one question. >> > > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name >> > and >> > > > the new CatalogDatabase after modified. Is it enough only pass the >> > origin >> > > > database name? Will it be better to contain the origin >> CatalogDatabase >> > so >> > > > that listener have ways to know what changes? >> > > > >> > > > Best regards, >> > > > Yuxia >> > > > >> > > > - 原始邮件 - >> > > > 发件人: "ron9 liu" >> > > > 收件人: "dev" >> > > > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 >> > > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data >> Listener >> > > > >> > > > Hi, Shammon >> > > > >> > > > Thanks for driving this FLIP, It will enforce the Flink metadata >> > > capability >> > > > from the platform produce perspective. The overall design looks >> good to >> > > me, >> > > > I just have some small question: >> > > > 1. Regarding CatalogModificationListenerFactory#createListener >> method, >> > I >> > > > think it would be better to pass Context as its parameter instead of >> > two >> > > > specific Object. In this way, we can easily extend it in the future >> and >> > > > there will be no compatibility pro
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi Martijn, Thanks for your attention, I will soon initiate a discussion about FLIP-314. Best, Shammon FY On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser wrote: > Hi Shammon, > > Just wanted to chip-in that I like the overall FLIP. Will be interesting to > see the follow-up discussion on FLIP-314. > > Best regards, > > Martijn > > On Thu, Jun 1, 2023 at 5:45 AM yuxia wrote: > > > Thanks for explanation. Make sense to me. > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Shammon FY" > > 收件人: "dev" > > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > Thanks yuxia, you're right and I'll add the new database to > > AlterDatabaseEvent. > > > > I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a > > parameter in the `Catalog.alterDatabase` method. Although this value is > > currently always false in `AlterDatabaseOperation`, I think it's better > > to stay consistent with `Catalog.alterDatabase`. What do you think? > > > > Best, > > Shammon FY > > > > On Thu, Jun 1, 2023 at 10:25 AM yuxia > wrote: > > > > > Hi, Shammon. > > > I mean do we need to contain the new database after alter in > > > AlterDatabaseEvent? So that the listener can know what has been > modified > > > for the database. Or the listener don't need to care about the actual > > > modification. > > > Also, I'm wondering whether AlterDatabaseEven need to include > > > ignoreIfNotExists method since alter database operation don't have such > > > syntax like 'alter database if exists xxx'. > > > > > > Best regards, > > > Yuxia > > > > > > - 原始邮件 - > > > 发件人: "Shammon FY" > > > 收件人: "dev" > > > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 > > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > > > Hi yuxia > > > > > > Thanks for your input. The `AlterDatabaseEvent` extends > > > `DatabaseModificationEvent` which has the original database. > > > > > > Best, > > > Shammon FY > > > > > > On Wed, May 31, 2023 at 2:24 PM yuxia > > wrote: > > > > > > > Thanks Shammon for driving it. > > > > The FLIP generally looks good to me. I only have one question. > > > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name > > and > > > > the new CatalogDatabase after modified. Is it enough only pass the > > origin > > > > database name? Will it be better to contain the origin > CatalogDatabase > > so > > > > that listener have ways to know what changes? > > > > > > > > Best regards, > > > > Yuxia > > > > > > > > - 原始邮件 - > > > > 发件人: "ron9 liu" > > > > 收件人: "dev" > > > > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > > > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > > > > > Hi, Shammon > > > > > > > > Thanks for driving this FLIP, It will enforce the Flink metadata > > > capability > > > > from the platform produce perspective. The overall design looks good > to > > > me, > > > > I just have some small question: > > > > 1. Regarding CatalogModificationListenerFactory#createListener > method, > > I > > > > think it would be better to pass Context as its parameter instead of > > two > > > > specific Object. In this way, we can easily extend it in the future > and > > > > there will be no compatibility problems. Refer to > > > > > > > > > > > > > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > > > > 2. In FLIP, you mentioned that multiple Flink tables may refer to the > > > same > > > > physical table, so does the Listener report this physical table > > > repeatedly？ > > > > 3. When registering a Listener object, will it connect to an external > > > > system such as Datahub? If the Listener object registration times out > > due > > > > to permission issues, it will affect the execution of all subsequent > > SQL, > &
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi Shammon, Just wanted to chip-in that I like the overall FLIP. Will be interesting to see the follow-up discussion on FLIP-314. Best regards, Martijn On Thu, Jun 1, 2023 at 5:45 AM yuxia wrote: > Thanks for explanation. Make sense to me. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Shammon FY" > 收件人: "dev" > 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > Thanks yuxia, you're right and I'll add the new database to > AlterDatabaseEvent. > > I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a > parameter in the `Catalog.alterDatabase` method. Although this value is > currently always false in `AlterDatabaseOperation`, I think it's better > to stay consistent with `Catalog.alterDatabase`. What do you think? > > Best, > Shammon FY > > On Thu, Jun 1, 2023 at 10:25 AM yuxia wrote: > > > Hi, Shammon. > > I mean do we need to contain the new database after alter in > > AlterDatabaseEvent? So that the listener can know what has been modified > > for the database. Or the listener don't need to care about the actual > > modification. > > Also, I'm wondering whether AlterDatabaseEven need to include > > ignoreIfNotExists method since alter database operation don't have such > > syntax like 'alter database if exists xxx'. > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Shammon FY" > > 收件人: "dev" > > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > Hi yuxia > > > > Thanks for your input. The `AlterDatabaseEvent` extends > > `DatabaseModificationEvent` which has the original database. > > > > Best, > > Shammon FY > > > > On Wed, May 31, 2023 at 2:24 PM yuxia > wrote: > > > > > Thanks Shammon for driving it. > > > The FLIP generally looks good to me. I only have one question. > > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name > and > > > the new CatalogDatabase after modified. Is it enough only pass the > origin > > > database name? Will it be better to contain the origin CatalogDatabase > so > > > that listener have ways to know what changes? > > > > > > Best regards, > > > Yuxia > > > > > > - 原始邮件 - > > > 发件人: "ron9 liu" > > > 收件人: "dev" > > > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > > > Hi, Shammon > > > > > > Thanks for driving this FLIP, It will enforce the Flink metadata > > capability > > > from the platform produce perspective. The overall design looks good to > > me, > > > I just have some small question: > > > 1. Regarding CatalogModificationListenerFactory#createListener method, > I > > > think it would be better to pass Context as its parameter instead of > two > > > specific Object. In this way, we can easily extend it in the future and > > > there will be no compatibility problems. Refer to > > > > > > > > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > > > 2. In FLIP, you mentioned that multiple Flink tables may refer to the > > same > > > physical table, so does the Listener report this physical table > > repeatedly？ > > > 3. When registering a Listener object, will it connect to an external > > > system such as Datahub? If the Listener object registration times out > due > > > to permission issues, it will affect the execution of all subsequent > SQL, > > > what should we do in this case? > > > > > > Best, > > > Ron > > > > > > Shammon FY 于2023年5月31日周三 08:53写道： > > > > > > > Thanks Feng, the catalog modification listener is only used to report > > > > read-only ddl information to other components or systems. > > > > > > > > > 1. Will an exception thrown by the listener affect the normal > > execution > > > > process? > > > > > > > > Users need to handle the exception in the listener themselves. Many > > DDLs > > > > such as drop tables and alter tables cannot be rolled back, Flink > > cannot > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks for explanation. Make sense to me. Best regards, Yuxia - 原始邮件 - 发件人: "Shammon FY" 收件人: "dev" 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener Thanks yuxia, you're right and I'll add the new database to AlterDatabaseEvent. I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a parameter in the `Catalog.alterDatabase` method. Although this value is currently always false in `AlterDatabaseOperation`, I think it's better to stay consistent with `Catalog.alterDatabase`. What do you think? Best, Shammon FY On Thu, Jun 1, 2023 at 10:25 AM yuxia wrote: > Hi, Shammon. > I mean do we need to contain the new database after alter in > AlterDatabaseEvent? So that the listener can know what has been modified > for the database. Or the listener don't need to care about the actual > modification. > Also, I'm wondering whether AlterDatabaseEven need to include > ignoreIfNotExists method since alter database operation don't have such > syntax like 'alter database if exists xxx'. > > Best regards, > Yuxia > > ----- 原始邮件 - > 发件人: "Shammon FY" > 收件人: "dev" > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > Hi yuxia > > Thanks for your input. The `AlterDatabaseEvent` extends > `DatabaseModificationEvent` which has the original database. > > Best, > Shammon FY > > On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > > > Thanks Shammon for driving it. > > The FLIP generally looks good to me. I only have one question. > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and > > the new CatalogDatabase after modified. Is it enough only pass the origin > > database name? Will it be better to contain the origin CatalogDatabase so > > that listener have ways to know what changes? > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "ron9 liu" > > 收件人: "dev" > > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > Hi, Shammon > > > > Thanks for driving this FLIP, It will enforce the Flink metadata > capability > > from the platform produce perspective. The overall design looks good to > me, > > I just have some small question: > > 1. Regarding CatalogModificationListenerFactory#createListener method, I > > think it would be better to pass Context as its parameter instead of two > > specific Object. In this way, we can easily extend it in the future and > > there will be no compatibility problems. Refer to > > > > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > > 2. In FLIP, you mentioned that multiple Flink tables may refer to the > same > > physical table, so does the Listener report this physical table > repeatedly？ > > 3. When registering a Listener object, will it connect to an external > > system such as Datahub? If the Listener object registration times out due > > to permission issues, it will affect the execution of all subsequent SQL, > > what should we do in this case? > > > > Best, > > Ron > > > > Shammon FY 于2023年5月31日周三 08:53写道： > > > > > Thanks Feng, the catalog modification listener is only used to report > > > read-only ddl information to other components or systems. > > > > > > > 1. Will an exception thrown by the listener affect the normal > execution > > > process? > > > > > > Users need to handle the exception in the listener themselves. Many > DDLs > > > such as drop tables and alter tables cannot be rolled back, Flink > cannot > > > handle these exceptions for the listener. It will cause the operation > to > > > exit if an exception is thrown, but the executed DDL will be > successful. > > > > > > > 2. What is the order of execution? Is the listener executed first or > > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > The listener will be notified to report catalog modification after DDLs > > are > > > successful, so you can not do permission verification for DDL in the > > > listener. As mentioned above, Flink will not roll back the
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks yuxia, you're right and I'll add the new database to AlterDatabaseEvent. I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a parameter in the `Catalog.alterDatabase` method. Although this value is currently always false in `AlterDatabaseOperation`, I think it's better to stay consistent with `Catalog.alterDatabase`. What do you think? Best, Shammon FY On Thu, Jun 1, 2023 at 10:25 AM yuxia wrote: > Hi, Shammon. > I mean do we need to contain the new database after alter in > AlterDatabaseEvent? So that the listener can know what has been modified > for the database. Or the listener don't need to care about the actual > modification. > Also, I'm wondering whether AlterDatabaseEven need to include > ignoreIfNotExists method since alter database operation don't have such > syntax like 'alter database if exists xxx'. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Shammon FY" > 收件人: "dev" > 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > Hi yuxia > > Thanks for your input. The `AlterDatabaseEvent` extends > `DatabaseModificationEvent` which has the original database. > > Best, > Shammon FY > > On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > > > Thanks Shammon for driving it. > > The FLIP generally looks good to me. I only have one question. > > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and > > the new CatalogDatabase after modified. Is it enough only pass the origin > > database name? Will it be better to contain the origin CatalogDatabase so > > that listener have ways to know what changes? > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "ron9 liu" > > 收件人: "dev" > > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > > > Hi, Shammon > > > > Thanks for driving this FLIP, It will enforce the Flink metadata > capability > > from the platform produce perspective. The overall design looks good to > me, > > I just have some small question: > > 1. Regarding CatalogModificationListenerFactory#createListener method, I > > think it would be better to pass Context as its parameter instead of two > > specific Object. In this way, we can easily extend it in the future and > > there will be no compatibility problems. Refer to > > > > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > > 2. In FLIP, you mentioned that multiple Flink tables may refer to the > same > > physical table, so does the Listener report this physical table > repeatedly？ > > 3. When registering a Listener object, will it connect to an external > > system such as Datahub? If the Listener object registration times out due > > to permission issues, it will affect the execution of all subsequent SQL, > > what should we do in this case? > > > > Best, > > Ron > > > > Shammon FY 于2023年5月31日周三 08:53写道： > > > > > Thanks Feng, the catalog modification listener is only used to report > > > read-only ddl information to other components or systems. > > > > > > > 1. Will an exception thrown by the listener affect the normal > execution > > > process? > > > > > > Users need to handle the exception in the listener themselves. Many > DDLs > > > such as drop tables and alter tables cannot be rolled back, Flink > cannot > > > handle these exceptions for the listener. It will cause the operation > to > > > exit if an exception is thrown, but the executed DDL will be > successful. > > > > > > > 2. What is the order of execution? Is the listener executed first or > > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > The listener will be notified to report catalog modification after DDLs > > are > > > successful, so you can not do permission verification for DDL in the > > > listener. As mentioned above, Flink will not roll back the DDL even > when > > > the listener throws an exception. I think permission verification is > > > another issue and can be discussed separately. > > > > > > > > > Best, > > > Shamm
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, Shammon. I mean do we need to contain the new database after alter in AlterDatabaseEvent? So that the listener can know what has been modified for the database. Or the listener don't need to care about the actual modification. Also, I'm wondering whether AlterDatabaseEven need to include ignoreIfNotExists method since alter database operation don't have such syntax like 'alter database if exists xxx'. Best regards, Yuxia - 原始邮件 - 发件人: "Shammon FY" 收件人: "dev" 发送时间: 星期三, 2023年 5 月 31日 下午 2:55:26 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener Hi yuxia Thanks for your input. The `AlterDatabaseEvent` extends `DatabaseModificationEvent` which has the original database. Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > Thanks Shammon for driving it. > The FLIP generally looks good to me. I only have one question. > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and > the new CatalogDatabase after modified. Is it enough only pass the origin > database name? Will it be better to contain the origin CatalogDatabase so > that listener have ways to know what changes? > > Best regards, > Yuxia > > - 原始邮件 ----- > 发件人: "ron9 liu" > 收件人: "dev" > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from the platform produce perspective. The overall design looks good to me, > I just have some small question: > 1. Regarding CatalogModificationListenerFactory#createListener method, I > think it would be better to pass Context as its parameter instead of two > specific Object. In this way, we can easily extend it in the future and > there will be no compatibility problems. Refer to > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same > physical table, so does the Listener report this physical table repeatedly？ > 3. When registering a Listener object, will it connect to an external > system such as Datahub? If the Listener object registration times out due > to permission issues, it will affect the execution of all subsequent SQL, > what should we do in this case? > > Best, > Ron > > Shammon FY 于2023年5月31日周三 08:53写道： > > > Thanks Feng, the catalog modification listener is only used to report > > read-only ddl information to other components or systems. > > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > Users need to handle the exception in the listener themselves. Many DDLs > > such as drop tables and alter tables cannot be rolled back, Flink cannot > > handle these exceptions for the listener. It will cause the operation to > > exit if an exception is thrown, but the executed DDL will be successful. > > > > > 2. What is the order of execution? Is the listener executed first or > are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > The listener will be notified to report catalog modification after DDLs > are > > successful, so you can not do permission verification for DDL in the > > listener. As mentioned above, Flink will not roll back the DDL even when > > the listener throws an exception. I think permission verification is > > another issue and can be discussed separately. > > > > > > Best, > > Shammon FY > > > > On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > > > > > Hi, Shammon > > > > > > Thanks for driving this Flip, [Support Customized Job Meta Data > Listener] > > > will make it easier for Flink to collect lineage information. > > > I fully agree with the overall solution and have a small question: > > > > > > 1. Will an exception thrown by the listener affect the normal execution > > > process? > > > > > > 2. What is the order of execution? Is the listener executed first or > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > > > > Best, > > > Feng > > > > > > On Fri, May 26, 2023 at 4:09 PM Sham
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi yuxia Thanks for your input. The `AlterDatabaseEvent` extends `DatabaseModificationEvent` which has the original database. Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > Thanks Shammon for driving it. > The FLIP generally looks good to me. I only have one question. > WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and > the new CatalogDatabase after modified. Is it enough only pass the origin > database name? Will it be better to contain the origin CatalogDatabase so > that listener have ways to know what changes? > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "ron9 liu" > 收件人: "dev" > 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 > 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener > > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from the platform produce perspective. The overall design looks good to me, > I just have some small question: > 1. Regarding CatalogModificationListenerFactory#createListener method, I > think it would be better to pass Context as its parameter instead of two > specific Object. In this way, we can easily extend it in the future and > there will be no compatibility problems. Refer to > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same > physical table, so does the Listener report this physical table repeatedly？ > 3. When registering a Listener object, will it connect to an external > system such as Datahub? If the Listener object registration times out due > to permission issues, it will affect the execution of all subsequent SQL, > what should we do in this case? > > Best, > Ron > > Shammon FY 于2023年5月31日周三 08:53写道： > > > Thanks Feng, the catalog modification listener is only used to report > > read-only ddl information to other components or systems. > > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > Users need to handle the exception in the listener themselves. Many DDLs > > such as drop tables and alter tables cannot be rolled back, Flink cannot > > handle these exceptions for the listener. It will cause the operation to > > exit if an exception is thrown, but the executed DDL will be successful. > > > > > 2. What is the order of execution? Is the listener executed first or > are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > The listener will be notified to report catalog modification after DDLs > are > > successful, so you can not do permission verification for DDL in the > > listener. As mentioned above, Flink will not roll back the DDL even when > > the listener throws an exception. I think permission verification is > > another issue and can be discussed separately. > > > > > > Best, > > Shammon FY > > > > On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > > > > > Hi, Shammon > > > > > > Thanks for driving this Flip, [Support Customized Job Meta Data > Listener] > > > will make it easier for Flink to collect lineage information. > > > I fully agree with the overall solution and have a small question: > > > > > > 1. Will an exception thrown by the listener affect the normal execution > > > process? > > > > > > 2. What is the order of execution? Is the listener executed first or > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > > > > Best, > > > Feng > > > > > > On Fri, May 26, 2023 at 4:09 PM Shammon FY wrote: > > > > > > > Hi devs, > > > > > > > > We would like to bring up a discussion about FLIP-294: Support > > Customized > > > > Job Meta Data Listener[1]. We have had several discussions with Jark > > Wu, > > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > > and > > > > interfaces, and thanks for their valuable advice. > > > > The overall job and connector information is divided into metadata > and > > > > lineage, this FLIP focuses on metadata and lineage will be
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi ron, Thanks for your feedback. > 1. Regarding CatalogModificationListenerFactory#createListener method, I think it would be better to pass Context as its parameter instead of two specific Object. In this way, we can easily extend it in the future and there will be no compatibility problems. It sounds good to me I will add Context in CatalogModificationListenerFactory, thanks > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same physical table, so does the Listener report this physical table repeatedly? Yes, the listeners for different jobs may receive the same physical table, users should check and update the table information based on the identifier. For example, users may create tables on the same kafka topic in different jobs, which will notify listeners for the same kafka topic. > 3. When registering a Listener object, will it connect to an external system such as Datahub? If the Listener object registration times out due to permission issues, it will affect the execution of all subsequent SQL, what should we do in this case? Users should establish connections to external systems when creating a listener as needed, and they should handle the exceptions too. If users fail to create a listener and throw an exception, Flink will throw the exception too. Best, Shammon FY On Wed, May 31, 2023 at 11:36 AM liu ron wrote: > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from the platform produce perspective. The overall design looks good to me, > I just have some small question: > 1. Regarding CatalogModificationListenerFactory#createListener method, I > think it would be better to pass Context as its parameter instead of two > specific Object. In this way, we can easily extend it in the future and > there will be no compatibility problems. Refer to > > https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 > 2. In FLIP, you mentioned that multiple Flink tables may refer to the same > physical table, so does the Listener report this physical table repeatedly？ > 3. When registering a Listener object, will it connect to an external > system such as Datahub? If the Listener object registration times out due > to permission issues, it will affect the execution of all subsequent SQL, > what should we do in this case? > > Best, > Ron > > Shammon FY 于2023年5月31日周三 08:53写道： > > > Thanks Feng, the catalog modification listener is only used to report > > read-only ddl information to other components or systems. > > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > Users need to handle the exception in the listener themselves. Many DDLs > > such as drop tables and alter tables cannot be rolled back, Flink cannot > > handle these exceptions for the listener. It will cause the operation to > > exit if an exception is thrown, but the executed DDL will be successful. > > > > > 2. What is the order of execution? Is the listener executed first or > are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > The listener will be notified to report catalog modification after DDLs > are > > successful, so you can not do permission verification for DDL in the > > listener. As mentioned above, Flink will not roll back the DDL even when > > the listener throws an exception. I think permission verification is > > another issue and can be discussed separately. > > > > > > Best, > > Shammon FY > > > > On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > > > > > Hi, Shammon > > > > > > Thanks for driving this Flip, [Support Customized Job Meta Data > Listener] > > > will make it easier for Flink to collect lineage information. > > > I fully agree with the overall solution and have a small question: > > > > > > 1. Will an exception thrown by the listener affect the normal execution > > > process? > > > > > > 2. What is the order of execution? Is the listener executed first or > are > > > specific operations executed first? If I want to perform DDL > permission > > > verification(such as integrating with Ranger based on the listener) , > is > > > that possible? > > > > > > > > > Best, > > > Feng > > > > > > On Fri, May 26, 2023 at 4:09 PM Shammon FY wrote: > > > > > > > Hi devs, > > > > > > > > We would like to bring up a discussion about FLIP-294: Support > > Customized > > > > Job Meta Data Listener[1]. We have had several discussions with Jark > > Wu, > > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > > and > > > > interfaces, and thanks for their valuable advice. > > > > The overall job and connector information is divided into metadata > and > > > > lineage, this FLIP focuses on metadata and lineage will be discussed > in >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks Shammon for driving it. The FLIP generally looks good to me. I only have one question. WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and the new CatalogDatabase after modified. Is it enough only pass the origin database name? Will it be better to contain the origin CatalogDatabase so that listener have ways to know what changes? Best regards, Yuxia - 原始邮件 - 发件人: "ron9 liu" 收件人: "dev" 发送时间: 星期三, 2023年 5 月 31日 上午 11:36:04 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener Hi, Shammon Thanks for driving this FLIP, It will enforce the Flink metadata capability from the platform produce perspective. The overall design looks good to me, I just have some small question: 1. Regarding CatalogModificationListenerFactory#createListener method, I think it would be better to pass Context as its parameter instead of two specific Object. In this way, we can easily extend it in the future and there will be no compatibility problems. Refer to https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 2. In FLIP, you mentioned that multiple Flink tables may refer to the same physical table, so does the Listener report this physical table repeatedly？ 3. When registering a Listener object, will it connect to an external system such as Datahub? If the Listener object registration times out due to permission issues, it will affect the execution of all subsequent SQL, what should we do in this case? Best, Ron Shammon FY 于2023年5月31日周三 08:53写道： > Thanks Feng, the catalog modification listener is only used to report > read-only ddl information to other components or systems. > > > 1. Will an exception thrown by the listener affect the normal execution > process? > > Users need to handle the exception in the listener themselves. Many DDLs > such as drop tables and alter tables cannot be rolled back, Flink cannot > handle these exceptions for the listener. It will cause the operation to > exit if an exception is thrown, but the executed DDL will be successful. > > > 2. What is the order of execution? Is the listener executed first or are > specific operations executed first? If I want to perform DDL permission > verification(such as integrating with Ranger based on the listener) , is > that possible? > > The listener will be notified to report catalog modification after DDLs are > successful, so you can not do permission verification for DDL in the > listener. As mentioned above, Flink will not roll back the DDL even when > the listener throws an exception. I think permission verification is > another issue and can be discussed separately. > > > Best, > Shammon FY > > On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > > > Hi, Shammon > > > > Thanks for driving this Flip, [Support Customized Job Meta Data Listener] > > will make it easier for Flink to collect lineage information. > > I fully agree with the overall solution and have a small question: > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > 2. What is the order of execution? Is the listener executed first or are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > > > Best, > > Feng > > > > On Fri, May 26, 2023 at 4:09 PM Shammon FY wrote: > > > > > Hi devs, > > > > > > We would like to bring up a discussion about FLIP-294: Support > Customized > > > Job Meta Data Listener[1]. We have had several discussions with Jark > Wu, > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > and > > > interfaces, and thanks for their valuable advice. > > > The overall job and connector information is divided into metadata and > > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > > another FLIP in the future. In this FLIP we want to add a customized > > > listener in Flink to report catalog modifications to external metadata > > > systems such as datahub[2] or atlas[3]. Users can view the specific > > > information of connectors such as source and sink for Flink jobs in > these > > > systems, including fields, watermarks, partitions, etc. > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > > [2] https://datahub.io/ > > > [3] https://atlas.apache.org/#/ > > > > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, Shammon Thanks for driving this FLIP, It will enforce the Flink metadata capability from the platform produce perspective. The overall design looks good to me, I just have some small question: 1. Regarding CatalogModificationListenerFactory#createListener method, I think it would be better to pass Context as its parameter instead of two specific Object. In this way, we can easily extend it in the future and there will be no compatibility problems. Refer to https://github.com/apache/flink/blob/9880ba5324d4a1252d6ae1a3f0f061e4469a05ac/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DynamicTableFactory.java#L81 2. In FLIP, you mentioned that multiple Flink tables may refer to the same physical table, so does the Listener report this physical table repeatedly？ 3. When registering a Listener object, will it connect to an external system such as Datahub? If the Listener object registration times out due to permission issues, it will affect the execution of all subsequent SQL, what should we do in this case? Best, Ron Shammon FY 于2023年5月31日周三 08:53写道： > Thanks Feng, the catalog modification listener is only used to report > read-only ddl information to other components or systems. > > > 1. Will an exception thrown by the listener affect the normal execution > process? > > Users need to handle the exception in the listener themselves. Many DDLs > such as drop tables and alter tables cannot be rolled back, Flink cannot > handle these exceptions for the listener. It will cause the operation to > exit if an exception is thrown, but the executed DDL will be successful. > > > 2. What is the order of execution? Is the listener executed first or are > specific operations executed first? If I want to perform DDL permission > verification(such as integrating with Ranger based on the listener) , is > that possible? > > The listener will be notified to report catalog modification after DDLs are > successful, so you can not do permission verification for DDL in the > listener. As mentioned above, Flink will not roll back the DDL even when > the listener throws an exception. I think permission verification is > another issue and can be discussed separately. > > > Best, > Shammon FY > > On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > > > Hi, Shammon > > > > Thanks for driving this Flip, [Support Customized Job Meta Data Listener] > > will make it easier for Flink to collect lineage information. > > I fully agree with the overall solution and have a small question: > > > > 1. Will an exception thrown by the listener affect the normal execution > > process? > > > > 2. What is the order of execution? Is the listener executed first or are > > specific operations executed first? If I want to perform DDL permission > > verification(such as integrating with Ranger based on the listener) , is > > that possible? > > > > > > Best, > > Feng > > > > On Fri, May 26, 2023 at 4:09 PM Shammon FY wrote: > > > > > Hi devs, > > > > > > We would like to bring up a discussion about FLIP-294: Support > Customized > > > Job Meta Data Listener[1]. We have had several discussions with Jark > Wu, > > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions > and > > > interfaces, and thanks for their valuable advice. > > > The overall job and connector information is divided into metadata and > > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > > another FLIP in the future. In this FLIP we want to add a customized > > > listener in Flink to report catalog modifications to external metadata > > > systems such as datahub[2] or atlas[3]. Users can view the specific > > > information of connectors such as source and sink for Flink jobs in > these > > > systems, including fields, watermarks, partitions, etc. > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > > [2] https://datahub.io/ > > > [3] https://atlas.apache.org/#/ > > > > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Thanks Feng, the catalog modification listener is only used to report read-only ddl information to other components or systems. > 1. Will an exception thrown by the listener affect the normal execution process? Users need to handle the exception in the listener themselves. Many DDLs such as drop tables and alter tables cannot be rolled back, Flink cannot handle these exceptions for the listener. It will cause the operation to exit if an exception is thrown, but the executed DDL will be successful. > 2. What is the order of execution? Is the listener executed first or are specific operations executed first? If I want to perform DDL permission verification(such as integrating with Ranger based on the listener) , is that possible? The listener will be notified to report catalog modification after DDLs are successful, so you can not do permission verification for DDL in the listener. As mentioned above, Flink will not roll back the DDL even when the listener throws an exception. I think permission verification is another issue and can be discussed separately. Best, Shammon FY On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > Hi, Shammon > > Thanks for driving this Flip, [Support Customized Job Meta Data Listener] > will make it easier for Flink to collect lineage information. > I fully agree with the overall solution and have a small question: > > 1. Will an exception thrown by the listener affect the normal execution > process? > > 2. What is the order of execution? Is the listener executed first or are > specific operations executed first? If I want to perform DDL permission > verification(such as integrating with Ranger based on the listener) , is > that possible? > > > Best, > Feng > > On Fri, May 26, 2023 at 4:09 PM Shammon FY wrote: > > > Hi devs, > > > > We would like to bring up a discussion about FLIP-294: Support Customized > > Job Meta Data Listener[1]. We have had several discussions with Jark Wu, > > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and > > interfaces, and thanks for their valuable advice. > > The overall job and connector information is divided into metadata and > > lineage, this FLIP focuses on metadata and lineage will be discussed in > > another FLIP in the future. In this FLIP we want to add a customized > > listener in Flink to report catalog modifications to external metadata > > systems such as datahub[2] or atlas[3]. Users can view the specific > > information of connectors such as source and sink for Flink jobs in these > > systems, including fields, watermarks, partitions, etc. > > > > Looking forward to hearing from you, thanks. > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > > [2] https://datahub.io/ > > [3] https://atlas.apache.org/#/ > > >
Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener
Hi, Shammon Thanks for driving this Flip, [Support Customized Job Meta Data Listener] will make it easier for Flink to collect lineage information. I fully agree with the overall solution and have a small question: 1. Will an exception thrown by the listener affect the normal execution process? 2. What is the order of execution? Is the listener executed first or are specific operations executed first? If I want to perform DDL permission verification(such as integrating with Ranger based on the listener) , is that possible? Best, Feng On Fri, May 26, 2023 at 4:09 PM Shammon FY wrote: > Hi devs, > > We would like to bring up a discussion about FLIP-294: Support Customized > Job Meta Data Listener[1]. We have had several discussions with Jark Wu, > Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and > interfaces, and thanks for their valuable advice. > The overall job and connector information is divided into metadata and > lineage, this FLIP focuses on metadata and lineage will be discussed in > another FLIP in the future. In this FLIP we want to add a customized > listener in Flink to report catalog modifications to external metadata > systems such as datahub[2] or atlas[3]. Users can view the specific > information of connectors such as source and sink for Flink jobs in these > systems, including fields, watermarks, partitions, etc. > > Looking forward to hearing from you, thanks. > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener > [2] https://datahub.io/ > [3] https://atlas.apache.org/#/ >
