Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-10 Thread Jark Wu
I have assigned permission to you. Best, Jark > 2023年2月10日 17:26,Feng Jin 写道: > > I am very happy to do it, please help me to add editing permission, my > jira id is hackergin > > Thanks > > Best, > Feng > > On Fri, Feb 10, 2023 at 4:02 PM Jark Wu wrote: >> >> Thank you Feng, >> >>

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-10 Thread Feng Jin
I am very happy to do it, please help me to add editing permission, my jira id is hackergin Thanks Best, Feng On Fri, Feb 10, 2023 at 4:02 PM Jark Wu wrote: > > Thank you Feng, > > Feel free to start a FLIP proposal if you are interested. Looking forward to > it! > > Best, > Jark > > >

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-10 Thread Jark Wu
Thank you Feng, Feel free to start a FLIP proposal if you are interested. Looking forward to it! Best, Jark > 2023年2月10日 15:44,Feng Jin 写道: > > @Shengkai >> About the catalog jar hot updates > > Currently we do not have a similar requirement, but if the catalog > management interface is

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-09 Thread Feng Jin
@Shengkai > About the catalog jar hot updates Currently we do not have a similar requirement, but if the catalog management interface is opened, this can indeed realize the hot loading of the catalog jar > do we need to instantiate the Catalog immediately or defer to the usage I think this

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-09 Thread Jark Wu
Hi Feng, It's still easy to conflict and be inconsistent even if we have only one CatalogProvider, because CatalogProvider only provides readable interfaces (listCatalogs, getCatalog). For example, you may register a catalog X, but can't list it because it's not in the external metadata service.

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-09 Thread Shengkai Fang
Hi Feng. I think your idea is very interesting! 1. I just wonder after initializing the Catalog, will the Session reuse the same Catalog instance or build a new one for later usage? If we reuse the same Catalog, I think it's more like lazy initialization. I am a little prone to rebuild a new one

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-09 Thread Feng Jin
Thanks for your reply. @Timo > 2) avoid the default in-memory catalog and offer their catalog before a > TableEnvironment session starts > 3) whether this can be disabled and SHOW CATALOGS can be used for listing > first without having a default catalog. Regarding 2 and 3, I think this

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-06 Thread Hang Ruan
Hi Feng, I agree with what Jark said. I think what you are looking for is lazy initialization. I don't think we should introduce the new interface CatalogProvider for lazy initialization. What we should do is to store the catalog properties and initialize the catalog when we need it. Could you

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-06 Thread Jark Wu
Hi Feng, I think this feature makes a lot of sense. If I understand correctly, what you are looking for is lazy catalog initialization. However, I have some concerns about introducing CatalogProvider, which delegates catalog management to users. It may be hard to avoid conflicts and duplicates

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-06 Thread Timo Walther
Hi Feng, this is indeed a good proposal. 1) It makes sense to improve the catalog listing for platform providers. 2) Other feedback from the past has shown that users would like to avoid the default in-memory catalog and offer their catalog before a TableEnvironment session starts. 3) Also

[Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

2023-02-06 Thread Feng Jin
Hi everyone, The original discussion address is https://issues.apache.org/jira/browse/FLINK-30126 Currently, Flink has access to many systems, including kafka, hive, iceberg, hudi, elasticsearch, mysql... The corresponding catalog name might be: kafka_cluster1, kafka_cluster2, hive_cluster1,