Hi Feng, Thank you for providing the proposal. I believe this feature will be highly valuable.
I have a couple of inquiries: 1. According to the documentation [1], the createCatalog method does not mention any exceptions being thrown. However, I would like to confirm if it is always true that there will be no failures in all scenarios. Please let me know if there is any additional information I may have missed. 2. Regarding the registration process using the `table.catalog-store.kind` configuration, could you please provide an exhaustive list of the supported kinds? It would be great to have a comprehensive understanding of the options available. Bests, Samrat [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <jinfeng1...@gmail.com> wrote: > Hi Hang and Jark > > Thank you very much for your reply. > > @Jark > > 1. Could you move the CatalogStore registration API to the "Public > Interface" section? > > Indeed, it is more reasonable. > > > 2. We should prefix "table." for the CatalogStore configuration. > > Sure, the names that you provided are indeed more logical and easier to > understand. > > > 3. About the open/close method in CatalogStoreFactory. > > I also agree with the proposed requirement scenario and design. I have > already made modifications to the interface. > > @Hang > > > 1. The `CatalogStore` need the `open`/`close` methods to init or close > the resource. > > I have already added the missing method. > > > 2. The `getCatalog` is misspelled as `optionalDescriptor`. > > Sorry for the misspelling, I updated this part. > > > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods > to init or close its resource > > This is indeed a scenario that needs to be considered, and I have added > relevant methods accordingly. > Additionally, I have included some relevant description information in the > documentation to help others understand > why the open/close methods need to be added. > > > Best, > Feng > > On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <imj...@gmail.com> wrote: > > > Hi Jing, > > > > Thank you for the update. > > > > 1. Could you move the CatalogStore registration API to the "Public > > Interface" section? > > "Proposed Changes" is more like a place to describe the implementation > > details. > > > > 2. We should prefix "table." for the CatalogStore configuration. Besides, > > the config key > > name should be hierarchical[1]. Therefore, it may be better to use: > > "table.catalog-store.kind" > > "table.catalog-store.file.path" > > > > 3. I think Hang's suggestions make sense. > > > > Others look good to me. > > > > Best, > > Jark > > > > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ruanhang1...@gmail.com> wrote: > > > > > Hi, Feng. > > > > > > Thanks for the update. > > > The current design looks good to me. I have some minor comments. > > > > > > 1. The `CatalogStore` need the `open`/`close` methods to init or close > > the > > > resource. For example, when storing the information in MySQL, the store > > > needs to open and close the connections. > > > > > > 2. The `getCatalog` is misspelled as `optionalDescriptor`. > > > > > > 3. About the usage in the sql gateway. > > > Considering the usage in sql gateway, the sql gateway may create a > > > CatalogStore for each session. > > > If we are using the MySqlCatalogStore, there would be so many > > connections. > > > How can we reuse the connection among these sessions? > > > I think sql gateway need to maintain a connection pool in > > > the CatalogStoreFactory and each session get its own connection from > the > > > pool when it is created. > > > Then the `CatalogStoreFactory` may need the `open`/`close` methods to > > init > > > or close its resource. > > > Or is there a better way? > > > > > > Best, > > > Hang > > > > > > Feng Jin <jinfeng1...@gmail.com> 于2023年6月2日周五 14:45写道： > > > > > > > Thanks Jingsong. > > > > > > > > > Just naming, maybe `createCatalog` in TableEnv > > > > > > > > +1 For this, I have already updated FLIP. > > > > > > > > > > > > Best, > > > > Feng > > > > > > > > > > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <jingsongl...@gmail.com> > > > > wrote: > > > > > > > > > Thanks Feng, > > > > > > > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many > > > > > functions are converted to createxxx from registerxxx. > > > > > > > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1...@gmail.com> > > > wrote: > > > > > > > > > > > > Hi jark, thanks for your suggestion. > > > > > > > > > > > > > 1. How to register the CatalogStore for Table API? I think the > > > > > > CatalogStore should be immutable once TableEnv is created. > > Otherwise, > > > > > there > > > > > > might be some data inconsistencies when CatalogStore is changed. > > > > > > > > > > > > Yes, We should initialize the CatalogStore when creating the > > > TableEnv. > > > > > > Therefore, my current proposal is to add a method to configure > the > > > > > > CatalogStore in EnvironmentSettings. > > > > > > > > > > > > // Initialize a catalog Store > > > > > > CatalogStore catalogStore = new FileCatalogStore(""); > > > > > > > > > > > > // set up the Table API > > > > > > final EnvironmentSettings settings = > > > > > > EnvironmentSettings.newInstance().inBatchMode() > > > > > > .withCatalogStore(catalogStore) > > > > > > .build(); > > > > > > > > > > > > final TableEnvironment tableEnv = > > > > > TableEnvironment.create(settings); > > > > > > > > > > > > > > > > > > > 2. Why does the CatalogStoreFactory interface only have a > default > > > > > method, > > > > > > not an interface method? > > > > > > > > > > > > Sorry, While I did refer to the Catalog interface, I agree that > as > > a > > > > new > > > > > > interface, the CatalogStoreFactory should not have a default > method > > > but > > > > > an > > > > > > interface method. I have already modified the interface. > > > > > > > > > > > > > 3. Please mention the alternative API in Javadoc for the > > > > > > deprecated`registerCatalog`. > > > > > > > 4. In the "Compatibility" section, would be better to mention > the > > > > > changed > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or > other > > > > > > persisted catalog store) is used. > > > > > > > > > > > > Thanks for the suggestion, I have updated the FLIP [1]. > > > > > > > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > > > > > > > > > > > > > > > > > Best, > > > > > > Feng > > > > > > > > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <imj...@gmail.com> wrote: > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > This is a useful FLIP. Thanks for starting this discussion. > > > > > > > The current design looks pretty good to me. I just have some > > minor > > > > > > > comments. > > > > > > > > > > > > > > 1. How to register the CatalogStore for Table API? I think the > > > > > CatalogStore > > > > > > > should be immutable once TableEnv is created. Otherwise, there > > > might > > > > be > > > > > > > some data inconsistencies when CatalogStore is changed. > > > > > > > > > > > > > > 2. Why does the CatalogStoreFactory interface only have a > default > > > > > method, > > > > > > > not an interface method? > > > > > > > > > > > > > > 3. Please mention the alternative API in Javadoc for the > > deprecated > > > > > > > `registerCatalog`. > > > > > > > > > > > > > > 4. In the "Compatibility" section, would be better to mention > the > > > > > changed > > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or > > other > > > > > > > persisted catalog store) is used. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Jark > > > > > > > > > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <jinfeng1...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > Hi ， thanks all for reviewing the flip. > > > > > > > > > > > > > > > > @Ron > > > > > > > > > > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore > method, > > > do > > > > we > > > > > > > need > > > > > > > > to provide a default implementation? > > > > > > > > > > > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to > > > > create > > > > > an > > > > > > > > InMemoryCatalogStore. > > > > > > > > > > > > > > > > > If we get a Catalog from CatalogStore, after initializing > > it, > > > > > whether > > > > > > > we > > > > > > > > put it in Map<String, Catalog> catalogs again? > > > > > > > > > > > > > > > > Yes, in the current design, catalogs are stored as snapshots, > > and > > > > > once > > > > > > > > initialized, the Catalog will be placed in the Map<String, > > > Catalog> > > > > > > > > catalogs. > > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the > > > primary > > > > > > > source > > > > > > > > for obtaining the corresponding Catalog. > > > > > > > > > > > > > > > > > how about renaming them to `catalog.store.type` and > > > > > > > > `catalog.store.path`? > > > > > > > > > > > > > > > > I think it is okay. Adding "sql" at the beginning may seem a > > bit > > > > > > > strange. I > > > > > > > > will update the FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon > > > > > > > > > > > > > > > > Thank you for the review. I have made the necessary > > corrections. > > > > > > > > Regarding the modifications made to the Public Interface, I > > have > > > > also > > > > > > > > included the relevant changes to the `TableEnvironment`. > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY < > zjur...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi feng, > > > > > > > > > > > > > > > > > > Thanks for updating, I have some minor comments > > > > > > > > > > > > > > > > > > 1. The modification of `CatalogManager` should not be in > > > `Public > > > > > > > > > Interfaces`, it is not a public interface. > > > > > > > > > > > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and > > > > > > > > > `CatalogStoreFactory` > > > > > > > > > > > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor = > > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager` > > should > > > be > > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor = > > > > > > > > > catalogStore.get(catalogName);` > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron < > ron9....@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, Feng > > > > > > > > > > > > > > > > > > > > Thanks for driving this FLIP, this proposal is very > useful > > > for > > > > > > > catalog > > > > > > > > > > management. > > > > > > > > > > I have some small questions: > > > > > > > > > > > > > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore > > > method, > > > > > do we > > > > > > > > > need > > > > > > > > > > to provide a default implementation? > > > > > > > > > > 2. If we get Catalog from CatalogStore, after > initializing > > > it, > > > > > > > whether > > > > > > > > we > > > > > > > > > > put it to Map<String, Catalog> catalogs again? > > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and > > > > > > > > > > `sql.catalog.store.file.path`, how about renaming them to > > > > > > > > > > `catalog.store.type` and `catalog.store.path`? > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Ron > > > > > > > > > > > > > > > > > > > > Feng Jin <jinfeng1...@gmail.com> 于2023年5月29日周一 21:19写道： > > > > > > > > > > > > > > > > > > > > > Hi yuxia > > > > > > > > > > > > > > > > > > > > > > > But from the code in Proposed Changes, once we > > register > > > > the > > > > > > > > Catalog, > > > > > > > > > > we > > > > > > > > > > > initialize it and open it. right? > > > > > > > > > > > > > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the > > > original > > > > > > > CREATE > > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in > > > > > > > registerCatalog > > > > > > > > so > > > > > > > > > > > that parameter validation can be performed. > > > > > > > > > > > > > > > > > > > > > > In the current design, lazy initialization is mainly > > > > reflected > > > > > in > > > > > > > > > > > getCatalog. If CatalogStore has already saved some > > catalog > > > > > > > > > > configurations, > > > > > > > > > > > only initialization is required in getCatalog. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia < > > > > > luoyu...@alumni.sjtu.edu.cn > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi, Feng. > > > > > > > > > > > > I'm trying to understanding the meaning of *lazy > > > > > initialization*. > > > > > > > > If > > > > > > > > > > i'm > > > > > > > > > > > > wrong, please correct me. > > > > > > > > > > > > > > > > > > > > > > > > IIUC, lazy initialization means only you need to > access > > > the > > > > > > > > catalog, > > > > > > > > > > then > > > > > > > > > > > > you initialize it. But from the code in Proposed > > Changes, > > > > > once we > > > > > > > > > > > register > > > > > > > > > > > > the Catalog, > > > > > > > > > > > > we initialize it and open it. right? > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Yuxia > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 ----- > > > > > > > > > > > > 发件人: "Jing Ge" <j...@ververica.com.INVALID> > > > > > > > > > > > > 收件人: "dev" <dev@flink.apache.org> > > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46 > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of > > > Catalog > > > > > > > > > > configuration > > > > > > > > > > > > and asynchronous registration > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your effort! +1 for the proposal. > > > > > > > > > > > > > > > > > > > > > > > > One of the major changes is that current design will > > > > provide > > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead > of a > > > > > cache, > > > > > > > > which > > > > > > > > > > > means > > > > > > > > > > > > once it has been initialized, any changes done by > other > > > > > sessions > > > > > > > > will > > > > > > > > > > not > > > > > > > > > > > > affect it. Point 6 described follow-up options for > > > further > > > > > > > > > improvement. > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin < > > > > > jinfeng1...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, I would like to update you on the latest > > > progress > > > > > of > > > > > > > the > > > > > > > > > > FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon > FY, > > > > > ShengKai > > > > > > > > Fang > > > > > > > > > > > and I > > > > > > > > > > > > > had an offline discussion regarding the overall > > > solution > > > > > for > > > > > > > > Flink > > > > > > > > > > > > > CatalogStore. We have reached a consensus and I > have > > > > > updated > > > > > > > the > > > > > > > > > > final > > > > > > > > > > > > > solution in FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > Next, let me briefly describe the entire design: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog > > > > > configuration > > > > > > > > > similar > > > > > > > > > > > to > > > > > > > > > > > > > TableDescriptor. > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > > > > > > > The two key functions of CatalogStore - void > > > > > > > > storeCatalog(String > > > > > > > > > > > > > catalogName, CatalogDescriptor) and > > > CatalogDescriptor > > > > > > > > > > > > getCatalog(String) > > > > > > > > > > > > > will both use CatalogDescriptor instead of > Catalog > > > > > instance. > > > > > > > > > This > > > > > > > > > > > way, > > > > > > > > > > > > > CatalogStore will only be responsible for saving > > and > > > > > > > > retrieving > > > > > > > > > > > > catalog > > > > > > > > > > > > > configurations without having to initialize > > > catalogs. > > > > > > > > > > > > > 3. > > > > > > > > > > > > > > > > > > > > > > > > > > The default registerCatalog(String catalogName, > > > > Catalog > > > > > > > > catalog) > > > > > > > > > > > > > function in CatalogManager will be marked as > > > > deprecated. > > > > > > > > > > > > > 4. > > > > > > > > > > > > > > > > > > > > > > > > > > A new function registerCatalog(String > catalogName, > > > > > > > > > > CatalogDescriptor > > > > > > > > > > > > > catalog) will be added to serve as the default > > > > > registration > > > > > > > > > > function > > > > > > > > > > > > for > > > > > > > > > > > > > catalogs in CatalogManager. > > > > > > > > > > > > > 5. > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager > will > > > > > remain > > > > > > > > > > unchanged > > > > > > > > > > > > and > > > > > > > > > > > > > save initialized catalogs.This means that > deletion > > > > > > > operations > > > > > > > > > from > > > > > > > > > > > one > > > > > > > > > > > > > session won't synchronize with other sessions. > > > > > > > > > > > > > 6. > > > > > > > > > > > > > > > > > > > > > > > > > > To support multi-session synchronization > scenarios > > > for > > > > > > > > deletions > > > > > > > > > > > later > > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs > > > > > > > > configurable.There > > > > > > > > > > may > > > > > > > > > > > > be > > > > > > > > > > > > > three possible situations: > > > > > > > > > > > > > > > > > > > > > > > > > > a.Default caching of all initialized catalogs > > > > > > > > > > > > > > > > > > > > > > > > > > b.Introduction of LRU cache logic which can > > > > > automatically > > > > > > > > clear > > > > > > > > > > > > > long-unused catalogs. > > > > > > > > > > > > > > > > > > > > > > > > > > c.No caching of any instances; each call to > > > getCatalog > > > > > > > > creates a > > > > > > > > > > new > > > > > > > > > > > > > instance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is the document for discussion: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit > > > > > > > > > > > > > > > > > > > > > > > > > > This is the final proposal document: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you very much for your attention and > > suggestions > > > on > > > > > this > > > > > > > > > > FLIP. A > > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions on > > the > > > > > entire > > > > > > > > > design > > > > > > > > > > > and > > > > > > > > > > > > > organizing offline discussions. > > > > > > > > > > > > > > > > > > > > > > > > > > If you have any further suggestions or feedback > about > > > > this > > > > > FLIP > > > > > > > > > > please > > > > > > > > > > > > feel > > > > > > > > > > > > > free to share. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge > > > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to > me. > > > We > > > > > could > > > > > > > > > still > > > > > > > > > > > > > > reconsider in the future how to provide more > common > > > > > built-in > > > > > > > > > cache > > > > > > > > > > > > > > functionality in CatalogManager so that not every > > > > > > > CatalogSotre > > > > > > > > > > > > > > implementation has to take care of it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin < > > > > > > > jinfeng1...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jing, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your reply. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There might be more such issues. I would > > suggest > > > > you > > > > > > > > > > completely > > > > > > > > > > > > walk > > > > > > > > > > > > > > > through the FLIP again and fix those issues > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am very sorry for my carelessness and at the > > same > > > > > time, I > > > > > > > > > > greatly > > > > > > > > > > > > > > > appreciate your careful review. I have > thoroughly > > > > > checked > > > > > > > the > > > > > > > > > > > entire > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > and made corrections to these issues. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If I am not mistaken, with the current FLIP > > > > design, > > > > > > > > > > > > CatalogManager > > > > > > > > > > > > > > > could work without Optional CatalogStore being > > > > > configured. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, in the original design, CatalogStore was > not > > > > > necessary > > > > > > > > > > because > > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog> > catalogs > > > to > > > > > store > > > > > > > > > > catalog > > > > > > > > > > > > > > > instances. > > > > > > > > > > > > > > > However, this caused inconsistency issues. > > > > Therefore, I > > > > > > > > > modified > > > > > > > > > > > this > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > of the design and removed Map<String, Catalog> > > > > catalogs > > > > > > > from > > > > > > > > > > > > > > > CatalogManager. > > > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve > as > > > the > > > > > > > default > > > > > > > > > > > > > CatalogStore > > > > > > > > > > > > > > > to save catalogs in memory and replace the > > > > > functionality of > > > > > > > > > > > > > > > Map<String,Catalog>catalogs. > > > > > > > > > > > > > > > The previous plan that kept > > > > > Map<String,Catalog>catalogs has > > > > > > > > > been > > > > > > > > > > > > moved > > > > > > > > > > > > > to > > > > > > > > > > > > > > > Rejected Alternatives. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge > > > > > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are still many places contain > > inconsistent > > > > > content, > > > > > > > > > e.g. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still used. > > > > > > > > > > > > > > > > 2. The java comment of the method > > > > > registerCatalog(String > > > > > > > > > > > > catalogName, > > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not > > tell > > > > > what the > > > > > > > > > > method > > > > > > > > > > > is > > > > > > > > > > > > > > > doing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There might be more such issues. I would > > suggest > > > > you > > > > > > > > > completely > > > > > > > > > > > > walk > > > > > > > > > > > > > > > > through the FLIP again and fix those issues. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean > > that > > > > you > > > > > will > > > > > > > > > build > > > > > > > > > > > the > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > functionality in the CatalogStore? This is a > > very > > > > > > > different > > > > > > > > > > > design > > > > > > > > > > > > > > > concept > > > > > > > > > > > > > > > > from what the current FLIP described. If I am > > not > > > > > > > mistaken, > > > > > > > > > > with > > > > > > > > > > > > the > > > > > > > > > > > > > > > > current FLIP design, CatalogManager could > work > > > > > without > > > > > > > > > Optional > > > > > > > > > > > > > > > > CatalogStore being configured. That is the > > reason > > > > > why I > > > > > > > > > > mentioned > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > last email that the example code wrt the > > Optional > > > > > > > > > CatalogStore > > > > > > > > > > is > > > > > > > > > > > > not > > > > > > > > > > > > > > > > correct. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin < > > > > > > > > > > jinfeng1...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are still some NIT issues in the > FLIP > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you very much for the careful > review. I > > > > have > > > > > > > > already > > > > > > > > > > made > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > relevant changes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the > > > > > > > multi-instance > > > > > > > > > > > > > scenarios, I > > > > > > > > > > > > > > > am > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > sure if this is the intended behaviour > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, there are conflicts in multiple > > > > > scenarios > > > > > > > with > > > > > > > > > the > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > design. I am thinking whether we should > > remove > > > > > > > > 'Map<String, > > > > > > > > > > > > > Catalog>' > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > make Cache the default behavior of > > > > > > > InMemoryCatalogStore. > > > > > > > > > This > > > > > > > > > > > > way, > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > can implement their own CatalogStore to > > achieve > > > > > > > > > > multi-instance > > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you > think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge > > > > > > > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are > > > still > > > > > some > > > > > > > > NIT > > > > > > > > > > > issues > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > like: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore > has > > > been > > > > > used > > > > > > > as > > > > > > > > > > > > > > CatalogStore > > > > > > > > > > > > > > > > > > instead of Optional in the code example. > It > > > > > should be > > > > > > > > > fine > > > > > > > > > > to > > > > > > > > > > > > use > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > pseudo code for now and update it after > you > > > > > submit > > > > > > > the > > > > > > > > > PR. > > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used > somewhere > > in > > > > the > > > > > > > > > rejected > > > > > > > > > > > > > section > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > should be persistContext(...) to keep it > > > > > consistent. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the > > > > > multi-instance > > > > > > > > > > > > scenarios, > > > > > > > > > > > > > I > > > > > > > > > > > > > > am > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > sure if this is the intended behaviour. > If > > > > > > > Map<String, > > > > > > > > > > > Catalog> > > > > > > > > > > > > > > > > catalogs > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > used as a cache, it should be invalid, > once > > > the > > > > > > > related > > > > > > > > > > > catalog > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > removed from the CatalogStore by another > > > > > instance. > > > > > > > Did > > > > > > > > I > > > > > > > > > > miss > > > > > > > > > > > > > > > > something? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin > < > > > > > > > > > > > > jinfeng1...@gmail.com> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jing,Shammon > > > > > > > > > > > > > > > > > > > Thanks for your reply. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How about persistCatalog()? > > > > > > > > > > > > > > > > > > > I think this is a good function name, I > > > have > > > > > > > updated > > > > > > > > it > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > documentation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some common cache features should > be > > > > > > > implemented > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If > > > alternative > > > > 1 > > > > > > > turns > > > > > > > > > out > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > appropriate later, I will improve this > > part > > > > of > > > > > the > > > > > > > > > > content. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As the above discussion moves > forward, > > > the > > > > > > > option 2 > > > > > > > > > > > > solution > > > > > > > > > > > > > > > looks > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > like a replacement of option 1 > > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon > > offline, > > > > we > > > > > > > think > > > > > > > > > that > > > > > > > > > > > > > > solution > > > > > > > > > > > > > > > 2 > > > > > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > > > > be more suitable and also avoid any > > > > > inconsistency > > > > > > > > > issues. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are some inconsistent > > descriptions > > > in > > > > > the > > > > > > > > > > content. > > > > > > > > > > > > > Would > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > to clean them up? > > > > > > > > > > > > > > > > > > > I will do my best to improve the > document > > > and > > > > > > > > > appreciate > > > > > > > > > > > your > > > > > > > > > > > > > > > > > > suggestions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon > > > > > > > > > > > > > > > > > > > > can you put the unselected option in > > > > > `Rejected > > > > > > > > > > > > Alternatives` > > > > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected > > > > > > > Alternatives`. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > > > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon > > FY > > > < > > > > > > > > > > > > zjur...@gmail.com> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your update. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I found there are two options in > > > `Proposed > > > > > > > > Changes`, > > > > > > > > > > can > > > > > > > > > > > > you > > > > > > > > > > > > > > put > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > unselected option in `Rejected > > > > > Alternatives`? I > > > > > > > > think > > > > > > > > > > > this > > > > > > > > > > > > > may > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > us > > > > > > > > > > > > > > > > > > > > better understand your proposal > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing > Ge > > > > > > > > > > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am > > > still > > > > > > > confused > > > > > > > > > > after > > > > > > > > > > > > > > > > completely > > > > > > > > > > > > > > > > > > > > reading > > > > > > > > > > > > > > > > > > > > > the thread with following > questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Naming confusion - > > registerCatalog() > > > > and > > > > > > > > > > > addCatalog() > > > > > > > > > > > > > have > > > > > > > > > > > > > > > no > > > > > > > > > > > > > > > > > big > > > > > > > > > > > > > > > > > > > > > difference based on their names. > One > > of > > > > > them is > > > > > > > > > > > > responsible > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > persistence. How about > > > persistCatalog()? > > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that > Map<String, > > > > > Catalog> > > > > > > > > > > catalogs > > > > > > > > > > > is > > > > > > > > > > > > > > used > > > > > > > > > > > > > > > > as a > > > > > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > > > and catalogStore is used for data > > > > > persistence. > > > > > > > I > > > > > > > > > > would > > > > > > > > > > > > > > suggest > > > > > > > > > > > > > > > > > > > describing > > > > > > > > > > > > > > > > > > > > > their purpose conceptually and > > clearly > > > in > > > > > the > > > > > > > > FLIP. > > > > > > > > > > > Some > > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > > > features should be implemented, > i.e. > > > data > > > > > in > > > > > > > the > > > > > > > > > > cache > > > > > > > > > > > > and > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > store > > > > > > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog > instance > > > > > should be > > > > > > > > > found > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > > store > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > the cache(either it has been > > > initialized > > > > > or it > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > > lazy > > > > > > > > > > > > > > > > > > > initialized) > > > > > > > > > > > > > > > > > > > > > for the same catalog name. The > > > > consistency > > > > > will > > > > > > > > be > > > > > > > > > > > taken > > > > > > > > > > > > > care > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > > > > > > > updating the catalog. > > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves > > > forward, > > > > > the > > > > > > > > > option > > > > > > > > > > 2 > > > > > > > > > > > > > > solution > > > > > > > > > > > > > > > > > looks > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > like a replacement of option 1, > > > because, > > > > > afaiu, > > > > > > > > > > issues > > > > > > > > > > > > > > > mentioned > > > > > > > > > > > > > > > > > > > > > previously with option 1 are not > > solved > > > > > yet. Do > > > > > > > > you > > > > > > > > > > > still > > > > > > > > > > > > > > want > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > propose > > > > > > > > > > > > > > > > > > > > > both options and ask for > suggestions > > > for > > > > > both > > > > > > > of > > > > > > > > > > them? > > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, > there > > > are > > > > > some > > > > > > > > > > > > inconsistent > > > > > > > > > > > > > > > > > > descriptions > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > the content. Would you like to > clean > > > > them > > > > > up? > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng > > > Jin < > > > > > > > > > > > > > > jinfeng1...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > hi Shammon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your response, and > I > > > > > completely > > > > > > > > > agree > > > > > > > > > > > > with > > > > > > > > > > > > > > your > > > > > > > > > > > > > > > > > point > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > view. > > > > > > > > > > > > > > > > > > > > > > Initially, I may have over > > > complicated > > > > > the > > > > > > > > whole > > > > > > > > > > > issue. > > > > > > > > > > > > > > First > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > foremost, > > > > > > > > > > > > > > > > > > > > > > we need to consider the > persistence > > > of > > > > > the > > > > > > > > > > Catalog's > > > > > > > > > > > > > > > > > Configuration. > > > > > > > > > > > > > > > > > > > > > > If we only need to provide > > > persistence > > > > > for > > > > > > > > > Catalog > > > > > > > > > > > > > > > > Configuration, > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to > the > > > > > Catalog > > > > > > > > > > > interface. > > > > > > > > > > > > > > > > > > > > > > This method can convert a Catalog > > > > > instance > > > > > > > to a > > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > > String> > > > > > > > > > > > > > > > > > > > > > > properties, and the default > > > > > implementation > > > > > > > will > > > > > > > > > > throw > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > exception. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public interface Catalog { > > > > > > > > > > > > > > > > > > > > > > /** > > > > > > > > > > > > > > > > > > > > > > * Returns a map containing the > > > > > properties > > > > > > > of > > > > > > > > > the > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > object. > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > * @return a map containing the > > > > > properties > > > > > > > of > > > > > > > > > the > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > > > > > > * @throws > > > > > UnsupportedOperationException if > > > > > > > > the > > > > > > > > > > > > > > > implementing > > > > > > > > > > > > > > > > > > class > > > > > > > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > > > > > > > > not override > > > > > > > > > > > > > > > > > > > > > > * the default implementation > of > > > this > > > > > > > method > > > > > > > > > > > > > > > > > > > > > > */ > > > > > > > > > > > > > > > > > > > > > > default Map<String, String> > > > > > toProperties() > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > throw new > > > > > > > > > UnsupportedOperationException("Please > > > > > > > > > > > > > > implement > > > > > > > > > > > > > > > > > > > > > toProperties > > > > > > > > > > > > > > > > > > > > > > for this catalog"); > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The specific process is as > follows: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. If the user has configured a > > > > > > > CatalogStore, > > > > > > > > > the > > > > > > > > > > > > > > > > > > toConfiguration() > > > > > > > > > > > > > > > > > > > > > method > > > > > > > > > > > > > > > > > > > > > > will be called when registering a > > > > Catalog > > > > > > > > > instance > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String > catalogName, > > > > > Catalog > > > > > > > > > > catalog). > > > > > > > > > > > > The > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > will be converted to a > Map<String, > > > > > String> > > > > > > > > > > properties > > > > > > > > > > > > and > > > > > > > > > > > > > > > saved > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog > > > instances > > > > > do > > > > > > > not > > > > > > > > > > > > implement > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > method, > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > exception will be thrown. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a > > > > > CatalogStore, > > > > > > > > the > > > > > > > > > > > > > > > > > toConfiguration() > > > > > > > > > > > > > > > > > > > > method > > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring > > > > consistency > > > > > with > > > > > > > > the > > > > > > > > > > > > > original > > > > > > > > > > > > > > > > > process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String, > > > Catalog> > > > > > > > > catalogs > > > > > > > > > > and > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > CatalogStore > > > > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > the same time can also avoid > > > conflicts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For lazy initialization: > > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog > > itself > > > > and > > > > > > > > provide > > > > > > > > > a > > > > > > > > > > > > > > dedicated > > > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > > > > implementation for lazy loading, > > such > > > > as > > > > > > > > > > > > > > ConfigurationCatalog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog > > > > > implements > > > > > > > > > > Catalog > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, > > > > String> > > > > > > > > > > properties) > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have added this design to the > > FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM > > > Shammon > > > > > FY < > > > > > > > > > > > > > > > zjur...@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have no questions about the > > > > > functionality > > > > > > > > of > > > > > > > > > > > > > > > > `CatalogStore`, > > > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > different operations of > multiple > > > > > > > > > > `registerCatalog` > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > `storeCatalog` > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The > > > implementation > > > > in > > > > > > > Trino > > > > > > > > > is > > > > > > > > > > > also > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > same, > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only > > has > > > > one > > > > > > > > > > > > `createCatalog`, > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > save > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > catalog to memory and > > > `CatalogStore`. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add > > > another > > > > > > > > > > > > `registerCatalog` > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > `addCatalog` > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String, > > > > Catalog> > > > > > > > > > catalogs` > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > `CatalogStore > > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you > mentioned > > > > > before, > > > > > > > the > > > > > > > > > > > > > `Map<String, > > > > > > > > > > > > > > > > > > Catalog> > > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in > > > > > `CatalogManager`. > > > > > > > How > > > > > > > > > > about > > > > > > > > > > > > > > saving > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` > > and > > > > > > > > > `CatalogStore > > > > > > > > > > > > > > > > catalogStore` > > > > > > > > > > > > > > > > > > > > > together > > > > > > > > > > > > > > > > > > > > > > > when it is registered or added > in > > > > > > > > > > `CatalogManager`? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM > > Feng > > > > > Jin < > > > > > > > > > > > > > > > > jinfeng1...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I > am > > > very > > > > > sorry > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > misunderstanding > > > > > > > > > > > > > > > > > > > > > > caused > > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the > > original > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a > > > pre-discussion > > > > > [1] > > > > > > > for > > > > > > > > > > this > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a > > > > > discussion > > > > > > > > > before. > > > > > > > > > > > > > > However, > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > designing > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found that > > the > > > > > logic of > > > > > > > > > > > > > > CatalogManager > > > > > > > > > > > > > > > > > itself > > > > > > > > > > > > > > > > > > > > > doesn't > > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We > cannot > > > > only > > > > > > > > persist > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > instances > > > > > > > > > > > > > > > > > > > > > > > themselves*, > > > > > > > > > > > > > > > > > > > > > > > > so exposing only > > > > > registerCatalog(String > > > > > > > > > > > > catalogName, > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > > catalog) > > > > > > > > > > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > > > > > > > > > not be enough to save > Catalogs, > > > > > because > > > > > > > in > > > > > > > > > the > > > > > > > > > > > end > > > > > > > > > > > > we > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > save > > > > > > > > > > > > > > > > > > > > > > > > the configurations > > corresponding > > > to > > > > > the > > > > > > > > > Catalog > > > > > > > > > > > > > > > instances. > > > > > > > > > > > > > > > > > > > > > Therefore, > > > > > > > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the > > > > CatalogStore > > > > > > > > > interface > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this > > > part, I > > > > > also > > > > > > > > took > > > > > > > > > > > > > > inspiration > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > Trino's > > > > > > > > > > > > > > > > > > > > > > > > implementation[1]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @yuxia > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of > handling > > > > > catalog > > > > > > > with > > > > > > > > > the > > > > > > > > > > > > same > > > > > > > > > > > > > > name > > > > > > > > > > > > > > > > > > looks a > > > > > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > complex to me. > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the > suggestion. I > > > > will > > > > > > > > provide > > > > > > > > > a > > > > > > > > > > > > > detailed > > > > > > > > > > > > > > > > > > > description > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > code examples for this part, > > and > > > > add > > > > > it > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name > > > > > `addCatalog` > > > > > > > > still > > > > > > > > > > > looks > > > > > > > > > > > > > > > confused > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > me. > > > > > > > > > > > > > > > > > > > > > > IIUC, > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing > > > catalog > > > > to > > > > > > > > > > > CatalogStore, > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > renaming > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very > > > > personal > > > > > > > > > opinion, > > > > > > > > > > > you > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > decide > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > take > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > > > > not by your self. > > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more > > intuitive > > > > to > > > > > me. > > > > > > > I > > > > > > > > > > don't > > > > > > > > > > > > see > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG > > statement, > > > > > which > > > > > > > > > method > > > > > > > > > > > will > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > called? > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or > > > `addCatalog`? > > > > > I'm > > > > > > > > > > wondering > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > add a > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL > > > stement. > > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my > original > > > > > design > > > > > > > was > > > > > > > > to > > > > > > > > > > add > > > > > > > > > > > > it > > > > > > > > > > > > > > > > directly > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would > > > > disrupt > > > > > the > > > > > > > > > > existing > > > > > > > > > > > > > > logic. > > > > > > > > > > > > > > > > > > > > Therefore, I > > > > > > > > > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the > > > > > configuration to > > > > > > > > the > > > > > > > > > > > > > > > CatalogStore > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > initialize > > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the > > same > > > > time > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary > to > > > > > provide a > > > > > > > > > > default > > > > > > > > > > > > > > > > > implmentation > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > interface > > `CatalogStoreFactory`? > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary, > > > otherwise > > > > we > > > > > > > would > > > > > > > > > > need > > > > > > > > > > > to > > > > > > > > > > > > > > > > introduce > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > additional > > > > > > > > > > > > > > > > > > > > > > > > Map to store the > configuration > > > for > > > > > lazy > > > > > > > > > > loading. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous > > > > registration > > > > > for > > > > > > > > > > catalog. > > > > > > > > > > > > > > > > > > > > > > > > I don't think > > > > registerCatalog(String > > > > > > > > > > catalogName, > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > catalog) > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous > > > interface > > > > > > > because > > > > > > > > > > > Catalog > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > already > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > instance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy > > > > > initialization > > > > > > > > for > > > > > > > > > > > > catalog > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > > > > > > > > > > registration, right? > > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was > > > > > inaccurate. It > > > > > > > > > > should > > > > > > > > > > > be > > > > > > > > > > > > > > lazy > > > > > > > > > > > > > > > > > > > > registration > > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous > > > > > registration. I > > > > > > > > have > > > > > > > > > > > > already > > > > > > > > > > > > > > > > updated > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > title > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > the FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at > 4:27 PM > > > > > Shammon > > > > > > > FY < > > > > > > > > > > > > > > > > zjur...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a > > > > > `registerCatalog` > > > > > > > > > > method > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > `CatalogManager`, > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a > > method > > > > > named > > > > > > > > > > > `addCatalog` > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > `storeCatalog` > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > added. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the > > > memory > > > > > > > catalog > > > > > > > > > is a > > > > > > > > > > > > > > `cache`, > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > concept > > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be > > > exposed > > > > to > > > > > > > > users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a > > > pre-discussion > > > > > [1] > > > > > > > for > > > > > > > > > > this > > > > > > > > > > > > > FLIP. > > > > > > > > > > > > > > > > Please > > > > > > > > > > > > > > > > > > > > correct > > > > > > > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the > > conclusion > > > > of > > > > > that > > > > > > > > > > > > discussion > > > > > > > > > > > > > is > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an > > > interface > > > > > and > > > > > > > > > > implement > > > > > > > > > > > it > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > > > > stores > > > > > > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and > external > > > > > system. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap > > between > > > > the > > > > > > > > current > > > > > > > > > > FLIP > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal of > > the > > > > > > > discussion > > > > > > > > > in > > > > > > > > > > > > thread > > > > > > > > > > > > > > > [1] ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at > > 3:41 PM > > > > > yuxia < > > > > > > > > > > > > > > > > > > > > luoyu...@alumni.sjtu.edu.cn> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving > > this > > > > > FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments: > > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of > > handling > > > > > catalog > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > > > same > > > > > > > > > > > > > > > name > > > > > > > > > > > > > > > > > > > looks a > > > > > > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think > > it'll > > > be > > > > > > > better > > > > > > > > to > > > > > > > > > > > > explain > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > java > > > > > > > > > > > > > > > > > > > > > > doc > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > these methods and give a > > > brief > > > > > > > example > > > > > > > > in > > > > > > > > > > > this > > > > > > > > > > > > > > FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name > > > > > `addCatalog` > > > > > > > > > still > > > > > > > > > > > > looks > > > > > > > > > > > > > > > > confused > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > me. > > > > > > > > > > > > > > > > > > > > > > > IIUC, > > > > > > > > > > > > > > > > > > > > > > > > > > this method is for > storing > > > > > catalog to > > > > > > > > > > > > > CatalogStore, > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > > renaming > > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's > > very > > > > > personal > > > > > > > > > > > opinion, > > > > > > > > > > > > > you > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > decide > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > take > > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > > > > > > not by your self. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG > > > > statement, > > > > > > > which > > > > > > > > > > method > > > > > > > > > > > > > will > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > called? > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or > > > > > `addCatalog`? > > > > > > > I'm > > > > > > > > > > > > wondering > > > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > add a > > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL > > > > > stement. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really > neccessary > > to > > > > > > > provide a > > > > > > > > > > > default > > > > > > > > > > > > > > > > > > implmentation > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > interface > > > > `CatalogStoreFactory`? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous > > > > > registration > > > > > > > for > > > > > > > > > > > catalog. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog > > > with > > > > > CREATE > > > > > > > > > > > CATALOG, > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is > used > > > by > > > > > > > default. > > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous > > registration > > > is > > > > > the > > > > > > > > > default > > > > > > > > > > > > > > behavior, > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > > > > > way > > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to > > > > > synchronous > > > > > > > > > > > > registration > > > > > > > > > > > > > > just > > > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > > > before? > > > > > > > > > > > > > > > > > > > > > > > > > > Will both method > > `addCatalog` > > > > and > > > > > > > > > > > > > `registerCatalog` > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > > > > > > > > > > > > registration? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous > > > > > registration, > > > > > > > it > > > > > > > > > may > > > > > > > > > > > well > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > CREATE > > > > > > > > > > > > > > > > > > > > > CATALOG > > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully, > but > > > then > > > > > the > > > > > > > > > > following > > > > > > > > > > > > > > CREATE > > > > > > > > > > > > > > > > > TABLE > > > > > > > > > > > > > > > > > > > > > > statement > > > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail > > to > > > > > open. > > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break > change > > > > which > > > > > > > > should > > > > > > > > > be > > > > > > > > > > > > > > > highlighted > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > > FLIP, > > > > > > > > > > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying > asynchronous > > > > > > > > > registration, I > > > > > > > > > > > > would > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > expect > > > > > > > > > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to > open > > > or > > > > > > > register > > > > > > > > > > > catalog > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > background, > > > > > > > > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > your previous comments, > > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance > will > > be > > > > > > > > initialized > > > > > > > > > > if > > > > > > > > > > > it > > > > > > > > > > > > > has > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > > > > > initialized > > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the > initialization > > > > > process > > > > > > > > fails, > > > > > > > > > > > these > > > > > > > > > > > > > > > > > statements > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > executed successfully." > > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy > > > > > > > initialization > > > > > > > > > for > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > > > > > > > > > > > > registration, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 ----- > > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" < > > > > > > > jinfeng1...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" < > > > > dev@flink.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 > 下午 > > > > > 3:27:45 > > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP > 295: > > > > > Support > > > > > > > > > > > persistence > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous > > registration > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you > all > > > for > > > > > your > > > > > > > > > > interest > > > > > > > > > > > in > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > DISCUSS. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon > > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog > > > with > > > > > the > > > > > > > same > > > > > > > > > > name > > > > > > > > > > > > that > > > > > > > > > > > > > > > > exists > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > them? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a > crucial > > > > > point. > > > > > > > > Based > > > > > > > > > on > > > > > > > > > > > my > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > personal > > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the > > > Map<String, > > > > > > > Catalog> > > > > > > > > > > > > catalogs > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > serve > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > cache > > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs > > and > > > > > have > > > > > > > the > > > > > > > > > > > highest > > > > > > > > > > > > > > > > priority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are three methods > > that > > > > can > > > > > > > affect > > > > > > > > > the > > > > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > > > > > Catalog> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String > > > > > > > catalogName, > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > catalog) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the > > catalog > > > > > instance > > > > > > > > > into > > > > > > > > > > > the > > > > > > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > > > > > > Catalog> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. > unregisterCatalog(String > > > > > > > > catalogName) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the > > > catalog > > > > > > > > instance > > > > > > > > > > > > > > > corresponding > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > catalogName > > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String, > > Catalog> > > > > > > > catalogs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String > > > > catalogName) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This method first > retrieves > > > the > > > > > > > > > > corresponding > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog> > > > > > catalogs. If > > > > > > > > the > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > exist, > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the > corresponding > > > > > > > > configuration > > > > > > > > > > > from > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > CatalogStore, > > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts > > the > > > > > > > > initialized > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> > > > catalogs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following two methods > > > only > > > > > modify > > > > > > > > the > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String > > > > catalogName, > > > > > > > > > > Map<String, > > > > > > > > > > > > > > String> > > > > > > > > > > > > > > > > > > > > properties) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the > > > > properties > > > > > to > > > > > > > the > > > > > > > > > > > > > > catalogStore > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > checks > > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a > > > catalogName > > > > > with > > > > > > > the > > > > > > > > > > same > > > > > > > > > > > > > name. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String > > > > > catalogName) > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the > > > > specified > > > > > > > > > > > configuration > > > > > > > > > > > > > of > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > specified > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the > > > > catalogStore. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following are > possible > > > > > conflict > > > > > > > > > > > scenarios: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding > > > > > catalogName > > > > > > > > > > already > > > > > > > > > > > > > > exists > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in > the > > > > > > > Map<String, > > > > > > > > > > > > Catalog>, > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String > > > > > catalogName, > > > > > > > > > Catalog > > > > > > > > > > > > > > catalog) > > > > > > > > > > > > > > > > > method > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly > > saved > > > > to > > > > > the > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > Catalog> > > > > > > > > > > > > > > > > > > > > > catalogs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding > > > > > catalogName > > > > > > > > > > already > > > > > > > > > > > > > > exists > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the > > > > Map<String, > > > > > > > > > Catalog>, > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog > > catalog) > > > > > method > > > > > > > > will > > > > > > > > > > > fail. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding > > > > > catalogName > > > > > > > > > > already > > > > > > > > > > > > > > exists > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the > > > > > > > > > addCatalog(String > > > > > > > > > > > > > > > > catalogName, > > > > > > > > > > > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties) > method > > > can > > > > > > > directly > > > > > > > > > > save > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > properties > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the > > > > > > > getCatalog(String > > > > > > > > > > > > > > catalogName) > > > > > > > > > > > > > > > > > method > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties > for > > > > > > > > initialization > > > > > > > > > > > > because > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > corresponding > > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance already > > > exists > > > > > in > > > > > > > > > catalogs > > > > > > > > > > > and > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > prioritized. > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the > > > > > > > > > > unregisterCatalog(String > > > > > > > > > > > > > > > > > catalogName) > > > > > > > > > > > > > > > > > > > > method > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance > > > > > corresponding to > > > > > > > > the > > > > > > > > > > > > original > > > > > > > > > > > > > > > > > > catalogName > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > necessary. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse > > > users > > > > > that > > > > > > > > > > > > > > > > > > `registerCatalog(String > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog > > catalog)` > > > > in > > > > > the > > > > > > > > > > > > `Map<String, > > > > > > > > > > > > > > > > > Catalog> > > > > > > > > > > > > > > > > > > > > > > catalogs` > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String > > > > > catalogName, > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > String> > > > > > > > > > > > > > > > > > > > > > > properties) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially > lead > > > to > > > > > > > > > confusion. I > > > > > > > > > > > > > suggest > > > > > > > > > > > > > > > > > > changing > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to > > > > > > > > addCatalog(String > > > > > > > > > > > > > > > catalogName, > > > > > > > > > > > > > > > > > > > > > Map<String, > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as > > > > > previously > > > > > > > > > > mentioned > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Hang > > > > > > > > > > > > > > > > > > > > > > > > > > > add > > `registerCatalog(String > > > > > > > > > > > > catalogName,Catalog > > > > > > > > > > > > > > > > > catalog, > > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is > already > > an > > > > > > > instance, > > > > > > > > > > > adding > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > "lazyInit" > > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the > > > > > > > registerCatalog(String > > > > > > > > > > > > > > catalogName, > > > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > > > > catalog) > > > > > > > > > > > > > > > > > > > > > > > > > > method may not > necessarily > > > > > result in > > > > > > > > lazy > > > > > > > > > > > > > > > > initialization. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think > about > > > > > > > encryption > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is > > > > necessary, > > > > > but > > > > > > > > > > perhaps > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > encryption > > > > > > > > > > > > > > > > > > > > logic > > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented in > > the > > > > > > > > CatalogStore > > > > > > > > > > > when > > > > > > > > > > > > > the > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > actually > > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the > > > > > > > > FileCatalogStore > > > > > > > > > > > could > > > > > > > > > > > > > > > encrypt > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file, > > while > > > > the > > > > > > > > > > > > > MemoryCatalogStore > > > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > > require > > > > > > > > > > > > > > > > > > > > > > > > > > encryption. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the > > > > > > > > > > > `MemoryCatalogStore`? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary > to > > > > have a > > > > > > > > > > > > > > MemoryCatalogStore > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > default > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which > saves > > > > > Catalog > > > > > > > > > > > > > configurations > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > memory. > > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to > > > > > implement > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > loading > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > Catalog, > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce > > > > > additional > > > > > > > > cache. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous > > > > > registration > > > > > > > > > failed? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical > > > > concern. > > > > > > > When > > > > > > > > > > > > executing > > > > > > > > > > > > > > DDL, > > > > > > > > > > > > > > > > > DQL, > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > DML > > > > > > > > > > > > > > > > > > > > > > > > > > statements that > reference a > > > > > specific > > > > > > > > > > Catalog, > > > > > > > > > > > > the > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it > > has > > > > not > > > > > > > been > > > > > > > > > > > > > initialized > > > > > > > > > > > > > > > yet. > > > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > initialization process > > fails, > > > > > these > > > > > > > > > > > statements > > > > > > > > > > > > > will > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > executed > > > > > > > > > > > > > > > > > > > > > > > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, > > > Catalog> > > > > > > > catalogs > > > > > > > > > and > > > > > > > > > > > > > > > > CatalogStore > > > > > > > > > > > > > > > > > > > > > > catalogstore > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all > have > > > the > > > > > same > > > > > > > > > > catalog > > > > > > > > > > > > > name, > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > > > > correponding > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog > > instances. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be > resolved > > by > > > > > > > referring > > > > > > > > > to > > > > > > > > > > > the > > > > > > > > > > > > > > > previous > > > > > > > > > > > > > > > > > > > > > responses. > > > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that the > > > > Catalog > > > > > > > that > > > > > > > > > has > > > > > > > > > > > been > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > recently > > > > > > > > > > > > > > > > > > > > > used > > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority > for > > > > > subsequent > > > > > > > > > use. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at > > > 10:58 PM > > > > > > > Xianxun > > > > > > > > > Ye > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > yesorno828...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this > > > Flip, > > > > > I do > > > > > > > > > > believe > > > > > > > > > > > > this > > > > > > > > > > > > > > > Flip > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > helpful > > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I > work > > > for > > > > > also > > > > > > > > > > manages a > > > > > > > > > > > > lot > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > catalogs, > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks > becomes > > > > slow > > > > > > > > because > > > > > > > > > of > > > > > > > > > > > > > > loading a > > > > > > > > > > > > > > > > > > number > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > catalogs. > > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in > > the > > > > > user's > > > > > > > > > Flink > > > > > > > > > > > SQL > > > > > > > > > > > > > > > through > > > > > > > > > > > > > > > > > > > regular > > > > > > > > > > > > > > > > > > > > > > > > expression > > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all > > catalogs > > > > to > > > > > > > > improve > > > > > > > > > > the > > > > > > > > > > > > > speed > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > task > > > > > > > > > > > > > > > > > > > > > > > submission. > > > > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I > > > have > > > > > some > > > > > > > > > > > questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a > > catalog > > > > with > > > > > > > > CREATE > > > > > > > > > > > > CATALOG, > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is > used > > > by > > > > > > > default. > > > > > > > > > > What > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > asynchronous > > > > > > > > > > > > > > > > > > > > > > > > registration > > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to > > implement > > > > > > > > asynchronous > > > > > > > > > > > > > > > registration? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same > > > > > question > > > > > > > with > > > > > > > > > > > > Shammon’s > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > > > > comment. > > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> > > catalogs > > > > and > > > > > > > > > > > CatalogStore > > > > > > > > > > > > > > > > > catalogstore > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have > the > > > > same > > > > > > > > catalog > > > > > > > > > > > name, > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > > > correponding > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog > > instances. > > > > For > > > > > > > > example: > > > > > > > > > > > > > > catalog1-> > > > > > > > > > > > > > > > > > > > > jdbcCatalog > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> > > > > > hiveCatalog > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > > > > > catalogstore. > > > > > > > > > > > > > > > > > > > > Would > > > > > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should > > > define > > > > > > > > `catalogs` > > > > > > > > > > and > > > > > > > > > > > > > > > > > > `catalogstore` > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > clearly. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang > > > Ruan > > > > < > > > > > > > > > > > > > > > ruanhang1...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
